Head-up display (HUD) content control system and method

Abstract

A system and method are provided for controlling the content of a head-up display to provide enhanced visibility of contours and surfaces that occur naturally in the environment. Depth information is provided along with predefined lighting conditions so that the HUD outputs an image with enhanced feature contours.

Description

BACKGROUND OF THE INVENTION

[0001] The present disclosure relates to systems, components, and methods for enhancing a driver's field of view. In particular, the present disclosure relates to systems, components, and methods that perform image contour rendering and project the rendered image in alignment with the naturally occurring field of view to enhance the visibility of features in the field of view. SUMMARY OF THE INVENTION

[0002] According to the present disclosure, systems, components, and methods are provided for controlling a HUD to enhance visibility within a driver's field of view.

[0003] According to the disclosed embodiments, structures and software are provided for controlling a HUD such that the display can project an enhanced contour-rendered image of a naturally occurring field of view or a portion thereof outside of the vehicle. The enhanced contour-rendered image can be a product of depth information, lighting conditions, and features within the field of view.

[0004] According to at least one embodiment, depth information is sensed using Light Detection and Ranging (LIDAR) and formatted as a depth image. Features within the field of view can be captured by a camera. Lighting conditions can be, for example, specular, diffuse, or ambient occlusion, and can be calculated using normal vectors computed from the depth image.

[0005] According to at least one embodiment, lighting conditions can be determined based on vehicle trajectory, time of day, and / or date of the year. The light contour-rendered image can be projected by the HUD in response to flat or low lighting conditions or sensed rain. The HUD and processor can be integrated within the vehicle to communicate with various vehicle sensors and inputs via a communication bus.

[0006] Additional features of the present disclosure will become apparent to those skilled in the art when considering the illustrative embodiments that exemplify the best mode currently understood for carrying out the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The detailed description refers specifically to the drawings, in which:

[0008] Figure 1 is a block diagram of the hardware structure and software used by the disclosed embodiments to provide a naturally enhanced field of view in a HUD;

[0009] Figure 2 is a diagram of manipulating various image data on a windshield field of view and projecting and aligning the manipulated data to provide a naturally enhanced field of view in a HUD; and

[0010] Figure 3 is a flow chart of a method for generating a naturally enhanced field of view. DETAILED DESCRIPTION

[0011] The accompanying drawings and description provided herein may have been simplified to illustrate aspects relevant to a clear understanding of the devices, systems, and methods described herein, while eliminating other aspects that may be found in a typical device, system, and method for the sake of clarity. Those of ordinary skill in the art will recognize that other elements and / or operations may be desirable and / or necessary for implementing the devices, systems, and methods described herein. Since such elements and operations are well known in the art and since they do not facilitate a better understanding of the present disclosure, a discussion of such elements and operations may not be provided herein. However, the present disclosure is considered to inherently include all such elements, variations, and modifications of the described aspects that would be known to those of ordinary skill in the art.

[0012] In an illustrative embodiment, Figure 1 an example of which is shown in FIG. 5, a structure and software for a HUD content control system 100 are provided for improving, enhancing, or increasing the visibility of a naturally occurring view outside a vehicle. The HUD system 100 is illustratively implemented to be integrated into a communication bus 102 of a vehicle 104. The HUD system illustratively includes one or more sensors 106, which are implemented to include sensors configured to sense a geographical area of features outside the vehicle. A camera 110 may also be provided to capture an image of the geographical area outside the vehicle. A processor 118 may receive information from the sensors 106, process the information, and control the output to the HUD 120 on the vehicle windshield. The processor 118 may further be provided with a local memory 124 for storing commands, historical sensed data, and / or generated HUD outputs. The vehicle may also include a database 114 of maps, a navigation system 116, and additional vehicle sensors 112 that may communicate with the processor 118. For this purpose, the HUD will be described and positioned according to the view of the driver of the vehicle 104 and the windshield. However, the system may generate multiple different displays separately or simultaneously at multiple different viewpoints within the vehicle for passengers, such as at different windows and views that can be seen from the vehicle.

[0013] The system 100 illustratively includes one or more sensors 106, which are implemented to include at least one LIDAR sensor for receiving distance data indicating the distance between the vehicle and various points in the field of view from the windshield. The vehicle may also include a database 114 of maps, a GPS unit 115, and a navigation system 116, as well as additional vehicle sensors 112 such as a compass, weather, and speed sensors, or an input interface 112 such as an infotainment touchscreen, which may communicate with the processor 118. For illustrative purposes, the HUD will be described and positioned according to the view of the driver of the vehicle 104 and the windshield. However, the HUD system may generate and project multiple different displays separately or simultaneously at multiple different viewpoints within the vehicle.

[0014] As Figure 2 seen, the processor 218 can be a graphics processing unit (GPU) and can process the optical image 224 and the camera image 226 of the windshield field of view during flat lighting conditions to generate a light profiled image 228. The light profiled image 228 can be transmitted to the HUD 220. The HUD can project the light profiled image 228 onto the windshield and the corresponding windshield field of view 230 such that the light profiled image 228 is aligned with the actual windshield field of view 230, thereby creating a natural enhanced field of view 232.

[0015] A method for generating a natural enhanced field of view in real time can be initiated 300 via a manual driver input, such as at a vehicle interface such as a center console or an infotainment interface. Additionally, the process can be programmed to automatically initiate when a low lighting condition is sensed and / or when the vehicle headlights are lit. Depth data of the field of view in front of the vehicle can be captured 302, and at the same time, real image data of the field of view in front of the vehicle can be captured 304. The depth data can be captured, for example, using LIDAR or other depth sensors. The depth data can be captured as a depth image into a buffer in an image format and then processed by image processing to provide information about the geometry of the field of view in front of the vehicle. The real image data can be captured using a camera or using a map image database (such as GOOGLE EARTH[TM], HERE[TM] Maps, or other street view map databases) together with the current geographical location and orientation information of the vehicle that can be provided, for example, via a vehicle navigation system.

[0016] A light profiled image can be generated 306 using the depth data and the real image data. First, the normal vector of each pixel of the geometry or feature within the depth image can be determined 308. To calculate the normal vector at each pixel on the geometric surface, a first tangent vector on the surface is calculated. Then, the cross product of the tangent vectors generates the normal vector. At each pixel in the depth image, a vector from the camera to the geometry is calculated in the directions of four pixels adjacent to the current pixel. Then, the difference vector is calculated using two adjacent horizontal pixel vectors. The normalization of this difference vector can be used as an estimate of the horizontal tangent vector on the geometric surface of the current pixel. This process can be repeated for two adjacent vertical pixels to determine an estimate of the vertical tangent vector. Then, the normal vector can be determined for the pixel as the cross product of the horizontal tangent vector and the vertical tangent vector. An example of this normal vector processing is provided in more detail below.

[0017] The depth image space can be defined according to the camera's field of view space in order to calculate each normal from the camera to the geometry. The z-axis can be defined as along the camera aiming direction, with positive z being from the scene towards the camera, the x-axis being the horizontal direction when viewing the depth image, positive x being to the right, the y-axis being vertical, and positive y being upwards. Thus, using the standard perspective projection of a CG camera, the unit vector from the camera to pixel x, y can be calculated as:

[0018]

[0019] where fovY = the field of view angle in the Y (vertical) direction, width = the width of the depth image in pixels, height = the height of the depth image in pixels, aspect = width / height, and x and y are the pixel coordinates ranging from [0…width - 1] to [0…height - 1] respectively. The vector passing through the center of each pixel is calculated by adding.5 to each of the x and y coordinate values.

[0020] Using the depth value at pixel x, y, the point at the nearest geometry at pixel x, y is:

[0021] .

[0022] Using the above "PointOnGeomitryAtPixel" formula, the system can calculate the vectors of the geometry at the current pixel and at four adjacent horizontal and vertical pixels.

[0023] The horizontal and vertical tangent vectors at the current pixel on the geometry surface are calculated by subtracting the vectors of the geometry at adjacent horizontal and vertical pixels:

[0024] .

[0025] Finally, the normal vector can be calculated as the cross product of the tangent vectors:

[0026] .

[0027] Using the determined normal vector data together with information about the light direction, lighting calculations can be performed to generate the light image 310. For example, the cosine of the angle between the light direction and the normal direction can be used to generate a diffuse illumination light image. The light direction information can be stored locally and be related to one or more of the date, the time of day, and the compass orientation, or can be sensed by sensors integrated in the vehicle.

[0028] Finally, the optically profiled image can be calculated by combining the optical image and the original captured image. The product of the optical image and the original captured image can be determined to generate the optically profiled image 306. Next, the optically profiled image can be projected to overlay and align with the live view through the windshield 314 to generate a natural enhanced view. In the natural enhanced view or enhanced view, natural lighting and contrast are used to realistically sharpen the outlines of features in front of the vehicle to enhance the driver's view. When operating the vehicle, as the view through the windshield changes and updates, this process can be repeated and updated. Further, lighting conditions such as specular, diffused, backlit, or particularly relevant to the time of day lighting can be stored in local memory and applied according to driver or user preferences.

[0029] Although the method has been described using diffused lighting techniques, specular lighting can also be used in conjunction with sensed driver gaze data to provide directional lighting and realistic light flashes in the view. Additionally, backlighting can be used to enhance the mood and create the perception of "driving towards the light". For example, cinematic lighting can be used to further magnify or enhance the outlines, guide the eye, and enhance the mood. Cinematic lighting typically implies creative interpretation but often utilizes a standard 3-point lighting setup, which includes three lights: a key light that is the brightest and illuminates from a side angle of approximately 45 to 60 degrees, a fill light that is dimmer and illuminates from the opposite side to light the area of the key light's shadow, and a backlight that illuminates from behind the object towards the camera to create a border-type appearance of lighting on the object.

[0030] The depth image space is defined according to the camera space. However, in some embodiments, a camera-to-world matrix can be used to transform values calculated in the camera space to other spaces such as the world space.

[0031] According to an alternative embodiment, LIDAR depth data can be represented in a point cloud format, and the normals can be calculated directly from the point cloud. Existing libraries (such as pointcludes.org) can be used to calculate normal vectors from the point cloud data. This calculation typically performs a principal component analysis on neighboring points to obtain the secondary eigenvector representing the normal. The two principal eigenvectors represent the tangent vectors lying on the surface of the geometric features in the view.

[0032] Low-light driving conditions can result in low-contrast images where the driver sees flat or unnatural lighting. This can lead to reduced situational awareness, misinterpretation, or delayed response times when maneuvering the vehicle as the driver is unable to see features including potholes, road bends, contours, and terrain. During nighttime or dark conditions, the vehicle headlights create a visibility tunnel and there is a lack of perception of the surrounding environment and horizon. Available HUD systems simply add unnatural graphics and icons to the driver's field of view, thus blurring the view and distracting the driver.

[0033] The disclosed HUD system is configured to improve the driver's visibility of features in the field of view. This is achieved by automatically sensing the features, processing their contours, and projecting a contour-enhanced image aligned with the contours of the features in the field of view. In this way, the HUD system enhances the naturally occurring contours of features in the field of view using realistic lighting to improve driving safety. The system avoids adding additional graphics or icons to the projection output to the field of view to identify these features, which may blur the field of view or distract the driver. By using natural lighting cues to enhance what the driver sees, the driving conditions are realistically enhanced. This enhanced realism permits manipulation of the contour information regarding the images presented to the driver to compensate for poor lighting, weather, and time of day. This allows the driver to observe optimal driving conditions even when it is dark, raining, or otherwise more difficult to see while driving.

[0034] It should be understood that one or more servers, processors, and associated memories can be executed on, utilized by, or accessed to execute some or all of the above methods. Unless otherwise specifically stated and as is clear from the above description, it should be understood that throughout the description of the specification, terms such as "processing", "computing", "calculating", "determining", etc. refer to the actions and / or processes of a computer or computing system or similar electronic computing device that manipulates and / or transforms data represented as physical quantities such as electronic in the registers and / or memories of the computing system into other data similarly represented as physical quantities in the memories, registers, or other such information storage, transmission, or display devices of the computing system.

[0035] In a similar manner, the term "processor" can refer to a GPU or any device or part of a device that processes electronic data from registers and / or memories to transform that electronic data into other electronic data that can be stored in the registers and / or memories.

[0036] References to "one embodiment", "an embodiment", "example embodiment", "various embodiments", etc. may indicate that one or more embodiments of the invention so described may include a particular feature, structure, or characteristic, but not every embodiment must include that particular feature, structure, or characteristic. Additionally, repeated use of the phrase "in one embodiment" or "in an exemplary embodiment" does not necessarily refer to the same embodiment, although it may.

Claims

1. A system (100) for enhancing the visibility of features in a driver's field of view in real time during vehicle operation, the system comprising: a camera (110) configured to capture an image in the driver's field of view, at least one depth sensor configured to capture depth data in the driver's field of view, and means for generating a light silhouette image (228) from the captured image and the depth data and superimposing the light silhouette image (228) on the driver's field of view to enhance the visibility of features in the driver's field of view, wherein the at least one depth sensor includes a LIDAR sensor (106) configured to capture depth data and convert the data into depth images in frames in an image format buffer, where each frame corresponds to an image captured by the camera (110), wherein the means is further configured to perform image processing on the depth images using a preselected lighting condition to create a light image (224), and combine the light image (224) with the corresponding captured camera image (226) to generate the light silhouette image (228), wherein normal vector data for determining a normal vector of each pixel of a feature within the depth image is determined, and wherein lighting calculations are performed using the determined normal vector data together with information about the light direction to generate the light image, the information about the light direction being stored locally and related to one or more of date, time of day, and compass orientation, or sensed by sensors integrated in the vehicle.

2. The system according to claim 1, wherein, the means for generating and superimposing includes a processor (118, 218) in communication with a head-up display unit (120).

3. The system according to claim 1, wherein, the preselected lighting condition is one of a diffused, specular, backlit, cinematic, or artistic lighting condition.

4. A vehicle, comprising: at least one depth sensor configured to capture depth data of the geography or environment external to the vehicle, a processor (118, 218) configured to generate a light silhouette image (228) from the captured depth data and image data, and a head-up display unit (120) configured to project the light silhouette image (228) on a window of the vehicle (104) and align the light silhouette image (228) with the view through the window (230) to increase the contrast and silhouette of features in the view through the window, wherein the processor (118, 218) is further configured to perform image processing on the depth images using a preselected lighting condition to create a light image (224), and combine the light image (224) with the corresponding captured camera image (226) to generate the light silhouette image (228), Normal vector data for determining the normal vector of each pixel of a feature within the depth image, and wherein illumination calculations are performed using the determined normal vector data together with information about the light direction to generate the light image, the information about the light direction being stored locally and related to one or more of date, time of day, and compass orientation, or sensed by a sensor integrated in the vehicle.

5. The vehicle according to claim 4, further comprising a camera (110), wherein, the camera generates the image data.

6. The vehicle according to claim 4, further comprising a map database (114), wherein, the map database generates the image data.

7. The vehicle according to claim 4, wherein, the processor is further configured to receive information from a vehicle communication bus (102), the information including time of day, direction of travel, and illumination direction, such that an image (228) of the light profile depiction is generated based on the time of day data and direction of travel data.

8. The vehicle according to claim 7, wherein, the processor (118, 218) is further configured to generate the image (228) of the light profile depiction based on a preselected illumination condition, the preselected illumination condition including diffused, specular, backlit, cinematic, or artistic illumination conditions.

9. The vehicle according to claim 8, wherein, the processor is configured to start (300) generating (306) the image (228) of the light profile depiction in response to a user input, an environmental condition sensed outside the vehicle, or vehicle headlight illumination.

10. The vehicle according to claim 4, wherein, the at least one depth sensor is a LIDAR sensor (106).

11. The vehicle according to claim 10, wherein, the LIDAR sensor (106) captures depth data in the form of a depth image frame or point cloud.

12. A method for enhancing the visibility of features in a driver's field of view when maneuvering a vehicle, comprising: capturing (304) an image of the field of view, sensing (302) the depth of features in the field of view while the image is being captured, wherein the image in the field of view is captured by a camera (110), and the depth of the features is sensed by at least one LIDAR sensor (106) as a depth image, the depth image comprising the distance from the camera to the nearest geometry along each vector passing through each pixel from the camera, determining normal vector data for the normal vector of each pixel of a feature within the depth image, performing illumination calculations using the determined normal vector data together with information about the light direction to generate a light image (224), the information about the light direction being stored locally and related to one or more of date, time of day, and compass orientation, or sensed by a sensor integrated in the vehicle, Process (306) the depth-sensing data and the captured images to generate a light-profile depicted image (228), wherein the light image (224) is combined with the corresponding captured camera image (226) to generate the light-profile depicted image (228), and project (314) the light-profile depicted image onto the field of view to align the light-profile depicted image with the field of view.

13. The method according to claim 12, wherein, the process further includes applying an illumination condition selected from one of diffuse illumination, specular illumination, backlight illumination, cinematic or artistic illumination to the sensing data.

14. The method according to claim 12, wherein, the alignment of the light-profile depicted image with the field of view results in an overall enhanced field of view with a realistic illumination condition.

15. The method according to claim 12, wherein, the alignment of the light-profile depicted image with the field of view results in an overall enhanced field of view with a cinematic illumination condition.

16. The method according to claim 12, wherein, the light-profile depicted image is processed and updated in real time to align with the updated field of view as the vehicle moves.

Images