Head-up display

Abstract

A head-up display for a vehicle. The head-up display includes a projector and a processor. The projector is arranged to project image content such that the image content is visible from an eyebox. The processor is arranged to receive a captured image of a scene visible from the eyebox. The processor is arranged to, at a first time: detect a first object in the scene, and instruct the image projector to project an icon (e.g., computer graphics) that appears to coincide with the first object from a viewing position. The processor is further arranged to, at a second time later than the first time: detect a second object in a line of sight from the eyebox position to the first object, and in response to detection of the second object, instruct the image projector to change a visual appearance of the projected icon.

Description

Technical Field

[0001] This disclosure relates to projectors and head-up displays. More specifically, this disclosure relates to holographic projectors and head-up displays for use in vehicles, such as motor vehicles. This disclosure also relates to methods of holographic projection, methods of projecting virtual images onto a head-up display, and methods of displaying virtual images on a window, such as a windshield, using a head-up display. Background Technology

[0002] Light scattered from an object contains amplitude and phase information. This amplitude and phase information can be captured on, for example, a photosensitive plate using well-known interferometry techniques to form a holographic record, or "hologram," including interference fringes. The hologram can be reconstructed by illuminating it with appropriate light to form a two-dimensional or three-dimensional holographic reconstruction or replay image representing the original object.

[0003] Computationally generated holograms (CGHs) can numerically simulate interference processes. Computer-generated holograms can be calculated using techniques based on mathematical transformations such as Fresnel or Fourier transforms. These types of holograms can be called Fresnel holograms or Fourier holograms. A Fourier hologram can be considered a Fourier domain representation or a frequency domain representation of an object. For example, CGHs can also be calculated using coherent ray tracing or point cloud techniques.

[0004] CGH can be encoded on a spatial light modulator (SLM) arranged to modulate the amplitude and / or phase of the incident light. For example, optical modulation can be achieved using electrically addressable liquid crystals, optically addressable liquid crystals, or micromirrors.

[0005] An SLM can include multiple individually addressable pixels, which may also be referred to as cells or elements. The optical modulation scheme can be binary, multi-level, or sequential. Alternatively, the device can be sequential (i.e., without pixels), so optical modulation can be sequential across the device. An SLM can be reflective, meaning that modulated light is reflected from the SLM. An SLM can also be transmissive, meaning that modulated light is transmitted from the SLM.

[0006] The described technology can provide a holographic projector for imaging. Such projectors are already used in head-up displays (HUDs) and head-mounted displays (HMDs), including near-eye devices. Traditionally, a rectangular area (referred to herein as the virtual image area) is defined within the driver's field of vision, and the head-up display can display image content within this rectangular area. Summary of the Invention

[0007] The aspects of this disclosure are defined in the appended independent claims.

[0008] In summary, this disclosure relates to image projection. It relates to methods of image projection and image projectors including display devices. This disclosure also relates to projection systems including an image projector and an observation system, wherein the image projector projects or relays light from the display device to the observation system. This disclosure is equally applicable to monocular and binocular observation systems. An observation system may include one or more eyes of an observer. An observation system includes optical elements with optical power (e.g., the lens of the human eye) and an observation plane (e.g., the retina of the human eye). The projector may be referred to as a “light engine.” The display device and the image formed (or perceived) using the display device are spatially separated from each other. The observer forms or perceives the image on the display plane. In some embodiments, the image is a virtual image, and the display plane may be referred to as a virtual image plane. In other embodiments, the image is a real image formed by holographic reconstruction, and this image is projected or relayed to the observation plane. The image is formed by illuminating a diffraction pattern (e.g., a hologram) displayed on the display device.

[0009] Display devices consist of pixels. Display pixels can show diffraction patterns or structures of diffracted light. Diffracted light can form an image on a plane spatially separate from the display device. According to well-known optical principles, the magnitude of the maximum diffraction angle is determined by the pixel size and other factors such as the wavelength of light.

[0010] An improved HUD for motor vehicles is also disclosed herein. The HUD includes an image generation unit. The image generation unit can be arranged to generate images containing informational content such as speed or navigation information. An optical system is also provided, arranged to form a virtual image of the informational content. The virtual image of the informational content can be formed at a suitable viewing position for the driver, such as within the driver's normal field of vision when operating the motor vehicle. For example, the virtual image of the informational content can appear below the vehicle's hood (or engine cover) at a distance from the driver. The virtual image of the informational content is positioned so as not to adversely affect the driver's normal observation of the scene. The virtual image of the informational content can overlay the driver's observation of the real world. The informational content is computer-generated and can be controlled or updated in real time to provide the driver with real-time information.

[0011] By way of example only, the embodiments relate to an image generation unit including a holographic projector. This disclosure is compatible with any display technology, including backlit liquid crystal displays, laser scanning displays, digital micromirror devices (DMDs), fluorescent displays, and plasma displays. In embodiments involving holographic projectors, the image is a holographic reconstruction of a computer-generated hologram. The holographic projector-based head-up display (HUD) fully described below is able to provide significantly greater contrast than currently available competing technologies due to the efficiency of the holographic process and its inherent suitability for use with laser light sources.

[0012] A head-up display may include a holographic processor. The image may be a holographic reconstruction. The holographic processor may be arranged to output a computer-generated hologram to a spatial light modulator. The computer-generated hologram may be arranged to at least partially compensate for the shape of a vehicle's windshield.

[0013] This system can be arranged to form a virtual image of a picture using a windshield by reflecting spatially modulated light from it. The light source can be a laser and / or the light in the picture can be a laser. The spatial light modulator can be a liquid crystal on silicon spatial light modulator. The picture can be formed by the interference process of spatially modulated light at a light-receiving surface. Each computer-generated hologram can be a mathematical transformation of the picture, optionally a Fourier or Fresnel transform. The computer-generated hologram can be a Fourier or Fresnel hologram. The computer-generated hologram can be a hologram generated by a point cloud method. The spatial light modulator can be arranged to spatially modulate the phase of the light from the light source. The spatial light modulator can be arranged to spatially modulate the amplitude of the light from the light source.

[0014] A head-up display (HUD) for a vehicle with windows is provided. The HUD includes a display device and an optical system. In some embodiments, the display device is arranged to display a hologram of an image or picture for projection. The image or picture may include image or picture content. The image or picture content may include multiple discrete computer graphics.

[0015] In the first set of embodiments, a holographic reconstruction of an image is formed on a screen, such as a diffuser, by illuminating a display device with light from a light source such as a laser diode. In these embodiments, the laser diode, display device, and screen form an image generation unit, which is familiar to those skilled in the art of holographic projection. In these embodiments, the optical system may include an optical relay system having at least one element with refractive power, the optical relay system being arranged to magnify the image on the screen and project it onto the windshield of a vehicle to form a magnified virtual image of that image. Such a configuration has been disclosed, for example, in WO2020 / 016214, the entirety of which is incorporated herein by reference.

[0016] In the second set of embodiments, the intermediate reconstruction of the image is not formed on the screen, but rather the hologram (more specifically, light encoded with the hologram or light spatially modulated according to the displayed hologram) is projected directly onto the observer. In these embodiments, it is sometimes said that the lens of the observer's eye performs the hologram-to-image conversion—for example, this could be a Fourier or Fresnel transformation. In these embodiments, pupil dilators (or a pair of orthogonal pupil dilators) can be used to expand the eye box. For example, such a configuration has been disclosed in GB2101666.2, filed February 5, 2021, which is incorporated herein by reference in its entirety.

[0017] According to a first aspect of this disclosure, there exists a head-up display for a vehicle. The head-up display includes a projector and a processor. The projector is arranged to project image content visible from an eye-box (making it visible). The processor is arranged to receive a captured image of a scene visible from the eye-box. The processor is arranged to: detect a first object in the scene at a first time, and instruct / drive the image projector to project an icon (e.g., computer graphics) that appears aligned / coinciding with the first object from the viewing position (within the eye-box, i.e., the eye-box position). The processor is also arranged to: detect a second object in the line of sight from the viewing position to the first object at a second time, later than the first time, and in response to detecting the second object, instruct / drive the image projector to change the visual appearance of the projected icon.

[0018] The steps to change the visual appearance of a projected icon may include changing at least one aspect of the physical form of the projected icon.

[0019] At least one physical form of a projected icon may be selected from the group including shape, color, size and brightness.

[0020] The image projector can be a holographic projector that includes spatial light modulators arranged to display a hologram of the projected image content. The hologram can be calculated in real time.

[0021] An optical combiner can be used to project image content, allowing the image content to supplement / add / cover the scene visible from the eyebox.

[0022] The processor can be configured to continuously receive the observer's observation position within the eyebox (e.g., eyebox position) and determine whether the second object is in the line of sight based on the received observation position (e.g., eyebox position).

[0023] The first object can be a moving object. The processor can be configured to determine whether the second object is in the line of sight to the first object based on the position of the first object at a second time and optionally the observer's observation position at the second time (e.g., eye-box position).

[0024] The processor can be configured to optionally maintain positional / visual alignment between the projected icon and the first object based on the received observation position (e.g., eyebox position).

[0025] According to a second aspect of this disclosure, there exists a driver assistance system comprising a head-up display, a camera, and a user tracking system. The camera is arranged to capture images of a scene and continuously outputs the captured images to the head-up display. The user tracking system is arranged to monitor the position of a user on the head-up display and continuously outputs the user's eye position to the head-up display.

[0026] According to a third aspect of this disclosure, there is a method for a head-up display. The method includes a first step of capturing a first image of a scene at a first time. The method includes a second step of detecting a first object in the first image of the scene. The method includes a third step of projecting an icon that appears aligned / overlapping with the first object from an eye-box position. The method includes a fourth step of capturing a second image of the scene at a second time. The method includes a fifth step of detecting a second object (e.g., at an eye-box position) in the user's line of sight to the first object. The method includes a sixth step of changing the visual appearance of the projected icon in response to the detection of the second object.

[0027] The method may also include altering the visual appearance of the projected icon, including changing at least one aspect of the physical form of the projected icon. At least one physical form of the projected icon may be selected from the group consisting of shape, color, size, and brightness.

[0028] The term "hologram" is used to refer to a record containing amplitude or phase information about an object, or a combination thereof. The term "holographic reconstruction" is used to refer to the optical reconstruction of an object formed by illuminating a hologram. The term "reproduction plane" as used herein refers to the spatial plane in which the holographic reconstruction is fully formed. The term "reproduction field" as used herein refers to a subregion of the reproduction plane from which spatially modulated light can be received from a spatial light modulator. The terms "image," "reproduced image," and "image region" refer to the region of the reproduction field illuminated by the light forming the holographic reconstruction. In embodiments, an "image" may include discrete points that may be referred to as "image pixels."

[0029] The terms “encoding,” “writing,” and “addressing” are used to describe the process of providing multiple corresponding control values ​​to multiple pixels of an SLM, each determining the modulation level of the pixel. In other words, the pixel configuration of an SLM “displays” an optical modulation distribution in response to receiving multiple control values. Therefore, it can be said that an SLM “displays” a hologram.

[0030] It has been found that acceptable-quality holographic reconstructions can be formed from "holograms" containing only phase information related to the original object. Such holographic records can be referred to as phase-only holograms. While the embodiments relate to phase-only holograms, this disclosure is equally applicable to amplitude-only holography.

[0031] This disclosure is equally applicable to forming holographic reconstructions using amplitude and phase information associated with the original object. In some embodiments, this is achieved by using complex modulation of a so-called fully complex hologram that incorporates amplitude and phase information associated with the original object. Because the value (gray level) assigned to each pixel of the hologram has amplitude and phase components, such a hologram may be called a fully complex hologram. The value (gray level) assigned to each pixel can be represented as a complex number with amplitude and phase components. In some embodiments, a fully complex computer-generated hologram is computed.

[0032] The term "phase delay" can be used as an abbreviation for the phase value, phase component, phase information, or simply phase of a pixel in a computer-generated hologram or spatial light modulator. That is, any phase value described is actually a number representing the amount of phase delay provided by that pixel (e.g., in the range of 0 to 2π). For example, a spatial light modulator is described as a pixel having a π / 2 phase value causing a π / 2 radian change in the phase of the received light. In some embodiments, each pixel of a spatial light modulator can operate on one of several possible modulation values ​​(e.g., phase delay values). The term "grayscale level" can be used to refer to multiple available modulation levels. For example, the term "grayscale level" can be used for convenience to refer to multiple available phase levels in a phase modulator, even if different phase levels do not provide different shades of gray. For convenience, the term "grayscale level" can also be used to refer to multiple available complex modulation levels in a complex modulator.

[0033] Although different embodiments and groups of embodiments may be disclosed separately in the detailed description below, any feature of any embodiment or group of embodiments may be combined with any other feature or combination of features of any embodiment or group of embodiments. That is, all possible combinations and substitutions of the features disclosed in this disclosure are contemplated.

[0034] Although reference has been made to head-up displays for vehicles, those skilled in the art will understand that this disclosure extends to head-up displays for other purposes, and that the device may be more generally referred to as a display system.

[0035] In this disclosure, the term "basic" when applied to a structural unit of a device can be interpreted as the technical feature of the structural unit produced within the technical tolerances of the method used to manufacture the structural unit. Attached Figure Description

[0036] Referring to the following figures, specific embodiments are described by way of example only:

[0037] Figure 1 This is a schematic diagram of a reflective SLM that generates holographic reconstruction on a screen;

[0038] Figure 2 A block diagram of a display system according to some embodiments is shown;

[0039] Figure 3 A block diagram of an AR application according to some embodiments is shown; and

[0040] Figures 4A to 4E An example AR environment is shown, in which the physical form of virtual content associated with the detected physical object is changed in response to the detection of an obstructed view.

[0041] In all the accompanying drawings, the same reference numerals will be used to refer to the same or similar parts. Detailed Implementation

[0042] This invention is not limited to the embodiments described below, but extends to the full scope of the appended claims. That is, the invention may be implemented in different forms and should not be construed as limited to the described embodiments, which are illustrated for illustrative purposes.

[0043] Unless otherwise stated, singular terms may include plural forms.

[0044] A structure described as being formed above or below another structure should be interpreted as including situations where the structures are in contact with each other, and also including situations where a third structure is placed between them.

[0045] When describing temporal relationships, such as when the chronological order of events is described as “after,” “following,” “next,” “before,” etc., this disclosure should be considered to include both consecutive and discontinuous events, unless otherwise stated. For example, unless terms such as “exactly,” “immediately,” or “directly” are used, the description should be considered to include discontinuous cases.

[0046] Although the terms “first,” “second,” etc., may be used herein to describe various elements, these elements are not limited by these terms. These terms are used only to distinguish individual elements. For example, without departing from the scope of the appended claims, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.

[0047] Features of different embodiments may be coupled or combined with each other in part or in whole, and may interoperate differently with each other. Some embodiments may be implemented independently of each other, or may be implemented together in a mutually dependent relationship.

[0048] Optical configuration

[0049] Figure 1 An embodiment is illustrated in which a computer-generated hologram is encoded on a single spatial light modulator. The computer-generated hologram is a Fourier transform of the object used for reconstruction. Therefore, a hologram can be described as a Fourier domain, frequency domain, or spectral domain representation of the object. In this embodiment, the spatial light modulator is a reflective liquid crystal on silicon (LCOS) device. The hologram is encoded on the spatial light modulator, and a holographic reconstruction is formed at the playback field, such as a light-receiving surface like a screen or diffuser.

[0050] A light source 110, such as a laser or laser diode, is configured to illuminate the SLM 140 via a collimating lens 111. The collimating lens causes the approximately plane wavefront of the light to be incident on the SLM. Figure 1In this embodiment, the wavefront is oriented off-normal (e.g., two or three degrees away from a plane that is truly orthogonal to the transparent layer). However, in other embodiments, a generally planar wavefront is provided with normal incidence, and beam splitters are arranged to separate the input and output optical paths. Figure 1 In the illustrated embodiment, the arrangement is such that light from the light source is reflected from the mirrored rear surface of the SLM and interacts with the light modulation layer to form an outgoing wavefront 112. The outgoing wavefront 112 is applied to an optics device including a Fourier transform lens 120, the focal point of which is located at screen 125. More specifically, the Fourier transform lens 120 receives the modulated beam from the SLM 140 and performs a frequency-space transformation to produce a holographic reconstruction at screen 125.

[0051] It is worth noting that in this type of hologram, each pixel of the hologram contributes to the overall reconstruction. There is no one-to-one correlation between a specific point (or image pixel) on the playback field and a specific optical modulation element (or hologram pixel). In other words, the modulated light leaving the optical modulation layer is distributed across the entire playback field.

[0052] In these embodiments, the spatial position of the holographic reconstruction is determined by the diopter (focusing) of the Fourier transform lens. Figure 1 In the illustrated embodiment, the Fourier transform lens is a physical lens. That is, the Fourier transform lens is an optical Fourier transform lens and performs a Fourier transform optically. Any lens can act as a Fourier transform lens, but the lens's performance will limit the accuracy of the Fourier transform it performs. Those skilled in the art will understand how to use lenses to perform optical Fourier transforms.

[0053] Holographic computation

[0054] In some embodiments, the computer-generated hologram is a Fourier transform hologram, or simply a Fourier hologram or a Fourier-based hologram, wherein the image is reconstructed in the far field by utilizing the Fourier transform properties of a positive lens. The Fourier hologram is computed by Fourier transforming the desired light field in the reproduction plane back to the lens plane. The Fourier transform can be used to compute computer-generated Fourier holograms. By way of example only, the embodiments relate to Fourier holography and Gerchberg-Saxton type algorithms. This disclosure is equally applicable to Fresnel holography and Fresnel holograms that can be computed using similar methods. In some embodiments, the hologram is a phase or pure phase hologram. However, this disclosure is also applicable to holograms computed using other techniques, such as point cloud-based methods. UK Patent Application GB2112213.0, filed August 26, 2021, discloses an example hologram computation method that can be combined with this disclosure, which is incorporated herein by reference.

[0055] In some embodiments, a real-time engine is provided, configured to receive image data and compute holograms in real time using algorithms. In some embodiments, the image data is video comprising a sequence of image frames. In other embodiments, the holograms are pre-computed, stored in computer memory, and retrieved when needed for display on an SLM. That is, in some embodiments, a library of predetermined holograms is provided.

[0056] Optical modulation

[0057] The display system includes a display device that defines the exit pupil of the display system. The display device is a spatial light modulator. Spatial light modulation can be a phase modulator. The display device can be a liquid crystal on silicon (LCOS) or a spatial light modulator.

[0058] AR-HUD

[0059] Augmented Reality (AR) systems can be used in a variety of situations. One exemplary use of AR is to assist users when operating a vehicle. For example, virtual content can be presented on the HUD to provide users with directions to their desired destination. Virtual arrows or other indicators can be presented on the HUD to enhance the user's physical world and provide the route the user should follow to reach their desired destination. As another example, informative text can be presented on the HUD describing nearby shops, vehicles, etc. While AR is expected to provide valuable information, presenting information on the HUD presents challenges due to the constantly changing environment. The distance between the vehicle and surrounding objects changes as the vehicle and / or surrounding objects move.

[0060] AR allows users to augment reality with virtual content. Virtual content can be displayed on a transparent display of the viewing device to enhance the user's real-world environment. As an example, virtual content displayed on a car's HUD can present users with arrows, shapes, 3D objects, other indicators, and / or other icons, providing directions to a desired destination and / or other information about the environment. As another example, virtual content describing a vehicle and / or business can be displayed on the HUD to provide users with additional information about their environment.

[0061] To enhance the user's sense of realism, virtual content can be displayed on the HUD to create the appearance of the virtual content being presented in the user's real-world environment rather than arbitrarily displayed on the screen. To accurately create this appearance, the viewing device adjusts the presentation of the virtual content to correspond to physical objects.

[0062] Display systems are used to enhance the realism of the user experience. Users can be human users (e.g., humans), machine users (e.g., computers configured by software programs to interact with viewing devices), or any suitable combination thereof (e.g., machine-assisted humans or human-supervised machines). Display systems are computing devices integrated into vehicles such as cars to provide virtual content on a head-up display (HUD).

[0063] The display system can include a transparent or semi-transparent screen, such as the windshield of a car housing the display system, or an optical combiner for a standalone head-up display, such as a pop-up combiner. Users can simultaneously view virtual content presented by the display system and physical objects in their field of vision within the real-world physical environment.

[0064] Display systems can provide users with augmented reality experiences. For example, in addition to physical objects in the user's field of vision in the real-world physical environment, display systems can also present virtual content that the user can observe. Virtual content can be any type of image, animation, etc., presented on the display. For example, virtual content can include virtual models of objects (such as 3D models) or simple markers, such as warning triangles and similar shapes.

[0065] Physical objects can include any type of identifiable object, such as 2D physical objects (e.g., pictures), 3D physical objects (e.g., vehicles, cyclists, pedestrians, buildings, streets, etc.), locations (e.g., the ground floor of a factory), or any reference in the real-world physical environment (e.g., a perceived corner of a wall or furniture).

[0066] A display system can render virtual content in response to the detection of one or more identified objects (e.g., physical objects) in a physical environment. For example, a display system may include optical sensors for capturing images of a real-world physical environment and computer vision recognition for identifying physical objects.

[0067] In one example embodiment, the display system uses a local content dataset or any other dataset previously stored by the display system to locally analyze captured images. The local content dataset may include a library of virtual content associated with real-world physical objects or references. For example, the local content dataset may include image data depicting real-world physical objects. The display system can use captured images of physical objects to search the local content dataset to identify the physical objects and their corresponding virtual content.

[0068] In one example, the display system can analyze images of physical objects to identify feature points of the physical objects. The display system can then use the identified feature points to identify the corresponding real-world physical objects from a local content dataset. The display system can also identify tracking data associated with the physical objects (e.g., the GPS location, orientation, and distance of the viewing device to the physical object).

[0069] If the captured image is not recognized locally, the display system can download additional information (e.g., virtual content) corresponding to the captured image from a server's database, for example, via a network.

[0070] In another example, physical objects in images are remotely tracked and identified at the server using a remote dataset or any other previously stored dataset on the server. The remote content dataset may include a library of virtual content or augmented information associated with real-world physical objects or references. In this type of embodiment, the display system may provide the server with captured images of the physical objects. The server can use the received images to identify the physical objects and their corresponding virtual content. The server can then return the virtual content to the viewing device.

[0071] Display systems can project virtual content to enhance the user's sense of realism. For example, a display system can present virtual content to allow a user to simultaneously observe the virtual content and the real-world physical environment within their field of vision.

[0072] As an example, when another vehicle crosses the user's field of vision or the cyclist's line of sight, the display system can change the visual attributes (e.g., shape) of the virtual content corresponding to the cyclist. As another example, the display system can change the color or size of the virtual content in response to the same scene.

[0073] The display system can present virtual content at locations corresponding to the perceived physical object positions. Therefore, the virtual content appears to the user to be near or overlapping the physical object.

[0074] The display system re-presents virtual content based on changes in location, continuously updating the presentation of virtual content based on the position of physical objects relative to the user. As a result, when the user moves, the user perceives the virtual content as being fixed in the user's real-world environment.

[0075] Any machine, database, or device disclosed herein can be implemented in a general-purpose computer modified (e.g., configured or programmed) as a special-purpose computer to perform one or more of the functions described herein with respect to that machine, database, or device. As used herein, a “database” is a data storage resource and can store data structured as text files, tables, spreadsheets, relational databases (e.g., object relational databases), ternary storage, hierarchical data storage, or any suitable combination thereof. Furthermore, any two or more machines, databases, or devices can be combined into a single machine, and the functions described herein with respect to any single machine, database, or device can be subdivided among multiple machines, databases, or devices.

[0076] A network can be any network capable of communicating between machines (e.g., servers), databases, and devices (e.g., heads-up displays). Therefore, a network can be a wired network, a wireless network (e.g., a mobile or cellular network), or any suitable combination thereof. A network can include one or more components constituting a private network, a public network (e.g., the Internet), or any suitable combination thereof.

[0077] Figure 2 A block diagram according to an embodiment is shown. The display system 102 includes a sensor 202, a transparent display 204, a computer processor 208, and a storage device 206. The display system 102 is integrated into a vehicle, such as a car, motorcycle, airplane, boat, recreational vehicle (RV), etc.

[0078] Sensor 202 may include any type of known sensor. Sensor 202 includes at least one infrared or visible light image capturing device (e.g., a camera) arranged to capture images of the scene at, for example, a video rate.

[0079] The transparent display 204 includes, for example, a display configured to display a hologram of a virtual image generated and computed by the processor 208. The transparent display 204 can be positioned so that a user can simultaneously view virtual content presented on the transparent display and physical objects in the user's field of vision. For example, the transparent display 204 can be a head-up display (HUD) in a car or other vehicle that presents virtual content on the vehicle's windshield while allowing the user to view physical objects through the windshield. For example, the HUD can be configured to display the virtual image itself, or alternatively, it can display a virtual image projected onto the HUD.

[0080] The processor 208 includes an AR application 210 configured to present virtual content on a transparent display 204 to enhance the realism for the user. The AR application 210 may receive data from the sensor 202 (e.g., images of physical objects, location data, etc.) and use the received data to identify at least one physical object (e.g., a cyclist) and project virtual content (e.g., a warning shape) using the transparent display 204.

[0081] To identify physical objects (such as cyclists), AR application 210 determines whether an image captured by display system 102 matches an image locally stored by display system 102 in storage device 206. Storage device 206 may include a local content dataset of images and corresponding virtual content. For example, display system 102 may receive a content dataset from server 110 and store the received content dataset in storage device 206.

[0082] AR application 210 can compare a captured image of a physical object with an image locally stored in storage device 206 to identify the physical object. For example, AR application 210 can analyze the captured image of the physical object to identify feature points of the physical object. AR application 210 can use the identified feature points to identify the physical object from a local content dataset. In some embodiments, AR application 210 can identify the physical object based on the object's representational features.

[0083] If AR application 210 cannot identify a matching image from the local content dataset, AR application 210 can provide server 110 with a captured image of the physical object. Server 110 uses the captured image to search a remote content dataset maintained by server 110.

[0084] The remote content dataset maintained by the server can be larger than the local content dataset maintained by the display system 102. For example, the local content dataset maintained by the display system 102 may include a subset of the data contained in the remote content dataset, such as a core image set or the most popular images determined by the server.

[0085] Once the display system 102 or server identifies a physical object (e.g., a cyclist), it can retrieve corresponding virtual content and project it onto the transparent display 204 to enhance the user's realism by displaying virtual content that overlays the virtual content onto the user's real-world view through the transparent display. The AR application 210 can present the virtual content on the transparent display 204 to, for example, highlight the physical object (e.g., the cyclist) to the user—drawing the user's attention to the cyclist. For example, the AR application 210 can present shapes or other indicators that overlap with the physical object (e.g., the cyclist).

[0086] Virtual content changes in response to hidden objects

[0087] As described below, AR application 210 adjusts one or more properties or parameters of virtual content based on the detection of a physical object (e.g., a cyclist) and another object between the observer and the physical object. When an intervening object is detected, adjusting the properties or parameters causes the virtual content to display with different attributes (e.g., shape or color). That is, the virtual content has a changed or modified appearance.

[0088] In one embodiment, AR application 210 changes the shape of the virtual content corresponding to the cyclist when a car obstructs the cyclist's view—for example, by blocking a view such as across the line of sight. Therefore, the physical form of the virtual content presented on transparent display 204 changes when the car interferes with the cyclist's view. As another example, AR application 210 may change the color of the virtual content corresponding to the cyclist as the car moves into the user's line of sight. The virtual content may have a first form when the user's view of the physical object (e.g., the cyclist) is unobstructed, and a second form when the user's view of the physical object (e.g., the cyclist) is obstructed.

[0089] AR application 210 can continuously update the presentation of virtual content based on the position of physical objects (such as cyclists) relative to other vehicles and / or users. As another vehicle and physical object moves relative to each other, AR application 210 can use the new data to re-present the virtual content at the display position on transparent display 204 corresponding to the new position data.

[0090] AR application 210 can update the presentation of virtual content as the vehicle and / or physical objects change position. For example, AR application 210 can collect updated sensor data from sensor 202 as the vehicle moves and determine the updated position of the physical object relative to the vehicle. AR application 210 updates the presentation of virtual content based on the determined updated position of the physical object relative to the vehicle. For example, AR application 210 adjusts the display shape of the virtual content based on the updated position of the physical object. AR application 210 presents the updated presentation of virtual content on transparent display 204, thereby providing the user with an illustration of the changes in virtual content.

[0091] Any one or more modules described herein can be implemented using hardware (e.g., a machine's processor) or a combination of hardware and software. For example, any module described herein can be configured to perform the operations described herein for that module. Furthermore, any two or more of these modules can be combined into a single module, and the functionality described herein for a single module can be subdivided across multiple modules. Additionally, modules described herein as being implemented within a single machine, database, or device, according to various example embodiments, can be distributed across multiple machines, databases, or devices.

[0092] Figure 3 A block diagram of an example embodiment of an AR application 210 according to some embodiments is shown. To avoid obscuring the subject matter of the invention with unnecessary detail, from... Figure 3 Various functional components (e.g., modules) not closely related to conveying an understanding of the subject matter of this invention have been omitted. However, those skilled in the art will readily recognize that the AR application 210 can support various additional functional components to facilitate additional functions not specifically described herein. Furthermore, Figure 3 The various functional modules described can reside on a single computing device, or they can be distributed across multiple computing devices in various arrangements, such as those used in cloud-based architectures.

[0093] As shown in the figure, the AR application 210 includes an input module 302, a recognition module 304, a position determination module 306, a gaze determination module 308, a content generation module 310, and a display module 312.

[0094] Input module 302 receives sensor data from sensor 202, such as, but not limited to, optical image data of physical objects, ToF data, imaged light patterns, position / orientation data, and other data related to the operation of various sensors, and combinations thereof. Input module 302 provides the received sensor data to any other modules included in AR application 210.

[0095] The recognition module 304 identifies physical objects and corresponding virtual content based on images of physical objects captured by the sensor 202 of the display system. For example, the recognition module 304 can determine whether the captured image matches or resembles an image stored locally by the display system in the storage device 206.

[0096] The recognition module 304 compares a captured image of a physical object with a local content dataset of images stored locally in storage device 206 to identify the physical object. For example, the recognition module 304 can analyze the captured image of the physical object to identify feature points of the physical object. The recognition module 304 can then use the identified feature points to identify the physical object from the local content dataset.

[0097] If the recognition module 304 cannot identify a matching image from the local content dataset, the recognition module 304 can provide the server with a captured image of the physical object, and the server can search the remote content dataset maintained by the server.

[0098] Once the physical object 104 is identified, the identification module 304 can access the corresponding virtual content to be displayed on the transparent display 204 to enhance the user's sense of realism.

[0099] The position determination module 306 determines the position of the physical object relative to the display system. The position determination module 306 can analyze an image of the physical object to determine its position relative to the display system. For example, the position determination module 306 can analyze an image captured by the sensor 202 and identify the physical object in the captured image. The position determination module 306 then determines the position of the physical object relative to the display system based on the physical object's position in the captured image.

[0100] The gaze module 308 uses the position determined by the position determination module 306 and, for example, the user's eye-tracking information to determine information related to the gaze from the user to a physical object (e.g., a cyclist). The gaze module 308 can use any suitable technology to identify the gaze and detect whether another object (e.g., a car) obstructs (e.g., blocks, partially blocks, or completely blocks) the user's view of the identified physical object (e.g., the cyclist). As an example only, if the identification module 304 is no longer able to detect the physical object, the gaze module 308 can determine that the gaze to the physical object is obstructed. If the gaze is unobstructed, the gaze module 308 can provide a first output; if the gaze is obstructed, it can provide a second output. The eye tracking or user tracking mentioned herein is merely an example of one method for determining whether the user's view is obstructed, and it is not necessary for the invention to utilize eye tracking or user tracking information to function as disclosed herein. In some embodiments, the step of determining whether the gaze is obstructed is based on a fixed observation position, such as a fixed eye box position, like the center of the eye box.

[0101] The content generation module 310 generates virtual content based on the output of the gaze module 308. For example, if the output of the gaze module 308 changes, the content generation module 310 changes the display form (e.g., shape) of the virtual content.

[0102] Display module 312 presents virtual content on transparent display 204. This may include virtual content designed to enhance physical objects visible through transparent display 204. In some embodiments, display module 312 calculates a hologram of the output of content generation module 310. Display module 312 may present virtual content based on the position of the physical object corresponding to the virtual content. For example, display module 312 may present virtual content at a display position on transparent display 204 such that the virtual content appears to the user to overlap with and / or be close to the physical object.

[0103] Display module 312 continuously updates the presentation of virtual content on transparent display 204. For example, when the depth and / or position of physical object 104 changes, display module 312 updates the display of virtual content. Therefore, the virtual content appears to be part of the user's real-world environment and is paired with its corresponding physical object. In some embodiments, the hologram is calculated in real time.

[0104] Figures 4A to 4E Embodiments of this disclosure are shown by way of example only. Figures 4A to 4E The upper boundary 401 and lower boundary 403 of the field of view 405 defined by the vehicle windshield are shown. Figures 4A to 4E Also shown is a cyclist 407 (which corresponds to the previously described physical object) and another car 409 moving toward the cyclist 407. Figure 4A The image shows events prior to the system detecting the cyclist. Figure 4B A virtual content 411 (with a diamond shape in this example) projected by the projection system of this disclosure is shown. The virtual content 411 corresponds to a cyclist 407. In this example, the virtual content 411 draws the user's attention to the cyclist 407 at the intersection. Figure 4C The car 409 is shown turning towards the cyclist 407, but this does not obstruct the user's view of the cyclist 407. Therefore, the virtual content 411 remains diamond-shaped. As the car 409 is further manipulated, it moves to one or more positions that obstruct the user's view of the cyclist 407. Figure 4D This is detected by the system disclosed herein, and in response, the physical form of the virtual content 411' is changed. The changed virtual content 411' may take the form of a blind spot icon, which is familiar to those skilled in the art of driver assistance systems. As the car is maneuvered away from the cyclist, visibility is restored, and optionally, the virtual content 411 returns to its original form. Figure 4E Therefore, it can be understood that when the corresponding line of sight is clear, the virtual content has a first form 411, and when the line of sight is unclear, the virtual content has a second form 411'. These features provide the user with a visual warning that a physical object is hidden behind the car 409 and cannot be seen. To avoid doubt, in Figure 4D In the image, the cyclist is invisible because he is behind the car (from the observer's perspective), but the virtual content 411' is visible because it is projected onto the real-world scene.

[0105] Additional features

[0106] The methods and processes described herein can be embodied on a computer-readable medium. The term "computer-readable medium" includes media arranged for temporary or permanent storage of data, such as random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, and cache memory. The term "computer-readable medium" should also be considered to include any medium or combination of media capable of storing instructions for machine execution, such that when the instructions are executed by one or more processors, the machine performs, wholly or partially, any or all of the methods described herein.

[0107] The term "computer-readable medium" also covers cloud-based storage systems. The term "computer-readable medium" includes, but is not limited to, one or more tangible and non-transitory data repositories (e.g., data volumes) in the example forms of solid-state storage chips, optical discs, disks, or any suitable combinations thereof. In some example embodiments, instructions for execution may be transmitted by a carrier medium. Examples of such carrier media include transient media (e.g., propagation signals for transmitting instructions).

[0108] It will be apparent to those skilled in the art that various modifications and variations can be made without departing from the scope of the appended claims. This disclosure covers all modifications and variations within the scope of the appended claims and their equivalents.

Claims

1. A head-up display for a vehicle, the head-up display comprising: A projector, arranged to project images visible from the eye box; The processor is configured to receive a captured image of the scene visible from the eyebox, and, in a first instant: detect a first object in the scene, and drive an image projector to project an icon that appears substantially aligned with the first object from the viewing position. The processor is further configured to: detect a second object in the line of sight from the observation position to the first object at a second time later than the first time; continuously receive the observer's observation position; determine whether the second object is in the line of sight based on the received observation position; and, in response to detecting the second object, drive the image projector to change the shape of the projected icon and maintain the positional alignment between the projected icon and the first object based on the received observation position.

2. The head-up display as claimed in claim 1, wherein, The at least one physical form of the projected icon is selected from the group consisting of color, size, and brightness.

3. The head-up display as claimed in any of the preceding claims, wherein, The image projector is a holographic projector, which includes a spatial light modulator arranged to display a hologram of the projected image content.

4. The head-up display as claimed in any of the preceding claims, wherein, An optical combiner is used to project the image content so that the image content supplements / adds / covers the scene visible from the eyebox.

5. The head-up display as claimed in claim 1, wherein, The first object is a moving object, and the processor is arranged to determine whether the second object is in the line of sight to the first object based on the position of the first object at the second time and optionally the observation position of an observer at the second time.

6. A driver assistance system, comprising: The head-up display as described in any of the preceding claims; A camera, which is arranged to capture images of the scene and continuously outputs the captured images to a head-up display; as well as A user tracking system is configured to monitor the position of the user on the head-up display and continuously output the observer's position to the head-up display.

7. A method of using a head-up display, comprising: Capture the first image of the scene in real time; Detect the first object in the first image of the scene; The projected icon appears to be roughly aligned with the first object from the viewing position; Capture a second image of the scene at a second time. Detect the second object in the line of sight from the observer to the first object; The observation position of the observer is continuously received; Determine whether the second object is in the line of sight based on the received observation position; as well as In response to the detection of a second object, the shape of the projected icon is changed, and the positional alignment between the projected icon and the first object is maintained based on the received observation position.

8. The head-up display method as claimed in claim 7, wherein, The at least one physical form of the projected icon is selected from the group consisting of color, size, and brightness.

Images