Processing apparatus and display system
Through the pupil expander and processing device, the problems of limited light angle range and polarization elements in the existing display system are solved, and a wider observation area and image quality improvement is achieved.
Patent Information
- Application Number
- CN202380025871.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-09
- Filing Date
- 2023-03-07
- Publication Date
- 2024-11-15
AI Technical Summary
The existing display systems are limited in the angle range of propagating light, which makes it possible for observers to see images in a small range, and the presence of polarization elements or filters will affect the image quality and make it difficult to effectively correct.
Using a pupil expander and processing device, the light angle range is increased through the waveguide pupil expander, and a correction data set is generated to correct the intensity distribution and color balance of the picture to ensure image quality according to the presence of polarization elements or filters.
The observation area is effectively expanded, allowing users to see the image in a larger range, and improve image quality when the polarization element or filter exists, ensuring image uniformity and color balance.
Smart Images

Figure CN118974632A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to processing devices for use in display systems, in particular for display systems using diffracted light fields including diverging light beams. More specifically, the present disclosure relates to processing devices for display systems including waveguide pupil expanders, and to pupil expansion methods using waveguides. Some embodiments relate to picture generation units and heads-up displays, such as automotive heads-up displays (HUDs). Background Art
[0002] Light scattered from an object contains both amplitude and phase information. This amplitude and phase information can be captured, for example, on a photographic plate by well-known interference techniques to form a holographic recording or "hologram" comprising interference fringes. The hologram can be reconstructed by illumination with appropriate light to form a two-dimensional or three-dimensional holographic reconstruction or replay image representing the original object.
[0003] Computer generated holography can numerically simulate interference processes. Computer generated holograms can be calculated by techniques based on mathematical transformations such as Fresnel or Fourier transforms. These types of holograms may be referred to as Fresnel / Fourier transform holograms or simply Fresnel / Fourier holograms. Fourier holograms may be considered as Fourier domain / plane representations of objects or frequency domain / plane representations of objects. Computer generated holograms may also be calculated, for example, by coherent ray tracing or point cloud techniques.
[0004] The computer generated hologram may be encoded on a spatial light modulator arranged to modulate the amplitude and / or phase of the incident light. For example, the light modulation may be achieved using electrically addressable liquid crystals, optically addressable liquid crystals or micromirrors.
[0005] A spatial light modulator typically includes a plurality of individually addressable pixels, which may also be referred to as cells or elements. The light modulation scheme may be binary, multi-level, or continuous. Alternatively, the device may be continuous (i.e., not including pixels), and thus the light modulation may be continuous across the device. A spatial light modulator may be reflective, meaning that the light is modulated to be output in reflection. A spatial light modulator may also be transmissive, meaning that the light is modulated to be output in transmission.
[0006] Using the system described here, a holographic projector can be provided. Such projectors are already used in head-up displays "HUD". Summary of the invention
[0007] Various aspects of the disclosure are defined in the accompanying independent claims.
[0008] In general, the present disclosure relates to image projection. It relates to a method of image projection and an image projector including a display device. The present disclosure also relates to a projection system including an image projector and an observation system, wherein the image projector projects or relays light from the display device to the observation system. The present disclosure is equally applicable to monocular and binocular observation systems. The observation system may include one or more eyes of an observer. The observation system includes an optical element with optical power (e.g., a lens of a human eye) and an observation plane (e.g., a retina of a human eye). The projector may be referred to as a "light engine". The display device and the image formed (or perceived) using the display device are spatially separated from each other. The observer forms or perceives the image on the display plane. In some embodiments, the image is a virtual image, and the display plane may be referred to as a virtual image plane. In other embodiments, the image is a real image formed by holographic reconstruction, and the image is projected or relayed to the observation plane. The image is formed by illuminating a diffraction pattern (e.g., a hologram) displayed on the display device.
[0009] The display device includes pixels. The pixels of the display can display a diffraction pattern or structure of diffracted light. The diffracted light can form an image on a plane that is spatially separated from the display device. According to well-known optical principles, the magnitude of the maximum diffraction angle is determined by the size of the pixel and other factors such as the wavelength of the light.
[0010] In an embodiment, the display device is a spatial light modulator, such as a liquid crystal on silicon ("LCOS") spatial light modulator (SLM). Light propagates from the LCOS to a viewing entity / system such as a camera or eye within a range of diffraction angles (e.g., from zero to a maximum diffraction angle). In some embodiments, amplification techniques may be used to increase the range of available diffraction angles beyond the conventional maximum diffraction angle of the LCOS.
[0011] In some examples, the image (formed by the displayed hologram) is propagated to the eye. For example, spatially modulated light of an intermediate holographic reconstruction / image formed in free space between the display device and the observer or on a screen or other light receiving surface can be propagated to the observer.
[0012] In some other examples, the hologram itself (the light of) is transmitted to the eye. For example, the spatially modulated light of the hologram (not yet fully converted to a holographic reconstruction, i.e., an image) - which can be informally referred to as being "encoded with / by the hologram" - is transmitted directly to the observer's eye. The observer can perceive a real or virtual image. In these embodiments, no intermediate holographic reconstruction / image is formed between the display device and the observer. It is sometimes said that in these embodiments, the lens of the eye performs the conversion or transformation of the hologram to the image. The projection system or light engine can be configured so that the observer effectively looks directly into the display device.
[0013] The "light field" mentioned here is a "composite light field". The term "light field" simply means a light pattern with finite size in at least two orthogonal spatial directions, such as x and y. The term "light field" simply means a light pattern with finite size in at least two orthogonal spatial directions (x and y). The word "complex" used here simply means that the light at each point in the light field can be defined by an amplitude value and a phase value, and can therefore be represented by a complex number or a pair of values. For the purpose of hologram calculation, the composite light field can be a two-dimensional array of complex numbers, where the complex numbers define the intensity and phase of light at multiple discrete locations within the light field.
[0014] According to well-known optical principles, the angular range of light propagating from a display device that can be observed by an eye or other viewing entity / system varies with the distance between the display device and the viewing entity. For example, at a viewing distance of 1 meter, only a small range of angles from the LCOS can propagate through the pupil of the eye to form an image on the retina at a given eye position. The angular range of light propagating from the display device that can successfully propagate through the pupil of the eye to form an image on the retina at a given eye position determines the portion of the image that is "visible" to the observer. In other words, not all portions of the image are visible from any one point on the viewing plane (e.g., any one eye position within a viewing window such as an eye box).
[0015] In some embodiments, the image perceived by the observer is a virtual image that appears upstream of the display device, that is, the observer perceives the image as being farther away from them than the display device. Thus, conceptually, the observer can be thought of as observing the virtual image through a "window the size of the display device," which can be very small, such as 1 cm in diameter, and at a relatively large distance, such as 1 meter. And the user will be observing the display device-sized window through the pupil of their eye, which can also be very small. Thus, at any given time, the field of view becomes smaller, and the specific range of angles that can be seen depends heavily on the eye position.
[0016] Pupil expanders address how to increase the angular range of light propagating from a display device and that light can successfully propagate through the pupil of the eye to form an image. Display devices are typically (relatively) small and have (relatively) large projection distances. In some embodiments, the projection distance is at least one order of magnitude, for example at least two orders of magnitude, larger than the diameter or width of the entrance pupil and / or aperture (i.e., the size of the pixel array) of the display device. Embodiments of the present disclosure relate to a configuration in which a hologram of an image is transmitted to the human eye rather than the image itself. In other words, the light received by the observer is modulated according to the hologram of the image. However, other embodiments of the present disclosure may relate to configurations in which an image rather than a hologram is transmitted to the human eye—for example, through so-called indirect observation, in which light of a holographic reconstruction or "replayed image" formed on a screen (or even in free space) is transmitted to the human eye.
[0017] The use of a pupil expander increases the viewing area (i.e., the user's eye box) laterally, thereby enabling some eye movement while still enabling the user to see the image. As will be appreciated by the skilled artisan, in an imaging system, the viewing area (the user's eye box) is the area where the viewer's eye can perceive the image. The present disclosure relates to non-infinite virtual image distances - i.e., near-field virtual images.
[0018] Traditionally, a two-dimensional pupil expander comprises one or more one-dimensional optical waveguides, each formed using a pair of opposing reflective surfaces, wherein the output light from one surface forms an observation window - such as an eye box or eye box for an observer to observe. Light received from a display device (such as spatially modulated light from an LCOS) is replicated by the or each waveguide to increase the field of view (or observation area) in at least one dimension. In particular, the waveguide enlarges the observation window because additional light or "replicas" are generated by dividing the amplitude of the incident wavefront.
[0019] The display device may have an active or pixel display area having a first dimension of less than 10 cm, such as less than 5 cm or less than 2 cm. The propagation distance between the display device and the viewing system may be greater than 1 m, such as greater than 1.5 m or greater than 2 m. The optical propagation distance within the waveguide may be up to 2 m, such as up to 1.5 m or up to 1 m. The method is capable of receiving an image and determining a corresponding hologram of sufficient quality in less than 20 ms, such as less than 15 ms or less than 10 ms.
[0020] In some embodiments described only by examples of diffraction or holographic light fields according to the present disclosure, the hologram is configured to route light into multiple channels, each channel corresponding to a different portion (i.e., sub-region) of the image. The hologram can be presented on a display device such as a spatial light modulator, such as a display. When displayed on an appropriate display device, the hologram can spatially modulate the light that can be converted into an image by the observation system. The channels formed by the diffraction structure are referred to here as "hologram channels" simply to reflect that they are light channels encoded by holograms with image information. It can be said that the light of each channel is in the holographic domain, rather than in the image or spatial domain. In some embodiments, the hologram is a Fourier or Fourier transform hologram, so the hologram domain is the Fourier or frequency domain. The hologram can also be a Fresnel or Fresnel transform hologram. The hologram is described herein as routing light into multiple hologram channels simply to reflect that the image that can be reconstructed from the hologram has a finite size and can be arbitrarily divided into multiple image sub-regions, where each hologram channel will correspond to each image sub-region. Importantly, the hologram of this example is characterized by how it distributes the image content when illuminated. Specifically, the hologram divides the image content by angle. That is, each point on the image is associated with a unique ray angle in the spatially modulated light formed by the hologram when illuminated - at least a unique pair of angles, because the hologram is two-dimensional. For the avoidance of doubt, this holographic behavior is not conventional. When illuminated, the spatially modulated light formed by this special type of hologram can be arbitrarily divided into multiple hologram channels, each of which is defined by a range of ray angles (in two dimensions). It can be understood from the above that any hologram channel (i.e., a sub-range of ray angles) that can be considered in the spatially modulated light will be associated with a corresponding portion or sub-region of the image. That is, all the information required to reconstruct that portion or sub-region of the image is contained within the angular sub-range of the spatially modulated light formed by the hologram of the image. When the spatially modulated light is observed as a whole, there is not necessarily any evidence of multiple discrete light channels. However, in some arrangements, multiple spatially separated hologram channels are formed by intentionally leaving areas of the target image from which the hologram is calculated blank or empty (ie, without image content).
[0021] Nevertheless, the hologram can still be recognized. For example, if only continuous portions or sub-regions of the spatially modulated light formed by the hologram are reconstructed, only sub-regions of the image should be visible. If different continuous portions or sub-regions of the spatially modulated light are reconstructed, different sub-regions of the image should be visible. Another identifying feature of this type of hologram is that the shape of the cross-section of any hologram channel substantially corresponds to the shape of the entrance pupil (i.e., substantially the same), although the size may be different—at least at the correct plane where the hologram is calculated. Each light / hologram channel propagates from the hologram at a different angle or range of angles. Although these are example ways to characterize or identify this type of hologram, other ways may also be used. In summary, the holograms disclosed herein are characterized and identified by how the image content is distributed within the light encoded by the hologram. In addition, to avoid any doubt, references herein to holograms configured to guide light or to angularly divide an image into multiple hologram channels are only by way of example, and the present disclosure is equally applicable to any type of holographic light field or even any type of diffraction or pupil expansion of a diffracted light field.
[0022] In summary, a processing device is disclosed herein that corrects an image received by a user at an eye box when a polarizing element or filter obscures the user's eyes. The processing device receives a signal indicating the presence of a polarizing element or filter at the eye box. The processing device determines a correction data set for an image to be displayed to the user based on the signal. The processing device outputs the correction data set to a display system to display the image to the user.
[0023] The use of such a processing device allows the picture to be corrected based on the presence (or absence) of a polarizing element or filter. A polarizing element or filter may cause a change (e.g., a decrease) in the image quality perceived by the user because the intensity of the light reaching the eyebox is a complex function of the angle of incidence on the windshield and the polarization direction of the light received by the combiner on the optical path to the eyebox (i.e., the light received from the picture generation unit). In some embodiments, the light received at the eyebox has a p-polarization component and an s-polarization component. In an embodiment, due to the geometry of the system, the s-polarization component is dominant. That is, the intensity of the s-polarization component reaching the eyebox is greater than the intensity of the p-polarization component. This is true regardless of the polarization direction of the light emitted by the picture generation unit. For convenience, it is sometimes said that there is a dominant polarization state. The present disclosure relates to a curved and angled combiner. In some embodiments, the combiner is a vehicle windshield that is tilted and curved relative to the forward direction. That is, the windshield is not directly in front of the driver.
[0024] The combiner may have a so-called "tilt angle" and the projection system has a field of view. The angle of incidence of the picture light received by the combiner is a function of the position within the field of view. For example, the angle of incidence of the picture light appearing at the bottom of the field of view may be, for example, 65 degrees, and the angle of incidence of the picture light appearing at the top of the field of view may be, for example, 55 degrees. Due to the range of angles of incidence associated with the field of view and the dependence of polarization on reflectivity, the intensity distribution of the field of view is distorted.
[0025] Image compensation may be performed. In examples involving propagation of a hologram to an eye box, the inventors have discovered that a correction should be made to the target image prior to or as part of the hologram calculation. Image compensation may be based on the eye box position. This is because picture light received at different positions within the eye box undergoes a different number of internal reflections within the waveguide pupil expander. Since internal reflections within the pupil expander are inherently lossy, this results in variations in light intensity at different eye box positions. Furthermore, such variations may depend on the wavelength of the light, as described below. Typically, image compensation is based on a preferred or dominant polarization state directed to the eye box by a combiner. If a polarizing element is incorporated at the eye box (e.g., by an observer), image compensation based on prior art techniques may be ineffective or counterproductive.
[0026] In some embodiments, the correction data set is a first set of values if the signal indicates the presence of a polarizing element or filter and a second set of values when there is no indication of a polarizing element or filter in the eye box. The second set of values is different from the first set of values. The first set of values may apply a correction to each pixel to be displayed, while the second set of values may not apply any correction, thereby applying a factor of 1 to all pixels. Thus, each value of the correction data set may include a scaling factor for the corresponding image pixel to correct or compensate for the properties of components of the projection or relay system of the display system (e.g., an optical combiner and a pupil expander), and in the case of the first set of values, the presence of a polarizing element or filter at the eye box. Those skilled in the art will appreciate that the correction data set may be applied to any target image before the display system performs hologram calculations and their display.
[0027] In some embodiments, the signal indicates the presence of a polarizing element or filter, and the correction data set modifies the intensity distribution of the image so that the image observed by the user has an expected intensity distribution after filtering by the polarizing element or filter. The expected intensity distribution may be substantially uniform.
[0028] In some embodiments, the correction data set includes at least one of a brightness correction function or distribution and a color balance correction function or distribution.
[0029] In summary, an augmented reality display system is disclosed. The augmented reality display system includes a display device, a combiner, and a processing device. The display device can be configured to receive visible light from a visible light source and modify the light so that light carrying a picture is relayed from the display device to a user at an eye box. The combiner can be configured to receive light carrying a picture from the display device and combine it with an external scene to present an augmented reality scene to the user. The processing device receives a signal indicating the presence of a polarizing filter in the eye box. The processing device determines a correction data set for the picture to be displayed to the user based on the signal. The correction data set can depend on the user's eye box position. The processing device outputs the correction data set to the display system to display the picture to the user.
[0030] The combiner may have different reflectivities for different polarization states. The processing means may be configured to modify the correction data set (or select a different correction data set) based on the reflectivities for different polarization states and the presence of a polarization filter to improve the quality of the image seen by the user.
[0031] In summary, a user tracking system, such as an eye tracking system, is disclosed. The eye tracking system includes a sensor that images at least one eye of a user, the eye being located at an eye box of a display system. The eye tracking system also includes a light source that illuminates the user's eye. The light source operates in a first mode and a second mode, the first mode using light of a first polarization state and the second mode using light of a second polarization state. The display system, such as a processing device thereof, receives a signal indicating the presence of a polarization filter in the eye box. The processing device determines a correction data set for a picture to be displayed to the user based on the signal. The correction data set can be further determined based on a position of the user's eye within the eye box sensed by the eye tracking system. The processing device outputs the correction data set to the display system to display the picture to the user.
[0032] The first polarization state may include a polarization state of light (e.g., a linear polarization state of light), and the second polarization state may include substantially unpolarized light. The first polarization state may enable the system to detect the presence of a polarizing filter in the eye box. The second polarization state is used to detect the eye regardless of whether a polarizing filter is present.
[0033] The duration of the first state may be less than the second state. The duration of the first state may be less than 10% of the duration of the entire cycle. The duration of the first state may be less than 5% of the duration of the entire cycle. The duration of the first state may be less than 1% of the duration of the entire cycle. This increases the time the system is able to track the eye.
[0034] The first polarization state may be horizontally polarized, thereby being blocked by typical polarized sunglasses (which are typically vertically polarized). The second polarization state may be substantially vertically polarized light. The first and second polarization states may be opposite or perpendicular polarization states (eg, linear polarization states).
[0035] Eye tracking systems can be used in vehicles.
[0036] Augmented reality systems can be used as part of a heads-up display.
[0037] In general, the present disclosure relates to a method of modifying, correcting or improving a picture from the perspective of an observer. The method includes: receiving an indication of the presence of a polarizing filter; determining a correction data set of the picture to be displayed to a user of a display system based on the indication of the presence of the polarizing filter; and outputting the correction data set to the display system to display the picture in an eye box using the correction data set.
[0038] The core projection system disclosed herein utilizes user tracking, such as eye tracking, to improve the augmented reality experience. Notably, a system for using a user tracking system for additional and supplementary purposes is disclosed herein. In some embodiments, the user tracking system operates in additional different operating modes. For example, the first operating mode may include traditional user tracking, and according to the present disclosure, the second operating mode may include detecting whether the user is wearing polarized glasses. The second operating mode includes operating the user tracking system in a manner different from the first operating mode. The user tracking system illuminates a user area (e.g., an observation window or an eye box) with light such as invisible light (e.g., infrared light). In some embodiments, the second operating mode of the user tracking system includes changing the properties of the light illuminating the user area, or using a different light source that emits light with a different property than the light used in the first operating mode. The property may be polarization. In the first operating mode, the user tracking system may illuminate the user area with invisible light having a first polarization state (e.g., a first linear polarization direction), and in the second operating mode, the user tracking system may illuminate the user area with invisible light having a second polarization state (e.g., a second linear polarization direction). According to the present disclosure, in the second operating mode, an image of the user area captured by the user tracking system is analyzed to determine or infer the polarization state of the light received by the observer.
[0039] The system can be provided in a compact and streamlined physical form. This makes the system suitable for a wide range of practical applications, including those where space is limited and asset value is high. For example, it can be implemented in a head-up display (HUD), such as a vehicle or automotive HUD.
[0040] According to the present disclosure, pupil expansion is provided for diffracted light, which may include diverging bundles of light rays. The diffracted light may be output by a display device, such as a pixelated display device, such as a spatial light modulator (SLM) arranged to display a diffractive structure (e.g., a hologram). The diffracted light field may be defined by a "light cone". Thus, the size of the diffracted light field (as defined in a two-dimensional plane) increases with the propagation distance from the corresponding diffractive structure (i.e., the display device).
[0041] The spatial light modulator may be arranged to display a hologram. The diffracted or diverged light may include light encoded with / by the hologram, rather than light reconstructed from the image or hologram. In such an embodiment, it may therefore be said that the pupil expander replicates the hologram or forms at least one replica of the hologram to convey that the light delivered to the observer is spatially modulated according to the hologram of the image, rather than the image itself. That is, the diffracted light field is propagated to the observer.
[0042] In some embodiments, two one-dimensional waveguide pupil expanders are provided, each of which is arranged to effectively increase the size of the exit pupil of the system by forming multiple copies or replicas of the exit pupil (or the light of the exit pupil) of the spatial light modulator. The exit pupil can be understood as the physical area where the system outputs light. It can also be said that each waveguide pupil expander is arranged to expand the size of the exit pupil of the system. It can also be said that each waveguide pupil expander is arranged to expand / increase the size of the eye box in which the observer's eyes can be located in order to see / receive the light output by the system.
[0043] In the present disclosure, the term "replica" is used only to reflect that the spatially modulated light is split so that the composite light field is directed along multiple different optical paths. The term "replica" is used to refer to each occurrence or instance of the composite light field after a replication event - such as a partial reflection-transmission of a pupil expander. Each replica propagates along a different optical path. Some embodiments of the present disclosure are related to the propagation of light encoded with a hologram rather than an image - that is, light that is spatially modulated with a hologram of an image rather than the image itself. Those skilled in the art of holography will understand that the composite light field associated with the propagation of light encoded with a hologram will vary with the propagation distance. The term "replica" as used herein is independent of the propagation distance, so two light branches or paths associated with a replication event are still referred to as "replicas" of each other, even if the branches have different lengths, so that the composite light field evolves differently along each path. That is, according to the present disclosure, even if two composite light fields are associated with different propagation distances, they are still considered "replicas" - assuming that they originate from the same replication event or series of replication events.
[0044] According to the present disclosure, a "diffraction light field" is a light field formed by diffraction. A diffraction light field can be formed by illuminating a corresponding diffraction pattern. According to the present disclosure, an example of a diffraction pattern is a hologram, and an example of a diffraction light field is a holographic light field or a light field of a holographic reconstruction that forms an image. The holographic light field forms a (holographic) reconstruction of an image on a playback plane. The holographic light field propagated from the hologram to the reproduction plane can be said to include light encoded with the hologram or light in the holographic domain. The diffraction light field is characterized by a diffraction angle determined by the minimum characteristic size of the diffraction structure and the wavelength of the light (of the diffraction light field). According to the present disclosure, it can also be said that the "diffraction light field" is a reconstructed light field formed on a plane spatially separated from the corresponding diffraction structure. An optical system for propagating a diffraction light field from a diffraction structure to an observer is disclosed herein. The diffraction light field can form an image.
[0045] Reference is made here to different "correction data sets" that are used to modify a picture (e.g., image data representing a pixel array of a picture) before a hologram is calculated and / or projected. Each correction data set includes a digital array, where each pixel of the image has a number that can be modified using the correction data set. The correction data set can be an array of pixel correction values. The array of pixel correction values can be the same size as the pixel array of the picture. Therefore, there may be a 1:1 correlation between the pixels of the correction data set and the pixels of the picture. Each pixel correction value can correspond to one pixel of the picture. However, those skilled in the art of image processing will understand that the correction data set can take other forms. The correction data set can compensate for variations introduced by optical components of the projection system (e.g., an optical combiner or a pupil expander) between the display device and the eye box of the optical system. For example, each pixel value of the correction data set can be calculated based on local properties of an optical combiner (e.g., a vehicle windshield) used as part of the projection system. In some embodiments described herein, the reflectivity of the optical components of the projection system (e.g., an optical combiner such as a vehicle windshield) is actually non-uniform due to different parts of the picture having different angles of incidence on the optical components. For example, the correction data set can be configured to correct for the non-uniform reflectivity. In some embodiments described herein, the reflectivity of the optical components of the projection system (e.g., a pupil expander) is wavelength dependent, resulting in an uneven color balance of a multicolor image (e.g., a monochromatic image including red, green, and blue). The correction data set can be configured to correct for the uneven color balance. The skilled person will appreciate that a variety of different measurement techniques or mathematical calculations using, for example, the Fresnel equations can be used to determine the reflectivity at multiple different points on the optical components. The present disclosure is not limited to any particular method of determining the correction data set.
[0046] In some embodiments, a first correction data set is used in a first operating mode (i.e., applied to the picture before further processing and / or projection), and a second (i.e., different) correction data set is used in a second operating mode. In some embodiments, the first correction data set is used in the first operating mode, and in the second operating mode, a correction data set according to the present disclosure is not used. In some embodiments, the operating mode is related to polarization, such as the polarization of the picture light. In some embodiments, the first operating mode is used for light with a first linear polarization direction (e.g., p-polarization) and the second operating mode is used for light with a second linear polarization direction (e.g., s-polarization), wherein the first direction and the second direction are perpendicular. In some embodiments, the operating mode is selectable by a user. In other embodiments, the operating mode is detected by the system. The skilled person will be familiar with the idea that a correction data set can be mathematically "applied to" or "used on" the picture by, for example, multiplication, such as multiplying each pixel value of the picture by the corresponding or corresponding pixel value of the correction data set. The selected correction data set can be used to modify the picture before other image processing steps and / or before calculating a hologram of the picture for the holographic projection system.
[0047] The term "hologram" is used to refer to a record containing amplitude information or phase information or some combination thereof about an object. The term "holographic reconstruction" is used to refer to the optical reconstruction of an object formed by illuminating a hologram. The system disclosed herein is described as a "holographic projector" because the holographic reconstruction is a real image and is spatially separated from the hologram. The term "replay field" is used to refer to the 2D area in which the holographic reconstruction is formed and is fully focused. If the hologram is displayed on a spatial light modulator including pixels, the replay field will be repeated in the form of multiple diffraction orders, each of which is a replica of the zero-order replay field. The zero-order replay field usually corresponds to the preferred or main replay field because it is the brightest replay field. Unless otherwise explicitly stated, the term "replay field" should be considered to refer to the zero-order replay field. The term "replay plane" is used to refer to a plane in space that contains all replay fields. The terms "image", "replay image" and "image area" refer to the area of the replay field illuminated by the light of the holographic reconstruction. In some embodiments, an "image" may include discrete points, which may be referred to as "image points," or, for convenience only, as "image pixels."
[0048] The terms "encoding", "writing" and "addressing" are used to describe the process of providing a plurality of pixels of the SLM with a corresponding plurality of control values that respectively determine the modulation level of each pixel. It can be said that the pixels of the SLM are configured to "display" a light modulation distribution in response to receiving the plurality of control values. Thus, the SLM can be said to "display" a hologram, and a hologram can be considered to be an array of light modulation values or levels.
[0049] It has been found that holographic reconstructions of acceptable quality can be formed from "holograms" that contain only phase information related to the Fourier transform of the original object. Such holographic recordings may be referred to as phase-only holograms. The embodiments relate to phase-only holograms, but the disclosure is equally applicable to amplitude-only holography.
[0050] The present disclosure is also equally applicable to forming a holographic reconstruction using amplitude and phase information related to the Fourier transform of the original object. In some embodiments, this is achieved by using complex modulation of a so-called fully complex hologram that contains amplitude and phase information related to the original object. Because the value (gray level) assigned to each pixel of the hologram has an amplitude and a phase component, such a hologram may be referred to as a fully complex hologram. The value (gray level) assigned to each pixel can be represented as a complex number with an amplitude and a phase component. In some embodiments, a fully complex computer-generated hologram is calculated.
[0051] Reference may be made to the phase value, phase component, phase information, or simply phase, of a pixel of a computer-generated hologram or spatial light modulator as shorthand for "phase delay". That is, any phase value described is actually a number (e.g., in the range of 0 to 2π) representing the amount of phase delay provided by that pixel. For example, a pixel of a spatial light modulator described as having a phase value of π / 2 will delay the phase of the received light by π / 2 radians. In some embodiments, each pixel of the spatial light modulator may be operated at one of a plurality of possible modulation values (e.g., phase delay values). The term "grayscale" may be used to refer to a plurality of available modulation levels. For example, the term "grayscale" may be used for convenience to refer to a plurality of available phase levels in a phase modulator alone, even if the different phase levels do not provide different shades of gray. For convenience, the term "grayscale" may also be used to refer to a plurality of available complex modulation levels in a complex modulator.
[0052] Thus, a hologram comprises an array of gray levels, i.e. an array of light modulation values, such as phase delay values or an array of complex modulation values. A hologram is also considered to be a diffraction pattern, since it is a pattern that causes diffraction when displayed on a spatial light modulator and illuminated with light of a wavelength relative to (usually less than) the pixel pitch of the spatial light modulator. Reference is made herein to combining a hologram with other diffraction patterns, such as a diffraction pattern used as a lens or a grating. For example, a diffraction pattern used as a grating may be combined with a hologram to shift the replay field on the replay plane, or a diffraction pattern used as a lens may be combined with a hologram to focus the holographic reconstruction on the replay plane in the near field.
[0053] Although different embodiments and embodiment groups may be disclosed separately in the following detailed description, any feature of any embodiment or embodiment group may be combined with any other feature or combination of features of any embodiment or embodiment group. That is, all possible combinations and permutations of the features disclosed in this disclosure are contemplated.
[0054] In the present disclosure, the term "substantially" when applied to a structural unit of a device may be interpreted as technical features of the structural unit produced within the technical tolerance of a method for manufacturing the structural unit. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Specific embodiments are described by way of example only with reference to the following drawings:
[0056] Figure 1 is a schematic diagram showing a reflective SLM producing a holographic reconstruction on a screen;
[0057] Figure 2 An image for projection is shown, comprising eight image regions / components V1 to V8;
[0058] Figure 3 shows a hologram displayed on an LCOS that directs light to multiple discrete areas;
[0059] Figure 4 Shown includes display such as Figure 2 and 3 A system for displaying a computer generated hologram as shown;
[0060] FIG5 shows a perspective view of a system comprising two replicators arranged to expand a light beam in two dimensions;
[0061] Fig. 6A shows a configuration according to an embodiment wherein the windshield is tilted relative to the forward direction of the observer;
[0062] Figure 6B and 6C shows the intensity of horizontally and vertically polarized light received by the observer;
[0063] Fig. 7A and 7B The perceived brightness distribution of the display observed with and without the polarizing filter is shown.
[0064] Fig. 8A and 8B The desired perceived brightness distribution of the display is shown, along with the corrected distribution observed in the presence of a polarizing filter.
[0065] Fig. 9A and 9B It shows the user's eyes being detected and not being detected.
[0066] Fig.10 A processing device according to some embodiments is shown.
[0067] Fig.11 An augmented reality display system is shown in accordance with some embodiments.
[0068] Fig.12 An eye tracking system in accordance with some embodiments is shown.
[0069] Fig.13 Methods according to some embodiments are shown.
[0070] The same reference numbers will be used throughout the drawings to refer to the same or like parts. DETAILED DESCRIPTION
[0071] The present invention is not limited to the embodiments described below, but extends to the full scope of the appended claims. That is, the present invention can be implemented in different forms and should not be construed as limited to the described embodiments, which are set forth for illustrative purposes.
[0072] Unless otherwise stated, terms in the singular may include plural forms.
[0073] A structure described as being formed on / under another structure or above / below another structure should be construed to include the case where the structures are in contact with each other and, further, also include the case where a third structure is provided therebetween.
[0074] When describing a temporal relationship, for example, when the temporal order of events is described as "after," "subsequently," "next," "before," etc., the present disclosure should be considered to include both sequential and non-sequential events unless otherwise specified. For example, unless terms such as "just," "immediately," or "directly" are used, the description should be considered to include non-sequential situations.
[0075] Although the terms "first", "second", etc. may be used herein to describe various elements, these elements are not limited by these terms. These terms are only used to distinguish between various elements. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element without departing from the scope of the appended claims.
[0076] The features of different embodiments may be coupled or combined with each other in part or in whole, and may interoperate with each other in different ways. Some embodiments may be performed independently of each other, or may be performed together in a mutually dependent relationship.
[0077] Optical Configuration
[0078] Figure 1An embodiment is shown in which a computer generated hologram is encoded on a single spatial light modulator. The computer generated hologram is a Fourier transform of the object used for reconstruction. Therefore, it can be said that the hologram is a Fourier domain or frequency domain or spectral domain representation of the object. In this embodiment, the spatial light modulator is a reflective liquid crystal on silicon "LCOS" device. The hologram is encoded on the spatial light modulator and a holographic reconstruction is formed at a playback field, such as a light receiving surface such as a screen or diffuser.
[0079] A light source 110, such as a laser or laser diode, is arranged to illuminate the SLM 140 via a collimating lens 111. The collimating lens causes a substantially planar wavefront of light to be incident on the SLM. Figure 1 In some embodiments, the direction of the wavefront is off-normal (e.g., two or three degrees from a plane that is truly normal to the transparent layer). However, in other embodiments, a substantially planar wavefront is provided at normal incidence and a beam splitter arrangement is used to separate the input and output optical paths. Figure 1 In the embodiment shown, the arrangement is such that light from the light source reflects from the mirrored rear surface of the SLM and interacts with the light modulating layer to form an exit wavefront 112. The exit wavefront 112 is applied to an optical device including a Fourier transform lens 120, the focus of which is located at a screen 125. More specifically, the Fourier transform lens 120 receives the modulated light beam from the SLM 140 and performs a frequency-space transform to produce a holographic reconstruction at the screen 125.
[0080] It is important to note that in this type of hologram, every pixel of the hologram contributes to the entire reconstruction. There is no one-to-one correlation between a specific point on the replay field (or image pixel) and a specific light modulation element (or hologram pixel). In other words, the modulated light leaving the light modulation layer is distributed over the entire replay field.
[0081] In these embodiments, the position of the holographic reconstruction in space is determined by the power (focus) of the Fourier transform lens. Figure 1 In the illustrated embodiment, the Fourier transform lens is a physical lens. That is, the Fourier transform lens is an optical Fourier transform lens and performs a Fourier transform optically. Any lens can act as a Fourier transform lens, but the performance of the lens will limit the accuracy of the Fourier transform it performs. The skilled person understands how to use a lens to perform an optical Fourier transform.
[0082] Hologram computing
[0083] In some embodiments, the computer-generated hologram is a Fourier transform hologram, or simply a Fourier hologram or a Fourier-based hologram, in which the image is reconstructed in the far field by exploiting the Fourier transform properties of a positive lens. The Fourier hologram is calculated by Fourier transforming the desired light field in the replay plane back to the lens plane. The computer-generated Fourier hologram can be calculated using the Fourier transform. By way of example only, the embodiments relate to Fourier holography and Gerchberg-Saxton type algorithms. The present disclosure is equally applicable to Fresnel holography and Fresnel holograms that can be calculated by similar methods. In some embodiments, the hologram is a phase or pure phase hologram. However, the present disclosure is also applicable to holograms calculated by other techniques, such as techniques based on point cloud methods. UK patent application GB2112213.0 filed on August 26, 2021 discloses an example hologram calculation method that can be combined with the present disclosure, and the patent application is incorporated herein by reference.
[0084] In some embodiments, a real-time engine is provided that is arranged to receive image data and calculate a hologram in real time using an algorithm. In some embodiments, the image data is a video comprising a sequence of image frames. In other embodiments, the hologram is pre-calculated, stored in a computer memory and called up as needed for display on the SLM. That is, in some embodiments, a repository of predetermined holograms is provided.
[0085] Light Modulation
[0086] The display system comprises a display device defining an exit pupil of the display system. The display device is a spatial light modulator. The spatial light modulation may be a phase modulator. The display device may be a liquid crystal on silicon "LCOS", a spatial light modulator.
[0087] Optical Channel
[0088] The optical systems disclosed herein are applicable to pupil expansion with any diffracted light field. In some embodiments, the diffracted light field is a holographic light field - that is, a composite light field that is spatially modulated according to a hologram of an image rather than the image itself. In some embodiments, the hologram is a special type of hologram that angularly divides / directs the image content. This type of hologram is further described herein merely as an example of a diffracted light field that is compatible with the present disclosure. Other types of holograms can be used in conjunction with the display systems and light engines disclosed herein.
[0089] A display system and method are described below that include a waveguide pupil expander. As will be familiar to the skilled reader, a waveguide may be configured as a "pupil expander" in that it may be used to increase the area over (or within) which light emitted by a relatively small light emitter (e.g., a relatively small SLM or other pixelated display device used in the apparatus described herein) is observable by a human observer or other observation system located at a distance (e.g., a relatively large distance) away from the light emitter. The waveguide achieves this by increasing the number of transmission points through which light is output to an observer. As a result, light may be seen from a number of different observer positions, e.g., an observer may move his or her head, and thereby his or her line of sight, while still being able to see light from the light emitter. Thus, it may be said that by using a waveguide pupil expander, the observer's "eye box" or "eye motion box" is enlarged. This has many useful applications, such as, but not limited to, heads-up displays, such as, but not limited to, automotive heads-up displays.
[0090] The display systems described herein may be configured to direct light, such as a diffracted light field, through a waveguide pupil expander so as to provide pupil expansion in at least one dimension, such as in two dimensions. The diffracted light field may include light output by a spatial light modulator (SLM) such as an LCOS SLM. For example, the diffracted light field may include light encoded by a hologram displayed by the SLM. For example, the diffracted light field may include light of a holographically reconstructed image corresponding to a hologram displayed by the SLM. The hologram may include a computer generated hologram (CGH), such as, but not limited to, a point cloud hologram, a Fresnel hologram, or a Fourier hologram. The hologram may be referred to as a "diffraction structure" or a "modulation pattern". The SLM or other display device may be arranged to display a diffraction pattern (or modulation pattern) in a manner familiar to a skilled reader, comprising a hologram and one or more other elements, such as a software lens or a diffraction grating.
[0091] The hologram can be calculated to provide guidance of the diffracted light field. This is described in detail in GB2101666.2, GB2101667.0 and GB2112213.0, all of which are incorporated herein by reference. In general, a hologram can be calculated to correspond to an image to be holographically reconstructed. The image to which the hologram corresponds may be referred to as an "input image" or "target image". The hologram can be calculated so that when it is displayed on the SLM and properly illuminated, it forms a light field (output by the SLM) comprising a spatially modulated light cone. In some embodiments, the cone comprises multiple continuous light channels of spatially modulated light, which correspond to corresponding continuous regions of the image. However, the present disclosure is not limited to this type of hologram.
[0092] Although referred to herein as "holograms" or "computer generated holograms (CGHs)", it should be understood that the SLM can be configured to dynamically display multiple different holograms in succession or according to a sequence. The systems and methods described herein are suitable for dynamic display of multiple different holograms.
[0093] Figure 2 and 3 An example of the type of hologram that can be displayed on a display device such as an SLM is shown, which can be used in conjunction with the pupil expander disclosed herein. However, this example should not be viewed as limiting the present disclosure.
[0094] Figure 2 An image 252 is shown for projection, comprising eight image regions / components V1 to V8. By way of example only, Figure 2 Eight image components are shown, and image 252 may be separated into any number of components. Figure 2 Also shown is a coded light pattern 254 (ie, a hologram) that can reconstruct the image 252—for example, when transformed by a lens of a suitable viewing system. The coded light pattern 454 includes first to eighth sub-holograms or components H1 to H8, corresponding to first to eighth image components / regions V1 to V8. Figure 2 It further shows how a hologram breaks down the image content by angle. Therefore, a hologram is characterized by its guidance of light. Figure 3 Specifically, the hologram in this example directs light into multiple discrete regions. In the example shown, the discrete regions are disks, but other shapes are also contemplated. After propagation through the waveguide, the size and shape of the optimal disk can be related to the size and shape of the entrance pupil of the viewing system.
[0095] Figure 4 A system 400 is shown, including a display such as Figure 2 and 3 A display device for a computed hologram is shown.
[0096] The system 400 comprises a display device which in this arrangement comprises an LCOS 402. The LCOS 402 is arranged to display a modulation pattern (or "diffraction pattern") comprising a hologram and projects light which has been holographically encoded to an eye 405 which comprises a pupil acting as an aperture 404, a lens 409 and a retina (not shown) acting as an observation plane. There is a light source (not shown) arranged to illuminate the LCOS 402. The lens 409 of the eye 405 performs the conversion of the hologram to an image. The light source may be of any suitable type. For example, it may comprise a laser source.
[0097] The viewing system 400 also includes a waveguide 408 located between the LCOS 402 and the eye 405. The presence of the waveguide 408 enables all angular content from the LCOS 402 to be received by the eye, even at the relatively large projection distance shown. This is because the waveguide 408 acts as a pupil expander in a well-known manner and is therefore only briefly described here.
[0098] in short, Figure 4 The waveguide 408 shown comprises a substantially elongated structure. In this example, the waveguide 408 comprises an optical plate of a refractive material, but other types of waveguides are also well known and can be used. The waveguide 408 is positioned to intersect the light cone (i.e., the diffracted light field) projected from the LCOS 402, for example, at an oblique angle. In this example, the size, position, and positioning of the waveguide 408 are configured to ensure that light from each of the eight light beams within the light cone enters the waveguide 408. The light from the light cone enters the waveguide 408 via a first planar surface of the waveguide 408 (positioned closest to the LCOS 402) and is guided at least partially along the length of the waveguide 408 before being emitted via a second planar surface of the waveguide 408 that is substantially opposite to the first surface (positioned closest to the eye). It is easy to understand that the second planar surface is partially reflective and partially transmissive. In other words, when each light ray propagates from the first planar surface within the waveguide 408 and strikes the second planar surface, some of the light will be transmitted out of the waveguide 408 and some of the light will be reflected back to the first planar surface by the second planar surface. The first planar surface is reflective so that all light striking it from within the waveguide 408 will be reflected back toward the second planar surface. Thus, some light may simply be refracted between the two planar surfaces of the waveguide 408 before being transmitted, while other light may be reflected and may therefore experience one or more reflections (or "bounces") between the planar surfaces of the waveguide 408 before being transmitted.
[0099] Figure 4 A total of nine "bounce" points B0 to B8 are shown along the length of waveguide 408. Figure 2 As shown, light associated with all points of the image (V1-V8) is transmitted out of the waveguide at each "bounce" from the second planar surface of the waveguide 408, but only light from one angular portion of the image (e.g., light from one of V1 to V8) has a trajectory that enables it to reach the eye 405 from each corresponding "bounce" point B0 to B8. In addition, light from different angular portions of the image (V1 to V8) reaches the eye 405 from each corresponding "bounce" point. Therefore, at Figure 4 In the example of FIG. 4 , each angular channel of coded light reaches the eye from waveguide 408 only once.
[0100] The above methods and apparatus can be implemented in a variety of different applications and viewing systems. For example, they can be implemented in a head-up display (HUD) or in a head or helmet mounted device (HMD) such as an augmented reality (AR) HMD.
[0101] Although virtual images have been discussed generally herein, which require the eye to convert received modulated light to form a perceived image, the methods and apparatus described herein can be applied to real images.
[0102] 2D pupil expansion
[0103] Although the arrangement shown in FIG5 includes a single waveguide that provides pupil expansion in one dimension, pupil expansion may be provided in more than one dimension, for example in two dimensions. Figure 4 The examples in use a hologram that has been calculated to create channels of light, each corresponding to a different portion of an image, but the present disclosure and the systems described below are not limited to this type of hologram.
[0104] FIG5 shows a perspective view of a system 500 comprising two replicators 504 , 506 arranged to expand a light beam 502 in two dimensions.
[0105] In the system 500 of FIG. 5 , the first replicator 504 includes a first pair of surfaces stacked parallel to each other, arranged in a manner similar to Figure 4 The first pair of surfaces are similar in size and shape to each other (in some cases identical), and are substantially elongated in one direction. A collimated light beam 502 is directed to an input on a first replicator 504. Due to internal reflection processes between the two surfaces, and partial transmission of light from each of a plurality of output points on one of the surfaces (the upper surface, as shown in FIG5 ), which will be familiar to the skilled reader, the light of the light beam 502 is replicated in a first direction along the length of the first replicator 504. Thus, a first plurality of replica light beams 508 are emitted from the first replicator 504 toward the second replicator 506.
[0106] The second replicator 506 includes a second pair of surfaces stacked parallel to each other, arranged to receive each collimated beam of the first plurality of beams 508, and further arranged to provide replication or pupil expansion by expanding each of these beams in a second direction substantially orthogonal to the first direction. The size and shape of the first pair of surfaces are similar to each other (in some cases the same), and are substantially rectangular. The rectangular shape is implemented for the second replicator so that it has a length along the first direction to receive the first plurality of beams 508, and has a length along the second orthogonal direction to provide replication in the second direction. Due to the internal reflection process between the two surfaces, and the partial transmission of light from each of the plurality of output points on one of the surfaces (the upper surface as shown in FIG. 5), the light of each beam within the first plurality of beams 508 is replicated in the second direction. Therefore, a second plurality of beams 510 is emitted from the second replicator 506, wherein the second plurality of beams 510 includes a replica of the input beam 502 along each of the first direction and the second direction. Therefore, the second plurality of beams 510 can be regarded as a two-dimensional grid or array including replica beams.
[0107] 5 can be said to combine to provide a two-dimensional replicator (or "two-dimensional pupil expander"). Thus, the replicated light beam 510 can be emitted along an optical path to an expanded eye box of a display system, such as a heads-up display.
[0108] Many types of augmented reality displays, such as heads-up displays, use at least one surface that has different reflectivity between polarization states. For example, an angled combiner, such as a windshield commonly used as a combiner in a vehicle, may have a different response to s-polarized light and p-polarized light. Thus, reflection of one type of light is generally favorable. This means that displays are generally optimized for essentially one type of polarized light.
[0109] In a first approximation, light with vertical linear polarization at the eyebox (visible through polarized sunglasses) is primarily light incident on the windshield or combiner with p-polarization. Light with horizontal linear polarization at the eyebox (not visible through polarized sunglasses) is primarily light incident on the windshield or combiner with s-polarization.
[0110] Existing driver monitoring systems (DMS) or eye tracking systems can detect whether a driver is wearing (sun) glasses. DMS typically uses an IR illuminator and an IR camera. IR light propagates through the sunglasses so the eyes can be imaged in the IR camera and the eye position or gaze is determined by software analysis of the camera image. The inventors have realized that the output from the DMS can be used to improve the image quality perceived by the user.
[0111] The HUD is typically configured to have good visibility both with and without polarized sunglasses. One way to achieve this is to have the light emitted from the pupil replicator or waveguide have linear polarization, so that after reflection by the combiner, there are components with vertical linear polarization (visible through polarized sunglasses, but with low reflection on the windshield because vertically polarized light is mainly p-polarized at the windshield) and horizontal linear polarization (not visible through polarized sunglasses, but with high reflection on the windshield because horizontally polarized light is mainly s-polarized at the windshield).
[0112] Fig. 6A A plan view of a head-up display system is shown, including an observer 600 and a vehicle windshield 602 used as a combiner. The vehicle windshield 602 is configured to receive a signal from under the vehicle dashboard ( Fig. 6A 604). The general gaze direction or forward direction 604 of the observer 600 is shown. The horizontal field of view of the picture light received by the vehicle windshield 602 from the picture generating unit is represented by arrow 606, and the picture light is expanded by the pupil expander (not shown) and relayed to the eye box. It is worth noting that the vehicle windshield 602 is tilted relative to the forward direction 604 of the observer 600, and is curved. That is, the vehicle windshield 602 is not directly in front of the observer 600. The vehicle windshield 602 is not planar.
[0113] Figure 6B shows the intensity of horizontally polarized light at the eye box (i.e. after reflection from the combiner), Figure 6C The intensity of vertically polarized light at the eye box is shown. Figure 6B The vertical proportions are different from Figure 6C In fact, the strength of the horizontal polarization is about ten times that of the vertical polarization. The horizontal polarization is sometimes considered to be dominant. The inventors have found that Figure 6B and 6C The general trend shown is independent of the linear polarization direction of the light emitted by the image generation unit. Figure 6B and 6C Relates to a representative direction of linear polarization from the picture generating unit.
[0114] Figure 6B and 6C It is obtained by taking the initial direction of the linear polarization of the light emitted from the picture generation unit of the HUD and tracking it to the eye box. At the windshield, the light is decomposed into s-polarization and p-polarization components, and the reflection intensity of the s-polarization and p-polarization components is calculated. After reflection, the reflected s-polarization and p-polarization components result in a new orientation of the linear polarization. The reflected light propagates to the eye box, where it is decomposed into vertical and horizontal polarizations.
[0115] In this case, due to the tilt of the windshield, light that reaches the eyebox with vertical polarization may already be s-polarized when incident on the windshield (i.e., some of the light reflected with s-polarization can be resolved into the vertical direction). The intensity at the eyebox is not solely related to one of the s-polarization and p-polarization reflectivities of the windshield.
[0116] Fig. 7A An example brightness distribution is shown for a user without polarized lenses or glasses, which is substantially uniform (ie, the gray level of each pixel is the same). Figure 7B The brightness distribution that a user wearing polarized lenses or glasses would perceive is shown such that the distribution varies undesirably with intensity.
[0117] The inventors have realized that this problem can be solved by using the processing device 1000. Fig.10 The processing device 1000 is described. The processing device includes a receiving device 1001, a determining device 1002 and an output device 1003.
[0118] The receiving means 1001 receives a signal indicating the presence of a polarizing filter or element in the eye box of the display system. The determining means 1002 determines a correction data set to be applied to the picture to be displayed to the user based on the signal. The output means 1003 outputs the correction data set to the display device of the display system so as to display the picture.
[0119] The signal indicating the presence of a polarizing filter may be input by the user, such that the user indicates to the system that they are wearing polarized glasses. Alternatively, the signal may be generated and received from an eye tracking system, which would refer to Fig.12 Give a description.
[0120] Eye tracking system 1200 includes sensor 1201, light source, and processing device 1203. The sensor is configured to image the user's eye and track the gaze direction of the eye and / or the position of the eye within the eye box. Sensor 1201 is illuminated using light source 1202. Light source 1202 is an invisible light source so as not to distract the user. Light source 1202 may include an infrared (IR) source.
[0121] Light source 1202 can be operated in a first mode, where horizontally polarized light is used, such that the light is blocked by the vertical polarizing filter and the user's eyes are no longer visible. In this way, eye tracking system 1200 is able to detect the presence of a polarizing filter (e.g., sunglasses) in the eye box. This Fig. 9B , where the eye is not visible.
[0122] The eye tracking system 1200 may also include a second mode, in which the light from the light source 1202 is either vertically polarized or unpolarized, so that it can detect the user's eyes even if the user is wearing (sun) glasses. Fig. 9A , where the eye is visible. Eye tracking system 1200 may be used primarily in the second mode, allowing eye tracking system 1200 to track the eye for as long as possible. The duration of the first mode will reflect the expected frequency of the user adding or removing polarizing filters and will therefore be relatively short compared to the second mode.
[0123] Although Fig.10 and 12 The eye tracking system 1200 is described with respect to the detection of vertical polarization filters, but it will be apparent that the described techniques are equally applicable to the detection of other types of polarization filters. The eye tracking system 1200 may include a plurality of states, each of which is intended to detect a particular type of polarization filter by using a corresponding cross-polarization state. Alternatively, the eye tracking system 1200 may be arranged to detect the presence of a single polarization filter, such as a horizontal polarization filter.
[0124] If the processing device 1000 receives an indication that a polarizing filter is present, the image may be corrected using the correction data set. The correction data set may correct for variations in the reflectivity of the combiner (e.g., as a function of angle) so that the image perceived by the user is more faithful to the intended image. This is Fig. 8A and 8B In Fig. 8A The expected brightness distribution for a user when viewing without a polarizing filter is shown in FIG. Figure 8B shows the brightness distribution perceived by a user wearing a polarizing filter after the correction from the correction data set has been applied. Figure 7B By comparison, it can be seen that the image now has the desired uniformity (uniform intensity in this example). For the avoidance of doubt, the image ( Fig. 7A and 8A ) are uniform and are used as an example only to illustrate the polarized sunglasses ( Figure 7B ) and the correction dataset ( Figure 8B ) - when a polarizing element is detected in the eye box (e.g. the observer is wearing sunglasses). In one embodiment, a first correction data set is used for a first operating state (e.g. detection of a polarizer) and a second correction data set is used for a second operating state (e.g. detection without a polarizer). In another embodiment, one of the operating states does not use a correction data set - i.e. only one operating state uses a correction data set.
[0125] The color balance in the picture may also be adjusted to compensate for any differences in the color balance of an image propagated through a pupil expander having a polarization suitable for viewing through polarized sunglasses. Those skilled in the art will appreciate that the reflectivity of light propagating through a pupil expander may also be wavelength dependent, and they will further appreciate from the foregoing description that the methods disclosed herein may be extended to alter the color balance of an image (i.e., the ratio of the relative intensities of the red, green, and blue image components) prior to further processing and / or hologram calculation in order to correct for any distortions in the color balance caused by different angles of incidence on the combiner.
[0126] In particular, in the holographic display system described herein, a multi-color or multi-color image is presented to the user by three single-wavelength holograms representing the corresponding color components (e.g., R, G, B) of the picture. The R, G, B holograms are reconstructed at the eye box simultaneously (or within the integration time of the human eye) so that the multi-color reconstruction is perceived. The color balance of the picture corresponds to the ratio of the color components R:G:B (i.e., the relative intensity of the pixels of the R, G, B color components of the image). Typically, the reflectivity of the partially transmissive / reflective surface of the waveguide pupil expander is highly dependent on the wavelength. This causes the color balance of the picture pixels to change when observed from different positions within the (expanded) eye box, because the light received at different positions within the eye box experiences different numbers of "bounces" on the partially transmissive / reflective surface of the waveguide. Therefore, the color balance / imbalance of the pixel may change as the observer moves within the eye box. In addition, in embodiments where the hologram is referred to as a "channel" (in the hologram domain) as described herein, different parts of the picture experience different numbers of "bounces" in the waveguide pupil expander. As a result, the left side of the image may experience more of a certain color imbalance / shift (e.g., R:B) than the right side. This problem can be addressed by applying a correction data set to one or more of the R, G, B color components of the target image to compensate for the color imbalance. As will be appreciated by those skilled in the art, the amount of compensation for the color imbalance is a function of the eye position within the eye box, and thus the correction data set used for hologram calculations depends on the eye box position. By correcting the color balance, the image advantageously appears consistent to the user at all positions within the eye box.
[0127] Thus, a correction data set may be determined for each eyebox position of the display system, which corrects for quality variations of the target image introduced by one or more components of the projection system between the picture generation unit and the eyebox. When a polarizing filter or element is present at the eyebox, a first correction data set may be determined, and when a polarizing filter or element is not present, a second correction data set may be determined. Each correction data set may correspond to a pixel of a picture and may be applied to the pixel of the image (e.g., as a scaling factor) before calculating a hologram of the image. The correction data set may correct for variations in the brightness distribution of the picture due to variations in the reflectivity of the light combiner (e.g., as a function of angle). Additionally or alternatively, the correction data set may correct for variations in the color balance of the picture due to variations in the reflectivity and / or transmittance of one or more surfaces of the pupil expander. As will be appreciated by those skilled in the art, the correction data sets described herein may be used for any target picture displayed to a user by the display system. Thus, in some embodiments, a data storage device (e.g., a memory) may store a matrix (e.g., a lookup table) comprising a plurality of first and second correction data sets for a corresponding plurality of eyebox positions. In this case, the processing device 1000 may be arranged to select and output an appropriate correction data set based on (i) a received signal indicating the presence or absence of a polarising filter at the eye box and (ii) a received signal indicating the detected eye box position of the user.
[0128] Fig.11 An augmented reality display system 1100 is shown according to some embodiments. The augmented reality display system 1100 includes a projection (eg, display or relay) device 1101 , a combiner 1102 , and a processing device 1103 .
[0129] Projection device 1101 can be any suitable system that receives visible light from a light source and modulates or otherwise modifies the light so that it carries a picture. In some embodiments, projection device 1101 can be configured for use with a non-holographic system so that there is a direct mapping between the pixels displayed on projection device 1101 and the picture observed by the user. In other embodiments, projection device 1101 can be configured for use with a holographic system so that a hologram is displayed on projection device 1101 and is illuminated so as to output light that can be transformed (e.g., Fourier or Fresnel transformed) so that the picture is seen (i.e., a holographic reconstruction or replayed image of the picture).
[0130] Combiner 1102 combines the picture-bearing light from projection device 1101 and combines it with visible light from an external scene to present an augmented reality display to the user. Combiner 1102 may have different responses to s-polarized light and p-polarized light.
[0131] If a polarizing filter is detected, the processing device 1103 can calculate an appropriate correction, which can be, for example, a change in the brightness distribution of the image, or a change in the RGB balance. The correction can be applied by outputting it to the projection device 1102. In the case of a non-holographic display, the correction can be calculated for each pixel of the display and applied to the display. However, in the case of a holographic display as described herein, the correction can be applied as part of the hologram calculation or before the hologram calculation (for example, changing the brightness distribution and / or RGB balance of the target image before input to the hologram calculation engine for hologram calculation).
[0132] Fig.13 A method 1300 according to some embodiments is shown. The method includes: receiving an indication of the presence of a polarizing filter 1301; determining a correction data set for a picture to be displayed to a user of a display system 1302 based on the indication of the presence of a polarizing filter; and outputting the correction data set to the display system to display the picture in an eye box using the correction data set 1303. In the embodiments described herein, the display system is a holographic display system that calculates one or more holograms of one or more color components of a target image representing the picture. Before or during the hologram calculation, the display system applies the correction data set to the target image (or each color component thereof). A picture generation unit (or display device) displays a diffraction pattern including the (or each) hologram and is illuminated to output picture light including a diffracted light field. In some embodiments, the diffracted light field includes a diverging bundle of light rays. In some embodiments, the image formed by the diffracted light field is a virtual image. For example, the virtual image can be formed by an optical combiner. In an embodiment, the diffracted light field is propagated through a pupil expander to expand the eye box of the display system.
[0133] A system is disclosed herein that uses diffracted light to form an image and provides an eyebox size and field of view suitable for real world applications - for example in the automotive industry via heads-up displays. Diffracted light is light that forms a holographic reconstruction of an image from a diffractive structure - for example a hologram such as a Fourier or Fresnel hologram. The use of diffraction and diffractive structures requires a high density of display devices with very small pixels (e.g. 1 micron) - which in practice means small display devices (e.g. 1 cm). The inventors have solved the problem of how to provide a diffracted light field for 2D pupil expansion, for example diffracted light comprising a divergent (non-collimated) beam of light rays.
[0134] In various aspects, a display system includes a display device—such as a pixelated display device, such as a spatial light modulator (SLM) or a liquid crystal on silicon (LCoS) SLM—that is arranged to provide or form diffracted or divergent light. In these aspects, the aperture of the spatial light modulator (SLM) is the limiting aperture of the system. That is, the aperture of the spatial light modulator—more specifically, the size of the area defining the array of light modulating pixels contained within the SLM—determines the size (e.g., spatial extent) of the bundle of light rays that can exit the system. According to the present disclosure, it is stated that by using at least one pupil expander, the exit pupil of the system is expanded to reflect that the exit pupil of the system (which is limited by the small display device having a pixel size for light diffraction) becomes larger in spatial extension.
[0135] Additional Features
[0136] The methods and processes described herein may be embodied on a computer-readable medium. The term "computer-readable medium" includes media arranged to store data temporarily or permanently, such as random access memory (RAM), read-only memory (ROM), buffer memory, flash memory, and cache memory. The term "computer-readable medium" should also be considered to include any medium or combination of media capable of storing instructions for execution by a machine, so that when the instructions are executed by one or more processors, the machine performs, in whole or in part, any one or more of the methods described herein.
[0137] The term "computer-readable medium" also encompasses cloud-based storage systems. The term "computer-readable medium" includes, but is not limited to, one or more tangible and non-transitory data storage repositories (e.g., data volumes) in the example form of solid-state memory chips, optical disks, magnetic disks, or any suitable combination thereof. In some example embodiments, instructions for execution may be conveyed by a carrier medium. Examples of such carrier media include transient media (e.g., propagated signals conveying instructions).
[0138] It will be apparent to those skilled in the art that various modifications and variations can be made without departing from the scope of the appended claims. The present disclosure covers all modifications and variations within the scope of the appended claims and their equivalents.
Claims
1. A processing device for displaying a picture in an eye box to a user in a display system, the processing device being configured to: receiving a signal indicating the presence of a polarization element in the eye box; determining a correction data set for the picture based on the signal, each data value of the correction data set corresponding to a pixel of the picture; and The correction data set is output to a display system so that the picture is displayed based on the presence of the polarizing element.
2. The processing device according to claim 1, wherein: The correction data set is a first set of values if the signal indicates the presence of the polarizing element, and a second set of values when there is no indication of the presence of the polarizing element in the eye box, the second set of values being different from the first set of values.
3. The processing device according to claim 1 or 2, wherein: When the signal indicates the presence of a polarizing filter, the correction data set modifies the intensity distribution of the image so that the image observed by the user has the expected intensity distribution after filtering through the polarizing element.
4. A processing device according to any preceding claim, wherein: The correction data set is arranged to change at least one of a brightness distribution of the picture and a color balance of the picture.
5. A processing device as claimed in any preceding claim, wherein: The signal is provided by a user tracking system operating in a secondary mode of operation, wherein the primary mode of operation includes user tracking and the secondary mode of operation includes detecting the presence of the polarizing element.
6. A processing device as claimed in any preceding claim, wherein: The polarizing element in the eye box is a pair of polarized glasses worn by the user.
7. A processing device as claimed in any preceding claim, wherein: Determining the correction data set comprises selecting the correction data set from a plurality of different correction data sets, optionally wherein there are only two different correction data sets.
8. A processing device as claimed in any preceding claim, wherein: The signal is indicative of the polarization state of light received by the user.
9. The processing device according to any preceding claim, further configured to: Determine the hologram of the image to be displayed, in, The processing means are arranged to apply the output correction data set to the image data set representing the picture before or during calculation of a hologram of the picture.
10. A display system comprising: a display device that may be configured to receive visible light from a visible light source and to modify the light so that light bearing the picture is relayed from the display device to a user; a combiner that may be configured to receive light carrying the picture from the display device and combine with an external scene to present an augmented reality scene to the user; an optional pupil expander, located between the combiner and the user, configurable to receive light from the combiner and expand an eye box of the display system within which the augmented reality scene is visible to the user; as well as A processing device according to any one of claims 1 to 9.
11. The display system according to claim 10, wherein: The display device and combiner are configured such that the intensity distribution of the picture-bearing light is non-uniformly attenuated by reflections off the combiner towards a user.
12. The display system according to claim 10 or 11, wherein: The non-uniform attenuation caused by reflections leaving the combiner is a function of polarization.
13. The display system according to any one of claims 10 to 12, wherein: The correction data set is determined based on a dominant polarization state of light reaching the user from the display device, optionally wherein the polarization state is a direction of linear polarization, such as horizontal polarization or vertical polarization.
14. The display system according to any one of claims 10 to 13, further comprising: a user tracking system arranged to determine the eye box position of the user, Therein, the processing means is arranged to determine a correction data set for the picture additionally based on the eye box position of the user.
15. The display system according to any one of claims 10 to 14, wherein: The processing means is arranged to determine correction data by selecting one of a plurality of correction data sets stored in a memory based on a signal indicative of the presence or absence of the polarisation element and / or a signal indicative of an eye box position of a user.
16. A user tracking system, comprising: an optical system arranged to illuminate an eyebox of a head-up display system with invisible light, wherein the optical system further comprises a polarization selector arranged to determine the polarization state of the invisible light illuminating the eyebox; a detection system arranged to form an image of the eye box using invisible light; as well as A controller is arranged to operate the optical system and the detection system in a primary mode of operation for detecting the position of an eye box of a user and a secondary mode of operation for detecting the presence or absence of a polarizing element, such as polarized glasses, associated with the eye box or the user.
17. The user tracking system according to claim 16, wherein: The invisible light is linearly polarized.
18. The user tracking system according to claim 16 or 17, wherein: In the secondary operation mode, the polarization state of the invisible light is vertically polarized.
19. The user tracking system according to any one of claims 16 or 18, wherein: In the main operation mode, the polarization state of the invisible light has a horizontal component.
20. The user tracking system according to any one of claims 16 or 19, wherein: In the main operation mode, the polarization state of the invisible light is horizontally polarized or unpolarized.
21. The user tracking system according to any one of claims 16 to 20, wherein: The optical system comprises a first illuminator arranged to output invisible light having a first polarization state and a second illuminator arranged to output invisible light having a second polarization state, wherein the polarization selector is arranged to cause the first illuminator to illuminate the eye box in a first operating mode and to cause the second illuminator to illuminate the eye box in a second operating mode.
22. The user tracking system according to any one of claims 16 to 21, wherein: The polarization selector is arranged to change the polarization state of the invisible light when the operation mode is changed.
23. The user tracking system according to any one of claims 16 to 22, wherein: During normal operation, the primary operating mode is used more frequently than the secondary operating mode.
24. The user tracking system according to any one of claims 16 to 23, wherein: In the secondary mode of operation, the controller is arranged to output a first control signal for the display system if the presence of a polarising element is detected, and to output a second control signal for the display system if the presence of a polarising element is not detected.
