Display device and system

Abstract

A display device and system are provided. The display device includes a display including a holographic optical element combiner; a modulator coupled to the display and configured to modulate a phase of light data to output modulated light data to the display, the modulated light data configured to compensate for geometric aberrations of the holographic optical element combiner; and the display configured to display an image based on the modulated light data.

Description

Technical Field

[0001] The examples in this article generally relate to the field of display technology, and in particular to display devices and systems. Background Technology

[0002] In Augmented Reality (AR) display technology, holographic optical elements (HOEs) are often used as combiners for near-eye displays due to their high transparency, wavelength selectivity, and ability to integrate complex optical functions. However, in head-mounted display devices, AR systems typically require large apertures, short focal lengths, and large eyeboxes for their slim and lightweight industrial designs. These design parameters physically limit the design freedom of HOEs, leading to a decrease in image quality. Therefore, it is desirable to improve the image quality of head-mounted display devices and eliminate various errors. Summary of the Invention

[0003] In a first aspect of this document, a head-mounted display device is provided, comprising: a display including a holographic optical element combiner; a modulator coupled to the display and configured to modulate the phase of optical data to output modulated optical data to the display, the modulated optical data being used to compensate for geometric aberrations of the holographic optical element combiner; and a display for displaying an image based on the modulated optical data.

[0004] In a second aspect of this document, a head-mounted display system is provided, comprising: a first head-mounted display device for displaying a first image to a first eye; and a second head-mounted display device for displaying a second image to a second eye, wherein the first head-mounted display device and the second head-mounted display device are head-mounted display devices according to the first aspect, and the first eye is either the left eye or the right eye, and the second eye is the other of the left eye or the right eye.

[0005] The above technical solutions can eliminate geometric aberrations in head-mounted display devices, thereby improving display accuracy.

[0006] It should be understood that the content described in this section is not intended to limit the key or important features of the examples in this article, nor is it intended to restrict the scope of the solution. Other features will become readily apparent from the following description. Attached Figure Description

[0007] The above and other features, advantages, and aspects of the various examples herein will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description. In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 Schematic diagrams of the environment in some scenarios are shown; Figure 2 Schematic diagrams of display systems in some scenarios are shown; Figure 3 The diagram illustrates the data processing flow for aberration compensation in some scenarios. Figure 4A and 4B Schematic diagrams of generating reverse phase data based on pupil information are shown for several scenarios. Figure 5 The diagram illustrates several scenarios of multi-color channel compensation. Figure 6 Schematic diagrams illustrating the coordinated changes between optical performance parameters in some scenarios are shown; and Figure 7 A block diagram of an electronic device capable of implementing multiple illustrative scenarios is shown. Detailed Implementation

[0008] The examples in the text will now be described in more detail with reference to the accompanying drawings. While some examples are shown in the drawings, it should be understood that solutions can be implemented in various forms and should not be construed as limited to the examples presented herein. Rather, these examples are provided to provide a more thorough and complete understanding of the solutions. It should be understood that the drawings and examples in this document are for illustrative purposes only and are not intended to limit the scope of protection of the solutions.

[0009] In the description of the examples in this document, the term "including" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "an example" or "the example" should be understood as "at least one example". The term "some examples" should be understood as "at least some examples". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. The term "trigger" refers to one or more interactive actions by a user on a terminal device. Further, these interactive actions may be triggered within the same user interface / pop-up window or within different user interfaces / pop-up windows. There is no limitation in this regard. Other explicit and implicit definitions may also be included below.

[0010] It should be noted that, unless explicitly stated otherwise, performing a step in response to A does not mean that the step is performed immediately after A, but may include one or more intermediate steps.

[0011] The examples in this article may involve user data, data acquisition, and / or use. All of these aspects comply with relevant laws, regulations, and rules. In the examples presented here, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, when implementing each example, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained through appropriate means, in accordance with relevant laws and regulations. The specific methods of notification and / or authorization can vary depending on the actual situation and application scenario; the scope of the solution is not limited in this regard.

[0012] In this manual and the sample solutions, any processing of personal information will be conducted only under legal grounds (such as obtaining the consent of the data subject or being necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0013] The term "model" or "machine learning model" as used in this article can refer to a computational model that performs tasks by learning patterns and rules from data. Machine learning models can include, but are not limited to, neural network models, deep learning models, and large language models.

[0014] As used in this article, the term "holographic optical element" refers to an optical element designed or fabricated using optical holography or computational holography methods, whose function is based on the principle of diffraction and has the ability to modulate the phase of incident light waves. Holographic optical elements have high transparency, wavelength selectivity, and the ability to integrate complex optical functions, and can be used as combiners for near-eye displays.

[0015] As used in this article, the term "combiner" refers to an optical element that combines multiple beams of light to produce an interference or superposition effect. For example, a combiner could be an optical element used to merge a beam of light from a virtual image with a beam of light from the real environment and direct it to the user's eye.

[0016] As used in this article, the term "eyebox" refers to the spatial area within which a user's eye can move while still observing the complete displayed image. This area defines the tolerance range of the display system for changes in pupil position.

[0017] The term "aberration" as used in this article refers to imaging defects in an optical system from the perspective of geometric optics. It is a deviation caused by the physical characteristics of optical elements (such as surface shape, material refractive index distribution, etc.) that prevents light rays of different apertures or fields of view from converging ideally. Geometric aberrations affect image quality in display systems. Optical distortion is a form of aberration, referring to distortion within geometric aberrations, primarily manifested as the warping of image shape.

[0018] The term "dispersion," also known as "chromatic aberration," as used in this article refers to wavelength-dependent aberrations caused by different wavelengths of light having different refractive indices or diffraction angles in an optical medium.

[0019] The term “pupil drift” as used in this article refers to a visual phenomenon in which the geometric position of an image becomes unstable due to aberrations as the user’s pupil moves within the eye box.

[0020] As used in this paper, the term "spatial light modulator" (SLM) refers to a device capable of modulating a light beam with a two-dimensional spatial distribution. The modulation methods include, but are not limited to, altering the spatial distribution of the light's amplitude (or intensity), phase, polarization state, and wavelength. In this paper, a spatial light modulator can be used to generate computational holographic wavefronts.

[0021] The term “computational holography” (CGH) as used in this article refers to the technique of generating holographic patterns through computational methods to produce the desired light wavefront.

[0022] The term "phase mask" as used in this paper refers to a technique that utilizes the coherence of light to alter the spatial intensity distribution of light by introducing a specific phase difference to improve image quality. In this paper, a phase mask refers to a phase modulation pattern used to cancel out geometric aberrations of optical elements at the physical wavefront level.

[0023] As used in this article, the term "dynamic aberration" refers to aberrations that change due to movement of the user's pupil position within the eyepiece or relative displacement of movable components. Dynamic aberrations differ from the static aberrations inherent in optical elements and change with movement of the human eye or components of the head-mounted device.

[0024] It should be understood that the imaging characteristics of large aperture, short focal length, and large eyebox may lead to geometric aberrations and chromatic aberration in head-mounted displays, causing a decrease in image quality at the eyebox edges, resulting in blurring, distortion, and pupil drift, thus affecting the user experience. Currently, solutions for aberrations in holographic optical elements mainly include static optical compensation and pure software pre-distortion compensation. The former corrects aberrations by adding additional physical correction lenses or optimizing the wavefront recording of the holographic optical element, but it cannot adapt to dynamic aberrations caused by changes in pupil position. The latter uses reverse distortion processing at the rendering end, but it is insufficient in compensating for image blurring caused by wavefront distortion. Neither approach can meet the clear imaging requirements of large-aperture, short-focal-length HOEs across the entire eyebox.

[0025] Therefore, the goal is to systematically address the aberration issues of large-aperture, short-focal-length HOEs while maintaining the slim and lightweight design of the HOE combiner, and to achieve compensation across the entire eyebox. Furthermore, building upon the resolution of static aberrations, the aim is to achieve dynamic aberration compensation that occurs as the visual axis of the human eye changes, ensuring that users can obtain high-quality, distortion-free AR images from different viewing positions.

[0026] In view of this, this paper proposes an improved scheme for a display device. According to this scheme, the display device includes: a display comprising a holographic optical element combiner; a modulator coupled to the display and used to modulate the phase of optical data to output modulated optical data to the display, the modulated optical data being used to compensate for geometric aberrations of the holographic optical element combiner; and a display for displaying an image based on the modulated optical data. Here, the display device can be a head-mounted display device, and the modulator can be an optical modulator.

[0027] Through the above technical solution, the modulator can phase-modulate the light data emitted by the light source, adding a wavefront distribution that compensates for aberrations in the modulated light data. This allows the modulated light data, after passing through the holographic optical element combiner, to counteract the impact of aberrations on image quality. Thus, without adding additional physical lenses, the clarity and realism of the displayed image are improved, thereby enhancing the user experience. Using the provided solution, head-mounted display devices can utilize the phase modulation capability of the light modulator to compensate for geometric aberrations of the holographic optical element combiner at the physical wavefront level. In this way, the imaging clarity of large-aperture, short-focal-length HOE systems can be improved across the entire field of view and the entire eyebox, thereby improving the image quality of head-mounted devices while maintaining a slim and lightweight industrial design.

[0028] The head-mounted display device described in this solution refers to an optical display device worn on a user's head that can present virtual images within their field of vision. The device includes a display and a light modulator. The display includes a holographic optical element combiner. In some cases, the device may also include components such as a controller and eye sensors. For example, the head-mounted display device can be AR glasses that combine computer-generated holography (CGH) technology with a holographic optical element combiner and actively modulate the wavefront using a spatial light modulator. The AR glasses can embed a HOE combiner in the lenses, and a light source and an SLM module can be integrated in the temples. The light source provides the light beam, and the SLM modulates the phase of the light data and outputs the modulated light data to the HOE combiner to present aberration-compensated virtual image in the user's field of vision.

[0029] The light source described in this solution refers to a component or module used to receive optical data and generate light beams, providing initial optical information for the head-mounted display device.

[0030] The optical data described in this solution refers to digital or electrical signals that contain information about the image to be displayed and are input into the head-mounted display device. For example, optical data can be image frame data or video streams from a graphics processing unit (GPU).

[0031] The optical modulator described in this scheme refers to an optical device capable of spatially and temporally modulating the phase, amplitude, or polarization state of incident light. For example, an optical modulator can be an SLM, which modulates optical data and outputs modulated optical data carrying inverse phase data. In this paper, the light source and the optical modulator together constitute an optomechanical engine, and the beam emitted by the light source is modulated by the optical modulator to form modulated light.

[0032] The modulated optical data described in this scheme refers to the physical optical wave signal output after optical data is modulated by an optical modulator. The modulated optical data carries a reverse phase mask or calculated holographic wavefront information used to compensate for the geometric aberrations of the holographic optical element combiner, and is the light wave actually projected onto the display (i.e., the HOE combiner).

[0033] The holographic optical element combiner described in this solution refers to an optical element designed or fabricated based on holographic principles, capable of simultaneously achieving wavefront modulation and beam combining. It typically exists as a photosensitive thin film with a thickness of micrometers, internally recording a precise interference fringe structure. In AR glasses, the HOE combiner functions as a diffractive optical element, diffracting the modulated light from the SLM and guiding it to the user's eyes; simultaneously, it acts as a combiner, maintaining high transparency to ambient light, allowing virtual images to be superimposed on real-world scenes.

[0034] It is understood that the solutions described in this article are also applicable to Virtual Reality (VR) display systems, Mixed Reality (MR) display systems, and other Extended Reality (XR) display systems. The solutions described in this article are based on AR as an example, but this should not be construed as a limitation on the scope of application of the solutions described herein.

[0035] The following will describe in detail various scenarios of head-mounted display devices and systems with reference to the accompanying drawings. Figure 1 A schematic diagram of environment 100 under some scenarios is shown. For example... Figure 1As shown, in environment 100, user 130 wears XR device 113. XR device 113 communicates with electronic device 110 to reconstruct virtual scenes for user 130 or to merge virtual content with real scenes. In this document, virtual scenes reconstructed based on VR technology, and scenes that merge virtual content with real scenes based on AR or MR technology, are collectively referred to as virtual scene 120.

[0036] XR device 113 can be a head-mounted near-eye display device, such as a head-mounted display or smart glasses, supporting technologies such as VR, AR, and MR. XR device 113 may include an image generation component and an optical display component for reconstructing a virtual scene 120 and displaying virtual objects in a monocular or binocular field of view. Virtual objects may include three-dimensional virtual objects and / or two-dimensional virtual objects. Two-dimensional virtual objects may include two-dimensional windows without thickness, used to present various content in the virtual scene 120, similar to an electronic screen. For example, a display block in the virtual scene 120. A display block can be a window used to load content such as web pages and documents, also referred to as a "panel".

[0037] In some embodiments, electronic device 110 may be a separate device capable of communicating with XR device 113 and / or other image capture devices, such as a server, computing node, etc., for image or data processing, or it may be integrated with XR device 113 and / or other image capture devices. In some embodiments, electronic device 110 may be integrated into XR device 113, that is, in this case, XR device 113 may have the functions of electronic device 110. It should be understood that the above description of electronic device 110 is merely exemplary and not limiting, and electronic device 110 can be implemented as a device of various forms, structures, or categories, and the embodiments herein are not limited thereto.

[0038] It should be understood that the structure and function of environment 100 are described for illustrative purposes only and do not imply any limitation on the scope of this document.

[0039] The outline of the proposed solution has been described below; see below. Figure 2 More details about head-mounted display devices. Figure 2 A schematic diagram 200 illustrates several scenarios of a head-mounted display device. It should be understood that, for ease of description, Figure 2 This example only uses a display device in a display system for presenting an image to the left eye. Alternatively and / or additionally, a head-mounted display device can be a display device in a display system for presenting an image to the right eye. Figure 2As shown, the head-mounted display device includes a light source 210, a display 220, a holographic optical element combiner 230, and a modulator (e.g., an optical modulator 240). The light source 210 and the optical modulator 240 may be referred to as an optomechanical system 245. Alternatively and / or additionally, the head-mounted display device may also include an eye sensor 250 and a controller 270.

[0040] Light source 210 is used to provide light data, which represents light signals carrying information about an image to be displayed. In some cases, light source 210 can receive light from... Figure 1 The digital signal of the electronic device 110 shown. The light source 210 generates a light beam in response to the received digital signal and directs the light beam to the light modulator 240.

[0041] Optical modulator 240 is coupled to light source 210 and display 220. Optical modulator 240 modulates the phase of optical data to output modulated optical data to display 220. This modulated optical data is used to compensate for geometric aberrations in holographic optical element combiner 230. Specifically, since the modulated optical data carries an inverse phase mask that is opposite to the phase of the geometric aberrations in holographic optical element combiner 230, geometric aberrations can be canceled after diffraction by holographic optical element combiner 230.

[0042] In some cases, the optical modulator 240 may include, but is not limited to, spatial light modulators and deformable mirrors. Spatial light modulators achieve high-resolution phase modulation through liquid crystals, while deformable mirrors achieve wavefront correction through mirror deformation. In this way, an appropriate type of optical modulator can be selected based on the specific application scenario and performance requirements.

[0043] Display 220 includes a holographic optical element combiner 230. The holographic optical element combiner 230 is typically a thin-film material that can be attached to a substrate (such as a lens) of display 220. Display 220 is used to display virtual images based on modulated light data. To achieve this display, the holographic optical element combiner 230 receives modulated light data from a light modulator 240 and diffracts the modulated light data towards the pupil 260 of the user's eye. In some cases, the holographic optical element combiner 230 may also remain highly transparent to real ambient light, thereby superimposing the virtual image onto the real scene in the user's field of vision.

[0044] In some cases, display 220 may include multiple optical element combiners, which may be stacked on the substrate of display 220. By physically stacking multiple layers of holographic optical element combiners, the display effect can be optimized in different areas. For example, different combiners can be used to display images to the human eye in different postures. In this case, a certain combiner can be activated according to different human eye postures, thereby providing a clear display image to the human eye.

[0045] The above describes the process by which an optical modulator outputs modulated light data to compensate for aberrations. The following will combine... Figure 3 From a data processing perspective, this explains how optical data is combined with inverse phase data generated based on aberration distribution to form modulated optical data carrying compensation information.

[0046] Figure 3 A schematic diagram 300 illustrates the data processing flow for aberration compensation in several scenarios. In Figure 3 In this context, aberration distribution 320 represents the aberration characteristics of an optical element (e.g., holographic optical element combiner 230). In some cases, during system design or production calibration, aberration distribution models of the holographic optical element combiner in the central and global fields of view can be obtained. Each location within the field of view can have its own aberrations; for example, geometric aberration 322 is an example of an aberration, and this geometric aberration 322 is included in aberration distribution 320, representing a specific type of aberration that affects image quality in the display system.

[0047] The reverse phase data 330 is phase information generated based on the geometric aberration 322 and is used to cancel or compensate for the aberrations corresponding to the aberration distribution 320. In some cases, the reverse phase data 330 can be implemented as a reverse phase mask to cancel the static geometric aberrations of the holographic optical element combiner 230 at the physical wavefront level.

[0048] Optical data 310 represents the original image information to be displayed. After applying reverse phase data 330 to optical data 310, modulated optical data 340 is generated. Modulated optical data 340 is the output result of combining optical data 310 and reverse phase data 330, which includes aberration compensation information provided by reverse phase data 330, thereby producing a corrected optical output that compensates for geometric aberration 322 and other aberrations.

[0049] Return to Figure 2 In some cases, controller 270 is coupled to optical modulator 240 to perform the aforementioned data processing flow: acquiring inverse phase data 330 based on a model, which includes the aberration distribution of the holographic optical element combiner; and applying the inverse phase data 330 to optical data 310 to generate modulated optical data 340. Optical modulator 240, based on modulated optical data 340, performs phase modulation on the beam emitted from light source 210 and outputs modulated optical data carrying an inverse phase mask to holographic optical element combiner 230. In this way, static geometric aberrations of the holographic optical element combiner can be canceled at the physical wavefront level, thereby improving imaging sharpness without adding additional physical correction lenses.

[0050] In real-world scenarios, a user's pupil moves within the eye box area, causing aberrations to dynamically change with pupil position. The following will combine... Figure 2 Describe a dynamic compensation method for generating reverse phase data based on real-time pupil position information. For example... Figure 2 As shown, eye sensor 250 is used to detect and track the pupil position of pupil 260. Pupil position represents the spatial location of pupil 260 within the head-mounted display device. Controller 270 is communicatively coupled to spatial light modulator 240 and eye sensor 250. Controller 270 receives pupil position from eye sensor 250 and generates phase modulation data for spatial light modulator 240 to perform aberration compensation.

[0051] Specifically, when the pupil position deviates from the center of the holographic optical element combiner 230, dynamic aberrations occur that vary with the pupil position. To compensate for this dynamic aberration, the controller 270 acquires inverse phase data that is opposite to the phase of the dynamic aberration based on the pupil position. This inverse phase data is loaded into the spatial light modulator 240, which applies corresponding phase modulation to the light data received from the light source 210 and outputs modulated light data. When this modulated light data is diffracted by the holographic optical element combiner 230, the inverse phase it carries cancels out the dynamic aberration, thereby compensating for the dynamic aberration caused by the pupil position deviating from the center.

[0052] In some cases, the eye sensor 250 acquires the pupil position, which can be represented, for example, as three-dimensional coordinates (X, Y, Z). The X and Y directions represent two orthogonal directions on the plane of the display 220 (e.g., horizontal and vertical directions), and the Z direction represents the distance from the display 220 to the pupil 260. Z may vary depending on factors such as the user's nose bridge height, wearing position, and face shape. The controller 270 acquires back-phase data corresponding to the pupil position. This back-phase data is used to compensate for dynamic aberrations caused by pupil position deviation.

[0053] In some cases, the specific implementation of obtaining reverse phase data based on pupil position may include the following: real-time generation and lookup of a table using a model (the lookup table includes the phase difference predetermined by the model).

[0054] In real-time generation scenarios, the controller 270 can input the pupil position into the model. The model then acquires the holographic wavefront in real time based on the pupil position and outputs the corresponding reverse phase data. This model can be an iterative algorithm based on optical theory or a machine learning model trained on a large amount of data. This method requires no pre-stored data, can adapt to any pupil position, and offers high flexibility.

[0055] In the application of a lookup table, the back-phase data from the model can be pre-stored using a lookup table. The controller 270 can retrieve the corresponding back-phase data from the pre-generated lookup table using the pupil position as an index. This lookup table stores multiple mapping relationships between different pupil positions and back-phase data, eliminating the need for real-time calculation and reducing the latency between eye tracking and phase compensation. The lookup table can be obtained based on experimental calibration, optical simulation, or pre-calculated using the aforementioned model; this paper does not impose any restrictions on this method.

[0056] In some cases, the eye sensor 250 further acquires the pupil radius. The controller 270 acquires back-phase data based on both the pupil position and the pupil radius. By incorporating the pupil radius, the local area where the light beam enters the pupil can be accurately determined, enabling high-precision dynamic aberration pre-compensation and thus improving the accuracy of compensation. Similarly, acquiring back-phase data based on the pupil radius can also be achieved through the aforementioned method of real-time generation using a model or by looking up a table, which will not be elaborated further here.

[0057] Figure 4A and 4B Schematic block diagrams 400A and 400B are shown for generating reverse phase data based on pupil information in several scenarios.

[0058] exist Figure 4A In the example scenario, reverse phase data is obtained based on the pupil position. For example... Figure 4A As shown, the eye sensor 250 detects the position of the pupil 260 and generates a pupil position 410, represented as coordinates (X, Y, Z). This pupil position 410 is used to generate back-phase data 330. One approach is to use model 420 to acquire the back-phase data 330 in real time based on the pupil position 410; another approach is to use the pupil position 410 as an index to retrieve pre-stored back-phase data 330 from a lookup table 430.

[0059] exist Figure 4B In the example scenario, inverse phase data is obtained based on the pupil range (i.e., pupil position and pupil radius). For example... Figure 4A As shown, the eye sensor 250 detects the position and radius of the pupil 260 to obtain the pupil range 440, represented as (X, Y, Z, R), where R is the pupil radius. This pupil range 440 is used to generate back-phase data 330. One approach is to use model 420 to acquire the back-phase data 330 in real time based on the pupil range 440; another approach is to use the pupil position 410 as an index to retrieve pre-stored back-phase data 330 from a lookup table 430.

[0060] The aforementioned eye-tracking-based dynamic compensation scheme allows for real-time acquisition of pupil coordinates, enabling high-precision dynamic aberration pre-compensation only for the local area where the light beam enters the pupil. This local compensation reduces the computational requirements of full-field-of-view compensation while ensuring the stability of the image's geometric position and image quality during eye movement, thus eliminating pupil drift.

[0061] It is understandable that the aforementioned static aberration pre-compensation and dynamic aberration real-time compensation can be used in combination. In some cases, a reverse phase mask for compensating static aberrations is generated by measuring the inherent aberration distribution of the holographic optical element combiner. Alternatively and / or additionally, based on the real-time pupil position obtained through eye tracking, additional phase data for compensating dynamic aberrations is dynamically generated and superimposed with the static reverse phase mask before being jointly loaded onto the optical modulator. Through this combination of static and dynamic methods, both the inherent static aberrations of the holographic optical element combiner and the dynamic aberrations caused by pupil movement can be compensated simultaneously without adding additional physical lenses, further improving the imaging sharpness and stability across the entire eyebox.

[0062] As diffractive optical elements, holographic optical elements also exhibit significant strong dispersion characteristics, meaning that light of different wavelengths is diffracted in different directions, resulting in color aberration phenomena such as red and blue stripes at the edges of virtual images. The following describes the dispersion compensation scheme presented in this paper.

[0063] Since the diffraction angle of holographic optical elements is wavelength-dependent, different colors of light will exhibit different dispersion shifts after passing through the holographic optical element combiner 230. To compensate for this dispersion shift, the controller 270 can acquire the dispersion shift of each color channel for multiple color channels of the light data; then, based on the aberration distribution model of the holographic optical element combiner 230, it acquires dispersion compensation data for each dispersion shift; finally, based on the dispersion compensation data, it updates the modulation light data so that the modulation light data carries compensation information that is opposite in phase to the dispersion shift of each color channel.

[0064] Figure 5 A schematic diagram 500 illustrates some scenarios of multi-color channel compensation. For example... Figure 5 As shown, in this scenario, the holographic optical element combiner 230 corresponds to three independent color channels, each processing red, green, and blue wavelengths of light. Specifically, the three color channels include a red channel 510, a green channel 520, and a blue channel 530. Each color channel is associated with a corresponding dispersion shift: the red channel 510 is associated with dispersion shift 512, the green channel 520 with dispersion shift 522, and the blue channel 530 with dispersion shift 532.

[0065] The controller 270 acquires the dispersion offset 512 of the red channel 510, the dispersion offset 522 of the green channel 520, and the dispersion offset 532 of the blue channel 530, respectively, and generates corresponding dispersion compensation data based on the aberration distribution model of the holographic optical element combiner 230. This dispersion compensation data is used to update the modulation light data, so that light of different wavelengths can be corrected after passing through the holographic optical element combiner 230, reducing the chromatic aberration effect.

[0066] In practical design, multiple performance parameters of holographic optical element combiners exhibit mutually restrictive and synergistic relationships, necessitating joint optimization. The goal is to optimize multiple performance parameters simultaneously; however, due to the aforementioned synergistic relationships, it is impossible to optimize all performance parameters at the same time.

[0067] In some cases, the HOE raster vector distribution and the SLM phase modulation dynamic range can be coupled and modeled to optimize their parameters collaboratively. This joint optimization allows for optimal division of labor between hardware and software, further improving overall display performance.

[0068] In some cases, the holographic optical element combiner 230 may include multiple attributes: field of view, diffraction efficiency, and sharpness, which are synergistically related. For example, increasing the field of view may decrease both diffraction efficiency and sharpness. Similarly, increasing diffraction efficiency may decrease both field of view and sharpness. In some cases, these attributes can be optimized in combination. For example, the field of view may be set based on the size of the head-mounted display device. Specifically, the field of view can be set to match a head-mounted display device with a large aperture, short focal length, and large eyebox to ensure a large field of view.

[0069] In some cases, diffraction efficiency can be optimized with a fixed field of view. For example, the diffraction efficiency can be set independently as needed, while the sharpness is adjusted by the optical modulator. It should be understood that since the phase modulation capability of the spatial light modulator can improve sharpness, the spatial light modulator can be used to independently optimize sharpness, that is, to improve sharpness without changing the diffraction efficiency.

[0070] Figure 6 A schematic diagram 600 illustrates the synergistic changes between optical performance parameters in some scenarios. For example... Figure 6 As shown, there is a synergistic relationship 630 between the large field of view 620, diffraction efficiency 622, and resolution 624. When one of them increases, the other two will decrease accordingly.

[0071] Specifically, the large-size eyebox 610 is directly related to the field of view 620. A large field of view 620 is required to achieve a good imaging range within the large-size eyebox 610. However, increasing the field of view 620 will constrain the diffraction efficiency 622 and the sharpness 624: on the one hand, the diffraction efficiency 622 of the holographic optical element will decrease under a large field of view; on the other hand, the aberrations of the holographic optical element will be more severe under a large field of view, leading to a decrease in the basic sharpness 624. The diffraction efficiency 622 is determined by the physical parameters of the holographic optical element itself and can be independently optimized according to the light energy utilization requirements. The sharpness 624 is compensated for and enhanced by the light modulator through phase modulation. Therefore, the sharpness 624 can be guaranteed through dynamic compensation of the light modulator without sacrificing the field of view 620 and the diffraction efficiency 622.

[0072] Through the above-mentioned collaborative optimization scheme, the limitations of mutual constraints among various performance parameters in traditional optical design are overcome, achieving a balance between large field of view, high diffraction efficiency and high definition, while improving imaging quality while maintaining a lightweight industrial design.

[0073] In some cases, head-mounted displays can be used to display images to a single eye. That is, a single device serves only one of the user's eyes, either left or right. For binocular displays, two head-mounted displays can be used, one for the left eye and one for the right eye, each displaying images independently to its corresponding eye. This configuration allows for independent aberration compensation for each eye, such as generating corresponding inverse phase data based on the pupil position and aberration distribution differences between the left and right eyes, thereby achieving more accurate binocular personalized compensation.

[0074] In some cases, the CGH generation algorithm can be optimized. Specifically, a customized term for the HOE distortion model (including the geometric distortion model and the point spread function model) can be introduced into the loss function of the CGH algorithm. In this way, the algorithm can actively adapt to the inherent aberration characteristics of HOE during iterative optimization, improving computational efficiency and image quality reproduction, and making the generated phase hologram more closely match the compensation requirements of actual optical systems.

[0075] In some cases, image post-processing methods can be used to sharpen pixels to improve image quality.

[0076] In some cases, the devices described above for monocular imaging can be used to provide head-mounted display systems. Specifically, this paper proposes an improved version of a head-mounted display system. According to this version, the head-mounted display system includes a first head-mounted display device and a second head-mounted display device. The first head-mounted display device displays a first image to a first eye. The second head-mounted display device displays a second image to a second eye. Both the first and second head-mounted display devices are the aforementioned head-mounted display devices. The first eye is either the left or right eye, and the second eye is the other of the left or right eye.

[0077] The provided solution addresses aberration issues in existing imaging systems and provides static or pre-compensation for inherent geometric aberrations and wavefront distortions of holographic optical elements through phase modulation of the spatial light modulator. Furthermore, it resolves intra-eyebox consistency issues, i.e., by combining eye tracking to eliminate image quality degradation and pupil drift caused by eye movement within the eyebox. Additionally, it addresses the dynamic balancing of display bandwidth, enabling efficient bandwidth allocation for imaging and aberration compensation even with a limited spatial bandwidth product of the spatial light modulator. In other words, it reduces system-level latency, specifically the perceived delay between eye tracking and dynamic phase compensation refresh.

[0078] The proposed solution employs a light modulator for phase modulation to compensate for geometric aberrations at the physical wavefront level. This improves the imaging clarity of large-aperture, short-focal-length HOEs without requiring additional physical lenses, overcoming the bulkiness of devices caused by static compensation schemes. Furthermore, aberration compensation at the physical wavefront level, rather than the image rendering level, fundamentally solves the image quality degradation problem caused by wavefront distortion, overcoming the limited capabilities of pure software pre-distortion compensation. Combined with eye-tracking technology, local dynamic compensation is performed based on real-time pupil position, ensuring image stability during eye movements and eliminating pupil drift. Local compensation logic is used, compensating only the local area where the light beam enters the pupil, significantly reducing the computational requirements for full-field-of-view compensation. Finally, independent head-mounted display devices are used for each eye, performing aberration compensation separately to ensure both eyes receive high-quality, consistent AR images.

[0079] This paper proposes a display device comprising: a display including a holographic optical element combiner; a modulator coupled to the display and used to modulate the phase of optical data to output modulated optical data to the display, the modulated optical data being used to compensate for geometric aberrations of the holographic optical element combiner; and a display for displaying an image based on the modulated optical data.

[0080] In some cases, the device further includes a controller coupled to the modulator and configured to: acquire inverse phase data based on a model, the model including the aberration distribution of the holographic optical element combiner; and apply the inverse phase data to the optical data to compensate for geometric aberrations.

[0081] In some cases, the device further includes: an eye sensor coupled to a controller and used to acquire a pupil position, the pupil position representing the spatial position of the pupil within the display device; and the controller is further used to: acquire inverse phase data based on the pupil position and the model, the inverse phase data further compensating for dynamic aberrations caused by the pupil position deviating from the center position of the holographic optical element combiner.

[0082] In some cases, the eye sensor is further used to: acquire the pupil radius, which represents the radius of the pupil, and the controller is further used to: acquire inverse phase data based on the pupil radius and the model.

[0083] In some cases, the controller is further used to: input pupil position to the model; and receive reverse phase data from the model.

[0084] In some cases, the controller is further used to: retrieve reverse phase data in a lookup table based on the pupil position, the lookup table including multiple reverse phase data, which are used to compensate for multiple dynamic aberrations caused by multiple pupil positions deviating from the center position of the holographic optical element combiner, and the multiple reverse data are obtained based on the model.

[0085] In some cases, the controller is further used to: acquire multiple dispersion offsets for multiple color channels of optical data; acquire multiple dispersion compensation data based on the model, wherein the multiple dispersion compensation data are used to compensate for the multiple dispersion offsets; and update the modulated optical data based on the multiple dispersion offsets.

[0086] In some cases, holographic optical element combiners include multiple attributes: field of view, diffraction efficiency, and sharpness. These attributes are synergistically related. The field of view is set based on the size of the display device, the diffraction efficiency is set independently of the sharpness, and the sharpness is adjusted by the modulator.

[0087] In some cases, display devices are used to display images to a single eye.

[0088] In some cases, the modulator includes at least one of the following types: spatial modulator or deformable mirror.

[0089] In some cases, the display includes multiple holographic optical element combiners, which are deployed on the substrate of the display in a stacked manner.

[0090] In some cases, the corresponding method can be performed by the display device described above. Specifically, the method can be performed by a controller in the display. The method includes: receiving optical data representing image information to be displayed; modulating the phase of the optical data to generate modulated optical data, the modulated optical data being used to compensate for geometric aberrations of the optical element combiner; and displaying the image based on the modulated optical data.

[0091] In some cases, the display device can acquire inverse phase data based on a model that may include the aberration distribution of the optical element combiner; and can apply inverse phase data to the light data to compensate for geometric aberrations.

[0092] In some cases, the display device can acquire the pupil position, which represents the spatial location of the pupil within the display device; and can acquire back-phase data based on the pupil position and the model, which can further compensate for dynamic aberrations that may be caused by the pupil position deviating from the center position of the optical element combiner.

[0093] In some cases, the display device can obtain the pupil radius, which can represent the radius of the pupil; and can obtain inverse phase data based on the pupil radius and the model.

[0094] In some cases, the display device can input pupil position into the model; and can receive inverse phase data from the model.

[0095] In some cases, the display device can retrieve back-phase data from a lookup table based on the pupil position. The lookup table can include multiple back-phase data, which can be used to compensate for multiple dynamic aberrations. These dynamic aberrations can be caused by multiple pupil positions deviating from the center position of the optical element combiner, and the multiple back-phase data can be obtained based on a model.

[0096] In some cases, the display device can obtain multiple chromatic dispersion shifts for multiple color channels of the light data; it can obtain multiple dispersion compensation data based on a model, which can be used to compensate for multiple chromatic dispersion shifts respectively; and it can update the modulated light data based on multiple chromatic dispersion shifts.

[0097] In some cases, an optical element combiner may include multiple attributes: field of view, diffraction efficiency, and sharpness. These attributes may vary synergistically. The field of view may be set based on the size of the display device, the diffraction efficiency may be set independently of the sharpness, and the sharpness may be adjusted by a modulator.

[0098] In some cases, display devices can be used to display images to a single eye.

[0099] In some cases, the modulator may include at least one of the following types: spatial light modulator, or deformable mirror.

[0100] In some cases, the display may include multiple optical element combiners, which may be deployed on the display substrate in a stacked manner.

[0101] In some cases, the method can be implemented by one or more modules using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units can be implemented at least partially by one or more hardware logic components. Exemplary types of hardware logic components that can be used, by way of example and not limitation, include Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Parts (ASSPs), Systems on a Chip (SOCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0102] In some cases, digital signals can be provided to the display device by electronic devices, and light signals can be provided by the light source 210. Figure 7 A block diagram of an electronic device 700 in which one or more examples may be implemented is shown. It should be understood that... Figure 7 The electronic device 700 shown is merely exemplary and should not be construed as limiting the functionality and scope of the examples described herein.

[0103] like Figure 7 As shown, electronic device 700 is in the form of a general-purpose electronic device. Components of electronic device 700 may include, but are not limited to, one or more processing units or processors 710, memory 720, storage devices 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processor 710 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 720. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 700.

[0104] Electronic device 700 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 720 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof). Storage device 730 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 700.

[0105] Electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 7 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 720 may include computer program product 725 having one or more program modules configured to perform various methods or actions of various examples.

[0106] The communication unit 740 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 700 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, networked personal computers, or another network node.

[0107] Input device 750 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 760 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 700 can also communicate with one or more external devices (not shown) via communication unit 740 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 700, or with any device that enables electronic device 700 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0108] A computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. A computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0109] The flowcharts and / or block diagrams of the methods, apparatus, devices, and computer program products referred to herein describe various aspects. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0110] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0111] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0112] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products under various scenarios. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0113] Various examples have been described above. The foregoing descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A display device, comprising: The display includes an optical element combiner; A modulator coupled to the display and used to modulate the phase of optical data to output modulated optical data to the display, the modulated optical data being used to compensate for geometric aberrations of the optical element combiner; as well as The display is used to display an image, which is displayed based on the modulated light data.

2. The device of claim 1, further comprising a controller coupled to the modulator and configured to: Reverse phase data is obtained based on a model, wherein the model includes the aberration distribution of the optical element combiner; and The reverse phase data is applied to the optical data to compensate for the geometric aberrations.

3. The device according to claim 2, further comprising: An eye sensor, coupled to the controller, is used to acquire the pupil position, which represents the spatial position of the pupil within the display device; as well as The controller is further configured to: acquire the reverse phase data based on the pupil position and the model, the reverse phase data further compensating for dynamic aberrations, the dynamic aberrations being caused by the pupil position deviating from the center position of the optical element combiner.

4. The device of claim 3, wherein the eye sensor is further configured to: acquire a pupil radius, the pupil radius representing the radius of the pupil, and the controller is further configured to: acquire the reverse phase data based on the pupil radius and the model.

5. The device according to claim 3, wherein the controller is further configured to: Input the pupil position into the model; and The reverse phase data is received from the model.

6. The device of claim 3, wherein the controller is further configured to: Based on the pupil position, the reverse phase data is retrieved from a lookup table, which includes multiple reverse phase data. These multiple reverse phase data are used to compensate for multiple dynamic aberrations, which are caused by the multiple pupil positions deviating from the center position of the optical element combiner. The multiple reverse phase data are obtained based on the model.

7. The device according to claim 2, wherein the controller is further configured to: For the multiple color channels of the light data, obtain multiple dispersion offsets for each of the multiple color channels; Based on the model, multiple dispersion compensation data are obtained, and the multiple dispersion compensation data are used to compensate for the multiple dispersion shifts respectively. as well as The modulation light data is updated based on the plurality of dispersion compensation data.

8. The device of claim 1, wherein the optical element combiner includes a plurality of attributes: field of view, diffraction efficiency, and sharpness, the plurality of attributes having a synergistic relationship, the field of view being set based on the size of the display device, the diffraction efficiency being set independently of the sharpness, and the sharpness being adjusted by the modulator.

9. The device of claim 1, wherein the display device is used to display the image to a single eye.

10. The device of claim 1, wherein the modulator comprises at least one of the following types: a spatial modulator, or a deformable mirror.

11. The device of claim 1, wherein the display includes a plurality of optical element combiners arranged in a stacked manner on the substrate of the display.

12. A display system, comprising: A first display device is used to display a first image to a first eye; as well as A second display device is used to display a second image to the second eye. The first display device and the second display device are display devices according to any one of claims 1 to 11, and the first eye is either the left eye or the right eye, and the second eye is the other of the left eye or the right eye.

Images