Display device

Abstract

An apparatus displays synthetic scene images for augmented reality, virtual reality, and other display types. One or more pairs of input beams (70, 76) are injected into a display panel comprising a layer of phase-matched non-linear optical (NLO) material (100). The input beams in each pair intersect each other in a non-collinear geometry in the NLO material to produce a combined beam (84) through a sum frequency effect that is directed to the viewer's eyes (160). Synthetic scene image information is applied to each input beam pair by amplitude and phase modulation to set the amplitude (i.e. Brightness), wave vector (i.e. Direction), and wavefront radius of curvature (i.e. Depth) of the synthetic scene image. For a color display, three input beam pairs with respective sum frequencies of three primary colors are used.

Description

Technical Field

[0001] The present invention relates to a display device. Background Art

[0002] Augmented Reality (AR) and Virtual Reality (VR) displays have been integrated into various wearable form factors including headsets. For AR headsets, glasses (i.e., Spectacles) are a common form factor. For VR headsets, ski goggles are a common form factor. A recent review article on AR and VR display technologies is the paper “Augmented Reality and Virtual Reality Displays: Perspectives and Challenges” by Zhan, Yin, Xiong, He, and Wu, https: / / doi.org / 10.1016 / j.isci.2020.101397, published in Perspective, Vol. 23, No. 8, 101397 (2020).

[0003] In AR, the image of the synthetic scene is superimposed on the natural scene that the viewer is simultaneously viewing. AR glasses are located at a vertex distance of about 12mm to 15mm, which is much closer than the near point of the eye, which gradually increases with age from about 7cm (for young people) to more than 50cm (for the elderly). The near point of the eye refers to the shortest distance at which the eye can focus. In AR, the synthetic scene is therefore represented by a virtual image that is set farther away from the eye than the AR glasses. VR imaging is simpler than AR because there is no natural scene and only a synthetic scene is generated.

[0004] For AR headsets in the form of glasses, the most common technical approach at present is to use a micro-projector system to project a virtual image that reproduces the synthetic scene onto one or two lenses of the glasses (for simplicity, the word "lens" is used in this article, but it should be understood that the lenses may be invisible, that is, they have no focusing effect in terms of normal vision correction for hyperopia or myopia, etc.). The lenses of the AR glasses are used as a medium to guide the light representing the synthetic scene into the path of the natural scene light through a suitable beam combiner. AR beam combiners can be subdivided into reflective types (such as mirrors or prisms) or diffraction types (such as some kind of grating). The virtual image of the synthetic scene should present one or more objects in the correct position in the natural scene (that is, not only in the correct position (roughly angle) in the field of view, but preferably also at a suitable distance from the eye).

[0005] Existing AR and VR display technologies have some problems to a greater or lesser extent.

[0006] One of the issues is how to avoid undesired light scattering on the glasses caused by the projected beam, which would impair the appearance of the scene and prevent the AR glasses from appearing completely transparent. For example, since the glasses are placed about 12mm to 15mm from the eyeball, which is close to the focal length of the eye, any undesired light scattered from the glasses will be focused by the eye and spread over a large area on the retina. In bright ambient light conditions, this scattering appears as glare. In dark ambient light conditions, this scattering can be much brighter than the light from the natural scene, thus severely impairing vision. Light scattering artifacts such as the rainbow effect are particularly problematic when using a grating-based beam combiner.

[0007] Another issue related to both AR and VR is how to avoid convergence-accommodation conflict, which is a characteristic of binocular vision, where the normal relationship between accommodation and convergence when viewing natural scenes is broken by the rendering of synthetic scenes, causing headaches and nausea. Convergence-accommodation conflict occurs when objects in synthetic scenes are perceived to be at a certain distance (and convergence is locked to this distance), while the light representing the objects originates from a different distance (and accommodation is locked to this distance). Even in monocular vision, the brain generates accommodation expectations based on the brain's understanding of the juxtaposition of objects in familiar scenes, so that headaches and nausea may also be caused by monocular vision of synthetic scenes. Such synthetic scenes cause the eyes to accommodate at a specific distance when viewing objects, where the distance does not match the brain's expectation of the distance of the object. Summary of the invention

[0008] The present invention proposes a display device based on a display panel that hosts phase-matched nonlinear mixing of input beam pairs that cross in a non-collinear geometry in the display panel. The crossing occurs in a nonlinear optical (NLO) material region within the display panel. Each crossed input beam pair produces a composite beam through Sum Frequency Generation (SFG) or other nonlinear mixing processes in the NLO material.

[0009] For both AR and VR, a display device according to the design can be provided for near-eye imaging. A display device according to the design can also be provided for large displays such as large screens for video conferencing or television.

[0010] According to one embodiment of the present invention, there is provided a display device for displaying a synthetic scene image in response to input of image information, the display device comprising:

[0011] A first beam source, for providing a first input beam of a first frequency and bandwidth;

[0012] A second beam source for providing a second input beam of a second frequency and bandwidth;

[0013] A display panel comprising an NLO material that is phase matched to a first input beam and a second input beam and a composite beam having a sum frequency equal to a sum of a first frequency and a second frequency;

[0014] an input beam guiding component configured to introduce the first input beam and the second input beam into the display panel so that they pass through the display panel in a first path and a second path, the first path and the second path intersecting each other in a non-collinear geometry in the NLO material to define an intersection volume, in which a composite beam is generated by a sum frequency effect; and

[0015] A controller can operate in response to the input of image information, and form a synthetic scene image by amplitude modulating at least one of the first input light beam and the second input light beam, and by phase modulating at least one of the first input light beam and the second input light beam to set each of the amplitude, wave vector and wavefront curvature radius of the synthetic light beam.

[0016] In some embodiments, the first input beam encodes image intensity information that defines the intensity of the synthetic beam generated at each intersection volume of a given synthetic scene image, and the second input beam encodes image depth information that defines the wavefront curvature radius of the synthetic beam generated at all intersection volumes of the given synthetic scene image. This is an advantageous configuration for associating each synthetic scene image with a different perceived depth by setting the wavefront curvature radius of each synthetic scene image formed. Specifically, when the second input beam has a substantially planar wavefront, meaning that the wavefront curvature radius is greater than 5 meters, the synthetic scene image is formed at a distance perceived as infinite; and when the second input beam has a wavefront curvature radius less than 5 meters, the synthetic scene image is formed at a distance perceived as finite. Thus, multiple synthetic scene images can be formed in rapid succession to establish different depth of field layers in an image frame, such as a background at infinity and one or more closer layers at which specific image elements are represented.

[0017] In some embodiments, the controller is operable to group the synthetic scene images into image frames, different synthetic scene images of a given image frame having different wavefront curvature radii to represent image elements located at respective different perceived distances.

[0018] In other embodiments, a display device is used to display a synthetic scene image in color and includes a plurality of beam sources including a first input beam source and a second input beam source, wherein the first input beam and the second input beam form one of three input beam pairs generated by the plurality of beam sources, and the first to third input beam pairs are combined to generate first to third composite beams having first to third sum frequencies providing first to third primary colors. In such embodiments, the plurality of beam sources used to form the three input beam pairs may include an emitter array and first to third lasers, wherein the emitter array provides one of the input beams for all three input beam pairs, the other input beam in each pair is provided by one of the first to third lasers, and the first to third primary colors are provided by time slicing, thereby sequentially generating the first to third composite beams. Specifically, the emitter array may be driven to amplitude modulate the input beams generated by it. For example, the NLO material may include first to third spatial modulation zones to provide quasi-phase matching for the first to third input beam pairs, respectively. Here, in one option, the first to third spatial modulation zones are formed in the NLO material at the first to third depth portions in the display panel. Here, in another option, the first to third spatial modulation regions are formed in the NLO material as an array of light spot clusters, each light spot cluster including adjacent light spots of each of the first to third spatial modulation regions.

[0019] For a color display in some embodiments, the controller is operable to group the synthetic scene images into image frames, different synthetic scene images of a given image frame presenting each of the first to third primary colors.

[0020] For a color display in some embodiments, the controller is operable to group synthetic scene images into image frames, with different synthetic scene images in a given image frame presenting each of the first to third primary colors and having different wavefront curvature radii, thereby representing color image elements located at respective different perceived distances.

[0021] In some embodiments, the input beam guiding component is adjustable to provide a change in at least one of the first path and the second path, thereby changing the angle at which the first input beam and the second input beam intersect in the NLO material, thereby changing the wave vector direction of the composite beam.

[0022] In some embodiments, the input beam directing component is adjustable to provide a change in at least one of the first path and the second path to thereby change a location where the first path and the second path intersect in the NLO material.

[0023] In some embodiments, the display device further comprises an amplitude modulator configured to amplitude modulate the first input light beam, and a phase modulator configured to phase modulate the second input light beam. In other embodiments, the display device further comprises a combined amplitude and phase modulator for amplitude and phase modulating one of the first input light beam and the second input light beam.

[0024] In some embodiments, the NLO material is spatially modulated with respect to its second order nonlinearity to provide phase matching by quasi-phase matching. In other embodiments, the NLO material is homogeneous with respect to its second order nonlinearity to provide phase matching by birefringence phase matching.

[0025] Regarding the first frequency, the second frequency and their sum frequency: the sum frequency is preferably in the visible light range, i.e. about 790 THz to 405 THz / 380 nm to 740 nm. The first frequency and the second frequency are in the infrared range, i.e. about 400 THz to 150 THz / 750 nm to 2 μm. The ratio of the first frequency to the second frequency is between 0.5 and 2.0.

[0026] For AR, display panels containing NLO materials remain transparent in the visible light range (i.e., approximately 790 THz to 405 THz / 380 nm to 740 nm), allowing natural scene light to pass through and reach the viewer's eyes.

[0027] In some embodiments, the display panel has a peripheral region containing a material that is absorptive to light at a first frequency and a second frequency, such that when a first input light beam and a second input light beam reach the peripheral region, the first input light beam and the second input light beam are absorbed after the first path and the second path have crossed each other in the NLO material.

[0028] In some embodiments, the display device further comprises one or more light sensors disposed outside the display panel to detect abnormal leakage of the input light beam from the display panel, wherein the abnormal leakage indicates structural damage of the display panel.

[0029] The NLO material may be included in the display panel, within the NLO material layer. In one embodiment, the display panel includes a front filter layer disposed in front of the NLO material layer, the front filter layer being opaque to the first frequency and the second frequency, but transparent to the sum frequency, so as to prevent the light of the first input light beam and the second input light beam from being emitted outwardly through the display panel. In another embodiment, the display panel includes a rear filter layer disposed behind the NLO material layer, the rear filter layer being opaque to the first frequency and the second frequency, so as to prevent the light of the first input light beam and the second input light beam from being emitted inwardly through the display panel. The rear filter layer may be transparent to visible frequencies. One option is to continuously distribute the NLO material on the NLO material layer so that phase matching and the resulting composite light beam may appear at any position in the NLO material layer. Another option is to distribute the NLO material as an array of light spots on the NLO material layer so that phase matching and the resulting composite light beam are confined to the positions of these light spots.

[0030] In some embodiments, the display panel includes a light shielding layer, which includes an array of pixels that can be individually addressed by electrically controlled lines so that the pixels can switch between a first state that is opaque to visible frequencies and a second state that is transmissive to visible frequencies, so that natural scene light can be mixed from selected areas of the light shielding layer.

[0031] In some embodiments, the first path and the second path pass through the display panel by continuous reflection. The continuous reflection may come from the front surface and the rear surface of the display panel by total internal reflection. Alternatively, the continuous reflection may come from a front mirror layer and a rear mirror layer disposed in the display panel, the front mirror layer and the rear mirror layer being disposed in front of and behind the NLO material, respectively. Specifically, the front mirror layer and the rear mirror layer may be reflective at the frequency of the input light beam and transmissive in visible frequencies.

[0032] In some embodiments, the NLO material is transparent in the visible range.

[0033] In some embodiments, the first frequency and the second frequency are different. In other embodiments, the first frequency and the second frequency are the same, so the sum frequency effect is a second harmonic effect. When the first frequency and the second frequency are the same, the first input beam source and the second input beam source can be the same beam source, and the first input beam and the second input beam both originate from this beam source.

[0034] The above-mentioned display device can be integrated into a wearable headset so that the display panel is set in front of the wearer's eyes with a vertex distance of less than 30mm for near-eye imaging.

[0035] Assuming SFG is the nonlinear process used, and assuming that the composite beam will be formed at a visible wavelength / frequency (approximately 380nm to 740nm / 790THz to 405THz) to create a composite scene image visible to the human eye, the input beams in each pair will be in the infrared range (e.g., 750nm to 2μm / 400THz to 150THz), at least with the constraint that they are not too different in wavelength / frequency (e.g., having a ratio of less than 2:1).

[0036] As an alternative nonlinear mixing process to SFG, three-photon additive mixing can be provided by a combination of collinear mixing Second Harmonic Generation (SHG) of one of the input beams in the pair and non-collinear mixing of the SHG with the other input beam.

[0037] Phase matching in NLO materials can occur without structuring the material, for example by using sufficiently thin layers of NLO material with high birefringence. However, in many embodiments, it will be advantageous to use Quasi-Phase Matching (QPM). To achieve QPM, the NLO material is structured to spatially modulate its nonlinear properties. The most common periodic poling is used to create a grating structure characterized by a single periodicity and alternating sign of the second-order nonlinearity X(2).

[0038] Each pair of two input beams is directed through the NLO material at an angle that ensures that the wave vector of the composite beam has a direction of propagation to the viewer's eye (ie, through the pupil).

[0039] The non-collinear mixing of a first input beam and a second input beam in an NLO material allows for the generation of a composite beam whose characteristics are controlled in terms of amplitude (i.e., image brightness), wave vector (i.e., directing the generated light toward the pupil), wavefront curvature (i.e., controlling the perceived distance of the image), and location of the image element in the field of view (i.e., position on the NLO material).

[0040] To create the realistic effect of binocular vision, the wave vector direction and image element location can be controlled to present slightly different images of the same synthetic object to the left and right eyes of the viewer, for example, on each lens of a pair of AR glasses, thereby providing natural convergence of the synthetic object and associated depth perception (stereoscopic vision). For synthetic objects that are close enough for monocular depth perception (i.e., less than about 4 to 5 meters), convergence simulation can be combined with setting the wavefront curvature of the synthetic object so that the focus (accommodation) of the eye is precisely matched or at least approximately matched to the same distance associated with convergence, for example, thereby avoiding convergence-accommodation conflicts. For synthetic objects at greater distances, a planar wavefront can be used.

[0041] The nonlinear process produces an oscillating polarization at the frequency of the composite beam (i.e. in the visible range). The process is essentially a wave mixing phenomenon, which means that the composite beam has a defined phase relationship between the different parts of the emitted waves, thereby exhibiting lateral (i.e. spatial) coherence. As a result, the composite beam produces a wavefront with a specific curvature. This is a fundamental property that is not possessed by the light produced by the pixel emitters in conventional microdisplays used in current VR headsets or the point emitters used in current AR glasses. In contrast, the non-collinear mixing of the input beam pairs in the NLO material produces a composite beam with properties that are more similar to traditional diffractive optical methods, which also emit wavefronts by diffraction. The lateral coherence method of producing light for synthesizing scene images is a method that allows the radius of curvature of the wavefront to be set, so that the composite scene image can be placed at a specific distance, which can be changed by changing the radius of curvature of one of the wavefronts of the input beam. A natural depth perception effect can therefore be presented on the composite scene image. Different synthetic objects can have different depth appearances, so that they have appropriate depth appearance relative to each other and, in the case of AR, also relative to objects in the natural scene. This not only contributes to the perceptual realism of the synthesized scene images, but also allows the avoidance of visual convergence-accommodation conflicts.

[0042] A display device may be provided that is capable of directing light for a synthetic scene image to an eye from a wide range of angles (i.e., from a wide range of positions across a field of view). The characteristic is advantageous because it takes into account a wide range of angular rotations of the eye. Furthermore, the characteristic is advantageous because it allows for the creation of a synthetic scene image that appears to emanate from an appropriate position within the field of view when the eye is looking directly in a certain direction. Furthermore, as described above, the synthetic light beams that form the synthetic scene image may be emitted from the display panel at a defined radius of curvature so that different portions of the synthetic scene image may be placed in the synthetic scene at appropriate perceived distances.

[0043] For AR headsets, the display panel can be in the form of lenses. In conventional glasses, a pair of lenses are set in the frames of the left and right eyes, and each lens corresponds to a physically separated display panel. In ski goggles in the form of a single lens, a single display panel can be used. For AR, the NLO material in the display panel is preferably transparent (in the visible range) so that it can be integrated into the display panel without hindering the light propagating from the natural scene from penetrating the display panel into the eyes. Therefore, an AR display panel can be provided, which appears completely transparent to the viewer, not only because the NLO material itself is transparent, but also because the wavelength of the input light beam is outside the visible light range (or at least can be flexibly selected to be outside the visible light range). Not only is the light from the input light beam invisible to the human eye, but more importantly, by configuring a suitable filter material layer between the NLO material layer and the viewer's eyes (for example, as a bandpass filter or an edge filter), the input light beam light of invisible frequency / wavelength is effectively prevented from entering the viewer's eyes. The filter material is selected to be opaque to the input light beam wavelength, but transparent in the visible wavelength range. For AR headsets, it is also possible to adjust the brightness of objects in the synthesized scene image. This can be achieved by simply adjusting the intensity of one or both input beams in the beam pair. For VR headsets, the NLO material does not have to be transparent (in the visible range), i.e., it can be opaque (in the visible range).

[0044] The display panel according to the present invention can be considered as a transparent microdisplay, which, like a microdisplay, is capable of generating visible light in a localized region (intersecting volume of input beam intersection) where a wavefront is generated on the display. According to the features of the present invention, only visible light is used for image generation, so there are no problems associated with stray light and scattering in conventional AR (or VR) microprojection systems. Unlike visible light beams that pass through lenses to form images in the field of view of the eye, the present invention uses infrared light beams, i.e., input light beams, which are at wavelengths outside the visible light range and can in any case be prevented from entering the eye by configuring a suitable filter layer between the eye and the NLO material layer. Therefore, there is no glare and bright scattering artifacts that interfere with night vision. Therefore, a display device can be realized, which is similar to a microdisplay VR display device in terms of its light generation, but unlike a microdisplay, the display device is transparent (in the visible range), so the display device can be used for AR.

[0045] Color display

[0046] The above description of the invention is limited to the following considerations: a single composite beam is generated based on a single input beam pair. This is suitable for describing a monochrome display, or alternatively, for describing one of the three color components of a color display. For a color display, three composite beams are required, one for each of the three primary colors, so there will be three input beam pairs.

[0047] According to another embodiment of the present invention, for a color display, there is provided a display device for displaying a synthetic scene image in response to input of image information, the display device comprising:

[0048] A plurality of beam sources having respective frequencies and bandwidths to provide first to third input beam pairs each having two input beams, each input beam pair having a pair of frequencies that is a sum of first to third primary color frequencies;

[0049] A display panel comprising an NLO material that is phase matched to the first to third input light beam pairs and the first to third composite light beams at first to third primary color frequencies equal to the sum of the frequencies of the first to third input light beam pairs;

[0050] an input beam guiding component configured to introduce two input beams of each input beam pair into the display panel so that the input beams of each pair pass through the display panel along a first path and a second path, wherein the first path and the second path intersect each other in a non-collinear geometry in the NLO material to define an intersection volume, wherein a composite beam of the input beam pair is generated by a sum frequency effect;

[0051] an amplitude modulator operable to amplitude modulate at least one beam of each input beam pair;

[0052] a phase modulator operable to phase modulate at least one beam in each input beam pair; and

[0053] A controller is operable to form color image frames, each image frame including first to third synthetic scene images generated by first to third input light beam pairs, respectively, wherein the controller is operable to set the synthetic beam amplitude, synthetic beam wave vector and synthetic beam wavefront curvature radius by controlling an amplitude modulator and a phase modulator in response to input of image information, thereby forming each of the first to third synthetic scene images.

[0054] For a color display, the multiple beam sources may include an emitter array and first to third lasers, wherein the emitter array provides one input beam in each pair for all three input beam pairs, the other input beam in each pair is provided by one of the first to third lasers, and the first to third primary colors are provided by time slicing, thereby generating first to third composite beams in sequence.

[0055] Although it is necessary to have three input beam pairs, meaning there are six input beams in total, this does not equate to the need to have six independently generated input beams. First, the same wavelength can be shared between different input beam pairs, so the total number of wavelengths required to form three beam pairs does not have to be six, but can be reduced to five, four, or three, with three being the minimum possible number. Second, time slicing (i.e., multiplexing) can be used so that a common first input beam is used in combination with a dedicated second input beam to generate all three colors, one input beam for each color, thereby reducing the total number of independently generated input beams required to four.

[0056] For color displays, if QPM is used, the NLO material can be structured to have a single grating period for all three colors. The lack of exact phase matching for at least two of the three colors is tolerable and can be compensated for by changing the total intensity of one or both input beams in each beam pair as needed or necessary. The angles at which the input beams pass through the QPM NLO material can also be selected to be different for different colors, thereby providing exact phase matching (or at least closer to exact phase matching) for all three beam pairs (i.e., all three colors). Alternatively, to provide quasi-phase matching, the NLO material can be periodically poled (or equivalently oriented patterned in the case of non-birefringent materials) with three different periods, so that there are three regions of QPM NLO material with different grating periods, each optimized for one of the three colors. Another alternative is to create a single synthetic QPM structure with three phase matching peaks optimized for the desired three colors of the display. The three color-specific QPM zones may be distributed in the imaging plane, for example, in the form of a regular or irregular 2D array of light spot clusters on the display panel, wherein each light spot cluster contains three laterally adjacent color-specific QPM zones, i.e., light spots. The light spots in each cluster are so close that it is almost impossible to detect the fact that they are not at the same position in the field of view. Alternatively, the three color-specific QPM zones may be distributed along an axis perpendicular to the imaging plane, for example, distributed over three different depth portions of the display panel. Here, similar to the above-mentioned clusters, the color-specific QPM zones may be arranged in a 2D array, or may be provided as a continuous area on the display panel as needed to cover the field of view required for synthetic scene imaging.

[0057] In the following, the invention is mainly described in terms of generating a single composite beam (ie in terms of generating a monochrome image or generating one color component of a color image). However, it should be understood that a color image can be generated from three such composite beams from three input beam pairs.

[0058] Image information for near-eye synthetic scene imaging

[0059] In order to create a realistic virtual image of a synthetic scene in an imaging plane close to the eye (especially required for AR glasses), it is necessary to accurately reproduce the image part information encoded in the light from the equivalent natural scene. The image part information is a subset of the entire image information and is what defines the 3D position perception of objects in the scene. As will be discussed in detail below, for monocular vision, the image part information in a natural scene is essentially a combination of the position where light from different parts of the natural scene appears in the field of view (i.e., the input angle of the light entering the eye) and the radius of curvature that the light wave from each point in the natural scene has when it reaches the eye. For nearby objects, the wavefront curvature is obvious, while for distant objects, there is no obvious curvature, that is, the wavefront is actually a plane wave. For synthetic scene imaging in a display panel close to the eye (e.g., at a vertex distance of about 12 mm to 15 mm, where eyeglass lenses are typically located), input angle and radius of curvature information from a natural scene can be mimicked by generating light at a specific location on the imaging plane (so it appears at the correct point in the field of view) and from that location with a specific light propagation direction (so light generated at that location in the imaging plane enters the eye). In addition to carrying the correct image location information, the light generated on the imaging plane must also have the correct intensity and, for color images, the correct color.

[0060] In the context of the present invention, correctly encoding the image location information of the synthesized virtual image means generating a composite beam of light in the NLO material at the correct location (viewing angle) with the correct wave vector direction and with the correct wavefront curvature radius. In order to produce the image, the image intensity information must also be reproduced, which can be done directly by appropriately amplitude modulating one or both of the first input beam and the second input beam. For color images, the image color information also needs to be reproduced, which is done by generating three composite beams of three primary colors (e.g., RGB) in the QPM NLO material at each location in the field of view (i.e., providing an appropriate color gamut in the same way as any color projection system).

[0061] In order to give a realistic binocular visual perception of the synthetic scene, the respective images formed for each of the left and right eyes will correspond, but will not be identical. Specifically, for the left and right eyes, light from the same point in the synthetic scene is formed at different locations in the monocular field of view of each eye with different wave vector directions, thereby emulating natural convergence and associated depth perception (stereoscopic vision). It should be noted that for any portion of the synthetic scene attributable to infinite distance, the images formed for each of the left and right eyes can be made identical, because in an equivalent natural scene, light from a distant object will arrive at both the left and right eyes as a plane wave, and arrive at the same location in the respective monocular fields of view of the left and right eyes. Although it should be recognized that when creating a realistic visual scene, any intermediate distance object may block or occlude the field of view of the left and right eyes in different ways, so although both are plane waves, they are emitted to the two eyes from different directions.

[0062] Imaging of synthetic scenes

[0063] Two basic methods are proposed for forming a synthetic virtual image in an imaging plane close to the eye. The first basic method is a scanning method that builds each image frame pixel by pixel, for example by rasterization. The second basic method forms complete image frames simultaneously.

[0064] Imaging in angular space (input beam scanning / retina scanning): The first input beam and the second input beam produce a composite beam by moving both in angle and position when passing through the NLO zone, and the composite beam is scanned within the monocular field of view of the viewer to form an image encoded in angular space by a retinal scanning method. Therefore, an image frame is established by rapidly scanning visible light over the retina. Specifically, a first input beam and a second input beam are provided, and the first input beam and the second input beam have respective beam cross sections, and the dimensions of these beam cross sections are set to make their intersection volume small when the first input beam and the second input beam intersect in the NLO zone to provide a plane wave that is not diffraction-limited and mapped to a point on the retina (focused by the eye), so that the intersection volume at any position in the imaging plane corresponds to a single shot of the angular field (i.e., wavefront) for establishing a synthetic virtual image on the eye. For each intersection volume, the first angle and the second angle at which the first input beam and the second input beam pass through the NLO zone define the wave vector direction of the composite beam. The wave vector direction is set to ensure that the composite beam is directed to the pupil of the eye. To build a complete image (for one eye), the angle of one or both of the first and second input beams is changed so that the angle of the nonlinearly generated light changes to different angular positions in the (monocular) field of view of the viewer's eye while the input beam intersection volume (i.e., the location where the composite beam is generated) remains substantially constant. Similarly, the intensity of the composite beam is changed to reproduce the image intensity information, which can be achieved by amplitude modulating the first and / or second input beams, preferably while taking into account the changes in the nonlinear mixing conversion efficiency caused by the changes in the lateral angle of the input beams that occur with the change in the intersection angle in the imaging plane, so as to gradually build up a complete image frame. As the input angle into the intersection volume changes, the wave vector direction changes accordingly to ensure that the composite beam is always directed to the pupil of the eye. This can be done by appropriately adjusting the angles at which the first and second input beams pass through the NLO material. The intersection volume is scanned within the monocular field of view to build up each image frame, for example, a line-by-line raster pattern can be followed. The raster can involve a serpentine movement, i.e., moving back and forth from left to right, then from right to left.

[0065] Overall imaging (upconversion simulation): This imaging method can be partially understood through simulation with a conventional upconversion imaging system. A first input beam (e.g., 1064nm) contains image intensity information, and a second input beam (e.g., 1550nm) intersects with the first input beam in the NLO material so that an upconverted version of the first input beam is reproduced (i.e., a composite beam). This simulation is helpful, but incomplete, because the simulation does not take into account the fact that non-collinear phase matching according to the present invention also allows control of the wavefront curvature of the composite beam, thereby incorporating image depth information into the composite virtual image. Therefore, by phase modulating the second input beam, the image depth information is "added" to the "upconverted version" of the intensity modulated first input beam (i.e., the composite beam). The wavefront engineering feature of the present invention has no analog in a conventional upconversion imaging system.

[0066] In one embodiment of the present invention, there is an implementation that is closest to upconversion imaging simulation. The first input beam is amplitude modulated on a beam cross section of the first input beam to carry image intensity information, which defines a specific brightness of the intersection volume in the composite beam. The second input beam is phase modulated on a beam cross section of the second input beam to carry image depth information, which defines a specific wavefront curvature of the intersection volume in the composite beam. In other implementations that deviate from upconversion imaging simulation, both amplitude and phase are modulated on a beam cross section of one or both of the first input beam and the second input beam. For example, the first beam can be amplitude modulated, and the second beam can be amplitude and phase modulated.

[0067] In a first specific embodiment, both the first input beam and the second input beam have a high degree of coherence in the transverse direction (i.e. in the cross section of the beams). This is true if the beam sources of both the first input beam and the second input beam are lasers and the beam modulation is performed using a spatial light modulator (SLM) in transmission or a liquid crystal on silicon (LCOS) device in reflection. It is noted here that both SLM and LCOS devices can be operated to modulate amplitude (only), phase (only), or both amplitude and phase.

[0068] In a second embodiment, the first input beam is incoherent and is generated by a 2D array of incoherent emitters (e.g., an LED or OLED array) that is capable of providing amplitude modulation by independently driving each emitter. The second input beam is coherent to meet the requirements of phase modulation, which is based on a laser source whose laser beam is phase modulated (e.g., by an SLM or LCOS device) and optionally amplitude modulated, as described in the first embodiment.

[0069] In the case of an RGB display requiring three input beam pairs for producing three colors, the same incoherent emitter array can be used to produce the first input beams for all three colors, while each color has its own dedicated second input beam, the three second input beams being produced by three different laser sources having three different wavelengths that are added to the wavelength of the incoherent emitter array to produce red, green, and blue light, respectively. Time division multiplexing (i.e., slicing) can then be used to sequentially image the red, green, and blue composite scenes by appropriate drive electronics that synchronize the outputs of the IR emitter array and the three laser sources. This sequential driving (or on / off modulation) of the three laser sources provides a temporal RGB imaging approach similar to the color wheel approach in a desktop projector. If the refresh rate of the three colors is high enough, the viewer will perceive a full RGB color image.

[0070] Since it is possible to apply both phase modulation and amplitude modulation to one beam to produce a single beam with combined phase and amplitude modulation (eg using an SLM or LCOS device), other embodiments are possible in which the other input beam is not spatially modulated at all.

[0071] Other features

[0072] Some additional features of certain embodiments are briefly summarized below.

[0073] In order to transmit the input beams into the display panel, a favorable method is to introduce the first input beam at or near one end of the display panel (e.g., the left end) and introduce the second input beam at or near the other end of the display panel (e.g., the right end). However, both the first input beam and the second input beam can be introduced at the same end of the display panel. This can be achieved by a more complex phase matching scheme that uses appropriate reciprocal lattice vectors for non-collinear phase matching. This can also be achieved by reflecting one of the first input beam and the second input beam from the other end of the display panel so that they intersect in the same non-collinear geometry as the introduction of the first input beam and the second input beam at the opposite end.

[0074] There are a number of options for the distribution of the NLO material within the NLO material layer. In certain configurations of the device, the NLO material is continuously distributed over the NLO material layer; in other configurations, it may be advantageous to have a spot produced by the NLO material arranged in a grid over the NLO material layer. In the case of the spot grid approach, it should be noted that the NLO material referred to refers to an NLO material that is phase matched to promote nonlinear mixing, so that if QPM is used, the spot can be a local area that is spatially structured for QPM in the same unstructured NLO material layer. It is still possible to produce realistic images using the spot grid approach of locally restricted areas when producing a synthetic beam of NLO material because the eye is constantly moving. As long as these local areas are close enough so that light from at least one of the local areas can be directed to the pupil, a realistic image can be formed.

[0075] In certain embodiments of the present invention, the first input beam and the second input beam are each introduced at an angle at or near one end of the display panel relative to the front and rear surfaces of the display panel. Each input beam then propagates across the display panel to a position where non-collinear mixing occurs through one or more reflections. In some embodiments, total internal reflection (TIR) ​​of the interface between the display panel and the air is used to make each input beam pass through the display panel, from the introduction position of each input beam to the position where each input beam intersects with another input beam paired with it to produce a composite beam. In other embodiments, a reflector layer is provided in front and behind the NLO material layer to reflect within the wavelength range of the input beam so that the input beam passes through the display panel by continuous mirror reflection. One or both of the front mirror layer and the rear mirror layer may be located on the front or rear surface of the display panel, respectively, or may be an embedded layer inside the display panel. Compared with TIR, mirror reflection allows reflection at a larger angle (i.e., closer to the normal), which is more advantageous when quasi-phase matching is used because the polarization period can be larger and therefore easier to manufacture.

[0076] Other functional layers may be added to the display panel. For example, it may be advantageous to add a front filter layer to the front of the NLO material layer, which absorbs in the wavelength range of the input beam (and transmits in the wavelength range of the composite beam), thereby preventing any scattered input beam light from reaching the eye. In the case where either of the two input beams is a laser beam, this also helps to ensure that laser safety requirements are met. Similarly, a rear filter layer may be added to the back of the NLO material layer to prevent the input beam light from escaping into the environment. If the display panel is used for AR, both the front filter layer and the rear filter layer should be transparent in the visible frequency range, so that natural scene light is transmitted through the display panel to the eye without attenuation.

[0077] Another safety measure is to provide one or more light sensors, such as photodiodes, outside the display panel to detect abnormal leakage of input light beams from the display panel, which indicates structural damage to the display panel. The light sensors may be arranged around the edge or outer edge of the display panel. Then, for example, when the display panel is damaged by a deep scratch, the operation of the display device may be stopped (as needed). Another possibility for AR is to include a photochromic layer or an electrochromic layer, so that the transmission of natural scene light can be reduced, for example, under bright ambient light conditions, so that the synthetic scene light is properly blended with the natural scene light. The darkening operation may be automatic (as in photochromic lenses) or electric. In addition, the darkening may be uniform darkening over the entire lens area, or it may be localized, such as to reduce the light from bright spots in the natural scene. A vision correction lens layer may also be provided in front of or behind the NLO material layer (or when two corrective lenses are required, a vision correction lens layer is provided in front of and behind the NLO material layer). If a corrective lens layer is provided, an input beam mirror layer is preferably inserted between the NLO material layer and the vision correction lens layer to prevent the input beam light from entering the vision correction lens layer so that the input beam path is not affected. In other words, when a vision correction lens is included, a TIR solution is not preferred. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0079] Figure 1 is an explanation of how imaging of distant objects requires wavefront information.

[0080] Figure 2 It shows how the image entering the eye through the lens assembly depends on both the wavefront direction and the wavefront curvature.

[0081] Figure 3 Clarifies why point light emitters cannot be used for near-eye displays.

[0082] Figure 4 Nonlinear optical methods are shown for producing near-eye artificial images that can be superimposed on natural scene light.

[0083] Figure 5 The concept of quasi-phase matching is shown.

[0084] Figure 6 Two-dimensional phase matching is shown, and different grating vectors can be constructed.

[0085] Figure 7 An implementation of NLO mixing for producing synthetic scene light with a bounce path within the lens is shown.

[0086] Figure 8 Non-collinear phase matching is shown.

[0087] Fig. 9 A more detailed schematic showing the nonlinear wavefront overlap in a layered NLO QPM material to generate image information.

[0088] Fig.10 Routes are demonstrated that allow pairs of beams to overlap at the same location at different crossing angles relative to the QPM structure.

[0089] Fig.11 A scheme for directing the light beam into the device is shown.

[0090] Fig.12a A scheme for deflecting a light beam without angle change is shown.

[0091] Figure 12b Components are shown for directing light to a desired location within a layer of NLO material while varying the angle at which the light approaches the spot.

[0092] Fig.13 The lens structure is shown along with the input beam and emitter, which are capable of creating an image with the desired depth of focus.

[0093] Fig.14 The rendering of the desired wavefront by nonlinear mixing of amplitude and phase structured beams is shown.

[0094] Fig.15 A system for generating a far-field pattern from an image to allow conversion through an NLO material is demonstrated.

[0095] Fig.16 Demonstrated multilayer optical composites for creating display systems.

[0096] Fig.17 Changing the local Kg position on a glasses-type display system is demonstrated.

[0097] Fig.18 Different lens designs with suitable advantages are shown in cross-section.

[0098] Fig.19 An eyeglass AR system is shown that includes multiple sensors and communicates with an external computing device.

[0099] Fig. 20 An eyewear system that includes a series of diode components on the periphery to detect system damage is demonstrated.

[0100] Fig.21 An electrically controlled dimming system built into the lens is shown.

[0101] Fig. 22A display device containing sub-regions of NLO material and having separate red, green and blue generating regions is shown.

[0102] Fig.23 The QPM period is shown as a function of the crossing angle in a simple second harmonic process, demonstrating the advantage of using larger crossing angles.

[0103] Fig.24 A method for making a lens in which light beams enter from one side but remain crossed in the opposite direction is shown.

[0104] Fig.25 A timing display for providing RGB and different ROC images to produce a realistic full color effect, as well as a depth of field realistic display is shown.

[0105] Fig.26 Demonstrating the internal camera used to track pupil position.

[0106] Fig. 27 Shows how interior camera based culling can be used to reduce unwanted visual effects.

[0107] Fig.28 A lens is shown with an absorbent layer distributed around the edge to prevent unwanted light reflections.

[0108] Fig.29 A lens having sub-regions of NLO material is shown, wherein each sub-region contains elements capable of producing red, green, and blue colors.

[0109] Fig.30 It is shown that the beam crossing angle on the mirror requires regional control of the QPM grating direction.

[0110] Fig.31 A Virtual Reality (VR) configuration according to the present invention is shown, wherein a near-eye image is generated without using complex thin lenses.

[0111] Fig.32 Schematic diagram of using an incoherent emitter array to generate one of the input beams for imaging in a layer of NLO material.

[0112] Fig.33a Various options for modulating two light beams interacting in an NLO material are shown.

[0113] Figure 33b Various options for phase controlling the input beam to impart radius of curvature information are shown.

[0114] Fig.34 A schematic diagram of the control system is presented, showing the inputs and outputs. DETAILED DESCRIPTION

[0115] Optical properties of the human eye and human vision

[0116] When designing display devices, it is important to take into account the optical properties of the eye and how the eye is controlled individually and in coordination by the brain, as well as how the brain processes image information from the eye.

[0117] In order to enter the eyeball, light passes through the pupil, cornea, eye lens and thus reaches the fundus where the retina is located. The effective radius of the pupil is controlled by the iris (pupillary muscle). The pupil expands and contracts according to the ambient brightness to increase and decrease the amount of light entering the eye. The pupil diameter varies within a range of several millimeters, of which the range of 2mm to 4mm is the most common. The diameter of the eyeball is about 25mm.

[0118] If wearing glasses, light from distant objects entering the eye passes through an area of ​​the eye lens, the size of which is substantially similar to the diameter of the iris. The distance from the eye lens to the front of the eyeball is about 12mm to 15mm and is called the back vertex distance or vertex distance.

[0119] The field of view of one eye is called the monocular visual field, and is generally defined as the angular range of what can be seen when the eye is looking straight in a specific direction and its field of view is fixed on a certain point. The monocular visual field extends laterally about 60 degrees inward (toward the nose) relative to the vertical meridian of the eye, and extends laterally about 107 degrees outward. It extends vertically about 70 degrees upward and about 80 degrees downward relative to the horizontal meridian of the eye. The binocular visual field is the superposition of the monocular visual fields of the left eye and the right eye, so it is significantly larger in the laterally. In this article, the "field of view" mentioned refers to the monocular visual field. If it refers to the binocular visual field, it will be clearly indicated.

[0120] Normal human vision involves a series of movements of the head centered on the spine, and the rotation of the eyes in the eye sockets. Head movement can be considered to correspond to roll, pitch and yaw angles relative to the front, wherein yaw is generated by the normal left and right rotation of the head, pitch is generated by the up and down nodding movement of the head, and roll is generated by the sideways movement of the head. The range of eyeball rotation of the human eye is about +25 degrees upward, about -30 degrees downward, and about ±45 degrees to the left and right. Eyeball rotation is used to project the retinal image of the most concerned natural scene part onto the fovea, which is the sub-area of ​​the retina with the highest resolution. The fovea provides an angle range of about 1 degree to 2 degrees in the visual field (as a reference, the visual field of the entire retina is about 18 degrees, i.e., the macula). In addition, the eyes move around naturally (saccade movement), which keeps the eyes moving to avoid the effects of saturation and unresponsiveness. Therefore, the eyes constantly look around the scene, while the brain processes the image to offset the blurring effect of eye movement.

[0121] For monocular vision, if we consider the light emitted by a given point in the natural scene, the light will actually be a spherical wave with the point as the origin. The spherical wave strikes the front of the eye, passes through the cornea, pupil and lens of the eye and reaches the retina. The refractive power of the lens of the eye is adjusted by the ciliary muscle to form a focused image of the scene light on the retina. The lens of the eye receives a wavefront, which has a specific central angle and radius of curvature. In the case of light from a far point in the natural scene (effectively infinite distance), the wave can be considered to enter the eye as a plane wave, that is, the wavefront is a plane wavefront. For monocular vision perception of most individuals, infinite distance refers to any distance above about 4 meters or 5 meters. The angle between the light incident from such a far point and the axis of the eye determines the position of the light falling on the retina. In the case of light from a closer point in the natural scene, the wave enters the eye with a finite radius of curvature, which will be perceived by monocular vision through the accommodation of the eye. The eye will accommodate, that is, adjust its focus to guide the light from the closer point to a sharp focus on the retina. The human brain associates the amount of adjustment with the distance of the object, thus producing distance perception (distance perception also takes into account the convergence effect of binocular vision).

[0122] Therefore, for monocular vision, a natural scene can be considered as a collection of point light sources at different positions in the monocular field of view, where the 3D position information of each point is encoded by the input angle of the light (i.e., the 2D position in the monocular field of view) and the radius of curvature of the light in the eye (i.e., depth). In addition to the 3D position information, the light from each point also carries intensity information and color information.

[0123] In near-eye display systems such as AR and VR, the term "eye range" (or eye motion range) refers to the spatial range within which the eye can receive an acceptable view of an image, whether the image is real or virtual. A simple definition of eye range is the 3D volume that can be seen across the Field Of View (FOV) for a standard pupil size, taking into account the fact that in human vision the eye is constantly moving within the FOV to focus on different areas. The specified size of the eye range in AR or VR goggles is usually larger than the theoretical range of movement of the pupil to accommodate alignment tolerances and differences in pupil distance between people.

[0124] Nonlinear Optical (NLO) Materials

[0125] One type of nonlinear mixing in NLO materials exploits the second-order nonlinearity X(2). For nonlinear mixing, energy conservation exists, which means that the energy of the photons generated by the nonlinear mixing depends on the sum of the energies of the photons of the input beams, which means that the frequency of the resultant light is determined by the sum or difference of the frequencies of the input beams, when the Planck relation holds.

[0126] Phase matching occurs due to the wavelength dependence of the refractive index of the NLO material. The type of phase matching used in embodiments of the present invention is called non-collinear phase matching. Non-collinear phase matching occurs when two (or more) beams with different propagation directions cross to form a beam crossing region located in the phase-matched NLO material. Phase matching ensures that the wave vector of the composite beam produced by nonlinear mixing is the vector sum of the wave vectors of the input beams (and the compensation grating k-vector). The term "non-collinear phase matching" is used to contrast with the term "collinear phase matching", which refers to the situation where the two input beams overlap in space and propagate together so that the wave vectors of both the input beams and the composite beam are in the same direction. The non-collinear mixing of the two input beams can be characterized by two subtended angles between the composite beam and each of the two input beams, the sum of which is the crossing angle between the two input beams, i.e., the beam crossing angle. In the case of two input beams with equal wave vectors (i.e., equal wavelengths), the two angles are equal, and the angles can be called the beam crossing half-angle.

[0127] In practice, phase matching is usually achieved using a method called Quasi-Phase Matching (QPM), which overcomes the fact that the phase mismatch of the refractive index dispersion in the birefringent material does not allow phase matching unless the birefringent material is extremely thin. NLO materials are structured to spatially modulate their nonlinear properties, typically using periodic poling to create a linear grating structure with alternating signs of the second-order nonlinearity X(2). For non-collinear mixing, the phase matching requirements of the QPM NLO material need to take into account the angle at which each input beam passes through the QPM NLO material. In this regard, it should be noted that the poling period required for non-collinear mixing will be smaller than that required for collinear mixing, because only one component of the wave vector of each input beam will be in the polarization direction.

[0128] Conventional birefringent NLO materials may be used. If the NLO material is quasi-phase matched, conventional inorganic NLO materials may be used, especially ferroelectric oxides, which may be patterned by periodic poling. For example, lithium niobate (LN) for producing periodically polarized (PP) lithium niobate (PPLN), periodically poled lithium niobate doped with magnesium oxide (PPMgOLN), lithium tantalate (LT) for producing PPLT, and potassium titanyl phosphate (KTP) for producing PPKTP. It is well known that ferroelectric NLO materials can be microstructured using QPM structures (e.g., linear gratings) by electric field poling. Certain non-ferroelectric materials, such as III-V zinc blende semiconductor crystals, such as GaAs, may also be processed using a recently developed technique called orientation patterning to spatially modulate their nonlinear properties. Although oriented and patterned GaAs is opaque in the visible range, other transparent materials can be used (e.g., oriented and patterned GaN). In addition, in addition to using the above-mentioned inorganic materials, the polarized NLO material can also be an organic material (or advantageously a non-centrosymmetric organic material layer). It is also worth noting that electric field polarization of the layer can be achieved.

[0129] Illustration

[0130] Figure 1 A diagram showing the process of receiving an image by a human eye, represented by an eyeball 160 with a pupil 170. The eye receives an image of an object 180, represented by a "stick man". Light scattered from the object 180 is received at the pupil 170 and then focused by the lens of the eye onto the retina at the back of the eye, where it is detected by rods and cones. The light can be represented as light ray 184. Typically, the distance between the eyeball 180 and the object is at least 15 cm (or so), which represents the near point of the eye. The distance can be much larger; for example, in the case of starlight, the distance is millions of kilometers or more. The fact that the distance can be "large" is represented by line 186 to indicate the discontinuity of the scale in the figure. The light scattered from the object can be represented by wavefronts 188, an illustration of which is shown at 190. Each wavefront is separated by one wavelength of light, λ. Any two points on a given wavefront (represented here as and ) there is a defined phase relationship between the wavefronts 192 and 180. In the figure, the wavefront is shown to bend with a wavefront curvature corresponding to the distance 192 from the object 180. The lens of the human eye changes shape under muscle control (ciliary muscle), and the shape adopted produces focusing of the incident (usually curved wavefront) so that the desired portion of the visual field is in sharp focus on the retina.

[0131] Figure 2 An eye 160 and an object 180 are shown, with the object 180 being disposed at a distance 192 from the eye 160, which is relatively large. An eyeglass 20 is disposed in front of the eye. Light scattered from the object 180 that has passed through the eyeglass is associated with a wavefront 188. Two sets of light rays are shown emanating from points A and B. Point A is on-axis, while point B is off-axis, relative to the primary visual axis of the eye 160. Light incident on the eye enters the eye through a pupil, such as controlled by an iris 172. Light rays emanating from points A and B are associated with respective wavefronts 188A and 188B, and are focused by the eye lens 166 onto the retina 168 at two different points 190A and 190B, respectively. The angles that the light rays make with the visual axis of the eye determine where the image is formed on the retina, while the wavefront curvature determines the location of the focal point.

[0132] Figure 3It is shown that light emitted from a point source close to the eyeball will have a wavefront curvature that cannot match the wavefront curvature of light from a natural scene. An eyeball 160 and a distant object 180 are shown. The wavefront curvature of the light from the object 180 corresponds to the distance from the object, which is "large". A piece of optical material (eyeglass lens) 20 is close to the eyeball and is shown to contain a point light source 192. The point source 192 can be a fluorescent or scattered point (such as from dust particles) from an optical material (such as a dye or a rare earth element). The point source will produce a wavefront 188 with an associated radius of curvature centered on the point source 192. Because the point source is extremely close to the eyeball (for example, about 1 cm), the radius of curvature of the light will be smaller when the light enters the eye. This creates two problems to be solved. First, the wavefront emitted from the point source 192 is too dispersed to be focused by the eye lens (that is, the eye cannot produce a clear image on the retina). Secondly, the radius of curvature of the wavefront emitted from the point source 188 will be very different from the radius of curvature of natural scene light from, for example, object 180. To address the first problem, the value of the radius of curvature must be higher than the threshold required to enable the eye to focus the image on the retina. To address the second problem, if the AR light source generates an object on an optical material, in order to obtain a realistic visual effect, the radius of curvature of the light representing the virtual object needs to be the same or approximately the same as the radius of curvature when the virtual object is a real object. For example, if the real scene is a golf course green with a hole, and the virtual object is a golf ball pushed into the hole, the radius of curvature of the wavefront representing the golf ball needs to match the radius of curvature of the real scene near the location where the golf ball is presented. This match makes the perception of distance realistic for the viewer. This match is equally important for a pleasant viewing experience, because it is known that if the natural match between the point where the eyes focus (accommodation) and the point where the visual axes of the two eyes intersect in binocular vision (convergence) is disrupted, then the huge difference between the wavefront curvatures presented to the viewer in a single scene will be unpleasant. This is called convergence-accommodation conflict, which can induce headaches and feelings of nausea, similar to motion sickness. In this context, it should be noted that in AR (or MR) systems, a combination of external cameras and distance measurement devices such as LiDAR can be used to survey the scene and obtain relevant information about the appropriate layout of virtual objects in the scene. AR headsets may need to superimpose images of new objects on natural scenes. This may be to superimpose objects that do not exist in the natural scene (such as superimposing an image of two tennis players playing on a natural scene of an empty tennis court). AR headsets may also need to replace images of real objects in the scene with virtual objects (such as replacing images of farm animals in a natural scene with images of dinosaurs).

[0133] Figure 4is a schematic diagram illustrating the operating principles used by an embodiment of the present invention. An eye 160 having a retina 168 is shown. A piece of transparent (in the visible range) NLO material 100 in the form of an eyeglass having a front (proximal end) and a back (distal end) is placed in front of the eye at a vertex distance. Light that leaves the eyeglass from the front toward the wearer's eye travels toward the proximal end or in an inward direction. Light that leaves the eyeglass from the back into the environment travels toward the distal end or in an outward direction. The lens is referred to as an eyeglass lens in terms of its design form, but it does not mean that the lens has any vision correction function.

[0134] The first input beam 70 and the second input beam 76 are input into the NLO material 100 and cross, thereby forming a crossover region within the NLO material 100. This is a non-collinear geometry. The first input beam 70 and the second input beam 76 have respective first and second frequencies. The first frequency and the second frequency may be the same or different from each other. In the crossover region, nonlinear mixing occurs through the sum frequency generation (SFG) effect, and a composite beam is generated. The composite beam has a sum frequency equal to the sum of the first frequency and the second frequency.

[0135] The composite beam produced by nonlinear mixing in the intersection region is effectively a point source and therefore produces its own wavefront 84 having a wave vector direction that sends the composite beam light to the pupil. The composite beam is only produced where the beams overlap (i.e., in the intersection region). The NLO material can be selected to be transparent to visible light so that the viewer can "see through" the material, which is required, for example, for AR to allow composite scene images to be superimposed on natural scenes, as schematically shown by object 180 with wavefront 188 that is farther away (the discontinuity is represented by 186).

[0136] As proposed, the physical properties of an extended source produced by nonlinear mixing differ from those of an extended source produced by projection or emission of fluorescent materials in that the nonlinear mixing of crossed beams allows wavefront generation and phase matching, which can be achieved, for example, by quasi-phase matching, thereby in principle producing a composite beam with any desired radius of curvature in the intersection region.

[0137] There are a number of different ways to produce a synthetic scene image in an NLO material. One option is to move the first input beam and the second input beam, as shown in the figure numerals 71 and 77, so that the angle at which the beams intersect is a function of the position on the imaging area to "write" the synthetic scene image into the NLO material. To achieve this, it may be necessary to change the entry point of the input beam into the NLO material. If the first input beam and the second input beam are appropriately modulated in terms of intersection angle and amplitude, a synthetic scene image can be established, for example, using some kind of rasterization. Another option is to provide a patterned cross-section for one or both input beams, such as 2D amplitude modulation, so that when the two input beams are mixed in the intersection area, a complete object is generated by the static superposition of the two input beam cross-sections and their angular spectra. A combination of the two methods can be used in conjunction with hybrid scanning and beam profile modulation. In the general concept of light that generates a synthetic scene image by a phase-matched nonlinear process, it should be further noted that the wavefront 84 generated by such a nonlinear mixing can be given a desired direction and a desired radius of curvature, the desired direction being toward the eyeball, as shown in the example.

[0138] In summary, an image is created on the retina 168 that combines the wavefront 188 from the natural scene with the wavefront 84 of the synthetic scene image generated in the lens, where both wavefronts have the same or similar radii of curvature.

[0139] Figure 5 The key concepts of Quasi-Phase Matching (QPM) are shown. QPM is a property that can be imposed in certain nonlinear crystals by a method called periodic poling. Ferroelectric nonlinear crystals are treated by periodic poling to create a periodic domain structure that modulates the sign of the nonlinear magnetic susceptibility of the crystal so that when light propagates in a certain direction within the crystal, positive and negative alternations are achieved. Therefore, periodic poling simulates the effect of stacking slices of birefringent material in an alternating crystallographic orientation, which is unrealistic in actual operation, at least except for extremely long wavelengths in the extreme far infrared or microwave range. The ferroelectric material most commonly used for periodic poling is lithium niobate, which is called periodically poled lithium niobate (PPLN) after periodic poling. Another material is periodically poled potassium titanyl phosphate (PPKTP). Recently, a technique called orientation patterning has been developed to allow periodic poling of non-ferroelectric materials, such as III-V zinc blende semiconductor crystals, such as GaAs. In addition to using the above-mentioned inorganic materials, the poled NLO materials can also be organic materials.

[0140] As shown in the upper portion of the figure, the QPM NLO material 200 comprises a series of alternative layers having an "upper" domain 202 and a "lower" domain 204, respectively. The material is structured in this way because the dispersion (variation of the refractive index with wavelength) of the material can hinder efficient nonlinear mixing due to phase mismatch. . This is demonstrated in the middle portion of the figure, which shows how the refractive index n depends on the wavelength λ. In general, the refractive index of commonly used NLO materials decreases with increasing wavelength (this is called normal dispersion). In SFG, three wavelengths are involved. A first wavelength λ_1 is mixed with a second wavelength λ_2 to produce a third wavelength λ_3 such that energy is conserved. The wavelength of the nonlinear mixing (in this case, sum frequency mixing) is set so that

[0141] 1 / λ_3=1 / λ_1+1 / λ_2

[0142] This corresponds to the conservation of energy.

[0143] Because the refractive index varies with wavelength, there is a phase mismatch between the three optical wave vectors. The wave vectors are given by:

[0144] K_i=2*πn_i / λ_i

[0145] in,

[0146] n_i is the refractive index of the NLO material at wavelength i, and

[0147] λ_i is the wavelength.

[0148] The wave vector matching diagram is shown by the lower portion of the diagram, where K_3 is longer than the vector sum of K_1 and K_2. This difference occurs because the refractive index n_3 at λ_3 is greater than the refractive indices n_1 and n_2 at wavelengths λ_1 and λ_2. The k_vector mismatch is K_G. In quasi-phase matching, the period of the QPM structure 200 is set so that

[0149] Lg=2*π / K_G.

[0150] where Lg is the QPM period. It should be emphasized that in the above discussion, it is assumed that the waves co-propagate in the same direction.

[0151] Figure 6More complex geometries of poled NLO materials are shown. Instead of a quasi-phase matching structure, which is a simple linear grating structure, the nonlinear crystal is processed by polarizing local areas so that the domains of these areas point upwards out of the plane of the picture ("up" domains), while the crystal itself naturally has "down" domains pointing in the opposite direction. The shape of the polarized regions can advantageously follow the crystal symmetry of the NLO material, and can therefore be hexagonal, for example in the case of LiNbO3. The pattern shown is a circular polarization domain in a hexagonal close packing (HCP) pattern, but any other distribution can be chosen, such as square or rectangular packing of square or rectangular domains. In addition, the patterning can be non-periodic. The figure shows the solid structure as having two vectors a_1 and a_2 with a period representing the two-dimensional polarization. In addition, two associated K_G basis vectors b_1 and b_2 are also shown. It should be noted that these vectors can be used for phase matching interactions in non-collinear wave mixing, so that other angle options can be used. The key is the presence of appropriate nonlinear grating vectors that allow the phase matching pattern to produce a composite beam that is properly directed (ie, directed to the pupil of the eye).

[0152] Figure 7 1 is a schematic diagram of a portion of a block of NLO material 100, wherein the NLO material is spatially modulated in a zone 108 with respect to its second order nonlinearity to provide quasi-phase matching with crossed input beams 70, 76. The QPM zone 108 is schematically shown as a linear grating. A first input beam 70 and a second input beam 76 are introduced into the lens from opposite ends and propagate through one or more reflections (e.g., total internal reflections from the inner and outer surfaces of the NLO material 100). For each of the first input beam 70 and the second input beam 76, one reflection from the outer surface is shown until the first input beam 70 and the second input beam 76 cross in a crossing zone overlapping the QPM zone 108. The quasi-phase matching condition is satisfied in the QPM zone 108, and a composite beam 84 is generated by nonlinear wave mixing to produce a wavefront that propagates toward the pupil 170 of the eyeball 160.

[0153] Figure 8Another representation of the input beam intersection region is shown. The first input beam 84 and the second input beam 85 have respective wave vectors k_1 and k_2, resulting in a composite beam with a wave vector k_g, which is the vector sum of k_1 and k_2. Ideally, in order to obtain maximum conversion efficiency, the vector sum of k_1 and k_2 (i.e., k_g) should be exactly equal to the wave vector associated with the QPM grating period K_G. The phase matching vector diagram is shown in the figure on the right. It should be noted that the period of the required QPM grating period can be calculated by simple trigonometry using the input wave vectors k_1 and k_2 and taking into account the dispersion characteristics of the NLO material.

[0154] Fig. 9 1 is a schematic diagram of a display panel including an NLO material layer 100. It can be seen that the first input light beam 70 and the second input light beam 76 are guided through the display panel 100 by multiple reflections from the mirror layers formed on the front and back sides of the NLO material layer, so that the light beams 70 and 76 cross each other in the intersection region 110 and produce a composite light beam wavefront 84 by nonlinear mixing. It should also be understood that the light beams 70 and 76 are focused at the intersection region 110. A distant object 180 is set at a distance of several meters from the eye, and the discontinuity is indicated by 186. Natural light is scattered from the distant object 180 toward the eye 160. Therefore, the synthetic light beam light representing the synthetic scene is superimposed on the light from the natural scene light. The light passes through the pupil 170 and enters the eye 160 to form a fused synthetic / natural scene image on the retina.

[0155] Fig.10A display panel having a multilayer structure is shown, the multilayer structure including an NLO material layer 100, a front mirror layer 136 and a back mirror layer 130. The front mirror layer and the back mirror layer are reflective at the frequency of the input beam and transmissive in the visible frequency. Two different input beam paths 70A, 70B of the first input beam are shown to illustrate how the angle and position (θ, z) at which the input beam is introduced into the display panel can be changed, thereby changing the angle at which the input beam intersects the intersection zone. The back mirror layer 138 covers the central portion of the display panel, but does not cover the left end portion of the front face of the display panel. The first input beam is introduced into the left end portion at an angle and position (θ1, z1) or alternatively at (θ2, z2). Then, the first input beam is scattered from the front mirror layer 136 to the back mirror layer 138, etc., thereby passing through the display panel to the intersection zone 108 containing the QPM grating structure 108. Similarly, a second input beam is introduced at the right end portion of the display panel (not shown) and follows beam path 76A or 76B to the intersection region 108. The first input beam and the second input beam then intersect each other in a non-collinear geometry at the QPM region 108 and produce a composite beam by the sum frequency effect. The required position and angle of introducing the input beams can be calculated by simply tracing the rays from the desired point of the intersection region 108 and determining the angle at which the input beams should intersect the QPM region 108. Since the path length of each beam will change due to changes in the angle and position at which each beam is injected into the display panel, a lens or other focusing component (not shown) located outside the display panel can be adjusted to ensure that both beams 70, 76 remain focused in the intersection region 108.

[0156] The provision of mirror layers 136, 138 allows reflection over a wider range of angles than would be possible if total internal reflection at the interface between the panel and the air were relied upon. Here, it should be noted that if the nonlinear mixing process is by frequency superposition, and assuming that the image formation is in the visible range, which is the usual case, and further assuming that the mixed beam frequencies are the same or not very different, the beams will be in the infrared (IR) wavelength range, so that the reflective coating will be designed to efficiently reflect the light of these beams at one (or more) infrared wavelengths.

[0157] Optionally, both the first input beam and the second input beam can be injected from the same end of the lens, and the lens structure can be designed so that the second beam is reflected from the reflective assembly to the right of the figure, thereby rebounding along path 76A or 76B, as shown. Therefore, it should be understood that by appropriately directing and / or scanning the first input beam and the second input beam, they can be introduced at different angles to propagate in the lens, wherein changes in angle require different introduction points to ensure that the intersection zone does not move.

[0158] It will be appreciated that the first input beam and the second input beam can only be accurately phase matched to the QPM NLO material at one particular combination of their respective angles. Therefore, scanning the input beams to vary their angles will result in the phase matching of the crossed input beams to the QPM NLO material becoming progressively less precise as the divergence of the angles from the specified ideal phase matching angles increases. Therefore, as these angles deviate from the specified angles, the conversion efficiency and therefore the intensity of the mixed composite beam light will progressively decrease. Since the conversion efficiency is a function of the beam angles, it will vary as each beam is scanned, for example by rasterization, which can be compensated by temporal variation of the beam intensity during the scan to ensure that the intensity scale produced by the composite beam light on the composite scene image is constant.

[0159] It is generally advantageous to make the distance between the front and back mirror layers large enough to keep the number of reflections (bounces) relatively low while allowing a relatively large bounce angle θ. This reduces the overlap between the light spots (i.e., the beam intersection area), thereby reducing unwanted waveguiding and mode excitation effects.

[0160] Fig.11 An exemplary optical component for directing an input light beam from a beam source (not shown) to a display panel (not shown) is shown. A light beam 220 from the beam source impinges on a first mirror 222A and is transmitted to a display panel 222A that can be rotated around an angle. The second rotating mirror 222B is then transmitted to the The third mirror 222C is rotated. Therefore, different beam paths 224A and 224B can be generated, which arrive at the display panel at different angles and positions. It should be understood that this is just an example of an optical component that can be guided, and other methods can also achieve the same function.

[0161] Fig.12aAnother exemplary optical component 230 is shown, which can be integrated into an optical component for directing an input beam from a beam source, in the figure, the component is a simple optical glass block. The optical component 230 allows the light beam to be translated, that is, its position is changed without changing the propagation angle of the light beam. The optical component 230 is mounted to be rotatable about an angle θ and is a block of glass or other optical material 230. The optical component 230 is transparent to the light beam 220. The light beam 220 enters the optical component 230, refracts at the input face and traces the path shown through the block. On exit, the beam path is translated downward to the beam path 224A, which is displaced according to the angle θ, but remains parallel to the beam path 220. Thus, a displacement with an angle change is achieved. For polarized light, the optical component (glass block) 230 can be placed to minimize unwanted reflections using the Brewster angle, or alternatively, it can have an anti-reflection coating. In miniaturized systems, there are various suitable implementation technologies, such as microelectromechanical system (MEMS) mirror technology, electro-optics and other known beam scanning technologies. The need for miniaturization and the need to limit the number of moving parts make solid-state or MEMS devices more attractive.

[0162] Figure 12bComponents for directing light to a desired location within a layer of NLO material while changing the angle at which the light approaches the spot are shown. The figure shows an input light beam (shown here as coming from the right side of the figure. The input light beam (labeled 76 in this case to match the other figures) is transmitted to the first lens 240A. The first lens 240A is positioned one focal length (f′) away from the rotatable mirror 222. Assuming that the input light beam 76 is roughly focused, the lens 240A will produce a small focused spot on the mirror 222. The light reflected by the mirror 222 is directed toward the second lens 240B. The second lens 240B is positioned a distance 2f away from the mirror 222 and images the small spot from the mirror 222 onto the desired refocused spot represented by 110, because the beam will be transmitted into the NLO material where it intersects with another beam. In the example, the distance from the lens 240B to the refocused light spot is 2f. The distance of the spot is also 2f. This focal length value meets traditional imaging requirements. When the mirror 222 is rotated about either axis (see rotation angle θ), the composite beam path will be deflected, but due to the imaging operation of the lens 240B, the spot will be emitted to the same part 110, but will be close at an angle set by the rotation of the mirror 222. Advantageously, the mirror 222 will be rotated so that its center of rotation coincides with the reflective surface of the mirror. It should be understood that different focal length values ​​can be used as long as the mirror is imaged into the beam intersection interaction area. In addition, other optical components can be added to the system to guide the intersection point to the target area of ​​the display component (AR glasses system). In addition, additional optical components can be used to ensure that the desired interaction area is of appropriate size wherever it is located within the display system.

[0163] Fig.13An AR display device according to an embodiment is shown. The first input beam has a phase-modulated beam profile, and the other input beam has an amplitude-modulated beam profile, so that in the intersection region 110 where the two input beams intersect, both amplitude modulation and phase modulation are encoded into a composite beam produced by nonlinear mixing of the first and second input beams in the phase-matched NLO material layer. The display panel produces an image of a synthetic object. The necessary information of the synthetic scene image is effectively encoded into the angular spectrum of the synthetic beam light irradiated on the eye. The focusing requirement of the eye lens 166 is to take the curved wavefront 181 from the natural scene object 180 and move it to a single focus on the retina 168. The eyeball 162 is shown as having a pupil 170, an eye lens 166, and a retina 168. The pupil 170 of the eye receives light from the natural scene object 180. The natural scene object 180 scatters the incident light, which produces a wavefront 181 having a spherical wave. The light of wavefront 181 passes through the display panel, shines on the eye, enters the pupil 170 and is focused by the eye lens 166 to form a sharply focused image on the retina 168. If light is emitted from other points on the natural scene object 180 and the light rays are tracked, the light rays will be at different angles to the visual axis of the eye and focus at different locations on the retina 168.

[0164] When the AR headset is worn, the display panel 88 is disposed in front of the eye 160. The display panel 88 is transparent to light scattered from the natural scene (e.g., light contained in the wavefront 188), while generating an artificial image in response to a crossed pair of imaging input beams (one pair is shown here). The image information is transmitted to the display panel 88 via a suitable transmission arrangement. The display panel 88 is curved as shown and has an outer surface 92 and an inner surface 90, which form an interface between the display panel 88 and the air.

[0165] The illustrated display panel 88 has a composite layered construction having a layer of NLO material and front and back mirror layers 136, 138. The mirror layers 136, 138 are reflective in the infrared wavelength range of the input light beam.

[0166] The NLO material layer of the display panel 88 has a QPM area 108, which is configured to cover a portion of the display panel 88, which is a relatively small portion as shown in the figure, but it should be understood that the QPM area 108 can extend over the entire field of view of the eye when the eye is looking straight ahead, and / or extend over the entire field of view of the eye at all possible rotation angles, and / or extend over the entire area of ​​the NLO material layer.

[0167] Referring to the left side of the figure, an amplitude modulated first input beam 70 is provided. The first input beam 70 is generated by the emitter array 16, which encodes image information in the beam cross section by driving the individual emitters of the array in a suitable manner to modulate the amplitude (i.e., intensity). The first input beam 70 is emitted by the emitter array, then reflected by the mirror 222A, and transmitted through the relay lens 240A to the left end face of the display panel 88. The mirror 222A can change the path of the first input beam 70, for example, by rotating along one or two axes, so it is deflectable. Similarly, the lens 240A can be mounted translationally and / or rotatably to change the light path of the first input beam 70. In addition, other mirrors, lenses or other optical components can be included for beam adjustment of the direction, position and divergence of the first input beam 70. Therefore, the position and angle at which the first input beam 70 is introduced into the end face of the display panel can be adjusted. As shown in the figure, the left end face of the display panel is set at a certain angle relative to the normal (i.e., the 90-degree cut angle of the display panel) to facilitate the incoupling of the first input beam 70. Alternatively, the first input beam 70 can be introduced into the front of the display panel. Another alternative is to use a grating incoupling of the first input beam 70, wherein the grating can cover a regional portion of the front or end face of the display panel. After being introduced into the display panel, the first input beam 70 passes through the display panel from left to right by continuous reflection from the mirror layers 136 and 138. With reference to the right side of the figure, a phase-modulated second input beam 76 is provided. The second input beam 76 is generated by the laser source 18. The second input beam 76 reaches the first plane mirror 222B from the laser source 18 through the beam expanders 240A and 240B, where the second input beam 76 is reflected onto the second mirror 222C. The second mirror in the beam path 222C is a phase retarder mirror such as a liquid crystal phase modulator (allowing a spatially varying phase to be imparted based on the input) and is used to apply a spatially structured phase modulation to the laser beam. With respect to the introduction of the second input beam 76 to the display panel 88 and the propagation on the display panel 88, the same explanations as described above for the first input beam 70 also apply to the second input beam 76. As shown in the figure, the second input beam 76 is introduced into the right end face of the display panel. The first input beam 70 and the second input beam 76 then cross in the intersection area 110 where the QPM NLO material is located, resulting in a composite beam having a wavefront 84 and a frequency at the sum frequency of the first input beam and the second input beam 76. The wavefront 84 enters the pupil 170 and is then focused on the retina 168. The composite beam light is superimposed on the natural scene light 181 from the object 180.By setting the radius of curvature of the wavefront of the second input beam 76 (by applying a spatially structured phase delay to the beam), the synthetic scene image can also be generated according to any desired radius of curvature, so that the synthetic scene image is a virtual image formed at any desired distance from the eye. The radius of curvature of the second input beam 76 can be changed by using Gaussian beam optics, thereby reducing the size of the laser beam. Reducing the cross-sectional area of ​​the laser beam will increase the beam divergence of the resulting nonlinear wavefront.

[0168] Thus, the virtual image distance of the synthetic scene image may be varied to provide a fused synthetic and natural scene image at the eye, wherein the synthetic scene image adapts to the natural scene in a realistic manner.

[0169] In the above embodiments, it is implicitly assumed that the QPM NLO material is a simple linear grating. However, in other embodiments, the QPM NLO material may have a more complex spatial profile including a specific curvature to produce a diverging (or converging) composite beam. In some other embodiments, both amplitude information and phase information may be applied to the first input beam 70, the second input beam 76, or both.

[0170] The emitter array 16 may be an OLED array, such as a micro-OLED array. There is a controller device 324 that controls the output of the OLED array to generate intensity information in the image. Other options for the emitter array include an inorganic LED array, a quantum dot emitter array, a liquid crystal device with appropriate backlighting, a Liquid Crystal On Silicon (LCOS) with appropriate illumination, a VCSEL array, and a tilting MEMS display device of the Digital Micromirror Device (DMD) type, such as a DLP device from Texas Instruments. It should be noted that the system does not require coherence of the amplitude structured beam, and this is advantageous in terms of allowing incoherent emitter light sources (such as micro-OLED arrays), and in terms of reducing speckle in the system.

[0171] Thus, a first input beam 70 is formed by applying different phase modulations to a beam generated by a single emitter, such as a laser, and has, for example, a Gaussian intensity profile. A second input beam 76 is generated by an array of emitters and thus has a beam cross-section that carries a spatially varying amplitude modulation. In other embodiments, the first input beam 70 may have a cross-section that is spatially modulated in both phase and amplitude, while the second input beam 76 may be modulated in phase or amplitude or both. A suitable source for combined phase and amplitude modulation would be a phase-controlled emitter array, such as a VCSEL array with external phase modulation or an independent emitter array with appropriate phase control.

[0172] An advantage of embodiments that allow both the first input beam and the second input beam to be spatially modulated in intensity is that this can be used to compensate for the fact that different portions of each beam cross-section are phase matched to the QPM NLO material with different precision. The display device can be designed so that the light along the principal optical axis of each beam is precisely phase matched to the QPM NLO material at the location where it crosses in the QPM region, while the phase matching of the crossed beam components to the QPM NLO material will become increasingly less precise with increasing distance from the optical axis of each beam. Therefore, the conversion efficiency will gradually decrease with distance from the optical axis of each beam. Therefore, the intensity profile of the beams can be modified to increase the intensity at the end of each beam so that the intensity scale of the composite beam light remains constant over the entire intersection region of the beams.

[0173] Fig.14 Schematic diagram of a beam intersection region 110 where a QPM NLO material 108 is located. The beam cross section of the first input beam 70 has a nearly Gaussian profile 81 to provide a simple flat phase front, i.e., there is no spatial phase variation across the beam, or there is only spatial phase variation inherent to the beam being a Gaussian beam. The second input beam 76 has a beam cross section that is, in general, amplitude modulated and phase modulated according to respective amplitude and phase modulation functions 75. The first input beam 70 and the second input beam 76 in the infrared range interact with each other through the QPM NLO material 108 to utilize the second-order nonlinear interaction as described above in conjunction with Figure 8 The phase matching scheme described further and the appropriate nonlinear grating K vector. The nonlinear interaction induces nonlinear polarization at a certain frequency (in this case, the frequency is the sum frequency of the first input beam and the second input beam, the sum frequency being in the visible range) to produce an output beam 82 having a wavefront 84 that varies in both amplitude and phase in a manner determined by the amplitude and phase of the input wave. Thus, the information encoded in the second input beam by the amplitude and phase modulation function 75 is transferred to the composite beam wavefront 84.

[0174] Fig.15 Is applicable to Fig.14 Schematic diagram of a setup. This means that there is a first input beam 70 whose beam cross section has an approximately Gaussian profile 81 to provide a plane wave front combined with a second input beam having an amplitude and / or phase modulated wave front. The second input beam is generated by a suitable emitter array 16, which is located at a focal length f from a relay lens 242, which is in turn located at a focal length f from the intersection area 110. The configuration places the image information contained in the amplitude and phase modulated wave front from the emitter array 16 in the Fourier plane of the imaging system. Therefore, the resulting visible light beam generated by the nonlinear mixing of the input beams will contain the same spatial image information as the spatial image information contained in the second input beam, which image information is then transmitted to the eye, essentially encoded in a plane wave. The visible composite beam light generated by the nonlinear mixing will have a wave front whose radius of curvature follows the radius of curvature of the first input beam 70, so that the composite beam light will have an equivalent imaging position determined by the radius of curvature of the first input beam 70. The foreshortening caused by the finite intersection angle of the two input beams can be compensated using simple trigonometry and taken into account in the beam profile or image construction, or both.

[0175] Fig.16 A more complex exemplary layer structure is shown for an AR display panel 88. Air 128 surrounds the display and has a refractive index very close to 1.

[0176] The figure shows the outer side of the display panel 88 farthest from the eye (also called the "front side" to match the popular description of the eyeglass lens), whose surface is located between the air and the display panel. The figure also shows the proximal surface located on the inner side of the display panel closest to the eyeball (also called the "back side" to match the popular description of the eyeglass lens).

[0177] The layers are described in order from distal to proximal:

[0178] Layer 146 is a front surface anti-reflection coating (ARC) layer AR_1 to reduce the reflection of visible light propagating from the interface between the panel and the air in the air 128. The outer anti-reflection coating layer 146 / AR_1 should preferably be anti-reflective over the entire visible wavelength range to accept light from natural scene sources. Such ARC coatings are typically deposited on optical components such as camera lenses. It is also necessary to make the layer "hard" to resist scratching.

[0179] Layer 142 is a front vision correction layer (labeled N1). It is made of an optically transparent material with refractive power (in the visible range) to correct the wearer's vision defects, such as myopia, hyperopia, astigmatism, etc. Vision correction may involve changing the curvature of the layer, or including a polarizing filter, such as is common in sunglasses. Alternatively, layer 142 may be optically neutral (i.e., providing no image enhancement effect), or be omitted. Another possible function of the rear vision enhancement layer 142 is to provide dimming in response to certain natural light conditions. For example, the vision enhancement layer 142 may include a UV-sensitive photochromic material, such as silver halide, as used in conventional dimming display panels. Another option for dimming is to use an electrochromic switchable material, such as is sometimes used for automatic dimming in automotive rearview mirrors. For example, the vision enhancement layer 142 may include tungsten trioxide. For electrochromic materials, a power source needs to be in contact with the vision enhancement layer 142 to power the dimming function.

[0180] Layer 130 is a front filter layer (denoted A1). The front filter layer absorbs the wavelength of the input beam, i.e. is opaque at the wavelength of the input beam, which is typically in the infrared range. This will prevent any light from the input beam from being emitted from the front of the display panel into the air 128. In the case where one or both of the first and second input beams are generated by a laser source, it may be necessary to provide a front filter layer 130 to comply with laser safety regulations. The front filter layer 130 may be made of an organic material that absorbs in the appropriate infrared wavelength range. In addition, the front filter layer needs to transmit visible light so as not to attenuate natural scene light. This type of filter is sometimes called a heat absorbing filter or a "short pass" filter and may be based on glass such as Schott KG1. Alternatively, the front filter layer 130 may be based on a thin film interference effect, which is advantageous when the light of the input beam is highly monochromatic and is narrow-band, such as generated by a laser.

[0181] Layer 150A is a front spacer layer of optically transparent material (in the visible range) and is used to separate the IR absorbing layer 150 from the next layer 136 .

[0182] Layer 136 is a front mirror layer for internally reflecting input light beams as they traverse the display panel 88. Providing the front mirror layer 136 avoids the use of total internal reflection at the far panel-to-air interface 146 and avoids any possible loss, scattering or other unwanted effects of the input light beams passing through the outer layers 146-150.

[0183] Layer 150B is another spacer layer that is optically transparent in the visible and infrared ranges (within the wavelength range of the input beams). The spacer layer 150B is used to increase the thickness of the material through which the input beams pass, thereby reducing the number of internal reflections ("bounces") required for each input beam to reach the intersection region.

[0184] • Layer 100 is a layer of NLO material that is quasi-phase matched to the input beams, at least in the region covered by the eye's field of view when looking straight ahead, taking into account the angles at which the input beams cross each other.

[0185] • Layer 150C is another spacer layer having similar functions and characteristics as spacer layer 150B.

[0186] Layer 138 is a rear mirror layer used to internally reflect the input light beam as it passes laterally through the display assembly 88 , and has similar functions and characteristics to the front mirror layer 136 .

[0187] Layer 150D is a rear spacer layer of optically transparent material (in the visible and infrared wavelength range of the input beam) and is used to separate the rear mirror layer 138 from the next layer 1814.

[0188] Layer 132 is a rear filter layer that is designed to absorb any stray light from the input light beam (i.e., the wavelength of the input light beam is typically in the infrared range) and has similar properties to the front filter layer 130. Thus, the rear filter layer 132 prevents light in the wavelength range of the input light beam from passing into the air 128, thereby protecting the wearer's eyes.

[0189] Layer 144 is a posterior vision correction layer that may have any of the functions or properties described above for anterior vision correction layer 142 .

[0190] Layer 148 is a rear anti-reflection coating (ARC) layer AR_2, which has similar properties to the front ARC layer AR_1 / 146. Its function is to reduce the reflection of visible light propagating in the air 128 from the panel-air interface.

[0191] One particular advantage of the described layer structure is that, by restricting the optical path of the input light beam to a specific portion of the layer stack between the front mirror layer 136 and the back mirror layer 138, vision correction can be performed without affecting the input light beam management and the nonlinear mixing of the resultant light beam. That is, vision correction of refractive anomalies in the eye lens, as performed in the described layer stack, can be achieved in one or more layers located outside the sub-stacks 136 to 138, preferably outside the sub-stacks in the distal (anterior) direction, such as in layer 142. In particular, the optical path of the input light beam can be managed without taking into account any vision correction layers.

[0192] It should be noted that some of the above layers may be omitted. The anti-reflection film layer is optional. In addition, the near-end reflection of the input beam can be performed by total internal reflection at the interface between the front panel and the air, in which case the rear mirror layer will be omitted. Similarly, if total internal reflection is used for the far-end reflection of the input beam, the front mirror layer will be omitted. Depending on the characteristics of the light source, such as whether the input beams are coherent, their wavelength and maximum output power, the rear filter layer may also be omitted, or both the rear filter layer and the front filter layer may be omitted. The front and rear vision correction layers are also optional.

[0193] It should be noted that additional layers may be added, such as providing anti-reflective coatings between the different optical materials involved, but it is worth noting that the largest unwanted reflections are likely to occur between the air and the display panel (due to the large refractive index difference to air) and between the nonlinear layer 100, which may have a considerable refractive index difference depending on the choice of NLO material.

[0194] Fig.17 An eyeball 162 connected to the brain via an optic nerve 174 is shown. The eyeball can rotate within the eye socket, so its main visual axis is aligned in different directions (e.g., A, B, and C as shown). The display panel 88 is constructed in the following manner: the quasi-phase matching grating vector Kg varies on the lens so that at all rotational positions of the eyeball (e.g., its visual axis is along A, B, or C), the QPMNLO material is oriented to produce a composite beam of light scattered toward the iris. This can be achieved by orienting the area portion of the NLO material to the appropriate position and orientation within the display panel. Alternatively, this can be achieved by using the display panel and the appropriate curvature of the front and back sides thereof. The front and back curvatures will affect both the wave vector direction of the composite beam within the display panel and the path of the input beam. It should be noted that, depending on how the light is introduced into the panel (the relative position within the panel and the curvature of the reflective layer), it may be advantageous to place Kg in a direction that achieves an ideal regional phase match (including the influence of the input angle and the optical axis).

[0195] Fig.18 It is shown that the display panel can have different shapes. Display panel 88A is planar, having parallel and flat front and back faces. This provides an input beam path that is easy to determine. Display panel 88B has a front and back face with the same radius of curvature (i.e., the front and back faces have respective curvatures associated with respective center points that are offset). Display panel 88C has a front and back face that are curved with different radii of curvature, so that the two circles have a common center point, thereby providing a display panel of constant thickness. In all cases, the curvature can be on the surface of a cylinder or on the surface of a sphere. Alternatively, different curvatures can exist in two planes to provide a more complex curved surface.

[0196] Fig.19is a schematic perspective view of a display device in the form of a pair of AR glasses 10. The glasses 10 include conventional parts, including left and right temples (i.e., temples) 32, 34, and a frame having a left frame and a right frame for accommodating a left eye lens and a right eye lens. The above-mentioned display panel forms each eye lens. The frame includes a nose bridge, a nose pad, and a left temple connection point and a right temple connection point that may be hinged or not. In order to support its AR function, compared with traditional glasses, the temples and the frame can be modified to accommodate necessary additional components through an internal housing and / or external accessories. Such components may include electronic circuit components, batteries, light sources, and optical components. A wireless transmitter or transceiver 40 may be included for communicating according to a wireless protocol such as a Bluetooth or WiFi protocol. The wireless transceiver 40 on the temple allows the AR glasses 10 to communicate wirelessly with an external control device 44 via a communication path 42, and the external control device 44 has its own wireless receiver or transceiver 43, and is processed by a processor 46 and associated memory 48 and external network communication 49, which may include WiFi, 4G / 5G, optical LAN, wired Ethernet, etc. The external control device may also provide a positioning device, such as a global positioning sensor (GPS), and position detection through a telecommunications wireless signal map and / or a wireless network signal map. The external control device may be a dedicated stand-alone unit specifically used to control the AR glasses 10, or a mobile phone, tablet computer, personal computer, etc. running a suitable application (computer program). In addition, the external control device can communicate data with other devices, and in particular, it may have access to remote computing resources in an ad hoc network (such as resources provided by cloud computing). Therefore, processing-intensive tasks for controlling imaging in AR glasses can be transferred to a control device outside the AR glasses, and can be selectively further delegated to more remote computing resources that communicate with the external control device. Therefore, by limiting the processing tasks performed by the processor integrated in the glasses to those with relatively low processing intensity, the processing power of the AR glasses themselves and the related power consumption can be kept low.

[0197] The frame houses an external camera 50 on the left and an external camera 52 on the right, which look outward at the natural scene. Using a pair of cameras arranged side by side at the same height allows stereoscopic images to be established, i.e., stereoscopic vision similar to the binocular vision of the wearer's eyes, thereby allowing the distance to objects of the natural scene to be determined, at least when the distance is relatively short. The frame 10 further houses a rangefinder for detecting objects in the natural scene and determining their distance from the wearer, such as a LiDAR device or a point cloud imaging system 58. In a LiDAR system, the LiDAR device includes at least one laser and a detector, and measures the transit time of the laser from the LiDAR device to the scattering source and back. The laser beam can be scanned to establish a map of objects in the natural scene, or multiple laser beams can be generated to allow information to be collected from multiple points in the natural scene in parallel. The movement of the laser beam caused by the natural rotation of the wearer's head can also be used to scan the natural scene. In a point cloud imaging system, the laser beam is projected into the natural scene imaged by the cameras 50, 52 to allow a model of the geometry of the natural scene to be established. The object information, especially the object distance information, collected by the camera and the rangefinder has a particular synergy with the AR glasses according to the present invention because the QPM nonlinear mixing process allows the radius of curvature of the wavefront generated in the wavefront of the synthetic scene image to be set to a desired value in response to the distance (and position) information measured by the rangefinder (and the camera). For example, if the image enhancement portion needs to superimpose features on objects in the natural scene (or completely replace objects in the natural scene), the image enhancement portion can be generated by wavefronts whose radius of curvature is equal to the radius of curvature that the image enhancement portion should have at the measured distance of the relevant natural scene object. Similarly, the enhanced synthetic scene image can be correctly positioned in the natural scene based on the position information collected by the camera. Of course, the position and distance information can also be combined to assist the segmentation process of the natural scene to identify objects in the natural scene.

[0198] One potentially useful remote computing resource is one that provides image information related to the wearer's real-time view of a natural scene, for example as an aid in segmenting the natural scene and providing realistic lighting when rendering augmented reality objects. For example, this image information could come from an image application, real-time satellite imaging, or live flight tracking.

[0199] The amount of potentially useful computing resources is almost unlimited. For example, computing resources can be used to reconstruct an entire simulated version of a natural scene in real time based on what is observed by the camera and range finder combined with the additional image information obtained from removing the computing resources. Augmented reality objects can then be positioned in the rendered simulated version of the natural scene and volume to provide realistic lighting based on textures, etc.

[0200] Fig. 20Another view showing additional components that can be combined into a display device in the form of AR glasses.

[0201] 100 is a transparent or nearly transparent optical component containing NLO material and generating the aforementioned IR beam of visible light directed to the eye.

[0202] 26 is a frame, which together with the temples (glasses legs) accommodates the necessary electronic and optical components.

[0203] · 50 and 52 are left and right external cameras.

[0204] 58 is a point cloud or LiDAR system.

[0205] 60 is an infrared light detector disposed on the frame around each lens. At least one light detector is required for each lens. The infrared light detectors are used to detect light leakage from the input beam, and these infrared light detectors should be substantially completely contained within the lens (and before being installed in the temple and frame portion of the glasses). Any increase or increase in the light detector signal indicates damage. The damage may be a deep scratch on the lens or physical damage to the frame or temple that exposes the end face of the lens. When infrared light leakage or excessive infrared light is detected, the glasses will be shut down, that is, the light source will be turned off, thereby ensuring safety, including laser safety in the case of using one or more laser sources.

[0206] 54 and 56 are other internal (i.e., toward the eye) cameras for tracking eye position. This can be used to assist in determining the optical image that needs to be transmitted, and to ensure that light is only directed when it needs to be received by the iris. This can save power, thereby extending battery life. Such cameras may also be able to collect information about the viewer's stereoscopic vision, so that convergence and accommodation conflicts can be handled and the artificial image can be correctly positioned relative to the natural scene. Importantly, the two cameras can be used to compensate for differences in interpupillary distance (equivalent to the distance between pupils) between different viewers, and ensure that light from the display is directed to the pupil of the viewer's eye. This has the effect of automatically increasing the eye movement range and overcomes the main difficulties of grating-based or other AR implementations.

[0207] 62 represents a magnetic compass assembly which may be included to assist in determining the compass direction the glasses and their wearer's head are facing. Again, this information may be processed in the electronics on the glasses or using remote processing and may be fused with the image information.

[0208] 64 is a set of accelerometers that can be used to track the movement of the glasses and determine the parameters of the roll, pitch and yaw angles of the wearer's head, assuming that the reference direction of such angles is when the head remains horizontal and the cervical spine is not twisted. The accelerometer data can be processed within the onboard electronics or by external processing to track the movement of the glasses and therefore the wearer's head.

[0209] Although not shown, the AR glasses may be provided with an ambient light sensor, such as a light detector sensitive in the visible range, for measuring the overall (non-directional) brightness of the natural environment. The output from the ambient light sensor may be used to adjust the overall brightness of the synthetic scene image so that it remains visible in bright light conditions while being less bright in low light conditions (e.g., when in a dim room or at night).

[0210] An internal camera may be used to compensate for differences in pupil distance between different users and the accompanying changes in the position of the eye relative to the glasses to ensure correct 3D composite scene creation and to ensure that light enters the pupil correctly.

[0211] Fig.21A display panel 88 is shown with an additional layer 140 for providing dimming in response to certain natural light conditions. For example, the dimming layer 140 may include a UV-sensitive photochromic material, such as silver halide, as used in conventional dimming display panels. Another option for dimming is to use an electrochromic switchable material, such as is sometimes used for automatic dimming of automotive rearview mirrors, train and commercial aircraft windows. For example, the electrochromic material may be tungsten trioxide. For electrochromic materials, a power source 139 (voltage source or current source) needs to be in contact with the dimming layer 140 through suitable electrical connections 141 to power the dimming function. Another option for the dimming layer is a liquid crystal layer with modulatable transmission, which may be pixel-based. The dimming effect may be applied to the entire display panel (such as in photochromic lenses). Another advantageous approach is to selectively apply the dimming effect only in certain areas of the image. A simple example is to form a dark rectangular area to provide a suitable background for a synthetic scene image component that displays written information, which would otherwise be difficult to read if the written information was superimposed on a bright natural scene portion. A more complex example is to provide selective regional dimming based on information obtained from the natural scene, in particular objects that have been marked by segmenting the natural scene using data collected by the glasses' forward-facing camera and rangefinder and / or other data collection sources, which can be obtained by vehicle systems, for example, when driving a car or flying an airplane. The regional dimming can be completely consistent with the range of objects in the natural scene to attenuate light (in whole or in part) from the natural scene objects, wherein local regional dimming can be combined with inserting a synthetic scene image into the same local area to be completed. There are many examples in which it is necessary to attenuate excessively bright components in the natural scene so that the rest of the natural scene cannot or is difficult to view. This includes, for example, attenuating oncoming car headlights and sunlight when looking up. Another example is suppressing light pollution when viewing the night sky. Another example is that AR glasses are designed for driver assistance facilities for night driving. At this point, other information from cameras and radars that are part of the car can also be incorporated into the image processing. A night vision augmented reality driving experience may be produced that can reveal or enhance natural scene objects that are normally visible during the day but are invisible at night or too dark to be clearly visible, thereby improving road safety.

[0212] Fig. 22A schematic diagram of a pair of AR glasses 10 is shown, wherein the NLO material in the lens 100 is formed in discrete areas (i.e., spots 104) over the image area, which spots are embedded in areas of material that are not NLO material, or that are NLO material, but as areas that have not been periodically polarized (or otherwise quasi-phase matched to the input beam). As a result, visible light generation is confined to the spots of QPM NLO material, wherein the remaining areas between the spots are inactive, i.e., unable to generate any significant flux of visible light by nonlinear mixing of the input beam. As the eye rotates in its orbit, only a relatively small amount of synthetic scene image light needs to be ensured to reach the eye, and considering that the iris width can be 4mm to 8mm and the lens surface is only about 12mm away from the eye, the NLO material can be confined to a relatively small local spot without image perception degradation compared to the full area coverage of the QPM NLO material. In other words, considering the total area covered by the QPM NLO material, the fill factor of the QPM NLO material can be relatively small, for example between 5% and 20%. For example, the spots may be circular, 0.3 mm to 0.5 mm in diameter, and distributed in a Hexagonal Close Packed (HCP) grid with a grid spacing of 1 mm to 2 mm. The lower portion of the figure shows details in a portion 104 of a lens according to one embodiment for RGB imaging. Each spot is actually a composite group of three spots 106R, 106G and 106B with different QPM structures to produce red, green and blue light, respectively, allowing the formation of an RGB image. Alternatively, RGB imaging can be achieved when the three QPM structures for red, green and blue are spatially superimposed (i.e., embedded in the lens at different depths z at the same xy location). Another alternative for RGB imaging is to polarize the NLO material according to a more complex single polarization pattern that is phase matched to all three colors (red, green and blue). There is an advantage in the approach of designing the lens as a grid of spots where mixing can occur, rather than a design with a continuous area of ​​QPM NLO material over the portion of the lens where the synthetic scene image is to be formed. That is, nonlinear mixing (i.e., the generation of visible light for the synthetic scene image) can only occur at specific locations where the light spot exists. This simplifies beam management, such as beam steering and beam modulation, because beams crossing outside the light spot do not generate any visible light and can therefore be allowed.

[0213] Fig.23is a graph showing the dependence of the QPM grating period P on the angle θ, which is 90° minus the half angle of reflection from the normal to the eye axis. The example is calculated for the nonlinear mixing of two 1064nm beams to produce green light at 532nm. It can be seen that if the display panel is traversed by the input beam by shallow angle reflection (right end of the graph), the required period of the QPM grating becomes quite small, which may make manufacturing more difficult and less accurate. For example, at 60 degrees (i.e. 30 degrees), the required periodicity is about 400nm in the UV, and about 300nm at 70 degrees. At angles close to the critical angle for total internal reflection (i.e. about 40 degrees in a typical glass-air interface), the grating period is about 800nm. If the lens is provided with an inner infrared reflecting material layer and an outer infrared reflecting material layer, the input beam passes through the lens by specular reflection that can use a larger angle reflection. For example, at a 2 micron grating periodicity, the angle is about 22 degrees. On the other hand, each reflection causes losses, for example through scattering, so the ideal angular range to be chosen is one that provides a good compromise between ease (and precision) of manufacturing and having sufficiently shallow reflections to keep the total number of reflections required to intersect the input beams in each beam pair relatively small.

[0214] Fig.24 Schematic diagram of a structure for introducing a first input beam 70 and a second input beam 76 into an eyeglass display lens. The first input beam 70 and the second input beam 76 are introduced into the respective distal and proximal lens layers in the dual-layer lens structure in the temple or temple side 264 of the lens. Each input beam 70, 76 passes through its lens layer. At the bridge side 266 of the lens, the first input beam 70 in the distal layer is reflected by the mirror 260 (the two mirrors are 90 degrees to each other) back into the proximal layer so that it propagates in the opposite direction to the second input beam 76 to reach the desired area 110 where the beams overlap in the NLO material layer. There is an additional mirror layer 262 in the display assembly, which is reflective in the infrared wavelength range of the first input beam 70 and the second input beam 76. Therefore, with the design described, the first input beam 70 and the second input beam 76 are crossed in the area 110 as required and are introduced at the same end of the lens at the same time, which end can preferably be the temple side as shown. It should be noted that other methods may be used to introduce the first and second input beams, including using a suitable light pipe or waveguide structure to guide them around the lens. Likewise, it is not necessary to use two separate layers (as shown), as a single layer may also be used to retroreflect the first input beam 70.

[0215] Fig.25A time slice sequence of a single example frame covering both RGB and ROC time multiplexing to produce a synthetic scene image is shown. To form an image frame, a series of time slices are provided within time t, each time slice being specific to a primary color, such as red, green, blue (Red, Green, Blue, RGB), and specific to a radius of curvature (Radius OfCurvature, ROC), such as 0.25 meters. Subframes of three time slices produce a synthetic color image at one ROC. In order to produce a color synthetic scene image containing objects at multiple distances in the scene, multiple such subframes are required, one for each ROC. In a full frame, a synthetic subimage is formed at multiple different perceived depths in the synthetic scene. Therefore, by displaying the relevant display fields in sequence, a realistic white light effect can be established and any color can be rendered, such as in a conventional display based on, for example, an RGB-based projector. The exemplary radii of curvature shown are 0.25m, 0.5m, 1.0m, 2.0m and 4.0m. In the example, they are shown in a monotonically increasing sequence, which will then be repeated. However, it should be understood that any given frame can use any sequence of time slices. In fact, a sequence different from the sequence shown can be more efficiently generated by hardware and / or produce images that are perceived by human vision as having higher quality. The five exemplary ROC values ​​shown have a minimum ROC value at 0.25 meters, which is approximately equal to or just below the near point of normal vision. The value enables very close, even slightly too close objects to be displayed in focus. The maximum distance in the example is 4 meters, which is actually equivalent to the maximum distance (or infinity) of a normal eye in monocular vision. It should be understood that a smaller or larger number of distances can be selected, and the distance and frame can also be adaptively controlled. In addition, the number of ROCs per frame can vary according to the content of the synthetic scene, for example, it can decrease or increase as certain objects leave or enter the synthetic scene. As previously described, the ROC value is set using a nonlinear mixing process and changing the intersection angle of the first input beam and the second input beam.

[0216] Fig.26A schematic cross-sectional view of a glasses-type system is shown, which includes the standard components of a frame 26, which accommodates left and right eye lenses 22, 24 separated by a nose bridge 28 and has left and right temples (eyeglass legs) 32, 34. In the embodiment, left and right internal cameras 54, 56 are mounted on or adjacent to the nose bridge 28 to capture images of the left and right eyes 162, which are shown in cross-sections through the eyeballs. The internal cameras 52, 54 are located on the nose wings of each lens in the figure. However, it should be understood that alternative layouts can also be used, such as on the temple side of each lens. The function of the internal cameras 54, 56 is to track the gaze direction of the person wearing the AR glasses. The provision of internal cameras for eye tracking allows for multiple advantageous features and functionality. First, the position of the pupil (and gaze) allows the system to illuminate the appropriate area of ​​the display so that the viewer sees the image in the correct position. With the help of an external camera and information from LiDAR or other predetermined scene information plus position and orientation (from a compass or accelerometer), appropriate artificial scene information can be displayed. Secondly, the eye position can be used to create a high-resolution image that will be received by the fovea area of ​​the retina, which is the part of the eye with the highest resolution. Therefore, a limited number of spatial pixels on the modulator or emitter array can be used to create a high-resolution image, and then when the gaze shifts, the input beam can be directed to a different area of ​​the eyeglass display lens and produce a high-resolution image at the new position. It should be noted that it may also be advantageous to display information (with lower resolution, but with the correct color and intensity) in a more peripheral part of the field of view (by directing appropriate beams to other areas of the lens) in terms of generating the ideal synthetic scene. A third feature of eye tracking is that it can be used to dim or blank the synthetic scene as the eye shifts its gaze (over a larger angle) so that the viewer does not perceive unnecessary ghosting. A fourth advantage of eye tracking with the aid of internal cameras 54 and 56 is that they ensure that all generated light is directed toward the pupil of the eye, thereby increasing the electro-visual efficiency of the display, but also avoiding visible light being directed to other parts of the viewer's eye (or other locations on their eyelids or surrounding skin), which can be perceived by another person observing the wearer of the AR glasses display system. A fifth advantage of eye tracking is that the system can be adapted to compensate for differences in pupillary distance (the spacing between the eyes). Since the distance between the eyes of different people may be different, it is advantageous to be able to adapt the system to direct light to the eyes of an individual. This can be achieved by changing the input angle of the two light beams entering the NLO material to guide the wavefront direction of the emitted light.

[0217] Fig. 27Shown is a time slice sequence of a single example frame covering RGB and ROC time multiplexing to produce a synthetic scene image, which also includes a glance movement blanking period 282. When the system is blanked, the illumination (one or both of the input beams) is turned off or switched to low intensity so that no synthetic scene image is formed during the blanking period. The decision about when to blank is made by a control system 280, which performs signal processing based on the eye tracking information obtained from internal cameras 54, 56. The purpose of blanking is to make no light be sent to the eye when the eye rotates. This avoids unnecessary ghosting and stripes in the visual field. This is particularly important in a pulse system because a pulse can cause an instantaneous image to be projected onto the retina. Pulses may occur because one or two beam sources are pulsed (e.g., pulsed lasers) or due to the pulse effect caused by the time slice of color and radius of curvature RGB, ROC.

[0218] Fig.28 An absorption layer disposed in the peripheral region of the display assembly lens is shown by the example of an eyeglass 20. The eyeglass 20 is surrounded by an absorption material zone 290 positioned to absorb the input beams after they have crossed in the NLO material and reached the peripheral zone, thereby preventing further propagation of the input beams by reflection from the edge of the lens after the input beams have reached their destination. The absorption material may preferably have a refractive index similar to that of the lens material (to avoid Fresnel reflections), while integrating the absorption function by molecular absorption, dye absorption, or black absorbing particle absorption. The same design can be used for display panels of other specifications, such as large area displays for conferences or single panels for both eyes such as can be used as VR headsets. The common point of this design is that there is a peripheral zone containing a material that is absorptive to light at a first frequency and a second frequency, so that when the first input beam and the second input beam reach the peripheral zone, the first input beam and the second input beam are absorbed after the first path and the second path have crossed each other in the NLO material. There are also photodiodes 60 disposed at the periphery of the lens 20, which, as previously described, can be used to check the integrity of the lens structure and provide a shut-off safety feature in the event of damage. The lens periphery also contains a non-absorbing region 292 through which the input beam 70 / 76 is injected into the lens 20, i.e., the region of absorbing material does not extend completely around the periphery of the lens 20.

[0219] Fig.29(Upper portion) shows a schematic diagram of a pair of AR glasses 10, wherein the NLO material in the lens 100 is formed in discrete areas (i.e., spots 104) over the image area, which spots are embedded in areas of material that are not NLO material, or that are NLO material, but as areas that have not been periodically polarized (or otherwise quasi-phase matched to the input beam). As a result, visible light generation is confined to the spots of QPM NLO material, wherein the remaining areas between the spots are inactive, i.e., unable to generate any significant flux of visible light by nonlinear mixing of the input beam. As the eye moves in its orbit, only a relatively small amount of synthetic scene image light needs to be ensured to reach the eye, and considering that the iris width can be 4mm to 8mm and the lens surface is only about 12mm away from the eye, the NLO material can be confined to a relatively small local spot without image perception degradation compared to the full area coverage of the QPM NLO material. In other words, considering the total area covered by the QPM NLO material, the fill factor of the QPM NLO material can be relatively small, for example between 5% and 20%. For example, the spot may be circular, 0.3 mm to 0.5 mm in diameter, and distributed in a Hexagonal Close Packed (HCP) grid with a 1 mm to 2 mm grid spacing. In the described approach, there is the advantage of designing the lens with a grid of spots where mixing can occur, rather than a continuous area design with QPM NLO material over the portion of the lens where the synthetic scene image is to be formed. That is, nonlinear mixing (i.e., visible light generation of the synthetic scene image) can only occur at the specific location where the spot exists. This simplifies beam management, such as beam steering and beam modulation, because beams crossing outside the spot do not produce any visible light and can therefore be allowed.

[0220] Fig.29 (Lower left) shows a first way to implement each spot, where spot 103A includes a stack of three different QPM periods, 206R, 206G, and 206B, designed to be phase matched with appropriate nonlinear mixing processes to produce three different wavelengths, such as red, green, and blue wavelengths, respectively. As shown, these layers remain in contact (as can be achieved by electric field poling), but it should be understood that gaps of inactive material can be interspersed.

[0221] Fig.29(Lower right) A second method for implementing each spot is shown, where the spot 103B is implemented as a superstructure grating 208 designed so that phase matching can be performed for three colors (e.g., red, green, and blue) simultaneously. This can be produced by polarizing the NLO material according to a more complex single polarization pattern that is phase matched to all three colors (red, green, and blue). A superstructure grating is also shown, with its layers tilted slightly away from the plane of the NLO material layer. The grating for each spot can be tilted in a manner that varies from one spot to another within the lens area, thereby directing a phase-matched output beam toward the pupil - as described above in conjunction with Fig.17 As further described, the same area tilting method can be used for the multi-stack light spot 103A described above.

[0222] Fig.30 Two schematic diagrams of a spectacle lens 20 for AR glasses are shown. The upper part of the figure shows a front view of the lens 20, and the lower part of the figure shows a top view, that is, a view as a cross section through the lens. In the front view, the first input beam 70A, 76A and the second input beam 70B, 76B enter the lens 20 from the left and right sides of the lens 20 via the entrance areas 292A and 292B, respectively. The first input beam 70A, 76A and the second input beam 70B, 76B cross at two different intersection positions 110A and 110B. When the eye behind the lens turns, the line of sight will be guided to multiple such positions, and it is necessary to create synthetic scene information received by the pupil of the eye. In the example, the intersection position 110A is a gaze toward the top, and the intersection position 110B is a gaze toward the bottom. As an example, two different arbitrary first input beam directions 70A, 76A and two different second input beam directions 70B, 76B are shown. The lower portion (top view) is a cross-sectional view of the lens showing the bounce path. It will be appreciated that the respective intersection angles in the nonlinear overlap regions 110A and 110B need to be considered in three dimensions, as can be clearly seen by combining the overview front view and the top view, where the input beam injection direction shown in the front view and the lens passing through due to bounce are related. From the figure, it will be appreciated that the direction of the vector that crosses at a given point on the lens 20 depends on the location on the lens towards which the gaze is directed. The synthesized wavefront needs to be directed to the viewer's pupil, so the phase matching diagram (which is now a 3-dimensional vector) must be designed so that the light is properly directed. This can be achieved by setting the QPM direction to the appropriate orientation of the lens surface so that phase matching in the desired direction can be achieved.

[0223] It should be noted that although the phase matching period and QPM grating vector will be determined when the lens is manufactured, there are still controllable degrees of freedom to ensure that the light resulting from the NLO process is directed to the eye. First, at a given crossover location 110A or 110B, the bounce angle θ of the first input beam and the second input beam can be independently controlled (see Fig.10 ). Secondly, the wavelength of the first input beam and / or the second input beam can be slightly changed to change the phase matching condition, but this requires that at least one of the beam sources is tunable. As previously described, by using an internal camera, it is possible to ensure that the light from the system is efficiently directed toward the pupil and to accommodate different pupil distances. As shown, the glasses can be designed so that the light beam enters each lens at an intermediate height position 292A and 292B from the opposite side of each lens. The light beams are then guided to cross at different positions 110A and 110B as needed. As described above, in order to ensure that light is efficiently directed from the lens to the wearer's pupil, the QPM grating vectors need to be locally adjusted at the crossing positions 110A and 110B to ensure that a phase-matched wavefront (or a wavefront that is close to phase-matched) is propagated to the pupil, that is, so that the wavefront has an appropriate wave vector.

[0224] Fig.31 An exemplary lens 300 suitable for VR goggles is shown. The lens 300 has an embedded NLO material layer 100. The first input beam 70 and the second input beam 76 enter the lens 300 from the distal side, which is possible for VR where there is no natural scene that is transmitted on the eye and crosses in the NLO material layer 100 at the intersection area 110. The fact that the lens 300 can be made much thicker in VR than is possible in AR also helps the input beam to be injected from the distal side of the lens 300. The thickness of the lens 300 can be, for example, 50mm. The illustrated form of the lens 300 is a roof shape with a first inclined facade and a second inclined facade, the surface of which forms an approximately orthogonal angle with the first input beam 70 and the second input beam 76 at the injection point. The other features of the system remain the same as described above for the end injection of the input beam in a bounce-through manner to reach the intersection area where the input beams cross.

[0225] Fig.32Schematically illustrates imaging in a NLO material layer with a nonlinear QPM grating 108 when the source of the first input beam is an incoherent emitter array 321 that emits multiple objects as incoherent infrared light (e.g., from a monochromatic micro-LED array), thus serving as a first input beam. A lens 325 is disposed between the emitter array 320 and the QPM grating 108, at a focal length f from each of them. The three emission regions are pixels 322, 323, and 324 in the figure. Each of them has the same radius of curvature. As shown, an input infrared laser wave 317 serving as a second input beam and having an intensity profile and a phase profile (corresponding to the radius of curvature) enters the QPM grating 108 (from the left). The input laser wave 317 may be given three different radii of curvature 318, 319, and 320 at different times. By varying the ROC of the input laser beam from the left and by providing different intensity patterns from the micro-LED incoherent emitters, continuous wavefronts can be generated that can be rapidly cycled in sequence so that the viewer perceives the presence of different objects at different focal positions of the visible nonlinear waves emitted by the QPM grating 108 as a result of nonlinear mixing of the input beams.

[0226] Overview of the options for phase and amplitude modulation between beams

[0227] The following will describe possible implementations of the amplitude and phase modulation functions, particularly around whether both are applied to a single input beam or to each of the input beams separately, and whether the beams are large area beams (or an equivalent array of beams produced by an array emitter) or rasterized thin line beams. While the following is intended to provide a comprehensive overview of the various schemes, it is not exhaustive of all possibilities, for example, other variations can be easily understood by combining the elements of the specific embodiments described below. In addition, it should be noted that the labeling of the two input beams as "Beam 1" and "Beam 2" is only arbitrary.

[0228] Table A: Example method for amplitude modulation

[0229]

[0230] Table B: Example method for phase modulation

[0231]

[0232] 1 A more sophisticated approach could use a Fresnel zone plate, which has the advantage of being able to use an absorptive spatial modulator and exploit the coherence of the laser-like input beam.

[0233]

[0234] Fig.33a A series of panels are shown, illustrating a specific approach in the method for controlling the amplitude and phase of two input beams described in Table A. The panels are labeled with reference marks taken from the corresponding columns in Table A. It should be understood that in the following, for convenience, the beam from the left side of the interaction zone 110 is labeled 70 and the beam from the right side is labeled 76, but the directions of the beams can of course be interchanged as long as the beams cross in the appropriate overlap zone 110. It is also possible that for the simplicity of the system design, the system is mirror-symmetric for the right lens and the left lens, thereby maintaining the symmetry of the two sides of the glasses. For example, in two glasses, one of the input beams (e.g., the first input beam carrying the phase modulation) is injected from the bridge of the nose, while the other input beam (e.g., the second input beam carrying the amplitude modulation) is injected from the temple (or temple) side.

[0235] Fig.33a A scheme suitable for raster scanning and overall beam modulation methods is shown in panel Ai). It shows two beams 70 and 76 interacting in a nonlinear region 110. The first beam is emitted by a laser 18, which is driven by a laser controller unit 19. The laser controller unit provides electrical energy to the laser and, by varying the current, can modulate the output power as the beam is scanned at a certain angle to create an image. The figure shows a discontinuity 186 to indicate the presence of other optical components located in the optical path (e.g., at a scanning angle, etc.). In the embodiment described, the power of the laser beam 70 is controlled by the laser controller unit 19. The second beam 76 has a phase profile 80 in the figure that is suitable for controlling the radius of curvature of the synthesized nonlinear beam (not shown). As described in Table A, the intensity of the overall beam 70 can also be controlled by placing a modulator in the beam path. The modulation component can be transmissive or reflective.

[0236] Fig.33aIn panel Aii) a beam generated by a laser is shown, to which a complex amplitude modulation is added. It is different from the previous example, in which the intensity of the entire beam is modulated, because a spatial light modulator is used to change the intensity of the beam. The figure shows two beams 70 and 76. The first beam 70 is amplitude modulated by a spatially varying reflective amplitude modulator 120 (e.g., a spatial light modulator based on reflective LC). The figure shows an amplitude modulator controller 121 electrically connected to the modulator 120. The example also shows a laser 18 and a laser controller 19. The laser controller can control the total amount of light in the beam, while the modulator controller 121 drives the modulator 120 to produce a spatially varying intensity profile on the laser beam. As previously described, discontinuity 186 indicates that there are other optical components in the beam path of 70. All controller components are controlled by a central controller (not shown) that controls imaging. The second beam 76 has a controlled phase profile 80.

[0237] Fig.33a In panel Aiii) a system is shown in which amplitude modulated light is produced by a device having multiple individual emitters (e.g. a micro LED device). In the method, a first light beam 70 is produced by a separate pixel emitter source from an emitter array 321. The emitter array is driven and controlled by a separate controller 324, which sets the intensity of each emission component. Light emitted from two pixel locations is shown. It should be noted that the array is typically a 2-dimensional array. As shown, the light emitted by the emitter array 321 is divergent (as is the case with a micro LED device), and a lens 240 is placed to collect the light and direct it in the direction of the first light beam 70. It should be understood that the first light beam 70 is no longer similar to a laser beam, but rather it is radiation emitted by an emitter pixel and directed to the interaction zone 110 by an optical system. The emitter array can be incoherent between pixels (e.g., this is the case with a micro LED array) or it can have coherence between pixels (e.g., this is the case with a single large laser spot that has an amplitude modulator disposed on its emitting surface). Alternatively, an array of emitting lasers (e.g., an array of VCSELs) may be used, and again may be coherent or incoherent between each emitter (note: a VCSEL is a vertical cavity surface emitting laser). The provision of a hybrid or composite emitter is described as Example Aiv in Table A, which describes placing a modulator (e.g., a liquid crystal modulator) on the emitting surface of a larger emitter (e.g., a laser or LED).

[0238] Fig.33aA combined phase and amplitude modulator is shown in panel Av). The example is a variation of the example shown in panel Aii. A laser 18 and a laser controller 19 are shown. The light from the laser is reflected by a combined amplitude and phase modulator 123a. This is controlled by a controller 123b, which assigns both amplitude information and phase information to the first light beam 70. The combined amplitude and phase modulator 123a can be a separate device, such as an improved LCOS device with a phase control layer, or can be implemented using adjacent separate components. Similarly, the device can operate in a transmission or reflection mode. The second light beam 76 is no longer assigned any controlled phase because the function of changing the radius of curvature of the emitted light is now achieved by applying phase information to the first light beam 70. Therefore, the phase 80 is now flat on the beam. However, the beam may still have some more complex phase function (for example, it can be focused), but it is not modulated when generating an image to generate a radius of curvature.

[0239] Figure 33b 1 shows a method of controlling the phase on the beam according to Table B. It shows a series of panels demonstrating different methods. The panels are labeled with reference numbers taken from the corresponding columns in Table B. The panel diagram will focus on the role of the second beam (labeled 76 in the figure), but it should be understood that the labeling of the first beam and the second beam as 70 and 76 is only arbitrary. Similar to Fig.33a Description, it should be recognized that for the convenience of system construction, the left lens and the right lens of the specific display device (AR glasses) have mirror symmetry.

[0240] Figure 33b In panel Bi) a pair of light beams 70 and 76 are shown interacting in the nonlinear region 110. The first light beam (from the left) is designated 70 and is based on Fig.33a and Table A. The second beam 76 is the beam from the right. It passes through a transmissive spatial phase modulator 122 that imparts a desired spatial phase profile to the beam 80 in the figure. There is a discontinuity 186 in the scale, which indicates that other components can be added to the optical path. The spatial phase modulator 122 is controlled by a phase modulator controller 123, which is used to create a phase pattern corresponding to the diverging beam to change the radius of curvature of the light emitted from the NLO region 110, thereby creating a visual effect of different focal points for a viewer of the display device.

[0241] In addition to reflecting the second light beam 76 by the phase modulator 122, Figure 33bIn panel Bii), similar to Bi), the phase modulator 122 is similarly controlled by the spatial phase modulator controller 123. For example, the phase modulator 122 may be a deformable MEMs mirror, a liquid crystal (LCOS) type device, or a thermally deformable mirror assembly. It should be noted that example Biii) in Table B is not shown in the figure because it is similar to the configuration of Bii).

[0242] Figure 33b Panel Biv) in Table B is not shown because it involves integrating amplitude and phase modulation in a single hybrid component and is the same as, for example, Av.

[0243] Figure 33b A diffraction intensity modulation method for changing the radius of curvature of the light beam 76 is shown in panel Bv). It superficially looks similar to Bi, except that the modulator is now a spatially controlled intensity modulator 120. In the figure, it is shown in transmission, but it will be appreciated that it can also be implemented in reflection. The intensity modulator 120 is controlled by an intensity modulator controller 121. In the simplest configuration, the radius of curvature of the light beam 76 (equivalent to the phase profile 80) can be changed simply by transmission through an aperture of controllable width (e.g., a pinhole). By reducing the size of the aperture, the beam divergence is increased due to diffraction. Similarly, a transmission pixel controlled aperture (e.g., a liquid crystal device) can produce a transmission pattern (e.g., a pinhole) that also increases diffraction as desired. A more preferred approach is to use a Fresnel zone plate pattern because it is more optically efficient.

[0244] Figure 33b In panel Bvi) a system is shown which is based on changing the characteristics of a laser-like beam 76 to control the phase 80 (equivalent to the radius of curvature) on the beam 76, thereby producing an output in a nonlinear crossover region 110 which produces the desired image depth for a display device. The figure shows a pair of lenses 240A and 240B for creating a beam expanding telescope. The two lenses are controllable (via a controller unit 241) which is used to modify the resulting optical power of the pair of lenses. This can be achieved in many different ways, including liquid lenses or by moving components to change the lens spacing (e.g. in a zoom lens of a camera). It should be understood that a different number of lenses can be used and selected based on the desired technical performance. Changing the size of the beam 76 (which will propagate over a certain distance and pass through other optical components represented by 186) will result in diffraction expansion, which is well known for Gaussian beam optics. Therefore, the radius of curvature (phase) of the beam will be controlled.

[0245] Example Control System Inputs, Outputs, and Functionality

[0246] Fig.34A simplified schematic block diagram of a controller for the above-mentioned display system is shown. The controller 13 has three main inputs as shown, namely, image data of the desired artificial scene information 11, eye monitoring data from internal cameras 54 and 56, and image data of the natural scene from external cameras 50 and 52. The controller 13 uses external cameras 50 and 52 to obtain information from the natural scene to determine where the artificial scene information should be located 11. The internal cameras 54 and 56 are used to determine the position of the line of sight, and then derive the position to which the overlapping input beams on the display component should be directed to overlap in the NLO material. The figure also shows the main outputs of the controller 13, including: laser power 19 for the radius of curvature determination beam; OLED controller 324 for generating an intensity structured beam; phase controller 123 (which sets the radius of curvature / phase on the laser beam); and control the output of the input beam manipulation optical component (e.g., rotatable mirror 222) for guiding and focusing the input beam. It should be understood that the described system is simplified and in more complex configurations will include other information, such as accelerometer data, magnetic compass data, LiDAR data, etc.

[0247] As examples, the display systems described herein may have any combination or options of the following inputs, controller functions, and outputs.

[0248] enter:

[0249] Any of the following inputs and associated controller input interfaces may be provided.

[0250] · WiFi or 4G / 5G connection for Internet

[0251] Bluetooth or equivalent (wired, wireless or optical)

[0252] Accelerometer and magnetometer

[0253] Front camera

[0254] Rear (eye-tracking) camera

[0255] Integrity sensor photodiode

[0256] Information used to determine the synthetic scene, including information derived locally and received from external networks

[0257] Coordinates and geometry of region space (regionally broadcast or from memory)

[0258] LiDAR / 3D image information

[0259] Calibration data in the installation agreement

[0260] Observe light scattering from the eye through a built-in camera

[0261] System controller functions:

[0262] Any of the following functions may be included in the controller.

[0263] Determine synthetic scenes based on external and internal sources

[0264] Determine local geometry and feature equality

[0265] Memory and signal processing

[0266] Current sight angle

[0267] Eye spacing and other visual characteristics, monocular dominance, blinking, etc.

[0268] Calculate overlap angle and phase control

[0269] Intensity modulation calculation

[0270] Determine display integrity

[0271] ·Exterior illumination

[0272] · Calculate the need to block blocks of real scene light

[0273] Output:

[0274] Any of the following outputs, which may be considered as control signals, and associated controller output interfaces may be provided.

[0275] Position, rotation and acceleration data for external users

[0276] Camera output for external processing

[0277] A controller for the first light beam and a controller for the second light beam

[0278] Amplitude modulation

[0279] Phase modulation for setting the radius of curvature

[0280] Color modulation

[0281] Guiding optics direct the beam to the desired spot

[0282] Control photochromic real scene shading

[0283] Wavelength combination

[0284] For color displays, it is necessary to provide red, green, and blue light. Assuming the nonlinear process is a three-beam SFG, the frequencies of the two input beams must be the sum of the frequencies of the composite beam of the desired color. There are a plethora of available laser and non-laser sources that provide an almost unlimited possible combination of two frequencies for the sum of red, green, and blue.

[0285] An exemplary combination is:

[0286] 1064nm+1550nm=630nm(red)

[0287] 1064nm+1064nm=532nm(green)

[0288] 1064nm+780nm=450nm (blue).

[0289] The example also shows a practical method for reducing the number of source wavelengths required. In the example, only three different wavelengths (frequencies) are used, i.e., less than two wavelengths (frequencies) per color.

[0290] A second exemplary combination is:

[0291] 1064nm+1560nm=632.5nm(red)

[0292] 1064nm+1064nm=532nm(green)

[0293] 1064nm+780nm=450nm (blue).

[0294] The example shows another method for reducing the number of source wavelengths required. In the example, only two laser sources are needed because 780nm light can be generated by the second harmonic effect of a 1560nm laser.

[0295] It will be appreciated that the choice of the number of lasers used depends on the availability, efficiency, price, lifetime and physical size of each technology. It should be noted that a single laser (e.g. 3144 nm) may be used which will provide 1572 nm by frequency doubling and its third harmonic providing 1048 nm.

[0296] Thus, a third exemplary combination using a single starting laser at 3144 nm is:

[0297] 1048nm+1572nm=628.8nm(red)

[0298] 1048nm+1048nm=524nm(green)

[0299] 1048nm+786nm=449.1nm (blue).

[0300] Another specific implementation is:

[0301] 950nm+1800nm=621nm(red)

[0302] 950nm+1210nm=532nm(green)

[0303] 950nm+880nm=456nm (blue).

[0304] The 950 nm beam may be generated by an OLED array (or other infrared light emitting diode array), while the other three beams at 1800 nm, 1210 nm and 880 nm are generated by respective laser sources. In the example described, there are four different wavelengths.

[0305] The core features of the present invention become apparent from the above examples. In other words, since the colors of the display are essentially in the visible range, the input beam will be in the infrared range and therefore invisible for most (if not all) practical combinations of the two wavelengths. Thus, the scattering problem of conventional micro-projector displays is essentially solved, because any scattered light from the source beam will be in the infrared range and therefore invisible.

[0306] The conversion efficiency in a nonlinear process depends on the product of the input beam intensities. For this reason, although a CW source can be used, it is preferred that the light source be operated in pulsed mode to produce high peak power, thereby achieving higher conversion efficiency. The laser source used to generate the input beam can be in any of the following operating modes: CW, long pulse, nanosecond pulse, picosecond pulse, or femtosecond pulse. The choice of which light source to use and the operating mode depends on a variety of factors, including the nonlinearity of the NLO material, the damage threshold of the NLO material, the required driving electrical power, form factor, weight, cost, lifetime, etc.

[0307] In addition, it should be understood that any specific beam source used will produce a beam with a specific frequency response, which is usually defined according to a single frequency (or wavelength) - peak frequency - and bandwidth, which is used to measure the emission spectrum range above and below the peak frequency. To the extent relevant to this document, the bandwidth of the beam source is defined as the full width at half maximum (FWHM). When a laser is used as the beam source, the bandwidth will be extremely narrow, and when a non-laser source is used as the beam source, the bandwidth can be significant. Considering the visible and near-infrared wavelengths described in the above examples, the typical bandwidth is as follows. In a single-mode edge-emitting LD, the bandwidth can be as high as a few nanometers, and for a distributed feedback (Distributed Feedback, DFB) or a distributed Bragg reflector (Distributed Bragg Reflector, DBR) edge-emitting LD, it is reduced to sub-pm. For non-laser sources, the bandwidth will usually be higher. For example, an OLED can have a bandwidth of tens of nanometers. In this article, the "frequency" of the beam source is usually referred to as the peak frequency, and the bandwidth is not mentioned unless it is relevant.

[0308] Other points and variations

[0309] Although SFG by nonlinear wave mixing is usually associated with the need to use an input beam with fairly high power (e.g., milliwatt to watt-level laser power), embodiments of the present invention generally do not require such high-power input beams due to the high sensitivity of the human eye. Sub-microwatt-level optical powers are bright enough for the human eye, so the maximum output power of the beam source (e.g., AR glasses or VR headsets) required for near-eye embodiments of the present invention will remain fairly modest.

[0310] Many conventional optical design methods known from existing AR and VR display devices can also be used to implement the display device of the present invention. For example, it is well known that relay lenses are used between modulated light sources (e.g., micro-OLED emitter arrays) and AR display panels to transfer light to emission points on the display panel, where the light effectively becomes an angular spectrum. In terms of Fourier optics, the concept is known, for example, in a four-aperture imaging system, where a simple lens is placed at a focal length away from the emitting screen and is arranged so that the position of the reflected plane wave is also outside a focal length. In the simplest case, and assuming that the focal length of the lens matches the focal length of the eye (e.g., about 16 mm), a symmetrical four-aperture imaging system is provided. The display panel is effectively located at the middle or Fourier plane and therefore contains angular information. It is possible for the relay lens to have a different focal length variation than the eye. In addition, the relay lens can also be modified in more complex ways, such as to compensate for unnecessary optical distortions, such as correcting chromatic aberrations and other aberrations or providing other effects, such as providing different magnifications. When designing the relay lenses, it should be noted that the non-collinear geometry results in a stretched spatial distribution of the image information in the composite beam compared to the corresponding image information in the first and optional second input beams that needs to be considered. A simple approach is to use the Scheimpflug principle, where the lenses are tilted so that the image is imaged at an angle to the plane. More complex optical lenses and combinations of lenses can be deployed to correct for distortions and aberrations, which is already common in optical design theory. There are also effects due to the curvature of the display panel, which can be corrected by determining a combination of zone angles of the QPM grating orientation and by selecting the image information.

[0311] If the AR display device is realistic, the person wearing the AR glasses can place any image anywhere within their field of view. For example, if the wearer looks at his wrist to view a watch, the dial is not actually needed for presentation. Instead, the dial can be superimposed on the dial. Similarly, mobile phones no longer need a display, but can have an arbitrary display area on which the AR display superimposes the content on the display. The same method can also be used for televisions or personal computers, which no longer need a screen, but only a designated area on the wall or device for the wearer of the AR display to perceive the display. Keyboards can also be projected.

[0312] It should be appreciated that any of the methods presented herein for AR are applicable to VR (simply by blocking all natural scene light).

[0313] It should be noted that when referring to SFG of two input light beams, frequency doubling, ie, second harmonic generation (SHG) is also included. SHG is a special case of SFG, in which the two input light beams have equal wavelengths.

[0314] While certain features and advantages of the proposed design are specific to AR vision systems, the proposed design is also advantageous for VR vision systems, especially VR goggles that can be made to have significantly less depth due to the ability to engineer the wavefront curvature of the VR image. The ability to place different synthetic objects at different perceived depths is as advantageous in the context of virtual images formed in VR goggles as it is for AR glasses, because it makes the VR image more realistic and allows for the avoidance of convergence accommodation conflicts.

[0315] In addition, the technical solutions proposed in this article can also be used to provide imaging solutions for other optical imaging systems in which the eye is very close to where the imaging occurs, including but not limited to binoculars, monoculars, telescopes, camera viewfinders, movie camera viewfinders, microscope eyepieces, medical imaging devices, endoscopes, magnification assemblies, rifle scopes and military aiming systems.

[0316] Reference numerals list

[0317] 10: AR glasses

[0318] 11: Artificial scene information

[0319] 12: Display Device

[0320] 13, 123b, 400: Controller

[0321] 14: Beam source

[0322] 16: Transmitter Array

[0323] 18: Laser source

[0324] 19: Laser controller unit

[0325] 20, 22, 24: Spectacle lenses

[0326] 26: Frame

[0327] 28: Nose bridge

[0328] 30: Nose pads

[0329] 32, 34: glasses legs

[0330] 36, 38: Eyeglass frames

[0331] 40: Wireless transmitters and transceivers

[0332] 42: Communication Path

[0333] 43: Wireless receivers and transceivers

[0334] 44: External control device

[0335] 46: Processor

[0336] 48: Memory

[0337] 49: External network communication

[0338] 50, 52: Internal camera

[0339] 54, 56: External camera

[0340] 58: Point cloud imaging system

[0341] 60: Photodiode

[0342] 62: Compass

[0343] 64: Accelerometer

[0344] 70: First input beam

[0345] 70A, 70B: Input beam path

[0346] 71, 77: Reference numerals

[0347] 72: First input beam wave vector

[0348] 74: First input beam wavefront

[0349] 75: Amplitude and phase modulation function

[0350] 76: Second input beam

[0351] 76A, 76B, 224A, 224B: Beam path

[0352] 78: Second input beam wave vector

[0353] 80: Phase Profile

[0354] 81: Gaussian Profile

[0355] 82: Output beam

[0356] 84: Synthetic beam wavefront

[0357] 86: Synthetic Beam Wave Vector

[0358] 88, 88A, 88B, 88C: Display panel

[0359] 90: Inner surface

[0360] 92: Outer surface

[0361] 94: Left end face

[0362] 96: Right end face

[0363] 100: NLO materials

[0364] 103: NLO material super structure spot

[0365] 103A, 103B, 104, 106B, 106G, 106R: Light spot

[0366] 106: Light spot cluster

[0367] 108: QPM Zone

[0368] 110: Intersection area

[0369] 110A, 110B: Crossover position

[0370] 120: Modulator

[0371] 121: Amplitude Modulator Controller

[0372] 122: Transmissive spatial phase modulator

[0373] 123: Phase Modulator Controller

[0374] 123a: Combined Amplitude and Phase Modulator

[0375] 124: Input beam guide components

[0376] 124b: Controller for a combined amplitude and phase modulator

[0377] 125: Rotatable mirror assembly

[0378] 128: Air

[0379] 130, A1: front filter layer

[0380] 132: Rear filter layer

[0381] 134: Light shielding layer

[0382] 136: Front mirror layer

[0383] 138: Rear mirror layer

[0384] 139: Power

[0385] 140: Dimming layer

[0386] 141: Electrical connection line

[0387] 142. N1: Anterior vision correction layer

[0388] 144: Rear vision correction layer

[0389] 146. AR_1: Front ARC layer

[0390] 148. AR_2: Post-ARC layer

[0391] 150: IR absorption layer

[0392] 150A, 150B, 150C, 150D: Layer

[0393] 152: Front spacer

[0394] 154: Rear spacer

[0395] 56: dimming layer, electrical connection line

[0396] 160: Eyes

[0397] 162: Eyeball

[0398] 164: Cornea

[0399] 166: Lens

[0400] 168: Retina

[0401] 170: Pupil

[0402] 172: Iris

[0403] 174: Optic nerve

[0404] 180: Object

[0405] 181: Curved wavefront

[0406] 182: Light scattered from an object

[0407] 184: Light

[0408] 186: Discontinuity

[0409] 188, 188A, 188B: Wavefront

[0410] 190: Illustration

[0411] 190A, 190B: Point

[0412] 192, 2f: Distance

[0413] 200: QPM NLO material

[0414] 202: "upper" domain

[0415] 204: "Down" domain

[0416] 206B, 206G, 206R: QPM cycle

[0417] 208: Superstructure Grating

[0418] 220: Beam

[0419] 222: Rotatable mirror

[0420] 222A: First Mirror

[0421] 222B: Second Mirror

[0422] 222C: The Third Mirror

[0423] 230: Optical components

[0424] 240, 300, 325: Lenses

[0425] 240A: First lens

[0426] 240B: Second lens

[0427] 241: Controller unit

[0428] 242: Relay lens

[0429] 260: Mirror

[0430] 262: Mirror layer

[0431] 264: temple or side of the glasses

[0432] 266: Nose bridge side

[0433] 280: Control System

[0434] 282: Blanking period

[0435] 290: Absorbent material area

[0436] 292: Non-absorption area

[0437] 292A, 292B: Entrance area

[0438] 317: Input infrared laser wave

[0439] 318, 319, 320: Radius of curvature

[0440] 321: Incoherent Transmitter Array

[0441] 322, 323, 324: pixels

[0442] 334: Controller for an Incoherent Transmitter Array

[0443] 402: Controller input

[0444] 404: Output of the controller

Claims

1. A display device (12) for displaying a synthetic scene image in response to input of image information, the display device comprising: A first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth; A second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth; A display panel (88) comprising a nonlinear optical NLO material (100) which is phase matched to the first input light beam, the second input light beam and a sum frequency synthesized light beam (82) having a frequency equal to the sum of the first frequency and the second frequency; An input beam guiding component (222, 240) configured to introduce the first input beam and the second input beam into the display panel so that they pass through the display panel in a first path and a second path, wherein the first path and the second path intersect each other in a non-collinear geometry in the NLO material to define an intersection volume (110), wherein the composite beam is generated in the intersection volume by a sum frequency effect; as well as A controller (400) is operable in response to the input of the image information to set each of the amplitude, wave vector and wavefront curvature radius of the synthetic light beam by amplitude modulating at least one of the first input light beam and the second input light beam, and by phase modulating at least one of the first input light beam and the second input light beam, so as to form the synthetic scene image.

2. The display device according to claim 1, wherein: The first input beam encodes image intensity information, which defines the intensity of the synthetic beam generated at each intersection volume of a given synthetic scene image, and the second input beam encodes image depth information, which defines the wavefront curvature radius of the synthetic beam generated at all intersection volumes of a given synthetic scene image.

3. The display device according to claim 1 or 2, wherein: The controller is operable to group the synthetic scene images into image frames, different synthetic scene images of a given image frame having different wavefront radii of curvature to represent image elements located at respective different perceived distances.

4. The display device according to claim 1 or 2, wherein: The display device is used to display a synthetic scene image in color and includes a plurality of beam sources, the beam sources including a first input beam source and a second input beam source, wherein the first input beam and the second input beam form one of three input beam pairs generated by the plurality of beam sources, and the first input beam pair to the third input beam pair are combined to generate first to third synthetic beams having first to third sum frequencies providing first to third primary colors.

5. The display device according to claim 4, wherein: The multiple beam sources for forming the three input beam pairs include an emitter array and a first laser to a third laser, wherein the emitter array provides one of the input beams for all three input beam pairs, the other input beam in each pair is provided by one of the first laser to the third laser, and the first primary color to the third primary color are provided by time slicing, so that the first synthetic beam to the third synthetic beam are generated in sequence.

6. The display device according to claim 5, wherein: The emitter array is driven to amplitude modulate the input light beam it generates.

7. The display device according to any one of claims 4 to 6, wherein: The NLO material includes first to third spatial modulation regions to provide quasi-phase matching for the first to third input light beam pairs, respectively.

8. The display device according to claim 7, wherein: The first to third spatial modulation regions are formed in the NLO material at first to third depth portions (206) in the display panel.

9. The display device according to claim 7, wherein: The first to third spatial modulation regions are formed in the NLO material as an array of light spot clusters (106), each light spot cluster including adjacent light spots of each of the first to third spatial modulation regions.

10. The display device according to any one of claims 4 or 9, wherein: The controller is operable to group the synthetic scene images into image frames, different synthetic scene images of a given image frame presenting each of the first to third primary colors.

11. The display device according to any one of claims 4 or 9, wherein: The controller is operable to group the synthetic scene images into image frames, different synthetic scene images of a given image frame presenting each of the first to third primary colors and having different wavefront curvature radii, thereby representing color image elements located at respective different perceived distances.

12. A display device as claimed in any preceding claim, wherein: These input beam guiding components are adjustable to provide a change in at least one of the first path and the second path, thereby changing the angle at which the first input beam and the second input beam intersect in the NLO material, thereby changing the wave vector direction of the composite beam.

13. A display device as claimed in any preceding claim, wherein: The input beam directing components are adjustable to provide a change in at least one of the first path and the second path, thereby changing the position where the first path and the second path intersect in the NLO material.

14. The display device according to any one of claims 1 to 13, further comprising: An amplitude modulator (120) configured to amplitude modulate the first input light beam, and a phase modulator (122) configured to phase modulate the second input light beam.

15. The display device of any one of claims 1 to 13, further comprising a combined amplitude and phase modulator (123a) for amplitude and phase modulating one of the first input light beam and the second input light beam.

16. The display device according to any one of claims 1 to 15, wherein: The NLO material is spatially modulated with respect to its second order nonlinearity to provide phase matching by quasi-phase matching.

17. The display device according to any one of claims 1 to 15, wherein: The NLO material is homogeneous with respect to its second order nonlinearity to provide phase matching through birefringence phase matching.

18. A display device as claimed in any preceding claim, wherein: The sum frequency lies in the visible range, and the first frequency and the second frequency lie in the infrared range.

19. A display device as claimed in any preceding claim, wherein: A ratio of the first frequency to the second frequency is between 0.5 and 2.

0.

20. A display device as claimed in any preceding claim, wherein: The display panel including the NLO material is transparent in the visible light range.

21. A display device as claimed in any preceding claim, wherein The NLO material is included in an NLO material layer in the display panel.

22. The display device according to claim 21, wherein: The display panel includes a front filter layer (130) arranged in front of the NLO material layer, the front filter layer is opaque to the first frequency and the second frequency and transparent to the sum frequency to prevent light from the first input light beam and the second input light beam from emitting outward from the display panel.

23. The display device according to claim 21, wherein: The display panel includes a rear filter layer (132) disposed behind the NLO material layer, the rear filter layer being opaque to the first frequency and the second frequency to prevent light from the first input light beam and the second input light beam from emitting inwardly from the display panel.

24. The display device according to claim 23, wherein: The rear filter layer is transparent to visible frequencies.

25. The display device according to any one of claims 21 to 24, wherein: The NLO material is continuously distributed over the NLO material layer such that phase matching and generation of a composite beam can therefore occur at any location in the NLO material layer.

26. The display device according to any one of claims 21 to 24, wherein: The NLO material is distributed in the form of an array of spots (106) on the NLO material layer so that the phases are matched and the generation of the composite beam is thus confined to the locations of these spots.

27. A display device as claimed in any preceding claim, wherein: The display panel includes a light shielding layer (134) including an array of pixels that can be individually addressed by electrical control lines to switch the pixels between a first state that is opaque to visible frequencies and a second state that is transmissive to visible frequencies, so that natural scene light can be mixed from selected areas of the light shielding layer.

28. A display device as claimed in any preceding claim, wherein: The first path and the second path pass through the display panel by continuous reflection.

29. The display device according to claim 28, wherein: The continuous reflections come from the front and rear surfaces of the display panel by total internal reflection.

30. The display device according to claim 28, wherein: The continuous reflection comes from the front mirror layer and the back mirror layer (136, 138) provided in the display panel, the front mirror layer and the back mirror layer are arranged in front of and behind the NLO material, respectively.

31. The display device according to claim 30, wherein: The front mirror layer and the back mirror layer are reflective at the frequency of the input light beam and transmissive in visible frequencies.

32. A display device as claimed in any preceding claim, wherein: The NLO material is transparent in the visible light range.

33. A display device as claimed in any preceding claim, wherein: The display panel has a peripheral region (290) containing a material that is absorptive to light of the first frequency and the second frequency, so that when the first input light beam and the second input light beam reach the peripheral region, the first input light beam and the second input light beam are absorbed after the first path and the second path have crossed each other in the NLO material.

34. A display device as claimed in any preceding claim, further comprising one or more light sensors (60) arranged outside the display panel to detect abnormal leakage of input beam light from the display panel, the abnormal leakage indicating structural damage to the display panel.

35. The display device according to any one of claims 1 to 34, wherein: The first frequency and the second frequency are different.

36. A display device as claimed in any one of claims 1 to 34, wherein: The first frequency and the second frequency are the same, so the sum frequency effect is a second harmonic effect.

37. The display device of claim 36, wherein: The first input beam source and the second input beam source are the same beam source, and both the first input beam and the second input beam originate from the same beam source.

38. A wearable headset (10) comprising a display device as claimed in any preceding claim, such that the display panel is arranged in front of the wearer's eyes at a vertex distance of less than 30 mm for near-eye imaging.

39. A display device (12) for displaying a synthetic scene image in response to input of image information, the display device comprising: A plurality of beam sources (16, 18) having respective frequencies and bandwidths to provide first to third input beam pairs each having two input beams (70, 76), each input beam pair having a pair of frequencies that is a sum of first to third primary color frequencies; A display panel (80) comprising a nonlinear optical (NLO) material (100) phase-matched to the first to third input beam pairs and first to third composite beams (82) at first to third primary color frequencies equal to the sum of the frequencies of the first to third input beam pairs; An input beam guiding component (222, 240) configured to introduce the two input beams in each input beam pair into the display panel so that the input beams in each pair pass through the display panel along a first path and a second path, wherein the first path and the second path intersect each other in the NLO material in a non-collinear geometry to define an intersection volume (110), wherein a composite beam of the input beam pair is generated in the intersection volume by a sum frequency effect; an amplitude modulator (120) operable to amplitude modulate at least one beam of each input beam pair; a phase modulator (122) operable to phase modulate at least one beam in each input beam pair; as well as A controller (400) is operable to form color image frames, each image frame comprising first to third synthetic scene images respectively generated by the first to third input beam pairs, wherein the controller is operable to control the amplitude modulator and the phase modulator in response to the input of the image information to set the synthetic beam amplitude, the synthetic beam wave vector and the synthetic beam wavefront curvature radius, thereby forming each of the first to third synthetic scene images.

40. The display device according to claim 39, wherein: The multiple beam sources include an emitter array (16) and first to third lasers (18), wherein the emitter array provides one of the input beams for all three input beam pairs, the other input beam in each pair is provided by one of the first to third lasers, and the first to third primary colors are provided by time slicing, so that the first to third composite beams are generated in sequence.

Images