Optical apparatus

Abstract

An apparatus that displays synthetic scene images for augmented reality, virtual reality and other display types. One or more pairs of input beams (70, 76) are injected into a display panel incorporating a layer (100) of phase matched nonlinear optical (NLO) material. The input beams of each pair are brought to cross each other in the NLO material in a noncollinear geometry to generate a product beam (84) by sum frequency generation that is directed to the eye (160) of a viewer. The synthetic scene image information is impressed on each input beam pair by amplitude and phase modulation to set the amplitude (i.e. brightness), wavevector (i.e. direction) and wavefront radius of curvature (i.e. depth) of the synthetic scene image. For colour display, three pairs of input beams with respective sum frequencies in three primary colours are used.

Description

[0001] TITLE OF THE INVENTION

[0002] OPTICAL APPARATUS

[0003] FIELD OF THE INVENTION

[0004] The invention relates to optical apparatus.

[0005] BACKGROUND OF THE INVENTION

[0006] Displays for different purposes vary in size from a few millimetres mm up to 10 metres or more.

[0007] Example smaller displays are for smart watches or AR spectacles or VR goggles. Augmented reality (AR) and virtual reality (VR) displays have been incorporated into a variety of wearable formats including headsets. For AR headsets a glasses (i.e. spectacles) format is common. For VR headsets, a ski goggles format is common. A recent review article on AR and VR display technologies is the paper by Zhan, Yin, Xiong, He and Wu "Augmented Reality and Virtual Reality Displays: Perspectives and Challenges" Perspective, Vol. 23, Issue 8, 101397 (2020) at https: / / doi.Org / 10.1016 / j.isci.2020.101397.

[0008] In AR, an image of a synthetic scene is superimposed on a natural scene that is simultaneously viewed by the viewer. AR spectacle lenses lie at a vertex distance of about 12-15 mm, which is much closer than the eye's near point, which is gradually increases with age from about 7 cm for a youth to over 50 cm for an old person. (The eye’s near point is the closest distance which the eye can bring into focus.) In AR, the synthetic scene is therefore be represented by a virtual image arranged further away from the eye than the AR spectacle lenses. VR image formation is simpler than AR, since there is no natural scene and only a synthetic scene is generated. For an AR headset in spectacles format, the most common technology approach at present is to use a microprojector system. A virtual image reproducing a synthetic scene is routed onto one or both of the spectacle lenses. (We use the term 'lens' for linguistic convenience but it will be understood that the lenses may be blanks, i.e. have no focusing effect in the sense of normal vision correction for long or short sightedness etc.) The lenses of the AR spectacles are used as a medium to route the light representing the synthetic scene into the path of the natural scene light via suitable beam combiners. AR beam combiners can be subclassified into reflective (e.g. a mirror or prism) or diffractive (i.e. some kind of grating). The virtual image of the synthetic scene should give the impression of portraying one or more objects at the correct location in the natural scene, which means not only at the correct position (essentially angle) in the visual field but preferably also at the correct distance from the eye. Compared with AR / VR displays, displays for smart phones and tablets are somewhat larger. Computer monitors and television screens are larger still, perhaps 10 inch to 90 inch diagonal length (25 cm to 225 cm). Even larger displays are found in cinemas, advertising hoardings and arenas, such as indoor and outdoor sport stadiums. Many different display technologies are used depending on the required brightness, the cost, the colour gamut, etc. These include liquid crystals, organic LED (light emitting diodes), and DLPs (digital light processors). Historical technologies include plasma TV, cathode ray tubes and laser back- projection.

[0009] Lack of brightness for outdoor use is an issue with both current AR spectacles and large area displays such as televisions that are placed outdoors or very large area displays for sports stadiums.

[0010] SUMMARY OF THE INVENTION

[0011] Various aspects of the invention are as specified in the attached claims.

[0012] The nonlinear optical (NLO) material is phase matched to first and second selected combinations of the frequencies of the first and second input beams, and to a frequency equal to the sum or difference of the selected combinations.

[0013] The simplest combination of the frequencies of the first and second input beams is to take those frequencies themselves, so that the product beam has a frequency equal to the sum of the first and second frequencies. Other frequency combinations that can be selected include an integer multiple of either or both of the first and second frequencies, and a sum or difference frequency between the first and second frequencies. For example, the product beam may be derived from mixing twice the frequency of the first input beam with the frequency of the second input beam. Another example would be that the product beam is derived from mixing the difference frequency between the first and second frequencies and the second frequency.

[0014] The above can be expressed scientifically as follows. The first and second input beams have two wavelengths A_1 and A_2. We label the more energetic one as A_1 (shorter wavelength). They will have associated frequencies v_1 and v_2. The selected combination of A_1 and A_2 output a product beam with wavelength A_0 as follows: i > 1 ° — -i-nA or frequency v_0 as follows: where m, n are integer values in the series: 1, 2, 3 ...

[0015] The simplest combination with the product beam having a frequency equal to the sum of the first and second frequencies is m = n = 1. A first specific embodiment of the invention is with m = n = 1. Another group of embodiments excludes the first specific embodiment and can be defined by at least one of m and n being not equal to 1. Selection of a suitable frequency combination from any of the above theoretically possible frequency combinations must also have regard to the absorption properties of the material, so large values of m, n - especially if the minus is used - can mean wavelengths that will be absorbed by the UV edge of the material and therefore be unsuitable in practice. The type of device into which a display panel according to the invention is incorporated will determine the vertex distance. A wearable headset will typically have a vertex distance of between 10 mm and 100 mm. A vertex distance range of between 10 mm and 20 mm is typical for spectacles, including AR spectacles, and a vertex distance range of between 50 mm and 80 mm is typical for a helmet visor. In a window panel structure aligned vertically, the vertex distance may typically be in the range 5 cm to 70 cm for a person standing or seated in front of the window panel and the viewer may have a fixation point that is relatively close, e.g., at a distance of 50 cm to 200 cm in the case of a fume cupboard, or a fixation point at any distance up to infinity, such as when driving and looking through the windscreen of a vehicle or aircraft.

[0016] BRIEF DESCRIPTION OF THE DRAWINGS

[0017] This invention will now be further described, by way of example only, with reference to the accompanying drawings.

[0018] Figure 1 is an explanation of how imaging of a distant object requires wavefront information.

[0019] Figure 2 is a diagram showing how imaging through a lens element and into the eye relies on both wavefront direction and wavefront curvature.

[0020] Figure 3 is a diagram explaining why a point light emitter cannot be used in a near-to-eye display.

[0021] Figure 4 shows a nonlinear optical approach for generating near-to-eye artificial images that can be superimposed onto natural scene light.

[0022] Figure 5 shows the concept of quasi phase matching.

[0023] Figure 6 shows two-dimensional phase matching and different grating vectors can be constructed.

[0024] Figure 7 shows an implementation of NLO mixing for generating synthetic scene light with bouncing paths within a lens.

[0025] Figure 8 shows non-collinear phase matching.

[0026] Figure 9 shows a more detailed figure showing nonlinear wavefronts overlapping in a layered NLO quasi-phase matching (QPM) material to create image information.

[0027] Figure 10 shows routes to allowing beams with different pairs of intersection angles relative to a QPM structure to overlap in a common place.

[0028] Figure 11 shows a scheme for steering beams into a device.

[0029] Figure 12a shows a scheme for deflecting a beam without angular change.

[0030] Figure 12b shows components to direct light to a desired point within a NLO material layer while altering the angles that the light approaches that spot.

[0031] Figure 13 shows a lens structure with input beams and emitters capable of creating image with desired apparent focus depth.

[0032] Figure 14 shows imprinting a desired wavefront through nonlinear mixing of amplitude and phase structured beams. Figure 15 shows a system for creating a far field pattern from an image to allow conversion via an NLO material.

[0033] Figure 16 shows a multi-layered optical composite to create a display system.

[0034] Figure 17 shows altering the local Kg position across a spectacle type display system.

[0035] Figure 18 shows different lens designs in terms of cross-section with appropriate advantages.

[0036] Figure 19 shows a spectacle lens AR system incorporating multiple sensors and communications to an external computing device.

[0037] Figure 20 shows a spectacle system incorporating a series of diode elements around the outside to detect damage to the system.

[0038] Figure 21 shows an electrically controlled light dimming system incorporated into a lens.

[0039] Figure 22 shows a display device incorporating sub regions of NLO material, and with separate red, green and blue generating regions.

[0040] Figure 23 shows QPM period as a function of crossing angle for a simple second harmonic process, indicating why it is advantageous to use larger crossing angles.

[0041] Figure 24 shows an approach for creating a lens in which beams enter from one side but maintain crossing in opposing directions.

[0042] Figure 25 shows a time sequenced displays to provide RGB and different ROC images to create a convincing full colour, and depth of field realistic display.

[0043] Figure 26 shows inwardly facing cameras to track pupil position.

[0044] Figure 27 shows how blanking based upon the inwardly facing cameras can be used to reduce unwanted visual effects.

[0045] Figure 28 shows a lens with an absorbing layer dispersed around the edge to prevent unwanted reflections of light.

[0046] Figure 29 shows a lens with sub regions of NLO material where each sub region contains elements capable of generating Red, Green and Blue.

[0047] Figure 30 shows that the beam crossing angles across a lens requires local control of the QPM grating direction. Figure 31 shows a Virtual Reality (VR) configuration according to this invention, in which a near-to-eye image is created without the complexity of using a thin lens.

[0048] Figure 32 is a schematic diagram of image generation in an NLO material layer using an incoherent emitter array to generate one of the input beams.

[0049] Figure 33a shows various options for modulating the two beams that interact in the NLO material.

[0050] Figure 33b shows various options for phase controlling across an input beam to impart radius of curvature information.

[0051] Figure 34 shows a schematic of the control system showing the inputs and outputs.

[0052] Figure 35 shows a multi-layered optical composite to create a display system.

[0053] Figure 36A is a schematic view of the front of a computing device with a display panel according to embodiments of the disclosure.

[0054] Figure 36B is a schematic view of the rear of the computing device of Figure 36A.

[0055] Figure 37 is a block diagram of the functional components of the computing device of Figure 36A and Figure 36B.

[0056] Figure 38 shows the differing principles of operation of a conventional NCTE display panel 910 (upper drawing) and an example NCTE display panel 911 embodying the invention (lower drawing).

[0057] Figure 39A is a schematic drawing of one example display apparatus according to the invention employing a variable focal length lens for phase control.

[0058] Figure 39B shows an example optical design to explain how apparent size of synthetic scene objects is determined.

[0059] Figures 39C and 39D shows, in more detail than in Figure 39A and Figure 39B, a 4f imaging system for a phase modulated input beam.

[0060] Figure 40A, Figure 40B, Figure 40C and Figure 40D are schematic drawings of different options for selecting the apparent depth in relation to the physical position of the display panel. Figure 41 is a schematic drawing of an NCTE display panel according to an embodiment of the invention employing a spatial light modulator (SLM) for amplitude modulation overlaid on a NLO material layer, thereby forming a hybrid NLO / SLM display panel.

[0061] Figure 42 illustrates use of the hybrid NLO / SLM display panel of Figure 41 to serve different display images to the left and right eyes of a viewer.

[0062] Figures 43A and 43B illustrate the comparative optical efficiency of the hybrid NLO / SLM display of Figure 41 (Figure 43B) and a conventional backlit LCD display panel (Figure 43A).

[0063] Figure 43C shows an example display stack of a an NLO / SLM display panel using an LCD as the SLM together with a touch sensor stack.

[0064] Figure 44 illustrates a person with a hand-held device being operated in a virtual depth display mode as shown in Figure 40C.

[0065] Figure 45 illustrates further design considerations for operating in a virtual depth display mode as shown in Figure 40C.

[0066] Figure 46 and Figure 47 are schematic drawings in perspective and plan view respectively of a first example video conferencing configuration.

[0067] Figure 48 is a schematic plan view of a second example video conferencing configuration.

[0068] Figure 49 is a schematic plan view of a third example video conferencing configuration.

[0069] Figure 50 is a schematic plan view of a fourth example video conferencing configuration.

[0070] Figure 51 shows a real-world video conference configuration (upper drawing) with an associated virtual video conference configuration (lower drawing) placing the participants on either side of a rectangular virtual table.

[0071] Figure 52 shows a real-world video conference configuration (upper drawing) with an associated virtual video conference configuration (lower drawing) placing the participants on three sides of a triangular virtual table.

[0072] Figure 53 shows a real-world video conference configuration (upper drawing) with an associated virtual video conference configuration (lower drawing) placing the participants around a round virtual table.

[0073] Figure 54 shows an example spectacles frame design for routing an input beam to the nose side of a lens rim. Figure 55 shows a composite spectacle lens with an inset display panel section according to embodiments of the invention.

[0074] Figure 56 is a schematic perspective view of a section of a composite spectacle lens as shown in Figure 55 with an inset display panel section.

[0075] Figure 57 A shows an example display panel according to the invention as suitable for an AR spectacle lens.

[0076] Figure 57B shows a further example display panel according to the invention as suitable for an AR spectacle lens together with an inset showing a wafer from which pieces for the display panel can be cut.

[0077] Figure 58 illustrates two different modes of image stabilisation through a collection of schematic diagrams of an AR spectacle frame 26 with a synthetic scene element shown as a five-pointed star.

[0078] Figure 59 is a schematic drawing of an AR spectacles frame with inbuilt light sources to mimic synthetic scene features that appear in the peripheral vision.

[0079] Figure 60A shows a set-up using display panels according to the invention in a contact lens format for AR representation of a synthetic scene combined with spectacles to provide the contact lenses with input beam pairs.

[0080] Figures 60B to 60E show further details of contact lens embodiments.

[0081] Figure 61 shows a display panel according to the invention set in a window of a fume cupboard.

[0082] Figure 62 is a schematic drawing of an apparatus for manufacturing a display panel using vapour phase deposition.

[0083] Figure 63 shows a layer stack for a phase matched grating in which first and second NLO material layers of opposite domain orientation are interposed with electrode layers formed of a conductive organic material to allow application of an electric field to the NLO material layers.

[0084] Figure 64 is a schematic drawing of an apparatus for rolling a display panel layer structure (upper drawing) to reduce its thickness and increase its area (lower drawing).

[0085] Figure 65 is a schematic drawing of an apparatus for manufacturing a display panel using a spin-coating process. Figure 66 is a schematic drawing of an apparatus for manufacturing a display panel using a Langmuir Blodgett film process.

[0086] Figure 67 is a schematic drawing of a display panel in which the nonlinear optical layers for forming the phase matched grating are made of liquid crystal cells.

[0087] Figures 68A to 68F show different possible poling arrangements for NLO materials according to the invention.

[0088] Figure 69 shows one half of a pair of spectacles according to an embodiment of the invention.

[0089] Figure 70 shows different view of the spectacles of Figure 69.

[0090] Figure 71 shows a mirror array structure and related components of the spectacles of Figure 69.

[0091] Figure 72 shows in plan view the mirror array structure of Figure 71 and related components.

[0092] Figure 73 shows an alternative optical arrangement for creating input beam illumination from the temple of the spectacles of Figure 69.

[0093] Figure 74 is a schematic drawing of a pair of spectacles according to an embodiment of the invention.

[0094] Figure 75 is a schematic plan view of the spectacles of Figure 74.

[0095] Figure 76 shows an enlarged version of a part of Figure 75.

[0096] Figure 77 show an optical arrangement for beam forming by noncollinear sum frequency mixing (or SHG) in a phase-matched region of material.

[0097] Figure 78 shows the same optical configuration as in Figure 77 but with the addition of a beam splitter 1251 arranged in the output beam path.

[0098] Figure 79 shows a concatenated pair of the previously described configurations, i.e., a two- stage beam combiner.

[0099] Figure 80 shows a hexagonal cross-section NLO material being used to combine 3 pairs of input beams. Figure 81 is a schematic diagram of a communication system including a headset or other device according to the invention.

[0100] Figure 82 is a block diagram illustrating an example computing apparatus that may be used in connection with various embodiments described herein.

[0101] DETAILED DESCRIPTION

[0102] INTRODUCTION

[0103] There are some problems with one or more of the existing AR and VR display technologies.

[0104] One problem is how to avoid unwanted light scattering on the spectacle lenses caused by the projected beams, which detracts from the appearance of the scene and results in the AR spectacles not giving the impression of being fully transparent. For example, since spectacle lenses are placed around 12-15 mm from the eyeball and since this distance is similar to the focal length of the eye, any unwanted light scattering from the spectacles becomes collimated by the eye and spreads widely across the retina. In bright ambient light conditions, such scattering is perceived as glare. In low ambient light conditions, such scattering can be much brighter than the light from the natural scene and hence severely impair vision. Light scatter artefacts, such as rainbow effects, are a particularly acute problem when a grating-based beam combiner is used.

[0105] Another problem, which is relevant for both AR and VR, is how to avoid vergenceaccommodation conflict, which is a characteristic of binocular vision where the normal relationship between accommodation and vergence that exists when viewing a natural scene is broken by the rendering of the synthetic scene, resulting in headaches and nausea. A vergence-accommodation conflict arises when an object in a synthetic scene is perceived to be arranged at one distance (and the vergence locks to that distance) whereas the light representing that object originates from a different distance (and accommodation locks to that distance). Even in monocular vision the brain has an accommodation expectation based on the brain's understanding of the juxtaposition of objects in a familiar scene, so that headaches and nausea can also arise from monocular vision of a synthetic scene that causes the eye to accommodate at a certain distance when viewing an object, where the distance is different from the brain's expectation of the object's distance.

[0106] Some problems are specific to AR displays.

[0107] One of the major technical limitations of current AR display technology is the use of AR outdoors. Ambient luminance on a sunny day is typically 3,000 candela per square metre (nit) or 1 ,000 lux. To be of sufficient contrast to be readable, text needs to be three times brighter than its background. The ratio for AR scene content is higher. Therefore, text information overlaid on an AR display panel needs a brightness of at least 30,000 nits or 10,000 lux to be readable by the wearer. These brightnesses are not readily achievable with current AR technology based on LED sources, such as micro-LEDs or LED plus modulator sources. Lack of brightness is therefore holding back widespread adoption of AR displays in daylight conditions. For example, a pilot may wish to use an AR display in a headset (e.g. helmet visor or glasses) or as part of a head-up display. An AR display using LED sources may be perfectly visible below cloud and then become effectively invisible when the pilot breaks through the cloud into sunlight. The challenge in bright ambient light is thus to ensure that the synthetic scene information is visible against the bright natural light scene.

[0108] Another problem with current AR display technology is stray light leakage out of the front of the display, which is undesirable in certain applications, such as when the wearer of AR glasses needs to remain unseen. An application example here is for infantry soldiers who need to remain undetectable when on night operations.

[0109] OPTICAL CHARACTERISTICS OF THE HUMAN EYE AND HUMAN VISION

[0110] When designing a display apparatus, it is important to have regard to the optical characteristics of the eye as well as how the eyes are controlled individually and jointly by the brain, and also how the brain processes image information from the eyes.

[0111] To enter the eyeball, light passes through the eye pupil, through the cornea, through the eye lens and thence to the back of the eyeball where the retina is located. The effective radius of the pupil is controlled by the iris (pupillae muscle). The pupil dilates and contracts according to ambient brightness to increase and decrease the amount of light entering the eye. The pupil diameter varies over a range of a few millimetres, with a range of 2-4 mm being the most common. The eyeball diameter is about 25 mm.

[0112] If spectacles are worn, then the light from a distant object that passes into the eye passes through an area of the spectacle lens which is essentially similar in size to the iris diameter. The distance from a spectacle lens to the front of the eyeball is about 12-15 mm and is referred to as the back vertex distance or vertex distance.

[0113] The visual field of a single eye is referred to as the monocular visual field and is usually defined in terms of the angular range of what can be seen by the eye when the eye is pointing in a particular direction to fix its vision to a certain point. The monocular visual field in relation to the eye's vertical meridian extends laterally inward (towards the nose) by approximately 60 degrees and laterally outward by approximately 107 degrees. In relation to the eye's horizontal meridian, it extends vertically upwards by approximately 70 degrees and vertically downward by about 80 degrees. The binocular visual field is the superposition of the monocular fields of the left eye and right eye, so is appreciably greater laterally. In this document, references to 'the visual field' refer to the monocular visual field. If the binocular visual field is referred to, this will be expressly stated. Normal human vision involves a series of movements of the head from the spine and swivelling of the eyes in their sockets. Head movements can be thought of corresponding to roll, pitch and yaw angles in relation to straight ahead, where yaw is produced by the normal left-to-right head swivel, pitch by the up-and-down head nodding motion and roll by side-to- side motion of the head. Eyeball swivel range for the human eye is about +25 degrees up, - 30 degrees down, and about ±45 degrees left and right. Eyeball swivelling is used to place the retinal image of the natural scene location that is of the most interest on the fovea, which is the subarea of the retina with the highest resolution, the fovea providing an angular range of about 1-2 degrees in the visual field (compared with about 18 degrees for the visual field of the whole retina, i.e. macula lutea). Furthermore, the eye moves around naturally (saccadic motion) which keeps the eyes moving to avoid effects of saturation and unresponsiveness. Thus, the eye is constantly roving the scene, with the brain carrying out image processing which cancels out the blurring effects of eye motion.

[0114] For monocular vision, if one considers light emanating from a given point in a natural scene, the light will effectively be a spherical wave with an origin at that point. The spherical wave impacts on the front of the eye, passes through the cornea, pupil and eye lens and onto the retina. The refractive power of the eye lens is adjusted by the ciliary muscles to form a focused image of the scene light on the retina. The eye lens receives a wavefront, this wavefront having a particular central angle and radius of curvature. In the case of light from a distant point in the natural scene (effectively infinite distance), the waves can be treated as entering the eye as plane waves, i.e. the wavefront is a planar wavefront. For monocular vision perception of most individuals, infinite distance is any distance above about 4 or 5 metres. The angle which light incident from such a distant point makes to the axis of the eye determines the position on the retina where that light falls. In the case of light from a closer point in the natural scene, then the waves enter the eye with a finite radius of curvature, which is perceived in monocular vision by an eye's accommodation. The eye will accommodate, i.e. adjust its focus, to bring the light from the closer point to a sharp focus on the retina. The human brain links the amount of accommodation to object distance, so a distance perception arises. (Distance perception also takes account of the vergence effect from binocular vision.)

[0115] For monocular vision, we can therefore consider the natural scene to be an aggregate of point light sources at different positions in the monocular visual field, in which the three- dimensional (3D) location information of each point is encoded by the light's input angle, i.e., two-dimensional (2D) position in the monocular visual field, and the light's radius of curvature at the eye (i.e. depth). As well as 3D location information, it is of course the case that the light from each point additionally carries intensity information and colour information. In near-eye display systems, such as AR, VR and contact lenses, the term 'eyebox' (or eye motion box) is used to refer to a volume within which the eye receives an acceptable view of an image, whether that be a real image or a virtual image. A simple definition of the eyebox is the 3D volume over which the entire field of view (FOV) is visible for a standard pupil size taking account of the fact that in human vision the eye is continually moving to focus on different areas within the FOV. The specified dimensions of the eyebox in AR or VR goggles are typically larger than the theoretical movement range of the eye pupil to cover for alignment tolerances and person-to-person variation in pupil distances.

[0116] OVERVIEW

[0117] This invention proposes a display apparatus based on a display panel which hosts phase- matched nonlinear mixing of pairs of input beams that cross in the display panel in a noncollinear geometry. The crossing takes place in a region of NLO material within the display panel. Each crossing input beam pair produces a product beam by sum frequency generation (SFG), or other nonlinear mixing process in the NLO material.

[0118] Display apparatus according to this design can be provided for close-to-eye image formation; both for AR and VR. Display apparatus according to this design can also be provided for large format displays such as large screens for video conferencing or television.

[0119] According to one aspect of the invention, there is provided a display apparatus for displaying synthetic scene images responsive to input of image information, the display apparatus comprising: a first beam source for providing a first input beam of a first frequency and bandwidth; a second beam source for providing a second input beam of a second frequency and bandwidth; a display panel containing NLO material that is phase matched to the first and second input beams and to a product beam of a sum frequency that is equal to the sum of the first and second frequencies; input beam routing components arranged to introduce the first and second input beams into the display panel so that they traverse the display panel in first and second paths that cross each other in the NLO material in a non-collinear geometry to define an intersection volume where the product beam is generated by sum frequency generation; and a controller operable responsive to the input of the image information to form the synthetic scene images by amplitude modulating at least one of the first and second input beams and by phase modulating at least one of the first and second input beams to set each of amplitude, wavevector and wavefront radius of curvature of the product beam. In some embodiments, the first input beam encodes image intensity information that defines the intensity of the product beam generated at each intersection volume of a given synthetic scene image and the second input beam encodes image depth information that defines the wavefront radius of curvature in the product beam generated at all intersection volumes for a given synthetic scene image. This is a convenient arrangement for allowing each synthetic scene image that is formed to be associated with a different perceived depth through setting its wavefront radius of curvature. Specifically, when the second input beam has a substantially planar wavefront, by which is meant a wavefront radius of curvature of greater than 5 metres, the synthetic scene image is formed at a distance perceived to be infinite, whereas when the second input beam has a wavefront radius of curvature of less than 5 metres, the synthetic scene image is formed at a distance perceived to be finite. It is thus possible to form several synthetic scene images in quick succession to build up different layers of scenery in an image frame, e.g. a background at infinity and one or more closer layers on which specific image elements are represented.

[0120] In some embodiments, the controller is operable to group the synthetic scene images into image frames, the different synthetic scene images of a given image frame having different wavefront radii of curvature to represent image elements that lie at different respective perceived distances.

[0121] In other embodiments, the display apparatus is for displaying a synthetic scene image in colour and comprises a plurality of beam sources which includes the first and second input beam sources, wherein the first and second input beams form one of three input beam pairs generated by the plurality of beam sources, the first to third input beam pairs combining to generate first to third product beams with first to third sum frequencies which provide first to third primary colours. In such embodiments, the plurality of beam sources to form the three input beam pairs may comprise an emitter array and first to third lasers, wherein the emitter array provides one of the input beams for all three input beam pairs, the other input beam of each pair being provided by one of the first to third lasers, the first to third primary colours being provided by time slicing so that the first to third product beams are generated sequentially. In particular, the emitter array may be driven to amplitude modulate the input beam which it generates. For example, the NLO material may comprise first to third spatially modulated regions to provide quasi-phase matching for the first to third input beam pairs respectively. One option here is when the first to third spatially modulated regions are formed in the NLO material at first to third depth portions in the display panel. Another option here is when the first to third spatially modulated regions are formed in the NLO material as an array of spot clusters, each spot cluster comprising adjacent spots of each of the first to third spatially modulated regions. For colour display in some embodiments, the controller may be operable to group the synthetic scene images into image frames, the different synthetic scene images of a given image frame being in each of the first to third primary colours.

[0122] For colour display in some embodiments, the controller may be operable to group the synthetic scene images into image frames, the different synthetic scene images of a given image frame being in each of the first to third primary colours and having different wavefront radii of curvature, thereby to represent colour image elements that lie at different respective perceived distances.

[0123] In some embodiments, the input beam routing components are adjustable to provide for variation of at least one of the first and second paths to vary the angle at which the first and second input beams cross in the NLO material and thereby vary the wavevector direction of the product beam.

[0124] In some embodiments, the input beam routing components are adjustable to provide for variation of at least one of the first and second paths to vary the location where the first and second paths cross in the NLO material.

[0125] In some embodiments, the display apparatus further comprises an amplitude modulator arranged to amplitude modulate the first input beam and a phase modulator arranged to phase modulate the second input beam. In other embodiments, the display apparatus further comprises a combined amplitude and phase modulator to amplitude and phase modulate one of the first and second input beams.

[0126] In some embodiments, the NLO material is spatially modulated in respect of its second order nonlinearity to provide phase matching through quasi-phase matching. In other embodiments, the NLO material is homogeneous in respect of its second order nonlinearity to provide phase matching through birefringent phase matching.

[0127] Regarding the first and second frequencies and their sum frequency: The sum frequency is preferably in the visible range, i.e. ca. 790-405 THz 1380-740 nm. The first and second frequencies are in the infrared range, i.e. ca. 400-150 THz 1750 nm to 2 pm. The ratio of the first and second frequencies is between 0.5 and 2.0.

[0128] For AR, the display panel, including the NLO material, is made transparent across the visible range, i.e. ca. 790-405 THz 1380-740 nm, so that natural scene light can pass through to the eyes of the viewer.

[0129] In some embodiments, the display panel has a peripheral region containing a material that is absorbent to light at the first and second frequencies so that the first and second input beams are absorbed after the first and second paths have crossed each other in the NLO material when they reach the peripheral region.

[0130] In some embodiments, the display apparatus further comprises one or more light sensors arranged outside the display panel to detect for abnormal leakage of input beam light out of the display panel indicative of structural damage of the display panel.

[0131] The NLO material may be contained in the display panel in a NLO material layer. In one specific implementation, the display panel comprises a back filter layer arranged to the back of the NLO material layer, the back filter layer being opaque to the first and second frequencies and transparent to the sum frequency, to block emission of light from the first and second input beams out of the display panel in an outward direction. In another specific implementation, the display panel comprises a front filter layer arranged to the front of the NLO material layer, the front filter layer being opaque to the first and second frequencies, to block emission of light from the first and second input beams out of the display panel in an inward direction. The front filter layer may be transparent to visible frequencies. One option is for the NLO material to be distributed across the NLO material layer continuously such that phase matching and hence product beam generation can occur at any location in the NLO material layer. Another option is for the NLO material to be distributed across the NLO material layer as an array of spots such that phase matching and hence product beam generation is confined to the locations of those spots.

[0132] In some embodiments, the display panel comprises a light blocking layer comprising an array of pixels which are individually addressable by electric control lines in order to switch the pixels between a first state which is opaque to visible frequencies and a second state which is transmissive to visible frequencies, so that natural scene light can be blended out of selected areas of the light blocking layer.

[0133] In some embodiments, the first and second paths traverse the display panel by successive reflections. The successive reflections may be from front and back surfaces of the display panel by total internal reflection. Alternatively, the successive reflections may be from front and back mirror layers arranged in the display panel that are respectively disposed to the front and back of the NLO material. Specifically, the front and back mirror layers may be reflective at the frequencies of the input beams and transmissive across visible frequencies.

[0134] In some embodiments, the NLO material is transparent across the visible.

[0135] In some embodiments, the first and second frequencies are different. In other embodiments, the first and second frequencies are equal so that the sum frequency generation is second harmonic generation. When the first and second frequencies are equal, the first and second input beam sources can be one and the same beam source and the first and second input beams are both derived from that beam source.

[0136] The above-specified display apparatus may be incorporated into a wearable headset such that the display panel is arranged in front of a wearer's eyes at a vertex distance of less than 30 mm for close-to-eye image formation.

[0137] Assuming SFG is the nonlinear process that is exploited and also assuming the product beams are to be formed at visible wavelengths / frequencies (ca. 380-740 nm I 790-405 THz) to create a synthetic scene image that is visible to the human eye, the input beams of each pair will both lie in the infrared (e.g. 750 nm to 2 pm / 400-150 THz), at least provided that they are not too dissimilar in wavelength / frequency (e.g. have a ratio of less than 2:1).

[0138] As an alternative nonlinear mixing process of SFG, three photon summative mixing could be provided by a combination of collinear mixing second harmonic generation (SHG) in one of the input beams and non-collinear mixing of the second harmonic with the other input beam of the pair.

[0139] The phase matching in the NLO material may occur without structuring the material, e.g. by using a sufficiently thin layer of NLO material with a high birefringence. However, the use of QPM will be convenient in many embodiments. To achieve QPM, the NLO material is structured to spatially modulate its nonlinear properties. Most commonly periodic poling is used to create a grating structure characterised by a single periodicity and alternating signs of the second order nonlinearity x(2).

[0140] The two input beams of a pair are directed to traverse the NLO material at angles which ensure that the wavevector of the product beam has a direction that propagates into the viewer's eye, i.e. through the pupil.

[0141] The non-collinear mixing of first and second input beams in the NLO material allows a product beam to be generated whose properties are controlled in amplitude (i.e. image brightness), in wavevector (i.e. to direct the generated light towards the pupil), in wavefront curvature (i.e. to control perceived distance of the image) and image element location in the visual field (i.e. position on the NLO material).

[0142] To create a realistic impression of binocular vision, it is possible to exploit the ability to control the wavevector direction and image element location in order to present slightly different images of the same synthetic object to the left and right eyes of a viewer, e.g. on each lens of a pair of AR spectacles, so that the natural vergence and the associated perception of depth (stereopsis) for the synthetic object is provided for. For synthetic objects that are close enough for monocular vision depth perception (i.e. less than about 4-5 metres), the vergence simulation can be combined with setting the wavefront curvature of the synthetic object so that the focus of the eye (accommodation) is matched exactly or at least approximately to the same distance as associated with the vergence, thereby avoiding vergence-accommodation conflict. For synthetic objects at greater distances, a planar wavefront can be used.

[0143] The nonlinear process creates an oscillating polarisation at the frequency of the product beam, i.e. in the visible. This process is inherently a wave-mixing phenomenon, which means that the product beam has transverse (i.e. spatial) coherence with a defined phase relationship between different parts of the emitted wave. The product beam thus creates a wavefront which has a specific curvature. This is fundamental property not possessed by light generated by pixel emitters in a conventional microdisplay as used in current VR headsets, or point emitters as used in current AR spectacles. Rather, the product beam generated by the non-collinear mixing of the pair of input beams in the NLO material has properties that are more associated with a conventional diffractive optics approach which also emits a wavefront through diffraction. A transversely coherent approach to generating the light for the synthetic scene image is what allows the radius of curvature of the wavefront to be set, thereby allowing the synthetic scene image to be placed at a certain distance which can be varied by varying the radius of curvature of one of the input beam's wavefront. A natural depth perception can therefore be impressed on the synthetic scene image. Different synthetic objects can be given different depth appearances so they appear at appropriate depths relative to each other and, in the case of AR, in relation to objects in the natural scene. This not only contributes to the perceived realism of the synthetic scene image but also allows convergence-accommodation conflicts to be avoided.

[0144] A display apparatus can be provided that is able to direct light for the synthetic scene image into the eye from across a wide range of angles (i.e. from a large range of positions across the visual field). This property is beneficial, since it takes account of the large angular swivel range of the eye. This property is also beneficial, since it allows creation of a synthetic scene image that appears to emanate from an appropriate position within the visual field when the eye is pointing in a certain direction. Moreover, as mentioned above, the product beam light that forms the synthetic scene image can be emitted from the display panel with a defined radius of curvature so the different parts of the synthetic scene image can be placed in the synthetic scene at appropriate perceived distances.

[0145] For an AR headset, the display panel can be realized in a lens format. In a traditional spectacles format, in which a pair of lenses are set in a frame for the left and right eyes, there will be a physically separate display panel for each lens. In a ski goggles format with a single lens sheet, a single display panel could be used. For AR, the NLO material in the display panel is preferably transparent (in the visible) so that it can be incorporated into the display panel without reducing the transmission of light travelling from the natural scene, through the display panel into the eye. An AR display panel can thus be provided that appears truly transparent to the viewer, not only since the NLO material is itself transparent but also since the wavelengths of the input beams are outside the visible range (or at least can be easily chosen to be). Not only is the light from the input beams invisible to the human eye but more importantly it is easy to prevent input beam light at non-visible frequencies / wavelengths from entering the viewer's eye by arranging a layer of a suitable filter material (e.g. as a bandpass or edge filter) between the NLO material layer and the viewer's eye. The filter material is chosen to be opaque to the input beam wavelengths and transparent across the visible range of wavelengths. For an AR headset, it is also possible to adjust the brightness of objects in the synthetic scene image. This may be done simply by adjusting the intensity of one or both of the input beams of a beam pair. For a VR headset, the NLO material need not be transparent (in the visible), i.e. could be opaque (in the visible).

[0146] A display panel according to the invention can be thought of as a transparent microdisplay in the sense that, like a microdisplay, the visible light is created in localised regions capable of creating wavefronts across the display (the crossing input beam intersection volumes). Because of the feature of the invention whereby the only visible light is that generated for the image, problems associated with stray light and scattering in conventional microprojection systems for AR (or VR) do not arise. Instead of having visible beams traversing the lens to form the image in the visual field of the eye, we have infrared beams, i.e. input beams that are at wavelengths outside the visible range and which can in any case be prevented from entering the eye by arranging a suitable filter layer between the eye and the NLO material layer. Consequently, there is no glare and no bright scattering artefacts to disturb night-time vision. A display apparatus can therefore be realised that in terms of its light generation is analogous to a microdisplay VR display apparatus but unlike a microdisplay is transparent (in the visible) so that it can be used for AR.

[0147] Colour Displays

[0148] The above discussion of the invention has been confined to considering the generation of a single product beam from a single pair of input beams. This can be thought of in terms of describing a monochrome display or alternatively one of three colour components of a colour display. For a colour display, three product beams are needed, one for each of the three primary colours, so there will be three pairs of input beams. According to another aspect of the invention for colour display, there is provided a display apparatus for displaying synthetic scene images responsive to input of image information, the display apparatus comprising: a plurality of beam sources with respective frequencies and bandwidths to provide first to third input beam pairs of two input beams, each input beam pair having a pair of frequencies that sum to first to third primary colour frequencies; a display panel containing NLO material that is phase matched to the first to third pairs of input beams and to first to third product beams at first to third primary colour frequencies which are equal to the sum of the frequencies of the first to third pairs of input beams; input beam routing components arranged to introduce the two input beams of each input beam pair into the display panel so that the input beams of each pair traverse the display panel in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume where the product beam for that input beam pair is generated by sum frequency generation; an amplitude modulator operable to amplitude modulate at least one beam of each input beam pair; a phase modulator operable to phase modulate at least one beam of each input beam pair; and a controller operable to form colour image frames, each image frame comprising first to third synthetic scene images generated by the first to third input beam pairs respectively, wherein the controller is operable to form each of the first to third synthetic scene images by controlling the amplitude and phase modulators responsive to the input of the image information to set product beam amplitude, product beam wavevector and product beam wavefront radius of curvature.

[0149] For colour display, the plurality of beam sources may comprise an emitter array and first to third lasers, wherein the emitter array provides one of the input beams for all three input beam pairs, the other input beam of each pair being provided by one of the first to third lasers, the first to third primary colours being provided by time slicing so that the first to third product beams are generated sequentially.

[0150] While the need to have three pairs of input beams implies there will be six input beams in total, this does not equate to a need to have six independently generated input beams. Firstly, the same wavelength can be shared between different input beam pairs, so the total number of wavelengths needed to form three pairs need not be as high as six but rather can be reduced to five, four or three, with three being the minimum possible number. Secondly, time slicing (i.e. multiplexing) can be used so that a common first input beam is used to generate all three colours in combination with dedicated second input beams, one for each colour, thereby reducing the total number of independently generated input beams that is required to four.

[0151] For colour display, if QPM is used, the NLO material can be structured with a single grating period that is used for all three colours. The lack of exact phase matching that will exist for at least two of the three colours can be tolerated and, if needed or desired, compensated for by changing the overall intensity of one or both of the input beams of each beam pair. The angles at which the input beams traverse the QPM NLO material can also be selected to be different for the different colours, so that exact phase matching (or at least closer to exact phase matching) is provided for all three beam pairs (i.e. all three colours). Alternatively, to provide quasi-phase matching, the NLO material can be periodically poled (or equivalently orientation patterned in the case of nonbirefringent materials) with three different periods so there are three regions of QPM NLO material with different grating periods, each period being optimized for one of the three colours. Another alternative is to create a single synthesised QPM structure with three phase matching peaks optimised for the desired three colours of the display. The three colour-specific QPM regions can be distributed in the image generation plane, e.g. across the display panel, in a regular or irregular 2D array of spot clusters, where each spot cluster contains three laterally adjacent colour-specific QPM regions, i.e. spots. The spots in each cluster are so close together that the fact they are not at the same position in the visual field will be imperceptible. Alternatively, the three colourspecific QPM regions can be distributed along an axis perpendicular to the image generation plane, e.g. over three different depth portions of the display panel. Here the colour-specific QPM regions could, like the above-described clusters, be arranged in a 2D array or could be continuous areas across the display panel as needed to cover the required visual field for synthetic scene image formation.

[0152] In the following, we mainly describe the invention in terms of generating a single product beam, i.e. in terms of generating a monochrome image or one colour component of a colour image. Nevertheless, it will be understood that a colour image can be generated from three such product beams from three pairs of input beams.

[0153] Image Information for Close-to-Eye Synthetic scene image Formation

[0154] To create a convincing virtual image of a synthetic scene in an image generation plane close to the eye, as is needed for AR spectacles in particular, it is necessary to accurately reproduce the image location information that would be encoded in light from an equivalent natural scene. The image location information is a subset of the overall image information and is what defines the perception of 3D placement of an object in a scene. As discussed in more detail below, for monocular vision, the image location information in a natural scene is essentially a combination of where the light from different parts of the natural scene appear in the visual field (i.e. the light's input angle into the eye) and the radius of curvature possessed by the light wave from each point in the natural scene when it reaches the eye. For close objects, the wavefront curvature is appreciable whereas for distant objects there is no appreciable curvature, i.e. the wavefront is effectively a plane wave. For synthetic scene image formation in a display panel that is close to the eye (e.g. at the vertex distance, which is ca. 12-15 mm, where a spectacle lens is typically situated), the input angle and radius of curvature information from a natural scene can be mimicked by generating light at a particular location on the image generation plane (so it appears in the correct point in the visual field) and with a particular direction of light propagation from that location (so the light generated at that location in the image generation plane enters the eye). As well as carrying the correct image location information, the light generated on the image generation plane must also have the correct intensity and, for a colour image, the correct colour.

[0155] In the context of the present invention, correctly encoding the image location information for the synthetic virtual image involves generating the product beam light in the NLO material at the correct locations (visual field angles), with the correct wavevector directions and with the correct radii of wavefront curvature. To generate the image, the image intensity information must of course also be reproduced, which can be done in a straightforward manner through appropriate amplitude modulation of one or both of the first and second input beams. For colour images, the image colour information also needs to be reproduced, which is done by producing three product beams of three primary colours (e.g. RGB) in the QPM NLO material at each location in the visual field, i.e. to provide an appropriate colour gamut in the same way as for any colour projection system.

[0156] To give a convincing binocular vision perception of the synthetic scene, the respective images formed for each of the left eye and right eye will correspond but will not be the same. In particular, for the left and right eyes, light from the same point in the synthetic scene is formed at different locations in the monocular visual fields of each eye and with a different wavevector direction, thereby to mimic the natural vergence and the associated perception of depth (stereopsis). It will be appreciated that, for any parts of the synthetic scene that can be ascribed infinite distance, the images formed for each of the left eye and right eye can be made the same, since in the equivalent natural scene light from distant objects will arrive at both the left eye and the right eye as plane waves and at the same location in the respective monocular visual fields of the left eye and right eye. Although it should be realised that in creating a realistic visual scene any intermediate distance objects can block or occlude differently between the left and right eye scenes, so although both are plane wave, they emanate from different direction to the two eyes.

[0157] Image Formation of the Synthetic Scene We disclose two basic approaches for forming a synthetic virtual image in an image generation plane close to the eye. The first is a scanning approach to build up each image frame pixel-by-pixel, e.g. by rastering. The second forms a whole image frame simultaneously.

[0158] Image Formation in Angular Space (Input Beam Scanning / Retinal Scanning): First and second input beams are moved in both angle and position in respect of their traversal of the NLO region to create a product beam that is scanned across the monocular visual field of the viewer to form an image encoded in angular space with a retinal scan approach. An image frame is thus built up by rapidly scanning the visible light over the retina. In more detail, first and second input beams are provided which have respective beam crosssections that are dimensioned so that where the first and second input beams cross in the NLO region their intersection volume is small to provide a plane wave that is not diffractively limited and which maps onto a point on the retina (through eye focusing), so the intersection volume at any location in the image generation plane corresponds to angular field single emission (as a wavefront) for building up the synthetic virtual image on the eye. For each intersection volume, the first and second angles at which the first and second input beams traverse the NLO region define the wavevector direction of the product beam. The wavevector direction is set to ensure that the product beam is directed to the pupil of the eye. To build up a whole image (for one eye), one or both of the first and second input beams are altered in angle, so that while the input beam intersection volume, i.e. where the product beam is generated, stays essentially constant the angle of the nonlinear generated light changes to a different angular position in the (monocular) visual field of the viewer's eye. At the same, the intensity of the product beam is varied to reproduce the image intensity information, which can be done by amplitude modulation of the first and / or second input beams, preferably also taking account of any variation in conversion efficiency of the nonlinear mixing as a result of the input beam traversal angles changing as the intersection angles alter over the image generation plane, so that over time a complete image frame is built up. As the input angles into the intersection volume are altered, the wavevector direction is also altered to ensure that the product beam remains directed to the pupil of the eye at all times. This can be done by appropriate adjustment of the angles at which the first and second input beams traverse the NLO material. The scanning of the intersection volume across the monocular visual field to build up each image frame may, for example, follow a line-by-line raster pattern. The raster may involve a back-and-forth movement meandering from left-to-right, then right-to-left etc. in a serpentine.

[0159] Whole Image Formation (upconversion analogy): This approach to image formation can be partially understood by analogy to a conventional up-conversion imaging system. A first input beam (e.g. at 1064 nm) contains the image intensity information and a second input beam (e.g. at 1550 nm) crosses with the first input beam in the NLO material to cause an upconverted version of the first input beam to be reproduced (as the product beam). This analogy is helpful but not complete, since it does not take account of the fact that the noncollinear phase matching according to the invention also allows control of the wavefront curvature of the product beam so that image depth information is incorporated into the synthetic virtual image. The image depth information is thus "added" to the "upconverted version" of the intensity modulated first input beam (i.e. the product beam) by phase modulation of the second input beam. This wavefront engineering feature of the present invention has no analogue in a conventional up-conversion imaging system.

[0160] In one embodiment of the invention, there is a implementation example which is closest to the up-conversion imaging analogue. The first input beam is amplitude modulated across its beam cross-section to carry the image intensity information that defines the intersection- volume-specific brightness in the product beam. The second input beam is phase modulated across its beam cross-section to carry image depth information that defines the intersection- volume-specific wavefront curvature in the product beam. In other implementation examples, which move further away from the up-conversion imaging analogue, both amplitude and phase are modulated in one or both of the first and second input beams across their beam cross-sections. For example, the first beam may be amplitude modulated and the second beam may be amplitude and phase modulated.

[0161] In a first specific example, both the first and second input beams are highly coherent transversely (i.e. in the cross-section of the beam). This is the case if the beam sources for both the first and second input beams are lasers and beam modulation is performed either with a SLM in transmission or a liquid crystal on silicon (LCOS) device in reflection. Here we note that both SLMs and LCOS devices can be operated to modulate amplitude (only), to modulate phase (only) or to modulate both amplitude and phase.

[0162] In a second specific example, the first input beam is incoherent and generated by a 2D array of incoherent emitters (e.g. an LED or OLED array) which provides amplitude modulation through the ability to drive each emitter independently. The second input beam is coherent, as is a requirement for phase modulation, being based on a laser source whose laser beam is phase modulated (e.g. by an SLM or LCOS device) and optionally also amplitude modulated as mentioned already for the first specific example.

[0163] In the context of an RGB display, which requires three input beam pairs for generating the three colours, the same incoherent emitter array can be used to generate a first input beam that is used for all three colours whereas each colour has its own dedicated second input beam, the three second input beams being generated by three different laser sources with three different wavelengths that sum with the wavelength of the incoherent emitter array to produce red, green and blue light respectively. Time division multiplexing (i.e. slicing) can then be used to sequence through red, green and blue synthetic scene image generation through appropriate drive electronics that synchronizes the outputs of the IR emitter array and the three laser sources. This kind of sequential driving (or ON / OFF modulation) of the three laser sources provides a kind of temporal RGB image formation analogous to the colour wheel approach in desktop projectors. Provided the refresh rate of the three colours is sufficiently high, then the viewer perceives a full colour palette RGB image.

[0164] Since it is possible to impress both phase modulation and amplitude modulation onto one beam to produce a combined phase and amplitude modulated single beam (e.g. with an SLM or LCOS device), further embodiments are also possible in which the other input beam is not spatially modulated at all.

[0165] Other Features

[0166] Some other features of certain embodiments are now briefly summarized.

[0167] For input beam delivery into the display panel, a convenient approach is to introduce the first input beam at or close to one end of the display panel, e.g. the left end, and the second input beam at or close to the other end of the display panel, e.g. the right. However, it would also be possible to introduce both first and second input beams into the same end of the display panel. This could be enabled by a more complex phase matching scheme that uses appropriate reciprocal lattice vectors for the non-collinear phase matching. This could also be enabled by reflecting one of the first and second input beams from the other end of the display panel so that they cross in the same non-collinear geometry as for opposite end introduction of the first and second input beams.

[0168] There are various options for distribution of NLO material within the NLO material layer. In some configurations of the device the NLO material is distributed continuously over the NLO material layer, while in others it can be attractive to have a spots of the NLO material arranged in a grid over the NLO material layer. In the context of a grid-of-spots approach, it is noted that by NLO material we mean the NLO material that is phase matched to facilitate the nonlinear mixing, so that if QPM is used, then the spots could be localized regions that have been spatial structured for QPM within a layer of the same NLO material that is unstructured. With a grid-of-spots approach with confined localized regions of product beam generating NLO material, it is still possible to generate a convincing image, since the eye is continually swivelling. So long as the localized regions are close enough together so that light from at least one such region can be directed into the eye pupil, a convincing image can be formed. In certain embodiments of the invention, the first and second input beams are each introduced at or close to one end of the display panel at a certain angle relative to the front and back faces of the display panel. Each input beam then propagates across the display panel to where the non-collinear mixing occurs by one or more reflections. In some embodiments, total internal reflection (TIR) from the display-panel-to-air interfaces is used to walk each input beams across the display panel from where it is introduced to where it crosses with the other input beam of its pair in order to generate the product beam. In other embodiments, reflective mirror layers are provided to the front and back of the NLO material layer to reflect at the wavelengths of the input beams, so that the input beams traverse the display panel by successive mirror reflections. One or both of the front and back mirror layers may be respectively at the front or back face of the display panel or be buried layers within the display panel. Mirror reflection allows for reflection at larger angles (i.e. closer to the normal) compared with TIR, which is beneficial if quasi-phase matching is used, since then the poling period can be larger, which is easier for fabrication.

[0169] Further functional layers can be added to the display panel. For example, it may be attractive to add a front filter layer to the front of the NLO material layer, which absorbs at the wavelengths of the input beams (and transmits at the wavelength of the product beam), so that any scattered input beam light is prevented from reaching the eye. In the case either of both of the input beams are laser beams, this will also help ensure laser safety requirements are met. A similar back filter layer can be added to the back of the NLO material layer to prevent input beam light escaping into the environment. If the display panel is for AR, then both the front and back filter layers should also be transparent across visible frequencies, so the natural scene light propagates through the display panel to the eye without attenuation.

[0170] A further safety measure is to provide one or more light sensors, such as photodiodes, outside the display panel to detect for abnormal leakage of input beam light out of the display panel that is indicative of the display panel being structural damaged. The light sensors may be arranged around the rim or outer edge of the display panel. Operation of the display apparatus can then be shut off if needed, e.g. when the display panel is damaged with a deep scratch.

[0171] A further possibility for AR is to include a photochromic or electrochromic layer so that the transmission of natural scene light can be reduced, e.g. in bright ambient light conditions, so that the synthetic scene light blends with the natural scene light appropriately. The darkening operation can either be automatic (as in photochromic lenses) or electrically operated. Moreover, the darkening can either be uniform darkening across the whole lens area or localized, e.g. to attenuate light from bright spots in the natural scene. Vision correction lens layers can also be provided, either to the front of or to the back of the NLO material layer (or both if two correction lenses are desired). If a correction lens layer is provided, there is preferably an input beam mirror layer interposed between the NLO material layer and the vision correction lens layer, so that the input beam light does not enter the vision correction lens layer and hence the input beam paths are not affected. In other words, the TIR solution is not preferred when vision correction lenses are included.

[0172] NONLINEAR OPTICAL (NLO) MATERIAL

[0173] One kind of nonlinear mixing in a NLO material exploits the second-order nonlinearity x(2). With nonlinear mixing there is conservation of energy, which means that the energy of the photons that are generated by the nonlinear mixing depends on addition of the energies of the photons of the input beams, which, given the Planck relation, means that the product light frequency is given by the sum or difference of the frequencies of the input beam frequencies.

[0174] Phase matching arises because of the wavelength dependence of the refractive index of a NLO material. The type of phase matching used in embodiments of the invention is known as non-collinear phase matching. Non-collinear phase matching occurs when two (or more) light beams with different directions of propagation cross to form a beam intersection region that lies in a phase matched NLO material. Phase matching ensures that the wavevector of the product beam generated by the nonlinear mixing is the vector sum of the wavevectors of the input light beams (and the compensating grating k-vector). The term non-collinear phase matching is used in contradistinction to the term collinear phase matching, which refers to the situation in which two input beams overlap in space and co-propagate so that the wavevectors of the input beams and the product beam are all in the same direction. Noncollinear mixing of two input beams can be characterized by the two angles subtended between the product beam and each of the two input beams which together sum to the crossing angle between the two input beams, i.e. the beam crossing angle. In the case of two input beams with wavevectors of equal magnitude, i.e. equal wavelengths, the two angles are equal and it is possible to refer to this angle as the beam crossing half-angle.

[0175] In practice, phase matching is commonly implemented with QPM, which overcomes the fact that the phase mismatch of the dispersion of the refractive index in a birefringent material does not permit phase matching unless the birefringent material is very thin. A NLO material is structured to spatially modulate its nonlinear properties, typically periodic poling is used to create a linear grating structure with alternating signs of the second order nonlinearity x(2). For non-collinear mixing, the phase matching requirement of a QPM NLO material needs to take account of the angles at which each of the input beams traverse the QPM NLO material. Here it is noted that the poling period that is required for noncolllinear mixing will be smaller than for collinear mixing, since only a component of the wavevector of each of the input beams will be in the poling direction.

[0176] Conventional birefringent NLO materials can be used. If the NLO material is quasi-phase matched then conventional inorganic NLO materials can be used, in particular oxides that are ferroelectric and which can be patterned by periodic poling. Examples are lithium niobate (LN) to create periodically poled (PP) lithium niobate (PPLN), magnesium oxide doped periodically poled lithium niobate (PPMgOLN), lithium tantalate (LT) to create PPLT and potassium titanyl phosphate (KTP) to create PPKTP. As is well known, microstructuring a ferroelectric NLO material with a QPM structure, e.g. a linear grating, is achieved by electric field poling. Some non-ferroelectric materials can also be processed to spatially modulate their nonlinear properties using a recently developed technique called orientation patterning, such as 11 l-V Zincblende semiconductor crystals, e.g. GaAs. While orientation patterned GaAs is not transparent in the visible, other materials which are transparent could be used (e.g. orientation patterned GaN) could be used. Moreover, as well as using inorganic material such as the above, the poled NLO material may also be an organic material (or layers of an appropriately non-centrosymmetric organic material).

[0177] DESCRIPTION OF FIGURES

[0178] Figure 1 shows a diagram of how an image is received by the human eye showing an eyeball 160 with a pupil 170. The eye receives an image of an object 180 illustrated as a ‘stick man’. The light scattered from the object 180 is received at the pupil 170 and is then focused by the eye's lens to the retina at the back of the eyeball where it is detected by rod and cone cells. The light can be considered as rays of light 184. Typically, the distance between the eyeball 180 and the object is at least 15 cm (or so), representing the near point of the eye. The distance can be much larger; in the case of starlight for example this is millions of kilometres or greater. The fact that this distance may be arge’ is indicated by the lines 186 to represent a discontinuity of scale in the figure. The light scattered from the object may be represented by wavefronts 188, an inset of which is shown as 190. Each wavefront is separated by one wavelength A of the light. There is a defined phase relationship between any two points on a given wavefront (indicated here as <j>_1 and In this diagram the wavefronts are shown to be curved with a wavefront curvature that corresponds to the distance 192 from the object 180. The lens of the human eye changes shape under muscular control (the ciliary muscles), and the shape taken creates a focusing of the incoming (generally curved wavefront) such that a desired part of the field of view is brought to a sharp focus on the retina.

[0179] Figure 2 shows an eye 160, an object 180, which is arranged at a distance 192 from the eye 160, this distance being large. A spectacle lens 20 is arranged in front of the eye. The light scattered from the object 180 that has passed through the spectacle lens is associated with a wavefront 188. Two sets of optical rays are shown emanating from points A and B. Point A is on-axis and Point B is off-axis in respect of the principal optical axis of the eye 160. The light incident on the eye enters the eye through the pupil as controlled by the iris 172. Light rays emanating from Points A and B are associated with respective wavefronts 188A and 188B and are focused at two different points 190A and 190B respectively on the retina 168 by the eye lens 166. The angle that the light rays make to the optical axis of the eye determines where on the retina the image is formed, while the wavefront curvature determines the position of focus.

[0180] Figure 3 shows that light emitted from a point source close to the eyeball will have a wavefront curvature that cannot match the wavefront curvature of light from a natural scene. An eyeball 160 and a distant object 180 are shown. The wavefront curvature of the light from the object 180 corresponds to the distance from the object, which is ‘large’. A piece of optical material (a spectacle lens) 20 is located close to the eyeball and is shown as hosting a point source of light 192. This point source 192 could be fluorescence from the optical material (such as a dye or a rare-earth element) or a scattering point (such as from a dust particle). The point source will create wavefronts 188 centred at the point source 192 with an associated radius of curvature. Because the point source is located very close to the eyeball (e.g. of the order 1 cm), the radius of curvature of the light will be small when it enters the eye. This creates two problems to be solved. Firstly, the wavefront emanating from the point source 192 is too divergent to be focused by the eye lens (i.e. the eye cannot create a sharp image on the retina). Secondly, the radius of curvature of the wavefront emanating from the point source 188 will be very different from that of the natural scene light from, for example, the object 180. To solve the first problem, the radius curvature must have a value above the threshold needed for the eye to be able to focus the image onto the retina. To solve the second problem, if the AR light source creates an object on the optical material, then, in order to obtain a convincing visual effect, it is necessary to make the radius of curvature of the light representing the virtual object the same, or approximately the same, as it would be if the virtual object were a real object. For example, if the real scene is of a golf course green with a hole, and the virtual object is a golf ball that is putted into the hole, then the radius of curvature of the wavefront representing the golf ball needs to be matched to the radii of curvature from the real scene in the proximity of where the golf ball is being presented. Such matching is what makes the distance perception realistic for the viewer. Such matching is also important for a pleasant viewing experience, since large discrepancies between wavefront curvatures presented to a viewer in a single scene is known to be unpleasant, since the natural match between the point on which the eye focuses (accommodation) and the point where the optical axes of the two eyes cross for binocular vision (vergence) is broken. This is referred to as the vergence-accommodation conflict which induces headaches and feelings of nausea similar to motion sickness. In this context, it is noted that in an AR (or MR) system outward facing cameras in combination with distance-measuring devices, such as LiDAR, can be used to survey the scene and obtain relevant information on an appropriate placement of virtual objects in the scene. An AR headset may wish to overlay an image of a new object onto a natural scene. This may be to overlay an object that is not present in the natural scene (e.g. to show two tennis players playing a match overlaid onto a natural scene of an empty tennis court). An AR headset may also wish to replace the image of a real object in a scene with a virtual object (e.g. to replace images of farm animals in the natural scene with images of dinosaurs).

[0181] Figure 4 is a schematic drawing illustrating operating principles used by embodiments of the invention. An eye 160 with retina 168 is shown. A piece of transparent (in the visible) NLO material 100 in the format of a spectacle lens with a front (proximal) face and a back (distal) face is arranged in front of the eye at a vertex distance. Light leaving the spectral lens from the front face towards the eye of the wearer travels in a proximal or inward direction. Light leaving the spectral lens from the back face away into the environment travels in a distal or outward direction. We refer to the lens as a spectacle lens by virtue of its format but this does not imply the lens has any vision correction function.

[0182] First and second input beams 70 and 76 are input into the NLO material 100 and cross so that an intersection region is formed within the NLO material 100. This is a non-collinear geometry. The first and second input beams 70, 76 have respective first and second frequencies. The first and second frequencies may be the same as each other or different. In the intersection region, nonlinear mixing occurs by sum frequency generation (SFG) and a product beam is generated. The product beam has a sum frequency that is equal to the sum of the first and second frequencies.

[0183] The product beam generated in the intersection region by the nonlinear mixing is effectively a point source and, as such, creates its own wavefront 84 with a wavevector direction that will send the product beam light to the pupil. The product beam is only generated where the beams overlap, i.e. in the intersection region. The NLO material can be chosen to be transparent to visible light and so the viewer can ‘see through’ the material, as would be needed for AR to allow a synthetic scene image to be superimposed onto the natural scene, as shown schematically by object 180 with wavefront 188, which is distant (represented by the discontinuity 186.

[0184] The physics of an extended source generated by nonlinear mixing, as proposed, is different from that of an extended source created by projection or emission from a fluorescent material in that nonlinear mixing by crossed beams allows for wavefront generation and phase matching, which can for example be realized with quasi-phase matching, so that the product beam can be generated in the intersection region with, in principle, any desired radius of curvature.

[0185] The synthetic scene image can be produced in the NLO material in a number of different ways. One option is to move the first and second input beams, as indicated by reference numerals 71 and 77, so that the angle of beam crossing is a function of position across the image generation area to "write" the synthetic scene image into the NLO material. To achieve this, the point of entry of the input beams into the NLO material may need to be varied. If the first and second input beams are appropriately modulated in intersection angle and amplitude, then a synthetic scene image can be built up, e.g. using some kind of rastering. Another option is to provide one or both input beams with a patterned crosssection, e.g. a 2D amplitude modulation, such that when the two input beams mix in the intersection region a complete object is created by the static overlay of the two input beam cross-sections and their angular spectrum. A combination of these two approaches may be used with a hybrid of scanning and beam-profile modulation. Within the general concept of generating the light of the synthetic scene image by a phase matched nonlinear process, it is further noted that the wavefront 84 generated by such nonlinear mixing can be given a desired direction as well as a desired radius of curvature, the desired direction being towards the eyeball as illustrated.

[0186] In summary, an image is created on the retina 168 which combines a wavefront 188 from the natural scene and a wavefront 84 of a synthetic scene image generated in the lens, wherein both wavefronts have the same or similar radii of curvature.

[0187] Figure 5 shows key concepts of QPM. QPM is a property that can be impressed in certain nonlinear crystals through a method known as periodic poling. A ferroelectric nonlinear crystal is processed by periodic poling in order to create a periodic domain structure which modulates the sign of the crystal's nonlinear susceptibility so that it alternates positive and negative when light propagates along a certain direction within the crystal. Periodic poling thus mimics the effect of bonding slices of a birefringent material together into a stack with alternating crystallographic orientations, which is of course not practical, at least not for anything other than extremely long wavelengths in the very far infrared or microwave regions. A ferroelectric material most commonly used for periodic poling is lithium niobate, which after periodic poling is referred to as periodically poled lithium niobate (PPLN). Another material is periodically poled potassium titanyl phosphate (PPKTP). More recently, a technique called orientation patterning has been developed to allow periodic poling of nonferroelectric materials, such as lll-V Zincblende semiconductor crystals such as GaAs. As well as using inorganic material such as the above, the poled NLO material may also be an organic material. As illustrated in the upper portion of the figure, a QPM NLO material 200 comprises an alternative series of layers with ‘up’ and ‘down’ domains 202 and 204 respectively. The material is structured this way because the dispersion of the material (the variation of refractive index with wavelength) prevents efficient nonlinear mixing due to a phase mismatch. This is shown in the middle part of the figure, which shows how the refractive index, n, depends upon wavelength, A. In general, the refractive index of a usual NLO material decreases with increasing wavelength (called normal dispersion). In SFG, three wavelengths are involved. First and second wavelengths A_1 and A_2 mix to create a third wavelength A_3 with a wavelength such that energy is conserved. The wavelength of the nonlinear mixing (in this case sum frequency mixing) is set such that

[0188] 1 / A 3 = 1 / A_1 + 1 / A_2 which corresponds to conservation of energy.

[0189] Because refractive index varies with wavelength, there is a phase mismatch between the three optical wavevectors. The wavevectors are given by

[0190] KJ = 2 * TT nJ I A_i where nJ is the refractive index of the NLO material at wavelength i, and A J is wavelength.

[0191] A wavevector matching diagram is shown in the lower part of the figure in which K_3 (is longer than the vector sum of K_1 and K_2. This difference occurs because the refractive index at A_3, n_3, is greater than the refractive indices n_1 and n_2 at wavelengths A_1 and A_2. The k_vector mismatch is K_G. In quasi-phase matching, the period of the QPM structure 200 is set such that

[0192] Lg = 2 * TT / K_G. where Lg is the QPM period. It should be emphasised that in this discussion it is assumed that the waves are co-propagating in the same direction.

[0193] Figure 6 shows a more complex geometry of a poled NLO material. Instead of the quasi- phase matching structure being a simple linear grating structure, the nonlinear crystal is processed by poling local regions to cause their domains to point upwards out the plane of the drawing (‘up’ domains) in the nonlinear crystal that naturally has 'down' domains pointing in the opposite direction. The shape of the poled regions may advantageously follow the crystal symmetry of the NLO material, so for example may be hexagonal in the case of LiNbO3. The illustrated pattern is of circular poled domains in a hexagonal close packing (HCP) pattern, but any other distribution may be chosen, e.g. a square or rectangular packing of square or rectangular domains. Moreover, the patterning could be aperiodic. The diagram shows the physical structure as having two vectors a_1 and a_2 which represent the period of the 2D poling. Also shown are the two associated K_G basis vectors b_1 and b_2. It should be noted that these can be used to phase match interactions in non-collinear wave mixing, so that other choices of angle can be used. All that is needed is for suitable nonlinear grating vectors to exist which permit the phase matching diagram to create a product beam that is appropriately directed, i.e. into the pupil of the eye.

[0194] Figure 7 is a schematic drawing of part of a piece of NLO material 100, wherein the NLO material is spatially modulated in a region 108 in respect of its second order nonlinearity to provide quasi-phase matching to the crossing input beams 70, 76. The QPM region 108 is schematically illustrated as a linear grating. The first and second input beams 70 and 76 are introduced into the lens from opposite ends and propagate via one or more reflections, e.g. total internal reflections from the inner and outer surfaces of the NLO material 100 (one reflection from the outer surface is illustrated for each of the first and second input beams 70, 76 until they cross in the intersection region which overlaps with the QPM region 108. The quasi-phase matching condition is satisfied in the QPM region 108 and product beam 84 is generated by the nonlinear wave mixing to generate a wavefront that propagates towards the pupil 170 of the eyeball 160.

[0195] Figure 8 shows another representation of the input beam intersection region. The first and second input beams 84 and 85 have respective wavevectors k_1 and k_2 producing a product beam having a wavevector k_g which is the vector sum of k_1 and k_2. Ideally the vector sum of k_1 and k_2, i.e. k_g, should be exactly equal to the wavevector associated with the QPM grating period, K_G, in order for greatest conversion efficiency. The phase matching vector diagram is shown in the right-hand figure. It is noted that the period of the QPM grating period that is needed can be calculated by simple trigonometry from the input wavevectors k_1 and k_2 and taking account of the dispersion characteristics of the NLO material.

[0196] Figure 9 is a schematic drawing of a display panel incorporating a NLO material layer 100. It can be seen that first and second input beams 70, 76 are directed to traverse the display panel 100 by multiple reflections from mirror layers formed to the front and back of the NLO material layer so that the beams 70, 76 cross each other in an intersection region 110 and generate a product beam wavefront 84 by nonlinear mixing. It will be understood also that the beams 70, 76 are brought to a focus at the intersection region 110. A distant object 180 is arranged some metres from the eye, indicated by the discontinuity 186. Natural light is scattered from the distant object 180 towards the eye 160. The product beam light representing a synthetic scene is therefore superimposed onto the light from the natural scene light. The light passes through the pupil 170 and into the eye 160 to form a fused synthetic / natural scene image on the retina.

[0197] Figure 10 shows a display panel with a multilayer structure comprising a NLO material layer 100, a front mirror layer 136 and a back mirror layer 130. The front and back mirror layers are reflective at the frequencies of the input beams and transmissive across visible frequencies. Two different input beam paths 70A, 70B for a first input beam are shown to illustrate how the angle and position (0, z) at which an input beam is introduced into the display panel can be varied to vary the angle at which an input beam crosses the intersection region. The back mirror layer 138 covers a central part of the display panel but not a left end portion of the display panel's front face. The first input beam is introduced into this left end portion at an angle and position (01, z1) or alternatively (02, z2). The first input beam is then scattered from the front mirror layer 136 onto the back mirror layer 138 etc. and thereby traverses the display panel to an intersection region containing a QPM grating structure 108. A second input beam is introduced similarly at a right end portion of the display panel (not shown) and arrives at the intersection region 108 along beam paths 76A or 76B. The first and second input beams then cross each other in a non-collinear geometry at the QPM region 108 and generate the product beam by sum frequency generation. The desired position and angle for introducing the input beam can be calculated simply by ray tracing back from the point where the intersection region 108 is desired and what angle the input beam should cross the QPM region 108. As the path length of each beam changes as a consequence of varying the angle and position of the injection of each beam into the display panel, lenses or other focusing elements external to the display panel (not shown) can be adjusted to ensure that both beams 70, 76 remain focused in the intersection region 108.

[0198] Provision of mirror layers 136, 138 allows for reflection over a wider range of angles than is possible when total internal reflection from the panel-to-air interface is relied upon. Here it is noted that, if the nonlinear mixing process is by frequency addition, and assuming the usual case where the image is formed in the visible region, and further assuming the frequencies of the light beams that mix are either the same or not too dissimilar, then the light beams will be in the infrared (IR) wavelength region, so that the reflective coatings will be designed to efficiently reflect light at the infrared wavelength (or wavelengths) of the light beams.

[0199] Alternatively, both the first and second input beams could be inserted from the same end of the lens and the lens structure could be designed so that the second beam is reflected from a reflective element to the right of the figure and thus bounce back along the paths 76A or 76B as illustrated. It will thus be appreciated that by appropriate routing and / or scanning of the first and second input beams they can be introduced to propagate in the lens at different angles where changes in the angle necessitate different points of introduction to ensure the intersection region does not move.

[0200] It will be appreciated that the first and second input beams can only be exactly phase matched to the QPM NLO material at one specific combination of their respective angles. Therefore, scanning the input beams to vary their angles will cause the phase matching of the crossed input beams with the QPM NLO material to become gradually less exact with increasing divergence of the angles from the specified perfect phase matching angles. The conversion efficiency, and hence the intensity of the mixed product beam light, will thus gradually reduce as these angles moves away from the specified angles. Since the conversion efficiency is a function of beam angle and will therefore vary while scanning each beam, e.g. by rastering, this can be compensated for by a temporal variation of the beam intensity during scanning to ensure a non-varying intensity scale for the product beam light generation across the synthetic scene image.

[0201] It is also generally preferred to make the distance between the front and back mirror layers sufficiently large that the number of reflections (bounces) is kept to a relatively low number while at the same time allowing relatively large bounce angles 0. This reduces overlap between spots (i.e. beam intersection regions) which in turn reduces unwanted waveguiding and modal excitation effects.

[0202] Figure 11 shows example optical components for routing an input beam from a beam source (not shown) to the display panel (not shown). A beam 220 from the beam source strikes a first mirror 222A and passes onto a second mirror 222B, which is rotatable by an angle <t>_1 , and then onto a third mirror 222C, which is rotatable by an angle <f_2. Different beam paths 224A and 224B can thus be produced which arrive at different angles and positions on the display panel. It will be understood that this is simply one example of optical components capable of carrying out the routing, and other approaches can fulfil the same function.

[0203] Figure 12a shows a further example optical component 230 which may be included in the optical components for routing the input beam from a beam source, in the figure the component is a simple optical glass block. The optical component 230 allows for translating a beam, i.e. changing its position, without changing its angle of propagation. The optical component 230 is mounted to be rotatable about an angle 0 and is a block of glass or other optical material 230. The optical component 230 is transparent for the beam 220. The beam 220 enters the optical component 230, refracts at the input face and traces the path shown through the block. Upon exit the beam path is translated downwards to beam path 224A which is displaced according to the angle 0 but remains parallel to beam path 220. Thus, a displacement with change in angle is achieved. With polarized light the optical element (glass block) 230 may be placed so as to exploit Brewster’s angle to minimise unwanted reflections, or alternatively it may be anti-reflection coated. In a miniaturised system various suitable implementation technologies exist, for example microelectromechanical systems (MEMS) mirror technology, electro-optics and other known beam scanning technologies. A desire for miniaturisation and also limiting the number of moving parts makes solid state or MEMS devices attractive.

[0204] Figure 12b shows components to direct light to a desired point within an NLO material layer while altering the angles that the light approaches that spot. The diagram shows an input beam (shown here coming from the right-hand side of the diagram. This input beam, in this case labelled 76 to match other diagrams, passes into a first lens 240A. This first lens 240A is placed one focal length (f’) from a rotatable mirror 222. Assuming the input beam 76 is approximately collimated, the lens 240A will create a small focused spot on the mirror 222. The light reflected from the mirror 222 is directed towards a second lens 240B. The second lens 240B is placed a distance 2f from the mirror 222 and images the small spot from the mirror 222 onto the desired refocused spot labelled 110 since this light beam is to be delivered into the NLO material where it crosses with the other beam. In this example, the distance from lens 240B to the refocused spot is also 2f. These focal length values satisfy the conventional imaging requirement. When the mirror 222 rotates (see rotation angle 0) around either axis, the resultant beam paths will be deflected, but, because of the imaging operation of lens 240B the spot of light will be sent to the same location 110, but will approach with an angle set by the rotation of the mirror 222. Advantageously the mirror 222 will be rotated so that its centre of rotation is coincident with the reflective surface of the mirror. It will be understood that various values of focal length can be used, so long as the mirror is imaged into the beam-crossing interaction region. Moreover, further optical elements may be inserted into the system so as to steer the crossing point to a desired region of the display element (AR spectacle system). Furthermore, additional optical elements may be used to ensure that, wherever within the display system the desired interaction region is placed, it will still have an appropriate size.

[0205] Figure 13 shows an AR display apparatus according to an embodiment. A first one of the input beams has a beam profile that is phase modulated and the other of the input beams has a beam profile that is amplitude modulated, so that in the intersection region 110 where the two input beams cross both amplitude and phase modulation is encoded into the product beam generated by the nonlinear mixing of the first and second input beams in the phase- matched NLO material layer. The display panel generates an image of a synthetic object. The necessary information for the synthetic scene image is effectively encoded in the angular spectrum of product beam light hitting the eye. The focus requirement of the eye lens 166 is such as to take the curved wavefront 181 from the natural scene object 180 and bring it to a single focus on the retina 168. An eyeball 162 is shown with a pupil 170, an eye lens 166 and a retina 168. The pupil 170 of the eye receives light from a natural scene object 180. The natural scene object 180 scatters incident light, which creates a wavefront 181 with spherical waves. The light of the wavefront 181 passes through the display panel, impinges upon the eye, enters the pupil 170, and is focused by the eye lens 166 to form a sharply focused image on the retina 168. If light were emitted and ray traced from other points on the natural scene object 180 it would present at different angles to the optical axis of the eye and be focused at different positions on the retina 168.

[0206] When an AR headset is worn, the display panel 88 is arranged in front of the eye 160. The display panel 88 is transparent to light scattered from the natural scene, such as contained in the wavefront 188, and additionally generates artificial images responsive to crossed pairs of image generating input beams, one pair being shown. Image information is delivered to the display panel 88 through a suitable delivery setup. The display panel 88 is shown as curved, and has an outer surface 92, and an inner surface 90 which from the interfaces from the display panel 88 to air.

[0207] The display panel 88 shown has a composite layered construction with a NLO material layer and front and back mirror layers 136, 138. The mirror layers 136, 138 are reflective at the infrared wavelengths of the input beams.

[0208] The NLO material layer of the display panel 88 has a QPM region 108 arranged to cover a portion of the display panel 88, which is illustrated as being quite a small portion, although it will be understood that the QPM region 108 could extend over the whole of the eye's visual field when the eye is pointing straight ahead and / or over the whole of the eye's visual field at all possible swivel angles and / or the whole area of the NLO material layer.

[0209] Referring to the left-hand side of the figure, a first input beam 70 is provided that is amplitude modulated. The first input beam 70 is generated by an emitter array 16 which encodes image information in the beam cross-section by amplitude, i.e. intensity, modulation by suitable driving of the individual emitters of the array. The first input beam 70 is emitted from the emitter array and then reflected by a mirror 222A and transmitted through a relay lens 240A onto a left end face of the display panel 88. The mirror 222A may be deflectable, e.g. by rotation in one or two axes, to alter the path of the first input beam 70. Similarly, the lens 240A may be translatably and / or rotatably mounted to alter the ray path of the first input beam 70. Moreover, other mirror, lens or other optical components may be included for beam manipulation of the direction, position and divergence of the first input beam 70. The position and angle at which the first input beam 70 is introduced into the end face of the display panel can thus be adjusted. The left end face of the display panel is illustrated as being set at an angle relative to normal, i.e. a 90 degree cut-off of the display panel, to facilitate in-coupling of the first input beam 70. Alternatively, the first input beam 70 could be introduced into the front face of the display panel. Another alternative is to use a grating incoupling of the first input beam 70, where the grating could cover an area portion of either the front face or the end face of the display panel. After introduction into the display panel, the first input beam 70 traverses the display panel from left to right by successive reflections from the mirror layers 136, 138. Referring to the right-hand side of the figure, a second input beam 76 is provided that is phase modulated. The second input beam 76 is generated by a laser source 18. The second input beam 76 passes from the laser source 18, through beam expanding lenses 240A and 240B, to a first plane mirror 222B where it is reflected onto a second mirror 222C. The second mirror in this beam path 222C is a phase retarding mirror (allowing a spatially varying phase to be imparted on the input), such as a liquid crystal phase modulator, and is used to impress the spatially structured phase modulation onto the laser beam. The same comments apply to the second input beam 76 as made above for the first input beam 70 in terms of its introduction into and propagation across the display panel 88. The second input beam 76 is illustrated as being introduced into a right end face of the display panel. The first and second input beams 70, 76 then cross in the intersection region 110 where the QPM NLO material is located resulting in generation of a product beam with a wavefront 84 and a frequency at the sum frequency of the first and second input beams 76. The wavefront 84 enters the pupil 170 and then is focused onto the retina 168. The product beam light is superimposed on the natural scene light 181 from object 180. By setting the radius of curvature of the wavefront of the second input beam 76 (by imposing a spatially structured phase retardation across the beam), the synthetic scene image can also be given any desired radius of curvature so that the synthetic scene image is a virtual image formed at any desired distance from the eye. The radius of curvature of the second input beam 76 can be altered by using Gaussian beam optics, so reducing the size of the laser beam. Reducing the laser beam's cross-sectional area, will increase the beam divergence of the resulting nonlinear wavefront.

[0210] It is thus possible to vary the virtual image distance of a synthetic scene image to provide a fused synthetic and natural scene image on the eye in which the synthetic scene image fits into the natural scene in a realistic manner.

[0211] In the above embodiment, it was implicitly assumed that the QPM NLO material is a simple linear grating. However, in further embodiments, the QPM NLO material may have a more complex spatial profile that incorporates some curvature so as to generate a diverging (or converging) product beam. In still further embodiments, it is possible to impress both amplitude information and phase information onto either or both of the first and second input beams 70, 76.

[0212] The emitter array 16 may be an OLED array such as a micro-OLED array. There is a controller device 324 which controls the output of the OLED array to create the intensity information in the image. Other options for the emitter array include an inorganic LED array, an array of quantum dot emitters, liquid crystal devices with appropriate back illumination, liquid crystal on silicon (LOOS) with appropriate illumination, VCSEL array, and tilting MEMS display devices of the digital micromirror device (DMD) type such as DLP devices from Texas Instruments. It is noted that the system does not require coherence in the amplitude structured beam and this is advantageous both in allowing incoherent emitter light sources (such as micro-OLED arrays) but also in reducing speckle in the system.

[0213] The first input beam 70 is thus formed by imposing a varying phase modulation on a beam generated by a single emitter, such as a laser, with for example a Gaussian intensity profile. The second input beam 76 is generated by an emitter array so as to have a beam crosssection that carries a spatially varying amplitude modulation. In other embodiments, the first input beam 70 could have a cross-section that is spatially modulated in both phase and amplitude while the second input beam 76 may be modulated in either phase or amplitude or both. A suitable source for combined phase and amplitude modulation would be a phase controlled emitter array, such as a VCSEL array with external phase modulation, or an array of independent emitters with appropriate phase control.

[0214] An advantage of embodiments which allow both the first and second input beams to have spatial modulation of their intensities is that this can be used to compensate for the fact that different parts of each beam cross-section will be more and less exactly phase matched to the QPM NLO material. The display apparatus may be designed such that light along the principal optical axis of each beam is exactly phase matched with the QPM NLO material where they cross in the QPM region, whereas the phase match of the crossed-beam components with the QPM NLO material will become gradually less exact with increasing distance away from the optical axis of each beam. The conversion efficiency will thus gradually reduce as one moves away from the optical axis of each beam. The intensity profile of the beam can therefore be modified to increase the intensity at the extremities of each beam so that the intensity scale of the product beam light is kept constant across the whole crossing region of the beams.

[0215] Figure 14 is a schematic drawing of the beam intersection region 110 where QPM NLO material 108 is located. A first input beam 70 has a beam cross-section with an approximately Gaussian profile 81 to provide a simple planar phase front, i.e. there is no spatial phase variation across the beam, or only that inherent to it being a Gaussian beam. A second input beam 76 has a beam cross-section which, in the general case, is both amplitude and phase modulated according to respective amplitude and phase modulation functions 75. The first and second input beams 70 and 76, which are in the infrared, interact via the second order nonlinearity of the QPM NLO material 108 to exploit a phase matching scheme as described further above with reference to Figure 8 with an appropriate nonlinear grating K-vector. The nonlinear interaction causes a nonlinear polarisation at a frequency, which in this case is at the sum frequency of the first and second input beams, the sum frequency being in the visible, to generate an output beam 82 having a wavefront 84 which varies in both amplitude and phase in a way determined by the amplitude and phase of the input waves. The information encoded into the second input beam by the amplitude and phase modulation functions 75 is thus transferred to the product beam wavefront 84.

[0216] Figure 15 is a schematic diagram applicable to the setup of Figure 14. Namely, there is a first input beam 70 with a beam cross-section of an approximately Gaussian profile 81 to provide a plane wave wavefront combined with a second input beam with an amplitude and / or phase modulated wavefront. The second input beam is generated by a suitable emitter array 16 which is located at one focal length, f, from a relay lens 242, which in turn is located one focal length, f, from the intersection region 110. This arrangement places the image information contained in the amplitude and phase modulated wavefront from the emitter array 16 in the Fourier plane of the imaging system. The resulting visible beam generated by the nonlinear mixing of the input beams will therefore contain the same spatial image information as contained in the second input beam, this image information then passing to the eye encoded essentially in plane waves. The visible product beam light generated by the nonlinear mixing will have wavefronts with radii of curvature that follow from the radius of curvature of the first input beam 70 so that the product beam light will have an equivalent imaging position determined by the radius of curvature of the first input beam 70. The foreshortening caused by the finite crossing angles of the two input beams can be compensated for using simple trigonometry, and taken into account in either or both of the beam profiles I and the construction of the image.

[0217] Figure 16 shows a more complex example layer structure for an AR display panel 88. Air 128 surrounds the display, and this has a refractive index of very close to 1.

[0218] The figure shows the distal, outer side of the display panel 88, which is furthest from the eye (also called the ‘back’) which has a surface between air and the display panel. The figure also shows a proximal surface on an inner side of the display panel closest to the eyeball (also called the ‘front.

[0219] Referring to each layer in turn in the distal to proximal direction: • Layer 146 is a back surface anti-reflection coating (ARC) layer AR_1 to reduce reflections of visible light propagating in the air 128 from the panel-to-air interface. The outer antireflection coating layer 146 / AR_1 should preferably be anti-reflective over the full visible range of wavelengths to accept light from natural scene sources. Such ARC coatings are routinely deposited on optical elements such as camera lenses. It is also desirable to make this layer ‘hard’ so as to resist scratches.

[0220] • Layer 142 is a back vision correction layer. This is made of an optically transparent material (in the visible) which has refractive power in order to correct for vision defects of the wearer, such as short-sightedness, long-sightedness, astigmatism etc. The vision correction may involve altering the curvature of this layer, or to incorporate a polarizing filter as is common in sunglasses. Alternatively, the layer 142 could be optically neutral, i.e. not provide an image enhancement effect, or be omitted. Another possible function of the front vision enhancement layer 142 is to provide dimming responsive to certain natural light conditions. For example, the vision enhancement layer 142 may incorporate UV-sensitive photochromic material, such as a silver halide, as used in conventional dimming display panels. Another option for dimming would be to use an electrochromic switchable material, as are sometimes used for automatic dimming of rear view mirrors in automobiles. For example, the vision enhancement layer 142 may incorporate tungsten trioxide. For an electrochromic material, an electrical source will be needed to be contacted to the vision enhancement layer 142 to power the dimming function.

[0221] • Layer 130 is a back filter layer (labelled A1). The back filter layer absorbs, i.e. is opaque at, the wavelengths of the input beams, which will generally be in the infrared. This will avoid any light from the input beams being emitted out of the back face of the display panel into the air 128. The provision of a back filter layer 130 may be required to comply with laser safety regulations in case one or both the first and second input beams are generated by a laser source. The back filter layer 130 can be made from an organic material with an absorption at appropriate infrared wavelengths. Moreover, it is desirable for the back filter layer to transmit visible light so as not to attenuate natural scene light. Such filters are sometimes known as heat absorbing filters or ‘short pass’ filters, and can be based on a glass such as Schott KG1. Alternatively, the back filter layer 130 can be based on thin film interference effects, which may be suitable when the light of the input beams is highly monochromatic and narrowband, as generated by lasers.

[0222] • Layer 150A is a spacing layer of optically transparent material (in the visible) and is provided to separate the IR absorbing layer 130 from the next layer 136.

[0223] • Layer 136 is a back mirror layer for internally reflecting the input beams as they laterally traverse the display panel 88. Provision of the back mirror layer 136 avoids using total internal reflection at the distal panel-to-air interface 146 and also avoids any possible losses, scattering or other undesired effects from the input beams traversing the outer layers 146 to 150.

[0224] • Layer 150B is a spacing layer which is optically transparent both in the visible and the infrared (at the wavelengths of the input beams). This spacing layer 150B is provided to increase the thickness of the material traversed by the input beams and thereby reduce the number of internal reflections (‘bounces’) needed for each input beam to reach the intersection region.

[0225] • Layer 100 is a layer of NLO material which, at least in the area which will be covered by the eye’s visual field when pointing straight ahead, is quasi-phase matched to the input beams taking account of the angles with which they will cross each other.

[0226] • Layer 150C is a spacing layer with similar function and properties to the spacing layer 150B.

[0227] • Layer 138 is a front mirror layer for internally reflecting the input beams as they laterally traverse the display element 88 with similar function and properties to the back mirror layer 136.

[0228] • Layer 150D is a spacing layer of an optically transparent material (in the visible and the infrared wavelengths of the input beams) and is provided to separate the front mirror layer 138 from the next layer 132.

[0229] • Layer 132 is a front filter layer that is configured to absorb any stray light from the input beams, i.e. at the wavelengths of the input beams, which will generally be in the infrared with similar properties to the back filter layer 130. The front filter layer 132 thus prevents light at the wavelengths of the input beams from passing into the air 128 on the proximal side, thereby protecting the wearer's eye.

[0230] • Layer 144 is a front vision correction layer which can have any of the functions or properties as described above for the back vision correction layer 142.

[0231] • Layer 148 is a front anti-reflection coating (ARC) layer AR_2 with similar properties to the back ARC layer AR_1 / 146. Its role is to reduce reflections of visible light propagating in the air 128 from the panel-to-air interface.

[0232] One particular advantage of this layer structure is that by confining the optical paths of the input beams to a certain portion of the layer stack between the front and back mirror layers 138, 136, vision correction can be applied independently of managing the input beams and generation of the product beam light nonlinear mixing. Namely, vision correction of refractive errors in the eye lens can be achieved in one or more layers that are situated outside the substack 136-138, preferably outside the substack in the distal (back) direction, e.g. in layer 142 as done in the illustrated layer stack. In particular, the optical paths of the input beams can be managed without taking account of any vision correction layers.

[0233] It is noted that some of the above layers could be omitted. The anti-reflection coating layers are optional. Moreover, the proximal reflection of the input beams may be performed by total internal reflection from the back panel-to-air interface in which case the back mirror layer would be omitted. Similarly, if total internal reflection is used for the distal reflection of the input beams, the front mirror layer would be omitted. Depending on the nature of the light sources, e.g. whether the input beams are coherent or not, and their wavelengths and maximum output powers, it may also be that the front filter layer could be omitted, or possibly both the back and front filter layers. The front and back vision correction layers are also optional.

[0234] It should be noted that additional layers could be added, for example to provide antireflection coatings between the different optical materials involved, although it is worth noting that the greatest unwanted reflections are likely to occur between air and the display panel (due to the large index difference to air) and also between the nonlinear layer 100, which, depending on the choice of NLO material might have a quite large refractive index difference.

[0235] Figure 17 shows an eyeball 162 which connects to the brain through the optic nerve 174. The eyeball can swivel within the eye socket and thus align its principal optical axis in different directions, for example A, B and C as illustrated. The display panel 88 is constructed in such as way that the quasi-phase matching grating vector Kg varies across the lens so that in all swivel positions of the eyeball, e.g. with its optical axis along A, B or C, the QPM NLO material is oriented so as to create product beam light that will scatter towards the iris. This may be achieved by orienting local sections of NLO material in appropriate positions and orientations within the display panel. Alternatively, this may be achieved by using a display panel with appropriate curvature of its front and back faces. Front and back face curvature will affect both the wavevector direction of the product beam and the paths of the input beams within the display panel. It is noted that depending on how light is introduced into the panel (the relative position within the panel and curvature of the reflecting layers) it may be advantageous to place the Kg in a direction that achieves the optimal local phase matching including the effect of input angles and optic axis.

[0236] Figure 18 shows that the display panel can have different shapes. A display panel 88A is planar with parallel and flat front and back faces. This provides input beam paths that are easy to determine. A display panel 88B has front and back faces with the same radii of curvature (i.e. the front and back faces have respective curvatures associated with respective centre points that are offset). A display panel 88C has front and back faces that are curved with different radii of curvature such that both circles have a common centre point, thereby providing a display panel of constant thickness. In all cases, the curvature may be on the surface of a cylinder or the surface of a sphere. Alternatively, there may be different curvatures in two planes to provide more complex curved faces.

[0237] Figure 19 is a schematic perspective view of a display apparatus in the format of a pair of AR spectacles 10. The spectacles 10 comprise conventional parts including left-hand and right-hand spectacle arms (i.e. temples) 32, 34 and a frame with left and right rims for accommodating left-eye and right-eye lenses. A display panel as described above forms each spectacle lens. The frame including a bridge, nose pad and left-arm and right-arm location points which may be hinged or unhinged. To support its AR functions, the spectacle arms and frame may be modified in comparison to conventional spectacles to accommodate the necessary additional components by internal housing and / or external attachment. These parts may include electronic circuit components, batteries, optical sources and optical elements. A wireless transmitter or transceiver 40 may be included for communicating according to a wireless protocol, such as Bluetooth or a WiFi protocol. The arm-mounted wireless transceiver 40 allows the AR spectacles 10 to communicate wirelessly over a communication path 42 with an external control device 44 which has its own wireless receiver or transceiver 43 as well as a processing capability via a processor 46 as well as associated memory 48 and external network communications 49 which could include WiFi, 4G / 5G, optical LAN, wired ethernet, etc. The external control device may also provide location devices, such as via a global positioning sensor (GPS) and position sensing via a map of telecom wireless signals and / or wireless network signals. The external control device may be a dedicated standalone unit specifically for controlling the AR spectacles 10 or it may be a mobile phone, tablet, personal computer etc. running with a suitable app (computer program). In turn, the external control device may be in data communication with further devices and, in particular, may have access to remote computing resources in an ad hoc network (e.g. as provided by cloud computing). Processing intensive tasks for controlling image formation in the AR spectacles may therefore be delegated away from the AR spectacles to the external control device and optionally also more remote computing resources with which the external control device communicates. The processing power on board the AR spectacles themselves, and therefore also the associated power consumption, can therefore be kept low, by confining the processing tasks that are carried out on processors integrated in the spectacles to those which are relatively less processing intensive.

[0238] The spectacles frame accommodates left- and right-side outwardly facing cameras 50 and 52 which face outward to view the natural scene. Using a pair of cameras arranged at the same height side-by-side allows a stereoscopic image to be built up, i.e. binocular vision, similar to that of the eyes of the wearer, thereby allowing distance to natural scene objects to be determined, at least when the distance is relatively short. The spectacle frame 10 further accommodates a range finder, such as a LiDAR device or point cloud imaging system 58, for detecting objects in the natural scene and determining their distance from the wearer. In a LiDAR system, the LiDAR device comprises at least one laser and detector and the time of flight of the laser light from the LiDAR device to a scattering source and back is measured. The laser beam can be scanned to build up a map of objects in the natural scene or multiple laser beams can be generated to allow information to be obtained from multiple points in the natural scene in parallel. The movement of the laser beam as a result of the natural swivelling of the wearer's head can also be exploited to scan the natural scene. In a point cloud imaging system laser beams are projected into natural scene which are imaged by the cameras 50, 52 to allow a model to be built of the geometry of the natural scene. The object information collected by the cameras and range finder, in particular the object distance information, has a particular synergy with AR spectacles according to the invention, since the QPM nonlinear mixing process allows the radius of curvature of the wavefront generated in the wavefronts of the synthetic scene image to be set to a desired value in response to the distance (and position) information measured by the range finder (and cameras). For example, if the augmented part of the image is to superimpose features onto (or wholly replace) an object in the natural scene, then the augmented part of the image can be generated with wavefronts that have a radius of curvature equal to those that they would have if the augmented part of the image was at the measured distance of the associated natural scene object. Similarly, the placement of the augmented, synthetic scene image in the natural scene can be done correctly based on the position information obtained from the cameras. The position and distance information may of course also be combined to aid segmentation processing of the natural scene in order to identify objects therein.

[0239] One remote computing resource that may be useful is to provide mapping information that is relevant to the wearer's live view of the natural scene, e.g. as an aid to segmenting the natural scene and also to provide realistic lighting when rendering augmented reality objects. Such mapping information may be from a mapping application, live satellite imaging or live flight tracking, for example.

[0240] The level of computing resources that is potentially useful is almost limitless. For example, the computing resources could be used to recreate an entire simulated version of the natural scene in real time based on what is observed by the cameras and range finder in combination with extra mapping information obtained from remove computing resources. The augmented reality objects could then be placed in the simulated version of the natural scene and volume rendered to provide realistic lighting based on textures and so forth. Figure 20 shows a further view of additional elements that can be combined into a display apparatus in an AR spectacles format.

[0241] • 100 is the transparent or near transparent optical element containing the NLO material and the aforementioned IR beams that create the visible light directed to the eye.

[0242] • 26 is the frame of the spectacles which together with the arms (temples) accommodate the necessary electronics and optics components.

[0243] • 50 and 52 are outward facing left-side and right-side cameras.

[0244] • 58 is a point cloud or LiDAR system.

[0245] • 60 are infrared photodetectors arranged on the rims around the periphery of each lens. A minimum of one photodetector per lens is needed. The infrared photodetectors are provided to detect leakage of light from the input beams, which should be contained substantially wholly within the lenses (and prior to introduction into the lenses in the arms and frame parts of the spectacles). Any measurement of, or increase in, photodetector signal would be an indicator of damage. Damage might be a deep lens scratch or physical damage to the frame or arms which exposes a lens end face. On detection of infrared light leakage, or excessive amounts thereof, the spectacles would be shut down, i.e. the light sources would be switched off, thereby to ensure safety, including laser safety in the case that one or more laser sources are used.

[0246] • 54 and 56 are further cameras that are facing inwardly, i.e. towards the eye, are used to track eyeball position. This can be used to help determine the optical image that needs to be transmitted and to ensure that the light is directed only as required to be picked up by the iris. This saves power and thus prolongs battery life. These cameras may also be able to gather information about the stereo vision of the viewer so that the vergence and accommodation conflict can be dealt with as well as to ensure that artificial images are correctly positioned in relation to the natural scene. Importantly, these two cameras can be used to compensate for different interpupillary distances of different viewers (or equivalently interocular distances), and to ensure that the light from the display is directed to the pupil of the viewer's eyes. This has the effect of automatically increasing the eyebox, and overcomes a major difficulty with grating-based or other implementations of AR.

[0247] • 62 indicates a magnetic compass element that can be included to help determine the compass direction in which the spectacles, and thus the wearer's head, is facing. Again this information can be processed either in the electronics onboard the spectacles or with remote processing and may be fused with mapping information. • 64 is a set of accelerometers that can be used to track movement of the spectacles and also determine parameters such as the roll, pitch and yaw angles of the wearer's head assuming that the reference direction for those angles is when the head is held horizontal and with the neck vertebrae untwisted. The accelerometer data can be processed either within the onboard electronics or by external processing in order to track the spectacles motion and thus the wearer's head motion.

[0248] The AR spectacles could be provided with an ambient light sensor 55 (not shown but mounted on spectacles frame or temple) to measure the overall (non-directional) brightness of the natural environment, such as a photodetector sensitive across the visible region. The output from the ambient light sensor could be used to adjust the overall brightness of the synthetic scene image so that it remains visible in bright light conditions while at the same time not being too bright in low light conditions, such as when in dimly lit rooms or at night.

[0249] The inwardly facing cameras can be used to compensate for the different interpupillary distances of different users, and the attendant variation in the eye position relative to the spectacles to ensure correct 3D synthetic scene creation and to ensure light enters the pupils correctly.

[0250] Figure 21 shows a display panel 88 provided with an additional layer 140 to provide dimming responsive to certain natural light conditions. For example, the dimming layer 140 may incorporate UV-sensitive photochromic material, such as a silver halide, as used in conventional dimming display panels. Another option for dimming would be to use an electrochromic switchable material, as are sometimes used for automatic dimming of rearview mirrors in automobiles, windows in trains and commercial aircraft. For example, the electrochromic material may be tungsten trioxide. For an electrochromic material, an electrical source 139 (either a voltage source or current source) will be needed to be contacted to the dimming layer 140 via a suitable electrical connection line 141 to power the dimming function. Another option for a dimming layer would be a liquid crystal layer with modulatable transmission, which may be pixel based. The dimming effect can be applied across the whole display panel (as in a photochromic lens). Another interesting approach is for the dimming effect to be applied selectively only in certain areas of the image. A simple example would be to form a dark rectangular area to provide a suitable background for a synthetic scene image component displaying written information, which would otherwise be illegible if superimposed on a part of the natural scene that is bright. A more sophisticated example is to provide selective local dimming based on information obtained from the natural scene, specifically objects that have been identified by segmentation of the natural scene using data collected by the spectacle's forward-facing cameras and range finder and / or other data collecting sources that may be available, e.g. from vehicle systems in the case of driving a car or flying an aircraft. The local dimming may be exactly coincident with the extent of an object in the natural scene, either to attenuate light from that natural scene object (either entirely or partially), where the local area dimming may be done in combination with insertion of a synthetic scene image in the same local area. There are various examples where it is desirable to attenuate overly bright elements in the natural scene that make it impossible or difficult to view the rest of the natural scene. These include attenuating light from oncoming car headlights, from the sun, e.g. when looking upwards. Another example would be to suppress light pollution when viewing the night sky. Another example is when the AR spectacles are configured as a driver aid for night driving. Here further information from cameras and radar that are part of the car can additionally be included in the image processing. A night vision augmented reality driving experience can be created that is able to reveal or enhance natural scene objects that are usually visible in the daytime but at night are either invisible or too dark to be clearly visible, thereby improving road safety.

[0251] Figure 22 shows a schematic of a pair of AR spectacles 10 in which the NLO material in the lenses 100 is formed in discrete regions, i.e. spots 104, across the image area, the spots being embedded in areas of material which are either not the NLO material or which is the NLO material but are areas thereof that have not been periodically poled (or otherwise quasi-phase matched to the input beams). Visible light generation is therefore confined to the spots of QPM NLO material with the remaining area between the spots being inactive, i.e. not able to generate any significant flux of visible light by nonlinear mixing of the input beams. As the eye swivels in its socket, it is only necessary to ensure that a quite small amount of synthetic scene image light reaches the eye, and given that the iris can be 4- 8 mm across, and the lens surface is only around 12 mm from the eye, the NLO material can be confined to quite small area spots without the image perception degrading compared with full area coverage by the QPM NLO material. In other words, considering the overall area covered by the QPM NLO material, the fill factor of the QPM NLO material may be quite small, e.g. between 5 to 20%. For example, the spots could be circular and 0.3-0.5 mm diameter and distributed in a hexagonal close packed (HOP) grid with a grid spacing of 1-2 mm. The lower part of the figure shows detail in one portion 104 of a lens according to one implementation option for RGB image generation. Each spot is in fact a composite group of three spots 106R, 106G and 106B having different QPM structures so as to generate light in the red, green and blue respectively and thereby allow formation of an RGB image. Alternatively, RGB image formation can be achieved when the three QPM structures for red, green and blue are spatially overlaid, i.e. embedded at different depths, z, within the lens at the same xy-location. Another alternative for RGB image formation is to perform poling of the NLO material according to a single, more complex poling pattern that phase matches all three colours red, green and blue. There is an advantage in this approach of designing the lenses with a grid of spots where mixing can occur instead of a design with a continuous area of QPM NLO material over the part of the lens where the synthetic scene image is to be formed. Namely, nonlinear mixing, i.e. visible light generation for the synthetic scene image, can only occur at the specific locations where spots exist. This simplifies the beam management, e.g. beam routing and beam modulation, since beam crossing outside the spots will not generate any visible light so can be permitted.

[0252] Figure 23 is a graph plotting the dependency of QPM grating period, P, with angle, 0, the angle being 90 minus the half angle of the reflection from eye axis normal. This example is calculated for the nonlinear mixing of two 1064 nm beams to produce green light at 532 nm. It can be seen that if the traversal of the display panel by the input light beams is by shallow angle reflection (right end of graph) then the required period of the QPM grating becomes quite small which is likely to make fabrication more difficult and less accurate. For example, at 60 degrees (i.e. 30 degrees) the required periodicity is in the UV at about 400 nm and at 70 degrees is about 300 nm. At angles close to the critical angle for total internal reflection, i.e. about 40 degrees for a typical glass-air interface, the grating period is about 800 nm. If the lenses are provided with inner and outer infrared reflective material layers, then the input beams traverse the lenses by mirror reflection enabling larger angle reflections to be used. For example, at a 2 micrometre grating periodicity, the angle is about 22 degrees. On the other hand, each reflection is a cause of loss, e.g. by scattering, so the best range of angles to choose from is where there is a good compromise between ease (and accuracy) of fabrication and having a shallow enough reflection to keep the total number of reflections needed for the input beams of each beam pair to cross relatively small.

[0253] Figure 24 is a schematic drawing of a structure for introducing the first and second input beams 70, 76 into a spectacle display lens. The first and second input beams 70, 76 are introduced in the spectacle arm or temple side 264 of the lens into respective distal and proximal lens layers in a dual layer lens structure. Each input beam 70, 76 traverses its lens layer. At the bridge side 266 of the lens, the first input beam 70 in the distal layer is reflected back by mirror surfaces 260 (two mirrors at 90 degrees to each other) into the proximal layer so that it counter-propagates with the second input beam 76 to arrive at the desired region 110 for the beams to overlap within a NLO material layer. There is an additional mirror layer 262 within the display element, which is reflective at the infrared wavelengths of the first and second input beams 70, 76. With this design, the first and second input beams 70, 76 are thus brought as desired to cross in the region 110 while being introduced into the same end of the lens, which may conveniently be the temple side as illustrated. It should be noted that other methods could be used to introduce the first and second input beams, including routing around the perimeter of the lens using a suitable light guide (see Figure 54 below). Equally it is not necessary that two separate layers be used (as in the is figure, as it is also possible with a single layer to retroreflect the first input beam 70. Figure 25 shows a time-slicing sequence covering both RGB and ROC time multiplexing to generate a single example frame of a synthetic scene image. To form an image frame, a series of time slices are provided over time, t, each time slice being specific to one primary colour - e.g. red, green, blue (RGB) - and one radius of curvature (ROC) - e.g. 0.25 metres. A subframe of three time slices produces a synthetic colour image at one ROC. To generate a colour synthetic scene image containing objects at multiple distances in the scene, multiple such subframes are required, one for each ROC. In a full frame, synthetic subimages are formed at multiple different perceived depths in the synthetic scene. By sequentially displaying the relevant display fields, it is therefore possible to build up a convincing white light effect and render any colour, as in conventional displays such as projectors based on RGB. The example radii of curvature shown are 0.25 m, 0.5 m, 1.0 m, 2.0 m and 4.0 m. In this example they are shown in a monotonically increasing sequence which will then repeat. However, it will be understood that any given frame may use an arbitrary sequence of time slices. Indeed, sequences other than that shown may be more efficient to generate by the hardware and / or generate images that are perceived to have higher quality by human vision. The five example ROC values shown has the smallest ROC value at 0.25 metres, which is approximately equal to, or just below, the near point of normal vision. This value enables very near, and even slightly too near objects to be displayed in focus. The largest distance in this example is 4 metres, which is effectively the same as large distance (or infinity) for the normal eye in monocular vision. It will be understood that a smaller or greater number of distances may be chosen, and also that it is possible to adaptively control distance and frame. Moreover, the number of ROCs per frame may be varied according to the synthetic scene content, e.g. may be reduced or increased as certain objects leave or enter the synthetic scene. As previously discussed, the ROC value is set using the nonlinear mixing process and varying the crossing angles of the first and second input beams.

[0254] Figure 26 shows a schematic cross-sectional view of a spectacle-format system comprising the standard components of a spectacle frame 26 accommodating left and right eye spectacle lenses 22, 24 separated by a nose bridge 28 and having left and right spectacle arms (temples) 32, 34. In this embodiment, left and right inwardly facing cameras 54, 56 are mounted on or adjacent the nose bridge 28 for capturing images of the left and right eyes 162, which are shown in cross-section through the eyeballs. The inwardly facing cameras 52, 54 are illustrated located at the nose side of each lens. However, it will be understood alternative placements could also be used, e.g. at the temple side of each lens. The function of the inwardly facing cameras 54, 56 is to track the direction of gaze of the person wearing the AR spectacles. The provision of inwardly facing cameras for eye tracking allows for a number of advantageous features and functionalities to be provided. Firstly, the position of the pupil (and hence the gaze) allows the system to illuminate the appropriate region of the display so that the viewer sees the image in the correct place. By means of the external facing camera and the information from LiDAR or other pre-determined scene information, plus location and orientation (from the compass, or accelerometers) the appropriate synthetic scene information can be displayed. Secondly, the eye position can be used to create a high-resolution image which will be received by the foveal region of the retina, which is the part of the eye with highest resolution. Thus, a limited number of spatial pixels on a modulator or emitter array can be used to create a high-resolution image, and then, when the eye gaze shifts, the input beams can be steered to a different region of the spectacle display lens, and in that new position generate a high-resolution image. It should be noted that in terms of creating a compelling synthetic scene it may also be helpful to display information (with lower resolution but the correct colours and intensity) into the more peripheral parts of the visual field (by appropriate beam steering into the other regions of the lens). A third feature of eye tracking is that it can be used to dim or extinguish the synthetic scene as the eye shifts gaze (over larger angles) so that unwanted ghost images are not perceived by the viewer. A fourth advantage of eye tracking by means of the inwardly facing cameras 54 and 56 is that they can ensure that all generated light is directed towards the pupil of the eye, both increasing electrical to visual efficiency for the display, but also to avoid visible light being directed onto other parts of the viewers eye (or elsewhere on their eye lids or surround skin) which can be perceived by another person looking at the wearer of the AR spectacle display system. A fifth advantage of eye tracking is that the system can adapt to compensate for different interpupillary spacing (separation between the eyes). As the eye spacing of different people can vary it is advantageous to be able to have system adapt to steer light towards the eye of the individual. This may be achieved by altering the input angles of the two beams into the NLO material so as to steer the wavefront direction of the emitted light.

[0255] Figure 27 shows a time-slicing sequence covering RGB and ROC time multiplexing to generate a single example frame of a synthetic scene image which additionally includes saccadic movement blanking periods 282. When the system is blanked, the illumination (either one or both of the input beams) are switched off or switched to low intensity so that during the blanking periods no synthetic scene image is formed. The decision on when to blank is made by a control system 280 which carries out signal processing based upon eye tracking information obtained from the inwardly facing cameras 54, 56. The purpose of the blanking is so that when the eye is swivelling no light is sent to the eye. This avoids unwanted ghost images and streaks on the visual field. This is particularly important in a pulsed system as the pulsing can cause transient images to be projected across the retina. Pulsing may be present either because one or both beam sources are pulsed, e.g. pulsed lasers, or because of a pulsing effect caused by the time slicing of the colour and radius of curvature RGB, ROC. Figure 28 shows an absorbing layer arranged in a peripheral region of a display element lens through the example of a spectacle lens 20. The spectacle lens 20 is surrounded by an absorbing material region 290 which is placed there to absorb input beams beams after they have crossed in the NLO material and reached the peripheral region, thereby stopping further propagation of the input beams by reflection from the lens edges after the input beams have served their purpose. The absorbing material may advantageously have a similar refractive index to the material of the lens (to avoid Fresnel reflection) while incorporating an absorbing function through molecular absorption, absorption by a dye, or absorption by black absorbing particles. The same design may be used for display panels of other format, e.g. large area displays for conferencing or a single panel for both eyes as may be used in a VR headset format. These designs have in common that there is a peripheral region containing a material that is absorbent to light at the first and second frequencies so that the first and second input beams are absorbed after the first and second paths have crossed each other in the NLO material when they reach the peripheral region. There are also photodiodes 60 arranged at the periphery of the lens 20, which, as previously described, can be used to check the integrity of the lens structure and provide a switch-off safety feature in the event of damage. The lens periphery also includes a non-absorbing region 292 through which the input beams 70 / 76 are injected into the lens 20, i.e. the absorbing material region does not extend fully around the periphery of the lens 20.

[0256] Figure 29 (upper part) shows a schematic of a pair of AR spectacles 10 in which the NLO material in the lenses 100 is formed in discrete regions, i.e. spots 104, across the image area, the spots being embedded in areas of material which are either not the NLO material or which is the NLO material but are areas thereof that have not been periodically poled (or otherwise quasi-phase matched to the input beams). Visible light generation is therefore confined to the spots of QPM NLO material with the remaining area between the spots being inactive, i.e. not able to generate any significant flux of visible light by nonlinear mixing of the input beams. As the eye swivels in its socket, it is only necessary to ensure that a quite small amount of synthetic scene image light reaches the eye, and given that the iris can be 4-8 mm across, and the lens surface is only around 12 mm from the eye, the NLO material can be confined to quite small area spots without the image perception degrading compared with full area coverage by the QPM NLO material. In other words, considering the overall area covered by the QPM NLO material, the fill factor of the QPM NLO material may be quite small, e.g. between 5 to 20%. For example, the spots could be circular and 0.3-0.5 mm diameter and distributed in a hexagonal close packed (HOP) grid with a grid spacing of 1-2 mm. There is an advantage in this approach of designing the lenses with a grid of spots where mixing can occur instead of a design with a continuous area of QPM NLO material over the part of the lens where the synthetic scene image is to be formed. Namely, nonlinear mixing, i.e. visible light generation for the synthetic scene image, can only occur at the specific locations where spots exist. This simplifies the beam management, e.g. beam routing and beam modulation, since beam crossing outside the spots will not generate any visible light so can be permitted.

[0257] Figure 29 (lower left part) shows a first way to realize each spot. Namely, a spot 103A is comprised of a stack of three different QPM periods, 206R, 206G and 206B respectively, designed to phase match the appropriate nonlinear mixing processes for generating three different wavelengths, e.g. red, green and blue light wavelengths respectively. In the illustration the layers are shown as being in contact (as could be achieved by electric field poling), but it will be understood that gaps of inactive material could be interspersed.

[0258] Figure 29 (lower right part) shows a second way to realize each spot. Namely, a spot 103B is realized as a superstructure grating 208, which is designed so that it can simultaneously phase match for three colours, e.g. red, green and blue. This can be fabricated by poling of the NLO material according to a single, more complex poling pattern that phase matches all three colours red, green and blue. The superstructure grating is also illustrated with its layers being tilted slightly away from the plane of the NLO material layer. The grating for each spot may be tilted in a way that varies from spot to spot over the lens area so as to direct the phase matching output beam towards the pupil - as was described further above with reference to Figure 17. The same local tilting approach may be used for the multi-stack spot 103A described above.

[0259] Figure 30 shows two schematic views of a spectacle lens 20 for AR spectacles. The upper portion shows a front view of a lens 20, while the lower portion of the figure shows a top view, i.e., a view which is a section through the lens. In the front view, the first and second input beams 70A, 70B and 76A, 76B enter the lens 20 from left and right sides thereof respectively through entry regions 292A and 292B. The first and second input beams 70A, 70B and 76A, 76B cross at two different crossing positions 110A and 110B. As the eye behind the lens swivels, the gaze will be directed to many such places and it is necessary that synthetic scene information be created that will be received by the pupil of the eye. In this example, the crossing position 110A involves the gaze directed upward, and the crossing position 110B involves the gaze directed downward. Two different arbitrary first input beam directions 70A, 76A and two different second input beam directions 70B, 76B are shown by way of example. The lower portion (top view) is a slice through the lens showing the bouncing paths. It will be appreciated the respective crossing angles in the nonlinear overlap regions 110A and 110B need to be considered in three dimensions, as is clear from considering the front and top views in combination, where the input beam injection directions shown in the front view and the traversal of the lens by bouncing are relevant. From the diagram it will be appreciated that the vector directions crossing at a given point on the lens 20 depends on the position on the lens at which the gaze is directed. The resultant wavefronts need to be directed to the viewer’s pupil, and so the phase matching diagram (which is now 3-dimensional vectorial) must be designed so that the light is appropriately directed. This can be achieved by setting the QPM direction in an appropriate orientation across the face of the lens so that phase matching in desired directions can be achieved.

[0260] It should be noted that, although the phase matching periods and QPM grating vector will be determined in manufacturing the lens, there are still degrees of freedom that can be controlled to ensure that the resulting light from the NLO process is directed to the eye. The first of these is that, at a given crossing position 110A or 110B, the bouncing angles 0 (see Figure 10) of the first and second input beams can be independently controlled. Secondly, the wavelengths of the first and / or second input beams can be altered somewhat to change the phase matching condition, although this would require that at least one of the beam sources be tuneable. As already discussed, by using the inwardly facing cameras it is possible to ensure that the light from the system is directed towards the pupil efficiently and to adapt to differing interpupillary distances. As illustrated, the spectacles can be configured so that the beams enter each lens at intermediate height positions 292A and 292B from opposing sides of each lens. The beams are then directed to cross at different positions 110A and 11 OB as required. As just discussed, to ensure light is directed efficiently from lens-to-pupil of the wearer it is required that the QPM grating vector be set locally at crossing positions 110A and 11 OB so as to ensure phase matched wavefronts (or nearly phase matched wavefronts) travel to the pupil, i.e. so that the wavefronts have appropriate wavevectors.

[0261] Figure 31 shows an example lens 300 suitable for VR goggles. The lens 300 has embedded therein an NLO material layer 100. The first and second input beams 70 and 76 enter the lens 300 from the distal side, which is possible for VR where there is no natural scene to transmit onto the eye, and cross in the NLO material layer 100 at an intersection region 110. The injection of the input beams from the distal side of the lens 300 is also facilitated by the fact that, in VR, the lens 300 can be made much thicker than is possible for AR. The thickness of the lens 300 may for example be 50 mm. The illustrated form of the lens 300 is a roof shape with first and second sloping facets whose surfaces form an approximate orthogonal angle with the first and second input beams 70, 76 at the points of injection. The other features of the system remain the same as previously discussed for end injection of the input beams with a bouncing traversal to arrive at the intersection region where the input beams cross.

[0262] Figure 32 schematically shows image generation in a NLO material layer having a nonlinear QPM grating 108 when the first input beam source is an incoherent emitter array 321 that emits multiple objects as incoherent infrared light (such as from a monochromatic microLED array), thereby acting as the first input beam. A lens 325 is arranged midway between the emitter array 320 and the QPM grating 108 one focal length, f, away from each. Three emitting regions are shown as pixels 326, 327 and 328. Each of these has the same radius of curvature. An input infrared laser wave 317 acting as the second input beam and having an intensity profile and a phase profile (corresponding to radius of curvature) is shown coming into the QPM grating 108 (from the left). The input laser wave 317 can be given three different radii of curvature 318, 319 and 320 at different times. By altering the ROC of the input laser beam from the left and by providing different intensity patterns from the microLED incoherent emitters, it is possible to create successive wavefronts that can be rapidly cycled in sequence so that the viewer perceives different objects to exist at different apparent focal positions in the visible nonlinear wave emitted from the QPM grating 108 as a result of the nonlinear mixing of the input beams.

[0263] SUMMARY OF OPTIONS FOR PHASE & AMPLITUDE MODULATION BETWEEN BEAMS

[0264] In the following, we summarize possible approaches for carrying out the amplitude and phase modulation functions, in particular in respect of whether both are applied to one of the input beams or one to each of the input beams as well as whether the beams are large area beams (or equivalent beam arrays as generated by an array emitter) or rastered pencil beams. While the following aims to provide a comprehensive overview of the options, it is not exhaustive and further variations will be readily understood, e.g. by combining elements from the following specific examples. It is also noted that the labelling of the two input beams as Beam 1 and Beam 2 is arbitrary.

[0265] Table A: Example approaches for amplitude modulation Table B: Example approaches for phase modulation

[0266] 1More complex approaches can make use of Fresnel zone plates with the advantage of being able to use an absorptive spatial modulator and exploiting the coherence of the laser like input beam Figure 33a shows a series of panels illustrating certain ones of the approaches summarized in Table A for controlling the amplitude and phase of the two input beams. The panels are labelled with references that are taken from the corresponding row of Table A. It will be understood in the following that the two beams here are labelled for convenience with 70 for the beam coming from the left to the interaction region 110, and labelled 76 for the beam coming from the right, but of course the directions of the beams may be swapped so long as the beams cross in the appropriate overlap region 110. It may also be convenient to design the system so that, for the right and left lens, the system is mirror symmetric, so as to maintain symmetry across the spectacles. For example, in both spectacle lenses one of the input beams (e.g. the first input beam carrying phase modulation) is injected from the bridge side whereas the other input beam (e.g. the second input beam carrying amplitude modulation) is injected from the temple (or arm) side.

[0267] Figure 33a at Panel Ai) shows a scheme appropriate to a raster scanned and whole beam modulated approach. It shows two beams 70 and 76 that interact in a nonlinear region 110. The first beam is emitted from a laser 18, which is driven by a laser controller unit 19. The laser controller unit provides electrical power to the laser, and by varying the current the output power can be modulated as the beams are scanned in angle to build up an image. The figure shows a discontinuity 186 to indicate that there will be other optical elements placed in the optical path (for example to scan angle, etc). In this implementation the power of the laser beam 70 is controlled by the laser controller unit 19. The second beam 76 is shown with a phase profile 80 appropriate for controlling the radius of curvature of the resultant nonlinear beam (not shown). As discussed in Table A, the intensity of the whole beam 70 can also be controlled by placing a modulator in the beam path. The modulation element can be either transmissive or reflective.

[0268] Figure 33a at Panel Aii) shows a beam created by a laser with a complex amplitude modulation imprinted across the beam. This differs from the previous example, where the intensity of the whole beam is modulated, because the intensity across the beam is varied using a spatial light modulator. The figure shows two beams 70 and 76. The first of these 70 is amplitude modulated by a spatially varying reflective amplitude modulator 120 (for example a reflective LC based spatial light modulator). The figure shows an amplitude modulator controller 121 which is connected electrically to the modulator 120. This example also shows the laser 18 and the laser controller 19. The laser controller can control the total amount of light in the beam, while the modulator controller 121 drives the modulator 120 to create a spatially varying intensity profile across the laser beam. As before, the discontinuity 186 indicates that other optical elements will be present in the beam path of 70. All of the controllers elements are controlled by a central controller which controls the image formation (not shown). The second beam 76 has a controlled phase profile 80. Figure 33a at Panel Aiii) shows a system in which the amplitude modulated light is created from a device with multiple separate emitters (such as a Micro-LED device). In this approach the first beam 70, is created from separate pixel emitter sources from the emitter array 321. The emitter array is driven and controlled by a separate controller 324 which sets the intensity of each emitting element. The light emitted from two pixel positions are shown. Note that in general the array will be a 2-dimensional array. In the diagram the light emitted from the emitter array 321 is shown to be diverging (as would be the case with a Micro-LED device) and a lens 240 is placed to collect the light and direct it into the first beam 70 direction. It will be appreciated that the first beam 70 is no longer like a laser beam, rather it is radiation emitted by the emitter pixels and directed by an optical system to the interaction region 110. The emitter array can be either incoherent between pixels such as would be the case with a Micro-LED array or could have coherence between pixels (such as would be the case if a single large laser spot were to have an amplitude modulator placed on its emitting surface). Alternatively, an array of emitting lasers (such as a VCSEL array) could be used, and again could either be coherent or incoherent between each emitter, (note that a VCSEL is a vertical cavity surface emitting laser.) Provision of a hybrid or composite emitter is described in Table A as example Aiv), which describes placing a modulator (such as a liquid crystal modulator) onto the emitting surface of a larger emitter (such as a laser or LED).

[0269] Figure 33a at Panel Av) shows a combined phase and amplitude modulator. This example is a variant of the example shown in Panel Aii. A laser 18 and laser controller 19 are shown. The light from the laser is reflected off a combined amplitude and phase modulator 123a. This is controlled by a controller 123b which imparts both amplitude and phase information onto the first beam 70. The combined amplitude and phase modulator 123a can either be a single device for example a modified LCOS device with a phase controlling layer, or it can be implemented using separate elements in proximity. Again, the device may operate in either transmission or reflection. The second beam 76 no longer has any controlled phase imparted across it because the function of altering the radius of curvature of the emitted light is now achieved by applying the phase information to the first beam 70. Thus, the phase 80 is now flat across the beam. However, this beam may still have some more complex phase function (e.g. it could be focused) but is not modulated as the image is produced to create the radius of curvature.

[0270] Figure 33b shows approaches to controlling the phase across a beam according to Table B. It shows a series of panels illustrating different approaches. The panels are labelled with references that are taken from the corresponding row of Table B. The panel schematics concentrate on illustrating the role of the second beam (labelled 76 in the figures) but it will be understood that the labelling of the first and second beams as 70 and 76 is arbitrary. Similar to the description of Figure 33a it will be recognized that it may be convenient to construct a system in which there has mirror symmetry in the left and right lens of a particular display apparatus (AR spectacles).

[0271] Figure 33b at Panel Bi) shows a pair of beam 70 and 76 the interact within a nonlinear region 110. The first beam (that coming from the left) is labelled 70 and is the one that is amplitude modulated as per Figure 33a and Table A. The second beam 76 is the beam that comes from the right-hand side. It is shown as passing through a transmissive spatial phase modulator 122 which imparts a desired spatial phase profile on the input beam 76. There is a discontinuity in scale 186 which indicates that other elements may be added to the optical path. The spatial phase modulator 122 is controlled by a phase modulator controller 123 which is used to create a phase pattern corresponding to a diverging beam so as to alter the radius of curvature of the emitted light from the NLO region 110, and thus create the appearance of differing focus to the viewer of the display device.

[0272] Figure 33b at Panel Bii) is similar to Bi) except that the second beam 76 is reflected from a phase modulator 122, which is similarly controlled by the spatial phase modulator controller 123. The phase modulator 122 could, for example, be a deformable MEMs mirrors, could be a liquid crystal (LCOS) type device, or a thermal distorted mirror element. Note that the example Biii) from Table B is not shown in figures due to the similarity to the Bii) configuration.

[0273] Figure 33b at Panel Biv) from Table B is not shown either because it involves combining amplitude and phase modulation into a single hybrid element and is the same as for example Av.

[0274] Figure 33b at Panel Bv) illustrates a diffractive intensity modulation approach to altering the radius of curvature of beam 76. It looks superficially similar to Bi except that the modulator is now a spatially controlled intensity modulator 120. In this diagram it is shown in transmission but it will be understood that the same could be achieved in reflection. The intensity modulator 120 is controlled by an intensity modulator controller 121. In the simplest configurations the radius of curvature (equivalent to phase profile 80) of input beam 76 could simply be altered by transmission through an aperture (for example a pinhole) of controllable width. By reducing the size of the aperture, the beam divergence will increase due to diffraction. Equally, a transmission pixel-controlled aperture (e.g. a liquid crystal device) can create a transmissive pattern (e.g. a pinhole) that will also increase diffraction as required. A more elegant approach would be to use a Fresnel Zone plate pattern as this would be more light efficient.

[0275] Figure 33b at Panel Bvi) shows a system based around altering the properties of the laser like beam 76 so as to control the phase 80 across the beam 76 (equivalent to radius of curvature) so as to create an output of the nonlinear interaction region 110 which creates a desired image depth for the display device. The figure shows a pair of lenses 240A and 240B which are used to create a beam expanding telescope. The two lenses are controllable (by a controller unit 241) which acts to modify the resulting optical power of the pair of lenses. This can be achieved in many different ways including a liquid lens, or by altering lens spacing by moving an element (such as in a zoom lens on a camera). It will be understood that different numbers of lenses could be used, and choice will be based on desired technical performance. Altering the size of the beam 76 (which will propagate over some distance (and other optical elements represented by 186) will result in diffractive spreading as is well known for Gaussian beam optics. Thus, the radius of curvature (phase) of the beam will be controlled.

[0276] EXAMPLE CONTROL SYSTEM INPUTS, OUTPUTS AND FUNCTIONS

[0277] Figure 34 shows a simplified schematic block diagram of a controller for a display system as described above. The controller 13 is shown having three prime inputs, namely image data of the desired synthetic scene information 11 , eye monitoring data from the inward facing cameras 54 & 56 and image data of the natural scene from the outward facing cameras 50 & 52. The controller 13 takes information from the natural scene using the external cameras 50 & 52 to determine where the synthetic scene information should be located 11. The internal facing cameras 54 & 56 are used to determine where the eye gaze is directed which in turn is used to derive where on the display element the overlapping input beams should be directed to overlap in the NLO material. The ambient light sensor 55 is used to obtain a measurement of ambient luminance. The figure also shows the primary outputs of the controller 13 which are: the laser power 19 for the radius of curvature determining beam; the OLED controller 324 to create the intensity structured beam; the phase controller 123 (which sets radius of curvature I phase across the laser beam); and outputs to control input beam manipulating optical elements for steering and focusing the input beams, e.g. the rotatable mirrors 222. A further output is provided to control the dimming layer 140 according to the ambient luminance detected by the ambient light sensor 55. It will be appreciated that the system described is simplified and in more complex configurations other sources of information will be incorporated, such as additional sensors 57 to provide accelerometer data, magnetic compass data, LiDAR data, etc.

[0278] By way of example a display system as described herein may have any combination or selection of the following inputs, controller functions and outputs.

[0279] Inputs:

[0280] Any of the following inputs and associated controller input interfaces may be provided. • WiFi or 4G / 5G connection to internet

[0281] • Bluetooth or equivalent (wired, wireless or optical)

[0282] • Accelerometers and magnetometers

[0283] • Outwardly facing cameras

[0284] • Inwardly facing (eye tracking) cameras

[0285] • Integrity sensors photodiodes

[0286] • Information to determine synthetic scene including locally derived and that received from external network

[0287] • Coordinates and geometry of local space (locally broadcast or from memory)

[0288] • Lidar 13D mapping information

[0289] • Calibration data from installation protocols

[0290] • Light scatter from eye viewed by inward camera

[0291] • Ambient light sensor to measure ambient luminance

[0292] System Controller Functions:

[0293] Any of the following functions may be subsumed in the controller.

[0294] Determination of synthetic scene from external and internal sources

[0295] Determination of local geometry and feature identity

[0296] Memory and signal processing

[0297] Current eye gaze angle

[0298] Eye separation & other ocular properties, eye dominance, blinking, etc.

[0299] Calculation of overlap angles and phase control

[0300] Intensity modulation calculation

[0301] Determining display integrity

[0302] External light levels

[0303] Controlling brightness of displayed AR content according to ambient luminance

[0304] Calculation of need to block real-scene light block

[0305] Outputs:

[0306] Any of the following outputs, which may be considered as control signals, and associated controller output interfaces may be provided.

[0307] • Positional, rotational and acceleration data to external users

[0308] • Camera outputs to external processing

[0309] • Controller for first beam & controller for second beam

[0310] • Amplitude modulation

[0311] • Phase modulation for setting radius of curvature Colour modulation

[0312] Steering optics to direct beams to desired spot

[0313] Control of photochromic real-scene light blocking by the dimming layer according to ambient luminance, e.g., through control of the dimming layer.

[0314] WAVELENGTH COMBINATIONS

[0315] For a colour display it will be necessary to provide red, green and blue light. Assuming the nonlinear process is three beam SFG, the frequencies of the two input beams must sum to the frequency of the desired colour of the product beam. There are a plethora of available laser and non-laser sources which provide almost limitless possible combinations of two frequencies that sum to red, green and blue.

[0316] One example combination is:

[0317] 1064 nm + 1550 nm = 630 nm (red)

[0318] 1064 nm + 1064 nm = 532 nm (green)

[0319] 1064 nm + 780 nm = 450 nm (blue).

[0320] This example also shows a practical approach to reduce the number of source wavelengths that are needed. In this example, only three different wavelengths (frequencies) are used, i.e. fewer than two per colour.

[0321] A second example combination is:

[0322] 1064 nm + 1560 nm = 632.5 nm (red)

[0323] 1064 nm + 1064 nm = 532 nm (green)

[0324] 1064 nm + 780 nm = 450 nm (blue).

[0325] This example shows a further approach to reduce the number of source wavelengths that are needed. In this example, only two laser sources are needed, as the 780 nm light can be created by second harmonic generation of the 1560 nm laser.

[0326] It will be appreciated that the choice of the number of lasers used will depend on the availability, efficiency, price, lifetime and physical volume of each technology. It should be noted that it is possible to use a single laser (for example at 3144 nm), which via frequency doubling would give 1572 nm and its third harmonic would give 1048 nm.

[0327] Thus, a third example combination using a single starting laser at 3144 nm is:

[0328] 1048 nm + 1572 nm = 628.8 nm (red)

[0329] 1048 nm + 1048 nm = 524 nm (green) 1048 nm + 786 nm = 449.1 nm (blue).

[0330] Another specific example would be:

[0331] 950 nm + 1800 nm = 621 nm (red)

[0332] 950 nm + 1210 nm = 532 nm (green)

[0333] 950 nm + 880 nm = 456 nm (blue).

[0334] Where the 950 nm beam could be generated by an OLED array (or other infrared light emitting diode array) and the other three beams at 1800, 1210 and 880 nm by respective laser sources. In this example, there are four different wavelengths.

[0335] A general desirable feature of the present invention is apparent from the above examples. Namely, since the colours for the display are inherently in the visible range, the input beams will be in the infrared and hence invisible for most if not all practical combinations of two wavelengths. The scatter problem with conventional microprojector displays is therefore inherently solved, since any scattered light from the source beams will be in the infrared and so cannot be seen.

[0336] The conversion efficiency in a nonlinear process depends on the products of the input beam intensities. For this reason, although CW sources could be used, it is preferable that the sources are operated in pulsed mode to produce high peak power and thus greater conversion efficiencies. Laser sources for generating the input beams may be operated in any of the following modes: CW, long-pulsed, nanosecond pulsed, picosecond pulsed or femtosecond pulsed. The choice of which sources to use and in which modes of operation will depend on various factors including the nonlinearity of the NLO material, the damage thresholds of the NLO material, the required drive electrical power, form factor, weight, cost, lifetime, etc.

[0337] Moreover, it will be understood that any particular beam source that is used will generate a beam with a certain frequency response that is typically defined in terms of a single frequency (or wavelength) - the peak frequency - and a bandwidth - as a measure of the spread of the emission above and below the peak frequency. To the extent that it is relevant in this document, we define the bandwidth of a beam source to be the full width at half maximum (FWHM). When a laser is used as the beam source, the bandwidth will be very narrow, whereas when a non-laser source is used as the beam source the bandwidth may be appreciable. Considering the visible and near infrared wavelengths discussed in the examples above, typical bandwidths are as follows. In a single-mode edge-emitting LD, the bandwidth may be as high as a few nanometres reducing to sub-pm for distributed feedback (DFB) or distributed Bragg reflector (DBR) edge-emitting LDs. For non-laser sources, the bandwidths will generally be higher. For example, an OLED may have a bandwidth of a few tens of nanometres. In this document, we usually refer to ‘the frequency’ of a beam source - meaning the peak frequency - and do not mention the bandwidth unless relevant.

[0338] OTHER POINTS AND VARIANTS

[0339] Although SFG by nonlinear wave mixing is typically associated with the need to use input beams with quite high power (e.g. mW to Watt level laser powers), such high power input beams will not generally be needed for embodiments of the present invention because of the human eye’s high sensitivity. For the human eye, optical powers at the sub-microwatt level are quite bright, so the maximum output powers of the beam sources needed by close-to- eye embodiments of the invention, such as AR spectacles or VR headsets, will remain quite modest.

[0340] Many of the conventional approaches to the optical design known from existing AR and VR display apparatus can also be used in display apparatus embodying the present invention. For example, it is well known to use a positive lens between a modulated light source (e.g. a micro-OLED emitter array) and the AR display panel to transfer the light to an emitting point on the display panel where it effectively becomes an angular spectrum. This concept is familiar in terms of Fourier optics, for example in a 4f imaging system, with a simple ‘positive’ lens placed one focal length from the emitting screen, and set such that the position the plane waves are reflected from is also one focal length away. In the simplest case, and assuming that the focal length of the positive lens matches the focal length of the eye lens (e.g. about 16 mm), a symmetric 4f imaging system is provided by the positive lens and the eye lens. The display panel is then located at the intermediate or Fourier plane between those two lenses and will thus contain the angular information. Modifications are possible in which the relay lens has a different focal length compared to the eye. Moreover, the positive lens may also be modified in a more sophisticated way, e.g. to compensate for undesired optical distortion, such as to correct chromatic and other aberrations, or to provide other effects, such as to provide different magnifications. In designing the relay lens, it should be noted that the non-collinear geometry causes the spatial distribution of image information in the product beam to be stretched compared with the corresponding image information in the first and optionally second input beams which needs to be taken account of. One simple approach is to use the Schleimpflug principle in which the lens is tilted so as to image to planes at an angle. More complex optical lenses and lens combinations can be deployed to correct for distortions and aberrations as are already commonplace in optical design theory. There are also effects due to the curvature of the display panel, and this can be corrected through a combination of determining the local angle of the QPM grating direction, and also through the selection of image information. If an AR display apparatus is convincing, then a person wearing AR spectacles can have any image placed anywhere within their visual field. For example, if the wearer looks at their wrist to see a wristwatch, they don’t actually need a watch face to be present. Rather, the watch face can be superimposed onto the watch face. Similarly, a mobile phone no longer requires a display, it can instead have an arbitrary display area on which the AR display superimposes what is on the display. The same approach can be taken for a television or personal computer, which no longer needs a screen, merely a designated area on the wall or the device for the wearer of the AR display to perceive the display to be. A keyboard can be projected too.

[0341] It will be realised that any of the approaches disclosed herein in the context of AR will also be applicable to VR (simply by blocking all natural scene light).

[0342] It is noted that when we refer to SFG of two input beams we include frequency doubling, i.e. second harmonic generation (SHG), SHG being a special case of SFG in which both input beams are of equal wavelength.

[0343] While certain features and advantages of the proposed design are specific to AR vision systems, the proposed design is also advantageous for VR vision systems, in particular VR goggles could be made with much smaller depth as a result of being able to engineer the wavefront curvature of the VR images. The ability to place different synthetic objects at different perceived depths is just as advantageous in the context of a virtual image formed in VR goggles as for AR spectacles in that it makes the VR image more convincing and also allows vergence-accommodation conflict to be avoided.

[0344] Moreover, the disclosure of this document can also be deployed to provide imaging solutions for other optically viewed systems where the eye is in close proximity to where image formation occurs, including but not limited to binoculars, monoculars, telescopes, camera viewfinders, cinema camera viewfinders, microscope eye pieces, medical imaging devices, endoscopes, magnifying elements, rifle scopes and military targeting systems.

[0345] LOW SIGNATURE

[0346] Conventional AR spectacles tend to emit significant amounts of stray visible light into the environment, which can be problematic, for example making the wearer visible.

[0347] In respect of stray light emission, the display panels according to the invention have several inherent advantages over conventional AR displays. Firstly, the product beam is highly directional and pointed towards the viewer’s pupil. Secondly the input beams are not in the visible wavelength range, whereas in conventional AR spectacles of the projection type the input beam travels over a relatively large distance from the spectacle arms where the light source module is mounted into the spectacle frame and the light beams are directed along multiple projection paths in different directions, including the forward direction, via various lens and mirror combinations. According to the invention, the visible light of the product beam is generated in the display panel itself not beforehand. Thirdly, the display panel may be operated such that the AR scene information is only generated in image regions where the viewer is looking as detected by inwardly facing cameras tracking pupil position.

[0348] The total amount of visible light generated is therefore much lower than in a conventional AR display device and the visible light that is generated is generated close-to-eye and is highly directional pointed towards the pupil. The total amount of visible stray light that is emitted out of the front of the AR display panel can therefore be kept much lower than in a conventional AR display device.

[0349] Still further, the design of the present invention allows for stray light suppression. This is because the visible light of the product beam is produced by nonlinear frequency mixing and has a well-defined wavelength extending only over a narrow wavelength band. It is therefore possible to place filters that are spectrally narrow on the front layer of the AR display panel to filter out any stray visible light that would otherwise be emitted from the front of the AR display panel. The filters may be absorbers or reflectors. Specifically, in a colour system, it is possible to block at the specific RGB wavelengths with three narrowband filters. The provision of a stray visible light suppression layer to the front of the NLO material layer can significantly reduce the unwanted light signature of the AR display panel while retaining a substantially transparent lens for natural scene light across the visible spectrum. This is achievable because the bandwidth of the filter can be made much narrower than the bandwidth of any specific colour as perceived by human vision, so the viewer’s perception of the natural scene will not be impaired.

[0350] Figure 35 shows an example layer structure for an AR display panel 88. The layer structure is the same as shown in Figure 16 apart from the provision of an additional layer 147 serving the purpose of preventing light at the (or each) product beam frequency, i.e. , the sum frequency, from being emitted in the distal direction out of the display panel. The common description of Figure 16 is omitted for brevity. Layer 147 is to be incorporated between the display panel layer in which the input beams are routed and the distal surface of the display panel. Specifically, the layer 147 is shown arranged to the front of the layer 142 and separated from the layer 130 by a further spacer layer 150E.

[0351] The layer 147 thus constitutes a notch filter centred around the sum frequency which blocks stray light scattered from the product beam, e.g. product beam light reflected from the eye, while passing all other frequencies around it. The layer 147 can either operate by absorption or reflection. It will be understood that although the paired input beams (which are at non- visible wavelengths) combine to generate a product beam 82 (having a visible wavelength) that has a wavevector directed towards the viewer (proximal direction), i.e., directed out of the front of the display panel, there will inevitably be some unwanted scatter and reflection of the product beam leading to a relatively small number of photons at the product beam frequency travelling in a direction pointed towards the back of the display panel as stray visible light. To capture these stray photons, which would otherwise be emitted out of the back (distal) face of the display panel into the air 128, either an absorbing or one option for the layer 147 is for it to be a light absorbing layer with selective light absorbance in a narrow frequency band centred at the product beam frequency. To capture these stray visible photons, which would otherwise be emitted out of the back face of the display panel into the air 128, another option for the layer 147 is for it to be a reflective layer configured to reflect light in a narrow frequency band centred at the product beam frequency. The reflective layer may comprise a multilayer stack deposited as multiple coating layers that reflect based on thin film interference effects. In the case of a colour display panel, layer 147 may be a composite of three such light absorbing or reflective layers with respective selective absorbances or reflectivity in narrow frequency bands centred at the three frequencies of the different primary colour components, i.e., the three product beams. The narrow frequency band of the absorbance of layer(s) 147 ensures that light from the natural scene passes through the display panel to the viewer without the view perceiving any impairment of the natural scene, since only a small proportion of natural scene photons of each primary colour is absorbed by layer 147.

[0352] The stray light suppression layer 147 contributes to creating a low visible signature display. The back (distal) filter layer 130 contributes to creating a low infra-red signature display. In combination, the layers 130 and 147 thus provide a low signature display in which substantially no stray light from either the input beams (in the infrared) or the product beam(s) (in the visible) escapes out of the distal side of the display panel. This is important for military use where the detectability of the user of the display panel needs to be low, i.e., low signature. The display panel can therefore remain undetectable to a hostile party using infra-red imaging, visible imaging or simply natural human vision, The display panel may be part of a wearable headset, such as a helmet visor or AR spectacles, or embedded in a contact lens. Low visible signature may also be of benefit for policing applications using AR spectacles, since, when information is being displayed to the police officer wearing the AR spectacles, persons in close proximity to the wearer will not be aware of that. A potential source of tension is therefore eliminated. Stray visible light may just simply be annoying or distracting for someone who is interacting with the wearer of the AR spectacles. Other uses can be readily appreciated, e.g., hunting or wildlife photographers who wish to remain invisible to animals or people involved in surveillance. To avoid impairment of the viewer’s perception of the natural scene, the or each notch filter layer 147 should each have a bandwidth significantly smaller than the colour band that they lie in. The spectral bandwidth of the primary colours as perceived in human vision is as approximately as follows: 130 nm for red; 70 nm for green; and 45 nm for blue. If follows that the frequency band of the visible stray light suppression layer for any of these primary colours should be small relative to the red, green or blue bands, so that only a small proportion of natural scene photons of each primary colour is absorbed as they travel through the display panel into the wearer’s eyes. For example, the ratio of the bandwidth of the notch filter to the colour band it lies in may be less than one of 1 :3, 1:4, 1:5, 1 :6, 1:7, 1:8, 1:9 or 1:10. The notch filter bandwidth may be defined using a 3dB-bandwidth. In any case, for a colour display panel with multiple notch filters, the stop bands of each notch filter do not overlap.

[0353] Stray light suppression layer 147 can be combined with other features. For example, it could be combined with the anti- refl ection coating 146. The placement of the stray light suppression layer 147 in the layer stack of the display panel may be different to that illustrated but of course needs to be placed on the distal side of the NLO material layer 100, i.e. , between the NLO material layer 100 and the distal (back) surface of the display panel 88.

[0354] HIGH BRIGHTNESS

[0355] The brightness of the AR image components can be dynamically adjusted to remain visible in all ambient light conditions without becoming too bright in low-light conditions. For example, the brightness of the ambient light can be detected with a suitable light sensor 55 integrated in the AR spectacles, and the brightness of the AR image components scaled linearly or according to some other function, to become brighter when the ambient light is brighter and darker when the ambient light is darker. For example, brightness may be controlled to maintain a certain ratio between measured ambient luminance and AR image luminance. The ambient light sensor 55 may be a photodetector arranged in a suitable position. For AR spectacles, the ambient light sensor 55 can be mounted in the spectacle frame 26, as illustrated in Figure 20. The photodetector may be made of silicon or cadmium sulphide material, for example.

[0356] An inherent advantage of the display panel of the invention is that AR image components can be generated with extremely high brightness compared to conventional AR display panels based on visible light sources that are in essence isotropic emitters. Moreover, the brightness can be varied over multiple orders of magnitude. Still further, since the product beam is highly directional, the total energy of the light that needs to be generated to create an AR image feature is orders of magnitude lower compared with a conventional visible light emitter that radiates photons over a large solid angle.

[0357] As discussed further above, operation in bright ambient light is not readily achievable with current AR display technology for the following reasons. With AR devices based on light emitting diode (LED) sources, it is difficult to transfer light from the LED (which is inherently emitted over a wide range of angles) to the lens, and then to couple this light efficiently to the viewer. This is because of the limited numerical aperture and limited etendue of the optical system and also because transparent lenses must be ‘transparent’ and this transparency is difficult to reconcile with high efficiency transfer to the viewer’s eye. One approach that has been considered to create brighter displays is to use laser sources instead of LEDs, since lasers are inherently much brighter than LEDs. By brightness in this context, we mean more photons per mode, with lasers being far brighter than any natural source of light, even the sun. However, with laser-based systems, the challenge still persists in how to make a transparent display.

[0358] The present invention solves these problems by providing an extremely high-brightness capability with a transparent display panel. Our approach uses pairs of input laser beams incident on a transparent NLO material layer where a directional product beam is created by frequency summation. The product beam can be made much brighter than conventional AR display using LED sources, the product beam of the present invention being both spectrally brighter but also brighter in terms of the light power emitted into a given solid angle. Through the use of suitable sensors, the brightness can be controlled so that the augmented reality scene information remains visible in the brightest conditions and is not overly bright in darker ambient light.

[0359] The present invention thus opens up new possibilities for using AR display panels outdoors in daylight, especially on sunny days.

[0360] A first high-brightness example applies to a pilot using a head-up-display while flying an aircraft such as a fixed-wing aircraft or a helicopter. The daytime sky above the clouds is extremely bright, especially if the pilot is looking towards the sun. Using the invention, the AR content of the head-up display is visible without the pilot needing to wear sunglasses.

[0361] A second high-brightness example would be in an automotive vehicle, e.g., a car, with a head-up display in the windscreen. AR content of the head-up display will remain visible even when driving towards the sun, especially when the sun is low in the sky at sunrise and sunset. A similar situation occurs when driving at night created by the headlights of oncoming vehicles. Brightness control in bright ambient light conditions may be further controlled as described above through provision of a dimming layer made of photochromic or electrochromic material. The overall brightness of the natural scene light can be reduced, e.g. in bright ambient light conditions, so that the brightness of the synthetic scene light does not need to be increased as much as would otherwise be necessary in order that AR scene features blend into the natural scene appropriately.

[0362] SELECTIVE BRIGHTENING AND DIMMING

[0363] Area-specific dimming in this way to attenuate bright spots in the natural scene has the effect of reducing the dynamic range in the overall image and therefore increases the quality of the image as viewed through the AR display panel.

[0364] Another way to reduce dynamic range is to increase the brightness of selected areas of the overall scene by reproducing parts of the natural scene in the synthetic scene. The same effect can be achieved by reproducing the whole of the natural scene in the synthetic scene and rendering selected areas in the synthetic scene with higher brightness. The synthetic scene is therefore used to increase the natural light level in selected areas of the scene. This approach can be used to enhance dark areas of the image to reduce the dynamic range within the image. The approach of brightening the natural scene may also be used uniformly across the whole image if desired and may for example be useful when the whole natural scene is dim.

[0365] Specifically, the controller can be configured to form the synthetic scene images to include a reproduction of the natural scene and to superpose the reproduction onto the natural scene thereby to increase perceived brightness of the natural scene as viewed through the display panel. This may also be done for only certain areas of the scene, so that the synthetic scene images reproduce one or more parts of the natural scene images and superpose them onto the natural scene thereby to enhance said at least part of the natural scene as viewed through the display panel.

[0366] Brightening the scene, either in certain areas or overall, will also tend to improve the quality of the image as perceived by the viewer for the following reasons. Brightening of the scene causes the viewer’s pupil to reduce in diameter, which is a natural advantage, since aberrations of an optical lens, specifically here the eye lens, reduce with smaller diameter lenses. This is because of the reduced range of angles, which in turn reduces aberrations such as chromatic aberration, coma, sagittal and tangential distortion, etc. Furthermore, a smaller pupil has a greater depth of field (rather like a pin-hole camera) and this reduces other natural eye defects such as astigmatism, short sight, long sight, etc. Creating a brighter scene therefore contracts the pupil which reduces the lens diameter and thereby provides a better quality image on the retina.

[0367] This selective brightening approach has advantages of selective dimming in low light conditions or other situations where the natural scene is relatively dark in some significant areas. One example would be night driving when the headlights of an oncoming vehicle are extremely bright but the rest of the scene is dim. In such situations selective dimming alone will still leave road features quite difficult to perceive well, whereas selective brightening away from the headlights will improve the perception of these other road features.

[0368] It will be understood that the selective dimming and selective brightening may advantageously be used in combination.

[0369] As well as generally enhancing the natural scene, e.g., to provide whole scene image enhancement in low light conditions, this approach of providing an enhanced image of the natural scene may also be used to highlight certain parts of the natural scene.

[0370] The controller may be configured to process the natural scene images to identify at least one element in the natural scene, and to arrange the synthetic scene or an element thereof on the display panel having regard to the at least one natural scene element.

[0371] The enhancement may include one or more of positioning, sizing, alignment, brightness, or colouring of the synthetic scene element. An example of synthetic scene elements in this context could be text labels to label natural scene elements, e.g., parts of a machine, engine or other apparatus being worked on. Another example of synthetic scene elements in this context could be driving or flying aids for a vehicle driver or aircraft pilot, such as lane assist markers of a road for a driver or crosshairs for a combat pilot. For example, in the context of driving an automotive vehicle, this may be to highlight a hazard, e.g., to show a pedestrian or cyclist more brightly. Another example would be for a pedestrian wearing AR spectacles to enhance certain parts of the natural scene to make advertisements more prominent.

[0372] Artificial colouration of an element that has been identified in the natural scene may be useful in low light conditions when the eyes of the viewer are reliant on rods for image formation, so that the natural scene appears monochrome to the viewer. Artificial colouration may also be useful in any light conditions to highlight an identified element, e.g., red shading to indicate that the identified element is a hazard, such as to mark a pedestrian or cyclist to a vehicle driver. The enhancement could be outlining at or around the perimeter of the identified element.

[0373] Generating a copy of the natural scene to incorporate partially or wholly into the synthetic scene can be achieved as follows. A synthetic scene can be generated from outwardly facing cameras. In the case of AR spectacles, these cameras may be incorporated in the spectacle frame. In the case of an automotive vehicle, the cameras incorporated in the vehicle may be used, optionally in combination with other imaging sensors built into the automotive vehicle such as LIDAR (Light Detecting and Ranging), thermal imaging cameras, or cameras external to the vehicle.

[0374] Computer processing can recognise bright features in the natural scene. For selective brightening, synthetic scene light is then generated in other areas. Optionally, the bright features can also be attenuated by selective dimming. For example, the sun can be dimmed (either in selective areas or across the whole display) while the areas of the scene away from the sun can be enhanced. Even if dimming is not used (e.g. in the example of automotive vehicle headlights at night) where it is not desirable to dim the overall AR display, the bright synthetic scene light (away from the dazzling headlight) can create the effect of increasing the visibility of the background scene thereby reducing the dazzling effect of the oncoming vehicle’s headlights.

[0375] The creation of the synthetic scene light can make use of machine learning, artificial intelligence. The AI / ML can be trained to recognise and act on bright (or saturating) elements of the natural scene, or it can be trained to recognise specific features in the natural scene to suppress or enhance in the image by selective dimming and / or selective brightening respectively.

[0376] NON-CLOSE-TO-EYE (NCTE) DISPLAYS

[0377] NOTE DISPLAYS INTRODUCTION

[0378] Displays for televisions, personal computers and many other devices are designed to be viewed at a distance. Specifically with the display positioned at a greater distance from the viewer’s eyes than the near point. The eye's near point is the closest distance which the eye can bring into focus, i.e. , accommodate. The near point is often taken to be nominally 25 cm. However, a human eye’s actual near point gradually increases with age from about 7 cm for a youth to over 50 cm for an old person. Specific examples of such displays would be laptop computer displays, televisions, mobile phone displays, smart watches, aircraft video screens, dashboard-mounted automotive displays, scientific instrument displays and medical instrument displays.

[0379] Common commercially available displays of this kind are liquid crystal displays (LCDs) and organic light emitting diode (OLED) displays, although other display types are known such as historically cathode ray tubes (CRTs), active LED modules, laser projectors, and digital micromirror devices (so-called Digital Light Processors, DLPs). In this document, such displays will be referred to as ‘non-close-to-eye’ (NCTE) displays in contradistinction to near- to-eye (NTE) displays such as augmented reality (AR) displays and virtual reality (VR) displays for headsets as well as smart contact lens displays.

[0380] One way to classify between NTE and NCTE displays is to consider whether the display distance is less than or greater than the near point distance (i.e., about 25 cm). For NCTE displays, taking the example of a television, the pixels on the LCD or OLED display panel are in effect point sources, so the wavefront radius of curvature at the viewer’s eyes is equal to the distance between the viewer and the television display panel, which will typically be several meters. For NTE displays, taking the example of AR spectacles, a point source approach will not work and instead wavefronts must be created with a radius of curvature many times greater than the distance between the wearer’s eyes to the transparent glass lenses of the AR spectacles, which will only be perhaps 10-30 mm.

[0381] NCTE displays generally operate using a light emitter. The light emission can be scatter from a matte screen or can be direct emission of light from an array of point sources (e.g. in an OLED display) or through an array of amplitude modulators operating in transmission to modulate transmission of backlit light (e.g. in a backlit LCD display). In general, a pixel of a conventional display emits a spherical wavefront of light centred at the pixel location into the half-space in front of the display. Since the radius of curvature of the wavefront emitted from the display matches the physical distance from the display, the viewer’s eyes focus on the physical distance to the display element. The viewer’s object plane, i.e. the plane in which the viewer perceives the object to be positioned, is thus coincident with the physical plane of the display panel. In other words, the viewer’s individual eye focal distances

[0382] (accommodation of monocular vision) and also their depth perception (vergence of binocular vision) are both congruent with the display position.

[0383] However, there are situations where it would be useful to have a display where the eye focus distance is different to the physical distance from the viewer to the display.

[0384] An example where it would be beneficial to provide an eye focus distance that is greater than the physical distance from the viewer to the display would be a dashboard-mounted automotive display for providing a driving aid, such as a speedometer or navigation image. The driver will be looking at the road ahead, so their eye focus will be essentially at infinity, where infinity in human vision is often taken to be any distance greater than 4 metres. When the driver glances briefly to the display, their eyes must refocus to the shorter distance to the display panel, which might typically be between 50 cm to 150 cm. This refocusing takes time (typically 0.25 to 0.50 seconds) which is effectively dead time during which the driver is neither able to view the road nor the display. An example where it would be beneficial to provide an eye focus distance that is shorter than the physical distance from the viewer to the display would be for very short-sighted individuals who are unable to focus on a display panel designed to be positioned at an eye- to-display distance of for example 50 cm or more. For example, a very short-sighted person may not be able to view the display of a wristwatch.

[0385] NCTE SUMMARY

[0386] This invention provides a display apparatus in which visible light for forming the display image is generated in an NLO material by sum frequency mixing (SFG) from a pair of infrared beams. The two infrared input beams are structured with amplitude and phase information to allow the visible light to be generated with any desired wavefront radius of curvature, thereby allowing an image to be presented to a viewer at any arbitrary object plane. The object plane may thus be chosen to be more distant, less distant or coincident with the physical plane of the display panel. In other words, the eye focus distance can be made different from the physical distance from the viewer to the display.

[0387] This invention also provides a display apparatus in which light can be generated in a nonlinear optical material with any desired wavevector, i.e. , directionality, by structuring the two infrared input beams with amplitude and phase information. This ability to control the wavevector of the visible light being generated allows different images to be presented to different viewing points through time slicing. Different images can be served to a viewer’s left eye and right eye to provide stereoscopic viewing. In other words, the perceived vergence angle can be controlled. This also allows different images to be served to different viewers as well as different images to the same viewer as the viewer moves from one viewing position to another to provide the impression of perspective and changing perspective in a 3D scene.

[0388] The ability to control jointly the wavefront radius of curvature and the wavevector of light generated in the NLO material also allows for both vergence (stereoscopic or binocular vision) and accommodation (eye focus) to be controlled and, moreover, to be controlled independently of each other. This allows the left and right eyes of a viewer to be served image information in which vergence and accommodation match or at least are kept close enough to avoid unpleasant vergence-accommodation conflict effects for the viewer.

[0389] A general requirement of any NCTE display is to allow the viewer to gaze at a display panel located within their visual field and bring any features in the visual field into sharp focus. In this respect, it is noted that focusing of the human eye (for those with perfect eyesight) allows sharp images to be formed for distant objects (at infinity) through to a nearest distance (the near point) which is typically around 25 to 30 cm from the face for a young adult. Thus, ‘normal’ vision ranges from infinity to the near point. Some people (long-sighted - presbyopic) have eye lenses that are not sufficiently powerful to focus near objects onto their retinas (or eyes that are too short to allow focus), thus their near-point is at a longer distance. Other people are short-sighted or myopic and have lenses that are too strong (or eyeballs too long) and they are unable to focus light from infinity - although their near-points will be closer than 30 cm). Another effect is age-related loss of accommodation caused by the eye lens becoming stiffer and eye muscles weaker, making it more difficult for older people to focus on closer objects.

[0390] Because our approach allows wavefronts to be generated in the nonlinear optical material with a controlled radius of curvature and wavevector as well as amplitude, it is possible to create novel types of display element. In particular, the ability to control the wavefront radius of curvature makes it possible to manufacture a display that provides images in which the eye focus distance is different to the physical distance from the viewer to the display.

[0391] Some specific use cases are now described by way of example only.

[0392] NOTE display panels for devices for older people that are designed to be positioned at an eye-to-display distance which may be shorter than the eye's near point for an older person who suffers from loss of accommodation with age (presbyopia). For example, a display panel for a smart phone can be provided which will emit light with a radius of curvature that corresponds to 100 cm or 150 cm even though the smart phone will be held at a much shorter distance of perhaps 50 cm. To create an appropriate display, it is also beneficial to deal with stereoscopic vision to avoid vergence-accommodation conflict, a ubiquitous effect known to cause nausea, headaches and other unpleasant effects when viewing images where the vergence angle and focal distance are not matched. Example devices which could benefit from this approach are laptop computers (e.g. 10 to 14 inch display diagonal), tablets, smart phones and smart watches as well as other displays in a variety of commercial and industrial settings, as well as in automotive and transport. Specific examples include automotive vehicle displays for instrumentation and navigation (which can be hard to read for older drivers), and airplane seat-back screens for passenger entertainment that may be uncomfortably close for older passengers. These are all examples where there is a benefit to having a display panel that presents an object plane at a greater distance than the physical plane of the display panel.

[0393] NOTE display panels for devices for people suffering from extreme short-sightedness. Here the ability to control wavefront curvature can be used to present an object plane at a shorter distance than the physical plane of the display panel. This provides a viewable image to people who are very short sighted and unable to focus on a display panel designed to be positioned at an eye-to-display distance which is too far away. For example, if the display to eye distance is 50 cm then the display could present an object plane that is 20 cm in front of the physical display panel, thereby to provide a reduced image to eye distance of 30 cm. The same examples may apply as above, i.e. , handheld devices such as tablets and mobile phones, displays of smart watches, and laptop computer displays.

[0394] NCTE display panels for dashboard-mounted automotive displays, as used to supply instrumentation and navigation images to the driver. The driver will be looking at the road ahead, so their eye focus will be essentially at infinity, where infinity in human vision is often taken to be any distance greater than 4 metres. With a conventional automotive instrument or navigation display, when the driver glances to the display, their eyes must refocus to the shorter distance to the display panel, which might typically be 50 cm. This refocusing takes time (typically 0.25 to 0.50 seconds) which is effectively dead time during which the driver is neither able to view the road ahead nor the display. The approach of the invention allows a dashboard-mounted automotive display to be provided in which the object plane presented to the viewer is also at infinity, so the driver can glance at the display without needing to refocus their eyes. Thus, the display allows the viewer to see with an eye focus and preferably also vergence angle corresponding to a greater physical distance than the eye-to- display distance. Another similar automotive example is that there is a trend to replace physical side mirrors (wing mirrors) and rear-view mirrors with displays fed with a camera feed. However, while physical mirrors preserve a far distant object plane, the effect of replacing physical mirrors with a conventional display panel provided with a camera feed is to bring the object plane into the near distance causing the same problems as described above in relation to instrument and navigation displays. This can be avoided by using a display according to the invention.

[0395] NCTE display panels can be provided that ensure privacy. The wavevector control available with displays according to the invention allows light to be directed specifically to a viewer, more especially each of the eyes of a viewer. This compares with a conventional display where a point source radiates light into essentially the whole half-space in front of the display panel and is viewable by anyone who is able to see the front of the display panel. For example, the authorized user of a smart phone or laptop computer can be recognized by facial recognition with a forward-facing camera and images can be served only to the eyes of the authorized user, so remain invisible to persons who are located at other viewing angles, who therefore are unable to snoop.

[0396] NCTE display panels can be provided that have much lower power consumption than conventional NCTE display panels with associated benefits for battery life and reduced energy usage. This advantage follows from the wavevector control being able to improve the ratio between the number of photons generated in the display panel to the number of photons entering the eyes compared to a conventional display based on an array of isotropically emitting point sources.

[0397] NCTE display panels can be provided that provide stereoscopic vision and depth of focus effects while avoiding the vergence-accommodation conflict opening up applications where 3D imaging is important. One such application is as a replacement for stereo- microscopes, e.g., for support of surgeons carrying out operations. Other applications of interest are in computer gaming to provide more realistic 3D experience and inspection and servicing of spatially complex apparatus such as engines or watch mechanisms.

[0398] NCTE display panels that remain viewable in bright background light, such as in daylight on a sunny day. This is a particular problem for aircraft cockpit displays, since aircraft are flown above the cloud base and sometimes towards the sun. The wavevector control available with displays according to the invention allows light to be directed specifically to a viewer, more especially each of the eyes of a viewer. This compares with a conventional display where a point source radiates light into essentially the whole half-space in front of the display panel. In a conventional display, the proportion of photons generated in the display that end up entering a viewer’s eyes is therefore tiny and decreases rapidly with viewing distance. With the invention, the display can be driven to direct light only in the direction of the viewer (or viewers). For example, cameras can be integrated in the display apparatus and used to locate viewer(s) by facial recognition and then image generation on the display can be controlled to direct light only to each viewer’s eyes. As a consequence, display apparatus according to the invention can provide images that are orders of magnitude brighter than conventional NCTE displays. Thus, displays according to the invention are suitable for use in daylight, sunlight, etc, and can be deployed for entertainment, advertising, providing safety and warning information, etc.

[0399] Generally, a NCTE display panel need not be transparent, since the viewer does not look through it like in an AR display panel, which needs to be transparent. In an NCTE display according to the invention, the display panel merely needs to be sufficiently transmissive to visible light between where the NLO material and the front of the display panel to allow the visible product beam light to travel from the intersection volume where it is generated to the front of the display panel without too much attenuation taking account of not only display brightness but also heating effects in the display panel. Since the NLO material for an NCTE display according to the invention need not be transparent, i.e., can either be transparent or translucent, there is a greater choice of NLO materials available. The same is true for the materials choice of other layers and coatings that lie between the NLO material layer and the front of the display panel. An additional consideration is also transparency to the wavelengths of the input beams, which will be in the infrared for generating a visible product beam. If the input beams are conveyed through the NLO material layer 100 to the intersection volumes via multiple total internal reflections then the optical paths of the input beams will of course need to have a high degree of transparency at the wavelengths of the input beams.

[0400] This also opens the route to other nonlinearities that create forward going light by wave mixing (e.g., through third harmonic generation, through chi-3 nonlinearities. Also surface based nonlinear generation could be used.

[0401] For larger NCTE display panels, organic QPM nonlinear materials are attractive to provide the NLO material. Moreover, it will be attractive to use short pulse lasers to increase the efficiency of the frequency mixing in the NLO material, e.g., nanosecond, picosecond or femtosecond laser pulse durations.

[0402] It is also noted that there is no requirement to use a single laser to generate each input beam. In particular, for an NCTE display panel, it may be attractive to use arrays of lasers to generate the input beams. For example, the array could be such that at each node of the array there is a plurality of lasers as required to generate the number of input beams that are needed, e.g. two, three, four, five or six per array node. In the case of an NCTE display panel, the spatial requirements are relaxed compared with a transparent AR display panel, since the whole back side of the NCTE display panel can be used to accommodate devices and components including laser arrays. For example, infra-red vertical cavity surface emitting lasers (VCSELs) could be used as laser sources for the laser arrays.

[0403] It may also be desirable to overlap multiple incoherent beams in given regions of the NLO material 108 to improve image uniformity and reduce laser speckle effects caused by transverse coherence.

[0404] Given the large size of the NLO material layer needed for NCTE displays embodying the invention, it will be attractive to use high peak power pulsed lasers as the excitation sources for creating one or both of the input beams of each input beam pair.

[0405] Two specific forms of NCTE display panel according to the invention are now summarized.

[0406] A first specific form of NCTE display panel according to the invention generates complete images in the NLO material according to the invention. Pixel-by-pixel amplitude control is provided by the input beam mixing, i.e. , amplitude and phase modulation of the input beams. Pixel-by-pixel wavevector control is also provided by the input beam mixing, i.e., amplitude and phase modulation of the input beams. Further, the focus of the viewer is directed at the plane of the display as in a conventional display panel, so there is no vergenceaccommodation conflict. Specifically, the input beam mixing sets the wavefront radius of curvature to the same value for all pixels in order to match the image plane closely to the physical plane of the display panel. The NLO material is used to create an image frame which only has a single wavefront radius of curvature, i.e., apparent depth of origin, across the display panel and which uses wavevector control, i.e., product beam directionality to direct separate image frames to each participant associated with that display panel and optionally separate image frames to the left and right eyes of each participant associated with that display. For this mode of operation, the amplitude modulation of the light field is generated within the NLO material. In this mode of operation, the phase modulation of one or both of the input beams is used to set the product beam’s wavefront radius of curvature to the same value for all pixels whereas the product beam’s amplitude and wavevector are set to different values on a pixel-by-pixel basis. A variant of this uses the same hardware as the first form of display panel but additionally the input beams are generated to provide pixel-by- pixel control of the product beam’s wavefront radius of curvature to provide the illusion of different depths for different parts of the scene, such as a more distant background and nearer participants. The degree of complexity of the processing and the drive signals is therefore increased compared to the first form of display panel. This variant drives the NLO material in a similar way as for a near-to-eye display panel as used for augmented reality glasses or smart contact lenses.

[0407] A second specific form of NOTE display panel according to the invention forms images through combined use of an NLO material and an overlaid transmissive SLM. NLO material is arranged in one plane and combined with a transmissive electrically addressed spatial light modulator, such as a pixelated LCD layer arranged in front of the plane of the NLO material to provide amplitude (i.e., intensity) modulation. The transmissive spatial light modulator may also optionally provide colour filtering, which could be relevant if the display apparatus is configured to generate all three primary colours simultaneously. The colour filters would then selectively block two of the three primary colours or selectively attenuate all three to provide the desired colour mix. Otherwise, colour images can be produced by generating the three primary colours in the NLO material by separate mixing combinations of three pairs of input beams as described in detail elsewhere in this document. The focus of the viewer is directed at the plane of the display as in a conventional display panel. Specifically, the wavefront radius of curvature is set to the same value for all pixels in order to match the image plane closely to the physical plane of the display panel. In this form of display panel, amplitude and phase modulation of the input beams is used to set the product beam’s amplitude and wavefront radius of curvature to the same value for all pixels and the product beam’s wavevector to different values on a pixel-by-pixel basis. The NLO material is deployed such that the two input beams or any given pair jointly provide directional illumination on a pixel-by-pixel basis through wavevector control. The directional illumination generated in the NLO material is then amplitude modulated and optionally also colour filtered with a transmissive spatial light modulator such as a conventional pixelated LCD display layer arranged in front of the NLO material layer. This may be considered to be a kind of modified conventional display panel as familiar from television displays in which the conventional backlight is replaced with a directional backlight formed by the NLO material layer which is drivable through adjustment of the input beam pairs to provide pixel-by-pixel directional control of the light generated in the backlight. This compares to a conventional LCD display panel, in which the visible light that is modulated by the liquid crystal is generally produced from a phosphor which has been excited with UV LEDs, this light generation part of the panel being referred to as the backlight.

[0408] The NCTE display panel adopting this hybrid NLO / SLM design may subsume full amplitude modulation to the SLM in which the NLO is driven so as to vary only wavevector and wavefront radius of curvature of the product beam on a frame-by-frame basis. Alternatively, amplitude modulation may be performed jointly by the NLO and SLM, in which case the NLO is driven to vary each of amplitude, wavevector and wavefront radius of curvature of the product beam on a frame-by-frame basis.

[0409] The NCTE display panel adopting this hybrid NLO / SLM design can be driven such that same scene may be served to both eyes of a viewer. Alternatively, different scenes may be served to the left eye and right eye of a viewer to create the illusion of 3D view, rather akin to a lenticular 3D display using the nonlinear material to create directed illumination. This hybrid NLO / SLM display panel is also capable of directing different scenes to different viewers by time slicing through an increased refresh rate of the SLM compared with a conventional display. A variant of this approach would be to subdivide the amplitude modulation between the NLO material and the SLM, e.g., so that an LCD modulator is responsible for amplitude modulation of active scene elements, e.g., participants in a video conference, and the NLO material is responsible for amplitude modulation of scene elements representing background, e.g., conference room features in a video conference.

[0410] NCTE EMBODIMENTS

[0411] Figure 36A and Figure 36B are schematic perspective views from the front and behind of a hand-held display apparatus 12, such as a smartphone or tablet, incorporating a display panel 88.

[0412] Figure 37 is a block diagram of the functional components of the computing device of Figure 36A and Figure 36B.

[0413] Referring to Figure 36A, the display apparatus 12 is shown incorporated in a hand-held device, such as a smartphone or tablet. More generally, a NCTE display apparatus according to the invention may be incorporated in any of the following devices. A mobile phone (smartphone), phablet, tablet including specialist drawing tablet, laptop computer, personal computer, smart television, media player, satellite navigation device, automotive display console, games console, kiosk computer, or point-of-sale device. Further, the device may be a user equipment as defined by one or more telecommunications standards. Still further, the device may be portable, e.g. a handheld display apparatus, or fixed. The display apparatus according to the invention may also form part of a household appliance or other appliance.

[0414] The illustrated hand-held display apparatus 12 is arranged in a housing with a front face (facing outwards from Figure 36A), a rear face and a bezel forming the edges of a substantially rectilinear object. The front face is mostly taken up with a touch screen display which combines a display panel 88 (shown displaying a scene with house, smoke and sun) with a touch sensitive area 912 (shown with the hatching). The front face also accommodates a mechanical key (or button) 917 and two touch sensor keys (or buttons) 918, 919, one either side of the mechanical key 917. The edges of the housing accommodate a mechanical rocker switch 921 for volume control and an on / off switch 922.

[0415] An inwardly facing camera 923 for capturing stills or video images is arranged on the front face near the top of the housing facing forwards and has adjacent to it a range finder 59 for measuring the distance to the face or eyes of someone holding the device and gazing at the display panel 88. A microphone 924 is arranged on the front face. A speaker (not shown) is arranged on the edge of the housing.

[0416] Referring to Figure 36B, the rear view, an outwardly facing camera 909 for capturing stills or video images is arranged near the top of the housing facing backwards. A battery 908 is accommodated within the housing and constitutes a power supply (shown with dashed lines). The power supply further includes an external power supply input socket 900 which may be used for powering the device as well as charging the battery. Alongside the power input socket 900 at the bottom of the device there is another external connector in the form of an audio jack 902 for audio output. Further external interfaces may be provided including various ports, holders and sockets for physical connections. With dotted lines we show two internal holders 904, 906 which may be for a SIM card and a memory card or further SIM card. The memory card is a kind of data storage device.

[0417] Referring to Figure 37, this shows selected functional components of the display apparatus 12 of Figures 36A and 36B. The display apparatus 12 has radio components 43, input / output (I / O) components 61, one or more controllers 13 associated with one or more processors 15 and one or more memories 48, a power supply 41 , sensor components 39 and external interfaces 45. The memory 48 is operable to store computer applications ('apps') 47 which comprise software code portions that are loadable into and executable by the processor 15. The controller(s) 13 may include a touch sensor controller and a display controller, or a combined touch and display controller.

[0418] The processor(s) may comprise separate processing units for specialist tasks such as touch sensing, display drive, video processing, speech / audio analysis and / or speech / audio synthesis. The controller(s) and associated processor(s) and memory(ies) have the task of controlling the display apparatus and executing computer programs stored in the memory(ies). The memory(ies) may store computer applications for running on the display apparatus as well as collecting data from the various I / O devices. The controller(s) typically functions to control overall operation of the display apparatus, in addition to the operations associated with the application programs. The controller(s) processes signals, data, information and the like input or output through the above-mentioned components and / or runs application programs saved in the memory, thereby processing or providing a user with appropriate information and / or functions.

[0419] The radio components 43 include a WLAN transceiver, an LTE transceiver, and a GPS module. The I / O components 61 include a display capable of displaying content and also acting as part of a graphical user interface, wherein the display may be based on a suitable technology such as liquid crystal or organic light emitting diodes, as well as a positionsensitive touch sensor area overlaid on, or formed as an integral part of, the display to serve as part of a graphical user interface in conjunction with the display with optionally other touch sensor areas or buttons (e.g. on the reverse side or edge (bezel) of the device housing. Further I / O components, as previously mentioned, are front and rear facing cameras for capturing stills or video images, a microphone for capturing audio, a speaker for outputting audio and a haptic output embedded in the touch screen to provide tactile feedback. The sensing components include a gyroscope, an accelerometer, an ambient light sensor and a temperature sensor, for example. The external interfaces may include various ports and sockets for physical connections, such as a SIM card, wired LAN connectors, memory cards, audio jack socket, USB ports and so forth.

[0420] Figure 38 shows the differing principles of operation of a conventional NOTE display panel 910 (upper drawing) and an example NOTE display panel 911 embodying the invention (lower drawing). In the conventional NOTE display 910, the radius of curvature 914 of the emitted light representing an object 920 (shown as text ABC) comes from the pixels and enters the eye 162, so the eye focus 162 must match the position of the emitting pixel, i.e. the plane of the display panel. This corresponds to a physical distance marked as L1 and the object is a real object. In the example NCTE display 911 according to the invention, the display emits a wavefront of light with a radius of curvature 916 to show the object 920 at an apparent distance L2 (926) that is different from the physical distance L1. The viewer then perceives the object to be at the apparent depth L2, i.e., the viewer’s eye will focus at the apparent distance L2, and the object is a virtual object. (In the illustration, L2>L1 but it is also possible for L2<L1.) Although the viewer sees a 1:1 mapping of points between the virtual object and their retina, the display 911 emits the wavefront that represents any given point in the virtual object over a spatially extended region of the display. In other words, multiple pixels in the display 911 are involved in forming the wavefront representing any given point of the image as formed on the retina. This differs from the conventional display where each pixel, i.e. , each point in the real object, emits light which is imaged to the retina as a single point of a real image. It is noted that the angles 914, 916 and also the pupil are shown enlarged for greater conceptual clarity and the object closer than is realistic. In the ‘real world’, the angles 914, 816 are limited by the small pupil diameter of the human eye.

[0421] INPUT BEAM DELIVERY

[0422] We now discuss in more detail input beam delivery considerations.

[0423] The basic idea of our approach is to have two beams intersect in a non-collinear configuration in a nonlinear material, and to control amplitude and phase to create a desired output that forms the image. Various combinations for amplitude and phase modulation are shown further above in Table A and Table B respectively. An important factor for image formation is whether the display panel is positioned NTE (inside the near point of the human eye - as would occur with AR spectacles) or NCTE (beyond the near point of the human eye - as for a personal computer display or television. It is important to consider how best to deliver the input beams to the NLO material with appropriately controlled amplitude and phase for image formation.

[0424] For NTE display panels, our preferred approach is to use an optical design that places the NLO material at a Fourier Plane. An example is shown in Figure 15 where the emitter array lies in the focal plane of lens 242. Another example is the 4f imaging system described further above with the display panel located at the Fourier plane. The optical field in the Fourier plane is essentially mapping spatial frequencies of the object (e.g. the light scattered from the DLP or other spatial light modulator) into spatial positions. This light of the first input beam is used to create the visible light in the display panel by sum-frequency mixing of the first input beam with the second input beam in a non-collinear geometry.

[0425] With the NLO material located at this Fourier plane and the amplitude modulation being impressed on the first input beam, if the second input beam is a plane wave, then the image is created at infinity.

[0426] As we have discussed elsewhere in this document, the wavefront curvature of the second input beam can be used to alter the wavefront curvature of the eye’s visible image, and this creates the effect of focal depth. If one thinks mathematically about the creation of a nearer- than-infinity image, it is equivalent to the formation of a Fresnel transform (akin to a nearfield diffraction pattern), and our approach of mixing allows us to create this Fresnel transform automatically using the laws of images and optics. The ability of our system to perform Fresnel transforms through the optical design is important, since Fresnel transforms are computationally intensive to perform numerically. In particular, Fast Fourier transform algorithms cannot be used to perform Fresnel transforms. The computational intensity of performing Fresnel transforms numerically is what makes it difficult to realise real-time holographic imaging using phase modulators and visible light (see Shi, L., Li, B., Kim, C. et al. Towards real-time photorealistic 3D holography with deep neural networks. Nature 591, 234-239 (2021). https: / / doi.org / 10.1038 / s41586-020-03152-0).

[0427] A particularly practical implementation of the invention from a cost and miniaturisation perspective that uses solely commercially available optical components is as follows. A first one of an input beam pair is amplitude modulated with a suitable spatial light modulator which may be a DLP (operating in reflection) or an LCOS (operating in reflection or transmission). The second one of the input beam pair provides whole-beam phase modulation with a variable focal length lens (often referred to as a varifocal lens). Varifocal lenses are, for example, available from Corning, Inc. under the registered trade mark ‘Varioptic’ and are based on modifying the interface between two liquids with an electric field with a response time of around 10 milliseconds which for real-time imaging corresponds to 100 frames per second (fps). A response time of 33 milliseconds corresponds to a frame rate of 30 fps, which is the slowest usual frame rate for video. The response time for the variable focal length lens is therefore preferably less than or equal to one of: 40, 35, 30, 25, 20, 15 and 10 milliseconds. The variable focus lens is used to set or vary the wavefront radius of curvature of the second input beam. For the avoidance of doubt, this is relevant for both NTE and NCTE display panels. Alternatively, if the amount of phase modulation needs to be controllable across the second input beam, then an LCOS operating in reflection or transmission can be placed in the path of the second input beam, which may be instead of or in addition to a variable focal length lens. We now discuss how input beam delivery can be suitably configured to control apparent size of the synthetic scene.

[0428] When the human eye looks at a scene, what is perceived as the size of an object, referred to as its apparent size, is essentially defined by the angular subtense of light from that object. In a NTE display, such as AR spectacles or contact lenses, the actual area on the display that light representing a certain scene element is emitted from has little bearing on the apparent size. Rather, apparent size follows from the range of angles of light emitted from the display that represent that scene element. In a display apparatus according to the invention in the case where one of the input beams of each pair carries all the phase information as a plane wave, the apparent size of an object will be determined by the range of angles of the input beams of each pair that are coupled into the NLO material (e.g. see Figure 10).

[0429] Consider a simple configuration with the first input beam performing the amplitude modulation, the amplitude modulator or emitter array is located one focal length (1f) from a single focusing lens, which is itself located one focal length (1f) from the NLO material. An example of this setup is shown in Figure 15. Here the range of angles that can be created is set by the size of the amplitude modulator and the focal length of the lens. Having a short focal length lens increases the range of angles involved, and therefore increases the apparent size of the object, i.e., the synthetic scene. Setting the focal length with which the amplitude-modulated input beam is focused on the NLO material is therefore an important design decision. Since the focal length of the eye is around 16 mm, this sets the overall size of the image that will be formed on the retina.

[0430] The apparent size of the synthetic scene will depend on the range of angles with which the light from the amplitude modulator converges on the NLO material. Since the NLO material is located at a Fourier plane, the size of the input beam intersection region 110 in the NLO material sets the resolution of the display. The requirement for reasonably efficient sum frequency or second harmonic generation also sets a limit to the range of angles. However, using a thinner QPM structure can reduce this limitation.

[0431] The apparent size of the synthetic scene as a whole, or synthetic scene elements within it, will therefore depend on the optical design choices and in particular on the specification of the lens or lens combination which relays the input light beam from the spatial light modulator used for amplitude modulation onto the NLO material.

[0432] The pixel size, p, of the spatial light modulator used for amplitude modulation will impose a minimum size on the relay lens. For example, for a DLP or LCOS, a typical pixel size may be 5 x 5 micrometres. The diffraction angle (sin theta) will be around Nd (1.064 micrometre 15 micrometre) which corresponds to approximately 12 degrees at a wavelength of 1064 nm, so if a focal length of 16 mm is chosen and a simple 1 f: 1 f configuration is used, then the relay lens will need a minimum diameter of around 6 mm. The size requirements on the imaging lens are thus quite modest and relatively small lenses can be used in most cases.

[0433] The phase modulation is used to define a small solid angle over which the product beam light is emitted. This directionality towards the viewer also provides an inherently very bright display. The high brightness combined with the high directionality means that the optical components used for manipulation the input beams, i.e., lenses, mirrors, filters and the like, can all be small. This is a significant advantage for miniaturisation compared with known display apparatus using DLPs or LCOS devices to amplitude modulate light from a whitelight phosphor LED or OLED array, where large optical components must be used to maintain brightness.

[0434] Figure 39A shows an example optical design according to the above with the first input beam 70 being impressed with amplitude modulation with a DLP 120 operating in reflection and the second input beam 76 providing whole-beam phase modulation with a variable focal length lens 928. The first and second input beams are directed to cross in an intersection region within a phase-matched grating of the NLO material layer 100, which in turn is a part of the display panel 88. For the second input beam 76, a 4f imaging system is used with a suitable lens pair with the same focal length, f. The lens pair is a collector lens 240a on the object side and a detector lens 240b on the image side. In a 4f imaging system, the distances between object, collector lens, detector lens, and image are in the ratio f:2f:f. A discontinuity of scale 186 is shown to convey the fact that there will be a greater distance to the eyeball of the viewer 162. A pair of input light beams 70, 76 is combined by sum frequency mixing (or SHG if both input beams 70, 76 are at the same wavelength) in the NLO material 100.

[0435] Input beam 70 is spatially modulated with the desired image as a pixel pattern using a spatial modulator 120 of the type that operates in reflection, such as a DLP or LCOS device, as schematically illustrated. The light reflected from the spatial modulator 120 (or passing through it in cases where the spatial modulator is of the type that operates in transmission, such as a spatial light modulator, SLM) is collected by a lens 240c and an image of the pixels is formed (as shown in the figure). The object plane is chosen to be formed not in the NLO material region 100 but rather at an additional distance away having an offset value 8 as shown also by the scale bars at the bottom of the drawing. The apparent depth to the eye L’ is equal to the sum or difference between the physical distance between the eye and the display, L, and the additional offset distance, 8, i.e. , L’=L ± 8, where the illustration is with this being the sum but in other configurations the difference could be used. When L’=L + 8 as illustrated, the spatial modulator 120 is imaged beyond the front of the physical display, so the viewer needs to gaze ‘deeper’ than the physical display 88. Alternatively, when L’=L - 8, the spatial modulator 120 is imaged before the physical display, so the viewer needs to focus ‘in front’ of the physical display 88. In either case, the viewer needs to focus their eye at the L’ distance to see a sharp image of the desired scene as set by the pixels on the spatial modulator 120. The spatial modulator 120 is driven with an appropriate video feed, such as a video feed according to the HDMI protocol. The DLP or other spatial modulator can simply be driven as conventionally with standard pixel encoding. No additional calculations are required to supply the image information to the display compared with a conventional display. This allows the display to be operated with standard computing hardware and avoids the need to calculate complex transfer functions such as Fresnel transforms which are computationally involved.

[0436] Input beam 76 is shown as having a controllable phase front 928. This phase front can be created by using various different elements as discussed elsewhere in this document. One convenient approach is to use a dynamically variable focal length lens. Variable focal length lenses are, for example, available from Corning, Inc. under the registered trade mark ‘Varioptic’ and are based on modifying the interface between two liquids with an electric field with a response time of around 10 milliseconds which for real-time imaging corresponds to 100 frames per second. In this diagram two lens are shown 240a and 240b which can be chosen so as to image the phase element 928 into the nonlinear material 100. In this way the intensity of the beam is unaffected as the phase is altered. Although the phase is illustrated schematically as a curved spatial feature, it could be a plane wavefront. A plane wavefront results in an image being formed with output curvature dependent on the imaging parameters of the beam 70 (as described elsewhere in this document). However, by controlling the phase element 928 further refinement and optical adjustment can be made.

[0437] In the diagram, the elements that transfer the radius of curvature of the input beam 76 are included (specifically 928, 240a and 240b). This option provides another way of modifying the radius of curvature of the output wavefronts of the visible product beams that form the image. In particular, manipulating the input beam 76 may be an attractive option for adding (or removing) additional wavefront curvature to allow for sight correction of the viewer, such as correction of astigmatism and other refractive errors of the viewer’s sight. Therefore, input beam 76 is used for vision correction while input beam 70 is used to carry the wavefront curvature data needed for digital image information. The input beam 76 can be manipulated to match an individual optical prescription of the viewer (for each of the left eye and right eye) to make a personalized wavefront to correct for vision defects. The individual optical prescription could for example be looked up after the viewer is recognized with facial recognition. The system may also be able to create its own optical prescription of each individual viewer based on guiding the viewer through an eye test process. Finally, it is noted that the imaging condition has aspects of tilt, which can be compensated for using the Schleimpflug approach.

[0438] Figure 39B shows how the magnification of the system, i.e. , the apparent size of synthetic scene objects, can be controlled. The optical set-up is the same as shown in Figure 39A and the apparent depth is set to coincide with the NLO material layer as shown in Figure 40A. In other words, the Fourier plane is located to lie in the NLO material layer. The relay lens 240 is in a 1f:1f arrangement with the DLP 120 and the NLO material layer 100. (It is noted that the choice of focal length for the first input beam is made independently from that of the second input beam.) Ray traces from each of two arbitrary points on the amplitude modulator are shown, i.e. two example pixels on the DLP chip. These are labelled P1 and P2. Light of the first input beam 70 as reflected from each of the different DLP pixels is collected by the relay lens 240 and sent into the NLO material 100, where light from each DLP pixel will mix with light from the second input beam 76 to create a sum frequency visible light beam, the product beam, that goes to the eye 162 to form the synthetic scene image. The diagram is not to scale and, moreover, a discontinuity in the distance scale is labelled 186.

[0439] An inset shows the wavefronts of the input beam light from the two example pixels P1 and P2. It is apparent how the light from the different pixels P1 and P2 will travel through the NLO material in different directions. The light rays of the first input beam are reflected at all pixels of the DLP so as to emerge from the DLP parallel to each other. This is because the first input beam as incident on the DLP is essentially a plane wave and the DLP pixels all orient at the same angle. However, since the rays from the different pixels are incident on the relay lens at different radial distances from its optical axis, they are focused onto the NLO material layer at a range of angles, i.e., with a range of incoming directions, with the angle mapping to positions in the Fourier plane. This means that light of the first input beam from all the DLP pixels travels to the same point in the NLO material. However, as will be seen in the diagram, the corresponding wavefronts (which are labelled in the enlarged section as 995 and 996 have different directions. Specifically, there is a difference in the crossing angles (p which depends upon the focal length of the relay lens 240.

[0440] If the separation between pixels P1 and P2 is t, then the equation for the angle (p becomes f . (p = t, so (p = t / f. Assuming the eye is focused at infinity, the viewer will then see the light from the two pixels as having an angular separation of t / f.

[0441] It will be clear that choosing a small value for the focal length of the relay lens 240 leads to a larger angular separation (p between any two points on the spatial light modulator, leading to the apparent size of the object becoming larger. Choosing a larger size DLP chip has the same effect. However, for miniaturisation, it will usually be more preferable to choose a smaller DLP chip and set the apparent size of the synthetic scene by an appropriate specification of the relay lens.

[0442] The same considerations as discussed above by way of example with reference to a DLP also applies to other types of spatial light modulator. Furthermore, in practice, the single relay lens 240 may well be exchanged for a suitable lens combination, as for example may be desirable for cancelling out chromatic aberrations. Figure 39C shows, in more detail than in Figure 39A and Figure 39B, a 4f imaging system to transfer a desired phase pattern into the NLO material. This scheme is attractive for transferring the desired radius of curvature phase structure into the NLO material. In the figure an input laser beam 76 with wavefront 80 which has a given intensity profile 930 is imaged into the NLO material 100 to yield an identically sized intensity profile 930. It is noted that the profile is inverted (but as this is illustrated by way of example as a Gaussian profile it appears the same before and after the imaging system). By placing the phase modulating element 122 (which may be a phase modulator acting in transmission or reflection, or a varifocal lens) at a conjugate image point in the 4f imaging system from the NLO material, the image of the light at the varifocal lens will be imaged across regardless of the phase added by the lens. Strictly, of course, this is only approximately true for finite thickness lenses, and only true for small angles, but in our approach the numerical aperture (NA) can be limited to small angles, and thus thin lenses can be used. The phase (in this case a radially varying phase is indicated as 928) is imparted on the beam 76 (which is initially shown to have an approximately plane wave wavefront 80). The phase is imaged across to the NLO material 100 and will retain the curvature. Thus, the use of the 4f imaging system allows us to transfer the intensity unaltered as the phase is altered, allowing us to achieve our desired optical function of altering the radius of curvature of the generated wave in the NLO material (as extensively discussed elsewhere in this document).

[0443] Figure 39D illustrates the same 4f imaging system as Figure 39C but with a linear phase and showing ray tracing to show how a linear phase 80 is transferred through the imaging system. The ray directions are also indicated on the inset phase representations. Figure 39D shows how rays of light associated with the linear phase are refracted by the two paired lenses 240a, 240b in the f:2f:f design. The output is still a linear phase but now reversed across the optical axis. Again, placing the phase modulator (in this case a transmissive phase modulator) as the object in the imaging system preserves the intensity profile 930 of the beam.

[0444] It is noted that the sizes of the spot formed in the NLO material (which determines the size of the intersection region together with its paired input beam) can be altered by altering the imaging lens focal lengths while maintaining the imaging formation condition. However, it is advantageous to use a two-lens imaging system as in the 4f design, since that maintains plane waves at the output, whereas a single lens in a 2f:2f design would not have this desirable property.

[0445] While simple singlet lenses are illustrated, it will be understood that more complex lenses can be used to correct for aberrations, particularly chromatic aberrations amongst others. Moreover, the same optical design approach can also be implemented with focusing mirrors instead of lenses. Still further, the phase modulator need not be transmissive as illustrated but may instead be reflective (e.g. LCOS).

[0446] Figure 40A, Figure 40B, Figure 40C and Figure 40D are schematic drawings of different object placements for the synthetic scene to vary the apparent depth in relation to the physical position of the display panel. It is noted that the set-up of Figure 39 can be used to switch between these different object placements through adjustment of the variable focal length lens.

[0447] Figure 40A shows a virtual object formed at infinity, as may be applicable for a dashboardmounted automotive NCTE display example or synthetic rear-view or side-view mirror example discussed further above. Input light beam 70 carries the digital image information and input beam 76 is a plane wave. The two input beams 70, 76 mix in the NLO material of the display to create product beams that collectively form a virtual object at infinity. The dotted lines of input beam 70 indicate that light specularly reflected from the spatial light modulator are focused into the plane of the display, i.e., the plane of the NLO material layer. The lens 240a is chosen so that it is placed one focal length from the NLO material 100 so that the NLO material 100 is placed at a Fourier plane of the lens (e.g., 1 f: 1 f set up), which in turn maps input angles to positions in the light from beam 70. When mixed with input beam 76 (assuming input beam 76 is plane wave) product beams will generate an image that the eye perceives to be localised at infinity, so the viewer will need to focus at infinity.

[0448] Figure 40B shows an object formed in the plane of the display panel, as may be suitable for a video conferencing or television display example where it is acceptable for the eye focus distance and the physical distance from the viewer to the display to be the same. As discussed above, one condition for this to be acceptable is that the display will be positioned at a greater distance than the viewer’s (or any potential viewer’s) near point, e.g. greater than 1.5 metres. This is referred to as localised display. With localised display, the intensity modulator is focused into the NLO material, so as to create the impression for the viewer that the image is being generated in the plane of the display panel, as in a conventional display panel. The emission pattern created by the product beams collectively is controllable by the second input beam 76 (to set additional radius of curvature values; setting the output direction of the product beams) and setting the effective brightness (numerical aperture) of the product beams. The pixels from the spatial modulator 120 are imaged directly into the NLO material 100 by lens 240c. The optical system is shown to be 2f:2f in terms of distances, which corresponds to the classical imaging condition (1 / f = 1 / u + 1 / v) where u and v are the object and image distance respectively, and f is the focal length of a simple lens. Specifically, distance u is the distance from the spatial modulator 120 to the lens 240c and distance v is the distance from the lens 240c to the NLO material. The dotted lines shown emerging from a single pixel of the spatial light modulator are to show imaging to points in the plane of the NLO material. When the input beam 70 is mixed with light from input beam 76 in the NLO material 100 to create the product beams this will generate pixels localised in the NLO material 100. Thus, the viewer will need to focus on the display itself to see a sharp image.

[0449] Figure 40C and Figure 40D shows a virtual object formed at a finite offset distance 8 from the plane of the display panel. We refer to the set-up option of Figure 40C with a positive value of the offset 8 as a virtual depth display and of Figure 40D with a negative value of the offset 8 as a closer display. The positive offset set-up option is applicable to NOTE display panels for devices for older people suffering from age-related loss of accommodation (using a positive value of offset 8) and the negative offset set-up option is applicable to people suffering from extreme short-sightedness (using a negative value of offset 8). The virtual depth display approach places the image plane at some intermediate plane which corresponds to ‘virtual’ pixels that appear to emerge from this virtual image plane. If mixed with a plane wave input beam 76, the output waveform formed by the product beams will appear to have a different radius of curvature than the real physical pixels of the spatial modulator. When the input beam 70 is mixed with the light of input beam 76 in the NLO material 100, the viewer must now look beyond (or in front of) the display element to see a sharp image. The image equation now makes the image distance, v, the sum of the physical distance from lens 240c (or 240d) to the NLO material 100 plus (or minus) the desired magnitude of the offset.

[0450] To simplify the above-described representations, details of the delivery optics for input beam 76 are not shown. However, it is noted the set-up shown in Figure 39A can be used for any of the options shown in Figure 40A, Figure 40B, Figure 40C and Figure 40D.

[0451] Importantly, the differences between the different options shown in each of Figure 40A, Figure 40B, Figure 40C and Figure 40D for selecting the apparent depth in these examples follow solely from optics of the input beam 70.

[0452] It is further noted that optical elements in the path of the input beam 70 may also provide dynamically variable focusing (e.g., using a vario-optic lens for lenses 240a, 240b, 240c, 240d or by adding a phase modulator). By providing dynamically variable focusing elements in the path of the input beam 70, dynamic adjustment of the apparent offset depth is possible, e.g., to move dynamically between the different options described above. For example, the display may be operated by default in localised display and then only adjusted to virtual depth display with positive offset when it is detected that the viewer suffers from age-related accommodation loss with a near-point greater than the display-to-eye distance. The adjustment is then preferably only by the minimum amount needed based on the optical prescription of the viewer, i.e. so that L + 8 is equal to the viewer’s near-point. This is desirable since it is easier for a viewer to adjust their vision to smaller offsets than larger ones. These adjustments may of course also be made responsive to changes in the display- to-eye distance as relative movement between the display and the viewer takes place, as may occur with a hand-held device or when a viewer moves their chair when sitting in front of a desk-mounted display of a personal computer or when a viewer moves within a conference room while watching a wall-mounted display panel.

[0453] Provision of dynamically variable focusing elements in the path of the input beam 70 also allows 3D effects to be incorporated in the image, for example by varying the wavefront radius of curvature within each image to create the illusion of depth using the techniques described elsewhere in this document for AR displays and other NTE displays.

[0454] In addition, the display apparatus may incorporate or be in data communication with cameras, LIDAR and other image information acquisition devices that allow the distance from the viewer’s face (or eyes) to the display to be tracked, so that the image presented to the viewer can be dynamically and automatically adjusted for maximum viewing comfort and visual acuity. Furthermore, by tracking the viewer’s pupil the system may adjust its operation to maximise the viewing comfort and visual acuity, for example to make personalized adjustments based on the viewer’s optical prescription.

[0455] While single element lenses are shown it will be understood that more complex lens systems can be deployed to reduce unwanted artifacts from aberrations such as spherical aberration, chromatic aberration, coma, field curvature, distortion, astigmatism. Creating compound lens elements made from materials with different refractive indices and dispersion are all approaches widely used in lens design and can be applied to this situation. Additional features of the lens system can include automatic adjustment of focus according to position within the display, active alignment of components, additional filters and absorbers to reduce stray light, the addition of antireflective coatings on lenses, etc such are already well-known within optical systems such as single-lens reflex cameras, binoculars, telescopes, microscopes and so forth.

[0456] HYBRID DISPLAY PANEL WITH SPATIAL LIGHT MODULATOR

[0457] Figure 41 is a schematic drawing of a display panel according to an embodiment of the invention comprising an NLO material layer with QPM grating 108 for visible light generation (i.e., the product beams generated with defined wavevector to define their direction) and an overlaid transmissive SLM 938 for image formation through amplitude modulation of the visible light generated in the NLO material 100. The SLM 938 is pixelated and electrically addressable, so it can be driven using a conventional drive signal delivered by a suitable controller 324. Since the hybrid NLO / SLM display panel still permits full directional control of the product beams through control of the angles 01 , 02 of the input beams 76, 70, it is possible to provide 3D effects to a viewer, which may be without stereoscopic imaging by showing perspective or with stereoscopic imaging by serving different images to the left and right eye of the viewer.

[0458] The NLO layer thus provides a directional backlight that is modulated by the SLM, which can be a conventional liquid crystal overlay, to create a pixel display but with directionally controlled light emission. In the hybrid NLO / SLM display panel, the visible light is generated by NLO mixing between pairs of non-collinear infrared input beams, which generates the visible light only in defined directions (wavevector control). It is thus possible to generate visible light not only in terms of beams with a desired optical axis but also with control of a beam’s numerical aperture and brightness. As described elsewhere in this document, wavevector control allows images to be served directly to specific viewers and separately to the left and right eyes of any given viewer. Among other things this increases the optical efficiency of the display panel by several orders of magnitude compared with conventional display panels that emit light isotropically over essentially the whole half-space in front of the display panel. To serve images directed to a viewer, suitable forward- facing cameras are need to track viewer position. A simpler approach for display panels embodying the invention, which still provides improved optical efficiency compared with a conventional display panel, is to use the wavevector control to emit light only over a controlled solid angle, i.e. with a specific numerical aperture, so that the images are only viewable to viewers located in some range of angles in front of the display panel. For example, for a television at home, viewers will generally always be seated on a sofa or chairs in front of the display within a limited solid angle, so a limited emission angle is acceptable, especially if the emission angle, i.e., the numerical aperture, is adjustable by the user.

[0459] The hybrid NLO / SLM display panel provides an extra option for generating colour images. The NLO / SLM display panel can still use the same approach as described elsewhere when there is no SLM overlay, namely providing three pairs of input beams with three different sum frequencies to provide the three primary colours. However, optionally the NLO / SLM display panel can use the SLM also to provide colour filtering as described further above.

[0460] It is further noted that some of the amplitude modulation may be performed in the NLO material. For example, for deep black pixels, it may be beneficial to reduce the brightness of the light generated in the NLO material. Similarly, for very bright pixels, the NLO material can generate with increased brightness. Images can thus be generated that have a larger dynamic range than is possible from using the SLM alone for amplitude modulation.

[0461] Figure 42 illustrates use of a hybrid NLO / SLM display panel as shown in Figure 41 operated to serve different display images to the left and right eye 951 L, 951 R of a viewer 950 and thereby create a 3D effect. The lower and upper parts of the drawing show product beam light being directed respectively solely to the left eye and the right eye of the viewer as beam 954 to the left eye 951 L and beam 952 to the right eye 951 R. The product beam directions 952 and 954 are determined by the angles, spatial phase and amplitude of the input beams 70 and 76. The angles (or equivalently phase functions) of the input beams are indicated as 01, 02, 03, 04. By alternating left and right images in rapid succession (e.g. using at least 60Hz to provide 30 fps to each eye, preferably at least 100Hz), the viewer will be able to see a stereoscopic effect. Increasing the display brightness can also be exploited to cause the viewer’s irises to contract and thereby increase the depth of field. The locations of the left and eyes can be determined by forward-facing cameras embedded in the display panel bezel, for example. The hybrid NLO / SLM display panel can also be operated to direct the emitted light so that images are only viewable to a targeted viewer to ensure privacy and avoid snooping.

[0462] Figures 43A and 43B illustrate how the hybrid NLO / SLM display panel as shown in Figure 41 provides higher brightness compared to a conventional display panel. The lower drawing, Figure 43B, shows the hybrid NLO / SLM display panel of Figure 41. The upper drawing, Figure 43A, shows a conventional LCD display panel comprising an LCD modulator layer 108 and a phosphorescent backlight layer 958. The control unit 324 determines the transmission of pixels in the LCD layer so as to create the desired pixel display pattern. As schematically shown in Figure 43A, in the conventional LCD display panel, the backlight emitting phosphor 958 emits light in all directions (over 4 Pi steradians) and nearly all of this light is wasted in the sense that only a tiny proportion of the backlight photons ultimately enter the eyes of the viewer. (Some commercial LCD displays have a reflective back layer behind the phosphor layer 958 to recover some of the backlight that is emitted backwards and direct it back to pass through the LCD layer.) The viewer 950 is shown located a distance R (960) away from the display. Figure 43B shows a display in which the backlight function is replaced by our nonlinear approach 108 with directional input beam pairs 70, 76 which are - jointly or separately - modulated in amplitude and phase. Here the product beams 962 forming the image are highly directional with a small solid angle, so that a much higher proportion of the visible photons generated in the NLO material ultimately enter the eyes of the viewer compared with the conventional case. This is what makes the hybrid NLO / SLM display panel able to provide higher brightness compared to a conventional display. Considering the optical efficiency of a conventional backlit LCD television, take the example of a viewer watching the television screen at a distance of 3 metres. The eye pupil has an area of perhaps 7.1 mmA2 (TT.rA2 where r = 1.5 mm) whereas the light from the television screen is essentially emitted isotropically, i.e., over 4TT steradians, over a sphere of area 4TT.rA2. A sphere of three metres radius has a surface area of about 113 square metres. The ratio between these two areas sets the effective photon collection efficiency between photons generated in the backlight phosphor of the LED and the coupling to the human eye. The ratio in this example is 16 million to 1, i.e., only about 1 photon in 16 million arrives at the eye. This ratio explains why conventional television screens based on phosphorescent backlights are inefficient in using optical power. In contrast, in display panels according to the invention, essentially all the product beam photons can be directed towards the viewer over a narrow angle, yielding improvements in photon coupling efficiency of many orders or magnitude compared to the conventional case, which offers the opportunity to improve display brightness, reduce power consumption (increase battery life), and increase display lifetime.

[0463] While this difference to a conventional display is illustrated by comparison with a hybrid NLO / SLM display panel according to the invention, the same is true of other NLO display panels according to the invention, i.e., ones that have no overlaid SLM.

[0464] Figure 43C shows an example display stack of a an NLO / SLM display panel using an LCD as the SLM. The display stack comprises an NLO material layer 100 according to the invention. Below the NLO material layer 100 there is an optional OLED layer, similar to what is provided in conventional displays, to provide backlighting of a selected brightness and colour. Above the NLO material layer 100 there are bottom polariser and a glass layer to act as a substrate for the thin film transistors (TFTs). The TFTs together with their addressing and other electrodes, referred to as metal, are arranged in top of the TFT glass. It will be understood that reference to metal includes any suitable metallically conductive material, it often being the case that ITO is used (i.e. a non-metal), since it is transparent and therefore suitable for display applications. The next layer is the liquid crystal layer acting as the spatial light modulator followed by a colour filter layer. It will be appreciated that suitable spacers will be provided to maintain a design thickness for the liquid crystal layer. Next there is a black matrix layer for contrast enhancement, colour filter glass, and finally a top polariser. It will be appreciated that additional layers, such as adhesive, coatings and so forth may also be included as desired. Moreover, some of the recited layers could be omitted, for example the black matrix. A simpler structure would also result if the display was monochrome. A touch sensor layer stack may optionally also be arranged on top of the display stack. The touch sensor layers commence with a substrate, made of a material such as a glass material (e.g. an alkali aluminosilicate glass) or a plastics material (e.g. PET) for example, on which X electrodes are deposited. The X electrodes may be embedded in a dielectric matrix, such as an adhesive layer. On top of the X electrodes there is a dielectric and then Y electrodes. There then follows an optional decoration layer followed by a cover panel. The cover panel may be a glass or a plastics material. Moreover, the cover panel may be coated, e.g. with an anti-scratch material for increased hardness and / or wear resistance. It will be appreciated that one or more adhesive layers (e.g., OCA) may be used to bind layers of the display and touch sensor stack together, but these are not illustrated.

[0465] Figure 44 is three schematic illustrations relating to a person 970 holding and viewing a hand-held device 972 with a display, such as a tablet or smart phone, on which a five- pointed star 974 is being shown. The display is being operated in a virtual depth display mode as shown in Figure 40C in which the apparent depth of the display (i.e. the virtual object distance) is different to the real eye-to-display distance with a positive value of offset 8. The star 974 is an example of a virtual object. As shown in the diagram, the virtual depth display mode creates a diplopia (or double vision) challenge. In effect the viewer needs to gaze beyond the plane of the display panel in order to see the virtual object in focus.

[0466] The viewer 950 has left and right eyes 951 L and 951 R. The viewer 950 is looking beyond the display at the virtual object 974 (the star). The light paths from the display panel that generate one image for the left eye and another image for the right eye are necessarily different and must emanate from different regions 976 and 978 of the display panel, because of the stereoscopic nature of vision. In other words, display region 976 is responsible for image formation for the left eye, and display region 978 is responsible for image formation for the left eye. Moreover, the two regions 976, 978 may have some overlap as illustrated.

[0467] The viewer needs their eye convergence and eye accommodation to be formed on the virtual object 974 at its apparent distance. While Figure 44 illustrates a positive offset 8, the same discussion applies to operating in a virtual depth display mode with negative offset 8 when the apparent depth is brought closer to the eye of the viewer than the physical location of the display panel.

[0468] ADJUSTING VISION TO APPARENT DISTANCE

[0469] Figure 45 illustrates further design considerations for operating a display panel according to the invention in a virtual depth display mode by further developing the example of Figure 44 of a star virtual object with positive offset. The upper illustration shows what the display will look like when the viewer is looking at the display itself, i.e., when their eye focus is set on the display and their stereo eye vergence is overlapped onto the physical display (VIEW CASE 1). The lower illustration shows what the display will look like when the viewer allows their eyes to refocus, and allows their stereo vergence to be appropriate to the apparent distance of the virtual depth object, which is beyond the physical display in this example (VIEW CASE 2). The upper illustration labels the display as 972 (VIEW CASE 1) and the lower illustration labels the display as 988 (VIEW CASE 2). The object is the star pattern which is labelled 990 and 992 in VIEW CASE 1, and is labelled 994 in VIEW CASE 2. In VIEW CASE 1, the physical display as represented by a patterned display bezel is in sharp focus while the virtual object is blurred, with the virtual object represented by the star appearing double and blurred. In VIEW CASE 2, the virtual object represented by the star is in sharp focus while the physical display as represented by the patterned display bezel appears double and blurred.

[0470] Measures can be taken to make it easier for the viewer to set their vergence angle and focus to the distance of the virtual object, i.e., at the apparent distance, so they are gazing at the distance of the image to be viewed rather than at the distance of the physical device.

[0471] One aid is to make the edges of the physical device chamfered or angled or rounded to avoid sharp lines and specular reflections. The middle side view of Figure 45 is an illustration of this, where the display 986 has chamfered edges 984. The purpose of rounding the edges is to confuse the visual cues of the edge of the display to allow the viewer to refocus beyond (or in front of) the display panel more easily. In the case that the display panel has a bezel, choosing a bezel design where the edges of the display are matt finish and in darker colours may further help the device to blend into its surroundings. Figure 45 also illustrates a bezel area 982 that is patterned. The pattern may be printed or otherwise permanently present on a physical bezel or in the case of a bezel-free device the pattern may be a kind of virtual bezel shown around the edges of the display. The bezel pattern is periodic with a period chosen to match a specific value of stereo-vergence correction making it easier for the viewer to refocus and change their eye vergence.

[0472] In the case of a virtual patterned bezel created by the display, the pattern period could be set to the apparent distance. Moreover, this could be dynamically varied to track changes in the apparent distance.

[0473] The edge of the display can also be made to appear invisible or less visible by matching its colour to the colour of what the viewer sees directly adjacent the display edge, similar to camouflaging by an octopus. This could be done using backward-facing cameras.

[0474] Another approach is to exploit the phenomena of double vision (diplopia) by encouraging the viewer to locate their stereo-convergence at the apparent depth of the display, i.e. the virtual object distance, rather than at the actual depth of the physical display panel.

[0475] Forward-facing cameras, e.g. bezel mounted cameras, can be used to assess the viewer’s vergence angle to determine whether the viewer’s binocular vision is matched to the apparent distance of the virtual object (as needed for viewing) or the physical display distance (which will cause blurry double vision).

[0476] When a display is switched on for the first time, or when the image being displayed changes from a localized display to a virtual depth display, the forward-facing cameras can track the vergence angle of the viewer and specifically whether the stereo gaze distance changes from the physical display distance to the virtual object distance.

[0477] Vergence angle can be determined from changes in pupil separation and compared with the vergence angle associated with the physical display distance, which may be determined by appropriate sensing, e.g. forward-facing cameras or LIDAR mounted on the device. It is therefore possible to sense if the viewer is focusing on the display panel or on the in-focus image plane. Every individual person will have a different functional mapping of pupil separation distance to vergence angle. The functional mapping will, for example, be a function of interpupi llary distance which is the distance between left and right pupils when the eyes are focused at infinity. The functional mapping could also be learnt by the display apparatus. Learning could be done by a calibration or test routine in which the viewer changes focus between the display panel and different offset distances from the display panel and provides feedback to the display apparatus. Learning could also be done adaptively by the display apparatus by the inwardly facing cameras observing a viewer’s face over time and determining their left and right gaze direction as they comfortably operate the display at different offset distances - positive, zero and possibly also negative.

[0478] Through the above measures, the display apparatus will be able to determine when a vergence-accommodation conflict exists and can provide a prompt or active assistance to the viewer to change their vergence to match accommodation (which will automatically take account of a positive or negative offset). One way to assist the viewer to match their vergence to a non-zero offset is to display a pattern (like the illustrated star pattern) and then alter the pattern so as to encourage the viewer’s eyes to seek the correct virtual depth (perhaps using eye tracking and feedback during the process). Another way to assist the viewer to match their vergence to a non-zero offset is to use a peripheral region of the display area (like an active bezel) to display a sliding pattern (again perhaps with feedback) to encourage their stereo vergence to move from being localised on the display to the offset distance (positive or negative).

[0479] VIDEO CONFERENCING

[0480] VIDEO CONFERENCING INTRODUCTION

[0481] Video conferencing is an important application for NCTE display apparatus embodying the invention. Video conferencing has become even more commonplace since the Covid Pandemic, with the widespread use of video conferencing software such as Teams, Zoom, etc. In video conferencing, a software application enables people at different physical locations to communicate with each other via suitable computing hardware located at each location. In other words, a communication node is provided at each location. A typical communication node includes: a computer apparatus, a data communication link, a display (video out), a camera (video in), a microphone (audio in) and a loudspeaker (audio out). The camera, microphone and loudspeaker are often ancillary devices to the computer apparatus. The microphone and loudspeaker are commonly combined in a single headset device to provide both audio in and out. Any of the display, camera, microphone and loudspeaker may be in wireless or wired communication with the computer apparatus. In the case of a receiver-only communication node, the video out and audio out channels may be omitted. The camera and display are for respectively transmitting and receiving a video image feed comprising successive image frames. The microphone and loudspeaker are for respectively transmitting and receiving an audio feed. The video conferencing software application presents a graphical user interface on the display to provide user controls and other features as well as the video feed. Data communication between the communication nodes is used to transfer the video and audio data between the communication nodes as well as other data. Such other data may be a share of the graphical user interface of a third-party computer application being run on the computer apparatus of one of the conference participants (e.g. a window of a presentation program, word processor program, video program etc.). Such other data may be a share of the whole screen of the computer apparatus of one of the conference participants (so-called screen share).

[0482] One of the major challenges with video conferencing is to create a more realistic experience. The aim should be to replicate as many of the features of an in-person meeting as possible and hence give a better impression of actually sitting in a meeting room with the other participants.

[0483] When only a single participant is present at a communication node, there is no sharing of a display between multiple participants at one location, the subjective experience is reasonable. However, when multiple participants share a display, then the fact that a conventional display lacks any 3D depth perception leads to a larger perceived difference between the quality of experience of an in-person meeting and a video conference. In particular, the perceived quality of experience is poor when a conference involves multiple participants sitting together at a conference table in a conference room at one location and viewing a single large display to communicate with remote participants. This poor experience is worsened further if this situation is replicated at another location, for example when the conference involves two groups of people in two conference rooms at different locations.

[0484] It is noted that, in video conferencing, the display will typically be positioned at a greater distance from the eye than the near point. The eye's near point is the closest distance which the eye can bring into focus, i.e. , accommodate. The near point is often taken to be nominally 25 cm. However, a human eye’s actual near point gradually increases with age from about 7 cm for a youth to over 50 cm for an old person.

[0485] For one person who is video conferencing on their personal computer, the eye-to-display distance may be a small multiple of the near point, e.g. 25 to 75 cm, i.e. , 1 to 3 times the near point distance. In a conference room for video conferencing which a group of participants using a single shared display, the eye-to-display distance may be much larger, e.g., in the range 2-10 meters for the different individuals sitting at different positions around a conference table. To be viable for such large viewing distances, a video-conferencing display therefore needs to be relatively large. For example, for a maximum viewing distance of 6 metres, a 90 inch (230 cm) display (diagonal) is the approximate minimum requirement.

[0486] VIDEO CONFERENCING DISPLAY PANELS

[0487] There are several specific ways to incorporate NLO display features according to the invention into a display panel suitable for video conferencing. The display panel will include NLO material arranged to receive pairs of input beams which have appropriate amplitude and phase profiles to create a product beam of the required amplitude, wavevector and wavefront radius of curvature.

[0488] VIDEO CONFERENCING SYSTEMS

[0489] A video conferencing system according to different embodiments of the invention have a set of common features. There will be a minimum of two physical locations with a minimum of one participant per location.

[0490] At least one display apparatus is provided at each location that comprises a display panel with NLO material according to the invention to display images of participant(s) at remote location(s) on the display panel at the local location.

[0491] At least one image capture device is provided at each location. The image capture device will typically be a camera operating in the visible, to capture video image data of the participant(s) at that location for transmission to partici pant(s) at remote location(s) after suitable image processing (see below). Preferably multiple image capture devices are provided at each location. Multiple image capture devices will assist construction of 3D models of the participants and the environment of the participants. Additional data capture devices may be provided for capturing other kinds of scene data that can be combined with the image data to build up more complete, accurate and / or reliable 3D models of the participants and / or their background scene. These additional data capture devices may include any of: range finders, LIDAR systems and thermal (infrared) cameras. These additional data capture devices may contribute further to the image information provided by, for example, identifying the position of the partici pant(s) in the room, and assisting in creation of an appropriate background scene 3D model, if required. Having multiple cameras and other additional data capture devices in the room will be helpful to determine the participant positions more accurately and will also assist creating a more accurate and complete 3D model of the location of the video conference participants in the room. The image capture device(s) may be integrated structurally in the display panel, e.g. mounted in corners or vertex mid-points of a display bezel, and / or independently mounted, e.g. desk- mounted or wall-mounted elsewhere in the room.

[0492] A data transmitter and receiver (data transceiver) is required at each location to send and receive video and audio data to and from the other participants via appropriate data communication links.

[0493] Appropriate computer processing resource is provided for image processing the video image data and optional additional data collected respectively by the image capture device(s) and additional data capture devices. One important task of the image and additional data processing is to determine the position(s) of the participant(s) relative to the display panel they are associated with (participant position). These positions are needed so that the display panel directs output light (product beam light), and thus images, in a targeted way to each participant. Optionally, this may include directing different images to the left eye and to the right eye of each participant. Another important task of the image and additional data processing is to generate a 3D model of the participant(s), most especially their head (3D participant model) so that perspective can be included in the images sent to the remote locations. The participant 3D model may be head-only or head-and-shoulders only, for example. A further task of the image and additional data processing is to generate a 3D model of the location to provide a backdrop (3D real local environment model), such as of a conference room or one side or other part thereof. The 3D real environment model will not include the participants. A 3D model of the locations’ real environments will not be needed if the conference is fully virtualized, by which is meant that all the participants are placed in a fully virtual conference room that is defined by a fully virtual 3D environment model. In general, a video conference will involve an arbitrary number of locations. It will thus be necessary to create an overall 3D environment model for the conference (3D conference environment model) so that participants can be allocated a place from which perspective can be calculated. Assuming the conference is not fully virtual, then the 3D environment model will need to merge one or more 3D real local environment models into the overall 3D conference environment model. A simple scenario would be a conference involving two locations with respective groups of participants, each group seated on the same side of an elongate conference table. The 3D conference environment model would then simply merge the two 3D real local environment models as two halves of one virtual conference room.

[0494] The 3D conference environment model and the placement of the 3D participant models within it enables participant-specific viewpoints (or pairs of viewpoints in case left and right eyes are to be served different images) for rendering images on the display panel at each location which have a participant-specific perspective. In other words, the images of remote participants that are formed on the display panel for each (local) participant are given a perspective that has been determined from the placement of the local participant (to whom the images are being served by the display panel) in relation to how each of the remote participants are placed in the 3D environment model. In the case that the remote participants are set in a 3D real local environment model, they will appear as if in their remote location as viewed from the perspective of the position in which the viewing participant is placed in the 3D environment model. In the case that the conference is fully virtualized, the perception of perspective will follow entirely from how the participants are placed in the fully virtual 3D environment model.

[0495] For ease of representation and discussion the following examples omit inclusion of the audio channel and include only the video channel. However, it will be appreciated that suitable audio equipment is included, namely microphones and loudspeakers for respectively transmitting and receiving an audio feed.

[0496] It will be understood that the data communication links, such as network connections, between and within video conferencing locations may comprise any of: copper transmission cables, optical fibres, wireless transmission, satellite communication, routers, firewalls, switches, gateway computers and / or edge servers. Moreover, the computers and their resources discussed in the following may be physical or virtual machines.

[0497] It will also be understood that while the following description concentrates on the video representation on the display panel, a commercial video conferencing system incorporating display panels according to the invention and associated video processing of scene image information will most probably include further ancillary hardware and software features as familiar from existing video conferencing systems. These may include: surround sound, multiple microphones, privacy filtering of backgrounds, provision of artificial backgrounds, various modes for scheduling calls, noise cancelling, the ability to remove certain people from the synthetic scene, blanking of the display for certain participants and so forth.

[0498] The image processing for the video conferencing system should take account of one or more of the following points. The video image frames served to each viewing participant (viewer) are specific to that viewer taking account of the viewer’s position in relation to the display panel and / or the position of remote participants in the 3D conference environment model. In this respect, if a remote participant is placed in a 3D location model, then that remote participant will appear to the viewer as being in their real environment. The viewer’s position at their location may change during the conference. The position of remote participants may also change during the conference. Both classes of positional change result in the video image frames served to that viewer changing to take account of one or both such classes of positional change. Optionally, the image frames served to the left and right eyes of a viewer will be different in which case the viewer’s position shall include separate positions for the left and right eyes.

[0499] Optionally, the viewer’s viewing direction may additionally be taken account of when forming the image frames for that viewer on the display panel. Head orientation may be taken as an approximate proxy of eye gaze direction of the viewer. Alternatively, eye tracking may be used to gain more precise information on eye gaze direction of the viewer. The synthetic scene served to that viewer is then adapted to take account of gaze direction, e.g. if the active speaker in a video conference changes from one remote participant to another remote participant, the viewer may change gaze direction to follow the active speaker, in which case the image served to the viewer could shift, e.g., to the right or left, accordingly - for example by a gradual panning movement from the previous gaze direction to the new gaze direction.

[0500] The imaging processing software will need calculate the scene that each viewer will see on the display panel at their location based upon the image data acquired at each of the remote locations. In general, the scene that is displayed on the display panel to a viewer will have to be synthesized, since the viewpoints of the cameras at each location that acquire the image data will not coincide the viewpoints of the scene required to be displayed to a viewer. The image processing software will therefore calculate an artificial 3D scene and then use appropriate rendering based on the desired viewpoint of each viewer (or pairs of viewpoints in the case of separate images being served to the left and right eyes of each viewer). The image element representing a remote participant or at least their head may either aim to look like a faithful representation of the participant (photo-realistic) even though they are actually synthesized. Alternatively, the image element representing a remote participant may be generated to look like an artificial avatar of the person (cartoon characters, animations, etc). For example, an active speaker or a moving participant may be represented photo- realistically whereas a silent or non-moving participant may be represented as an avatar.

[0501] The most sophisticated and realistic 3D effects during video conferencing will be created if the system uses all of: full pixel-by-pixel control of wavefront radius of curvature of the product beam as well as wavevector and amplitude; viewer eye tracking of each viewer; provision of different depth of field views for different elements in the scene being displayed (e.g. for each participant and the background); separate image frames served to left and right eyes of each viewer; full colour rendering; and photo-realistic rendering of participants and background. However, in most practical systems, the specification will be reduced to select those features which make the largest contributions to perceived quality of participant experience. In particular, driving the display panel to provide only a single depth of field (single image plane), which may be matched to be coincident with the physical plane of the display panel, may be acceptable, even though an accommodative focusing effect is not provided. Similarly, not catering for left and right eyes may be acceptable, so that the same image is delivered to both...

Claims

CLAIMS1 . A display apparatus (12) for displaying images responsive to input of image frames, each image frame being defined by an array of pixels, the display apparatus comprising: a first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth; a second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth; a directional backlight panel (88) containing nonlinear optical, NLO, material (100) that is phase matched to first and second selected combinations of the frequencies of the first and second input beams, and to a frequency equal to the sum of the selected combinations; input beam routing components (222, 240) arranged to introduce the first and second input beams into the directional backlight panel so that they traverse the directional backlight panel in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume (110) where the first and second input beams mix to generate a product beam having the sum frequency; an input beam modulator (120, 122) arranged to modulate one or both of the first and second input beams in amplitude and phase to set at least the wavevector of the product beam on a pixel-by-pixel basis; a spatial light modulator (938) which is arranged over the directional backlight panel (88) and is controllable to amplitude modulate the product beams on a pixel-by-pixel basis as they pass through the spatial light modulator (938); and a controller (13) operable to load the image frames onto the input beam modulator (120, 122) and the spatial light modulator (938).

2. The display apparatus of claim 1 , wherein the controller is operable to control the input beam modulator so as to set the product beam’s wavefront radius of curvature to a constant value for all pixels of any given image frame.

3. The display apparatus of claim 2, wherein the constant value is set so that image localisation is at or closely adjacent to the physical plane of the spatial light modulator.

4. The display apparatus of claim 1 , 2 or 3, wherein the controller is operable to control the input beam modulator so as to set the product beam’s amplitude to a constant value for all pixels in each image frame.

5. The display apparatus of any one of the preceding claims, wherein the display apparatus is viewable from a range of possible viewing angles and further comprises aforward-facing camera having a field of view directed to capture images within the range of possible viewing angles, wherein the controller is configured to process the camera images to identify a viewer of the display and locate their head position.

6. The display apparatus of claim 5, wherein the controller is further configured to vary the range of actual viewing angles for image frames being displayed within the range of possible viewing angles through wavevector control.

7. The display apparatus of claim 6, wherein wavevector control is used to direct the product beam light to the head position.

8. The display apparatus of claim 7, wherein the range of actual viewing angles is restricted such that image frames are only viewable at the head position.

9. The display apparatus of any one of claims 5 to 8, wherein the controller is further configured to process the camera images to identify left and right eye locations of the viewer, and to serve different images of each image frame to the left and right eyes to provide a three-dimensional effect.

10. A display apparatus (10) comprising: a first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth; a second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth; a display panel (88) containing nonlinear optical, NLO, material (100) that is phase matched to first and second selected combinations of the frequencies of the first and second input beams, and to a frequency equal to the sum of the selected combinations (82); input beam routing components (222, 240) arranged to introduce the first and second input beams into the display panel at first and second injection positions situated on opposing sides of the display panel and with first and second beam directions so that they traverse the display panel in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume (110) where the first and second input beams mix to generate a product beam having the sum frequency, wherein the input beam routing components (220, 240) comprise a light guide with one of a graded refractive index and stepped refractive index profile arranged to route one of the first and second input beams to its injection position; and a controller (13) operable responsive to the input of the image information to form synthetic scene images on the display panel by amplitude modulating at least one of the first and second input beams and by phase modulating at least one of the first and second input beams to set each of amplitude, wavevector and wavefront radius of curvature of the product beam.11 . The display apparatus of claim 10, wherein the display panel, including the NLO material, is transparent across the visible range.

12. The display apparatus of claim 11 , wherein the display apparatus comprises a frame that extends around at least a part of the display panel and said light guide is accommodated in the frame.

13. The display apparatus of claim 11 , wherein the display panel is configured to lie in front of a wearer's eyes and has separate first and second injection positions for each of a left eye area and a right eye area of the display panel, and said light guide is arranged to route one of the first and second input beams to its left eye injection position and a further light guide is accommodated in the frame and arranged to route one of the first and second input beams to its right eye injection position.

14. A wearable headset comprising : a first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth; a second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth; a display panel which is shaped and sized to cover a wearer’s eyes, the display panel containing a nonlinear optical, NLO, material that has a phase matched region that is phase matched to first and second selected combinations of first and second input beam frequencies, and to a visible product beam frequency equal to the sum of the selected combinations; and input beam routing components (222, 240) arranged to introduce the first and second input beams into the display panel to traverse the display panel in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume where the first and second input beams mix to generate a product beam having the sum frequency; a controller (13) operable responsive to input of image information to render synthetic scene images on the display panel by amplitude modulating at least one of the first and second input beams and by phase modulating at least one of the first and second input beams to set each of amplitude, wavevector and wavefront radius of curvature of the product beam; and a set of left eye and right eye visible light sources arranged angularly distributed around each of the left eye and right eye of a wearer so as to lie in the peripheral vision of a wearer’s line of sight for normal forward vision, wherein the controller is further operable to control the visible light sources in colour and brightness to simulate areas of the synthetic scene that are determined to lie in the peripheral vision of the wearer instead of rendering those areas of the synthetic scene.

15. The headset of claim 14, further comprising left eye and right eye inwardly facing cameras to provide eye tracking image information, wherein the controller is operable to determine eye tracking of the left and right eyes based on the image information provided by the left eye and right eye inwardly facing cameras and further operable to control the visible light sources in colour and brightness responsive to the eye tracking.

16. The headset of claim 14 or 15, wherein the light sources are arranged recessed in a mounting to shroud the light sources from view from a distal side of the wearable headset.

17. A wearable headset comprising : a first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth; a second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth; a display panel which is transparent across the visible range and which is shaped and sized to cover a wearer’s eyes, the display panel containing a nonlinear optical, NLO, material that has a phase matched region that is phase matched to first and second selected combinations of first and second input beam frequencies, and to a visible product beam frequency equal to the sum of the selected combinations; and input beam routing components (222, 240) arranged to introduce the first and second input beams into the display panel to traverse the display panel in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume where the first and second input beams mix to generate a product beam having the sum frequency; a controller (13) operable responsive to input of image information to render synthetic scene elements on the display panel by amplitude modulating at least one of the first and second input beams and by phase modulating at least one of the first and second input beams to set each of amplitude, wavevector and wavefront radius of curvature of the product beam; and an outwardly facing image acquisition device to capture images of the natural scene as viewed by the wearer, wherein the headset has access to computing resource configured to process the capture images of the natural scene and create a model thereof, wherein the controller has at least one of: a first mode of operation in which the synthetic scene elements are rendered on the display panel in a position referenced to the display panel; and a second mode of operation in which the synthetic scene elements are rendered on the display panel in a position fixed referenced to the natural scene model.

18. The headset of claim 17, wherein the computing resource is configured to identify instability motion of the headset through shake or jitter in the captured images of the natural scene relative to the natural scene model and to vary the position on the display panel where the synthetic scene elements are rendered to counteract the instability motion.

19. The headset of claim 17 or 18, wherein, in the first mode of operation, the wavefront radius of curvature is increased to reduce the depth of field of the synthetic scene elements, so that the natural scene will tend to lie beyond the depth of field of the wearer’s eyes and so become blurred.

20. The headset of claim 17, 18 or 19, wherein, in the second mode of operation, the wavefront radius of curvature is set having regard to the natural scene model such that the depth of field encompasses both the natural scene and the synthetic scene elements so both are in focus simultaneously to the wearer.21 . The headset of any one of claims 17 to 20, wherein the brightness of the synthetic scene elements is varied such that they retain sufficient contrast relative to the natural scene, with a luminance of at least three times that of the natural scene.

22. The headset of any one of claims 17 to 21 , wherein the brightness of the synthetic scene elements is varied such that they do not exceed a desired contrast relative to the natural scene, with a luminance of no more than one of 10, 20, 30, 40, 50 or 100 times that of the natural scene.

23. The headset of any one of claims 17 to 22, further comprising: one or more types of sensors for collecting additional data relevant to the natural scene and / or the wearer, the sensor types being selected from the group: LiDAR sensor, accelerometer, magnetometer, tilt sensor, MEMs sensor, ECG sensor, microphone, wherein the additional data is used by the computing resource to create the natural scene model.

24. The headset of any one of claims 17 to 23, further comprising: left and right eye inwardly facing image acquisition devices to capture images of the left and right eyes of the wearer.

25. The headset of claim 24, wherein, in the first mode of operation, the computing resource is configured to identify when the wearer is actively viewing one or more of the synthetic scene elements from the images captured by the inwardly facing image acquisition devices and in response increase the brightness of those synthetic scene elements.

26. The headset of claim 24 or 25, wherein the computing resource is configured to identify periods of saccadic motion of the eyes of the wearer from the images captured by the inwardly facing image acquisition devices and in response cause temporary suspension of the rendering of synthetic scene elements on the display panel or dimming thereof.

21. k headset (10) in a spectacles or goggles format for displaying synthetic scene images responsive to input of image information, the headset comprising: a frame (26) accommodating a display panel (88); and left and right temples (32, 34) accommodating beam generation optics and amplitude and phase modulators (120, 122), wherein the beam generation optics comprises: a first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth; and a second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth; wherein the display panel (88) comprises: a display area of nonlinear optical, NLO, material (100) with phase matched regions that are phase matched to first and second selected combinations of the frequencies of the first and second input beams, and to a frequency equal to the sum of the selected combinations (82), wherein a controller (13) is provided that is operable responsive to the input of the image information to form the synthetic scene images by amplitude modulating at least one of the first and second input beams with the amplitude modulator (120) and by phase modulating at least one of the first and second input beams with the phase modulator (122) to set each of amplitude, wavevector and wavefront radius of curvature of the product beam, and wherein the frame (26) and temples (32, 34) jointly accommodate: input beam routing components (222, 240) arranged to introduce the first and second input beams into the display panel so that they traverse the display panel in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume (110) where the first and second input beams mix to generate a product beam having the sum frequency, wherein the input beam routing components for each of the first and second input beams are actuatable by a controller to direct each of the first and second input beams so that they cross in an intersection volume (110) that is located at an addressable two-dimensional coordinate in the display area.

28. The headset of claim 27, further comprising: left and right eye inwardly facing image acquisition devices to capture images of the left and right eyes of the wearer, wherein the headset has access to computing resource configured to process the captured images to determine whether the wearer’s gaze is directed onto the display area and if so which two-dimensional coordinate on the display area the wearer’s gaze is directed,the controller being operable, responsive to determination of the two-dimensional coordinate of the wearer’s gaze, to direct the first and second input beams to cross at that two-dimensional coordinate.

29. The headset of claim 27 or 28, wherein the input beam routing components are actuatable under control of the controller to direct each of the first and second input beams to any one of a plurality of spatially separated optical channels (1124, 1126) via which they enter the display panel at different positions to select a vertical coordinate of the display area.

30. The headset of claim 27, 28 or 29, further comprising: a mirror array (1142) arranged in each optical channel, the mirror array comprising a plurality of mirrors of different focal lengths, wherein the input beam routing components are actuatable under control of the controller to direct each of the first and second input beams to any one of the mirrors of the mirror array to select a horizontal coordinate of the display area.31 . The headset of any one of claims 27 to 30, wherein the first input beam encodes image intensity information that defines the intensity of the product beam generated at each intersection volume of a given synthetic scene image and the second input beam encodes image depth information that defines the wavefront radius of curvature in the product beam generated at all intersection volumes for a given synthetic scene image.

32. A display apparatus (12) for displaying images responsive to input of image information, the display apparatus comprising: a first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth; a second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth; a display panel (88) containing nonlinear optical, NLO, material (100) that is phase matched to first and second selected combinations of the frequencies of the first and second input beams, and to a frequency equal to the sum of the selected combinations (82); input beam routing components (222, 240) arranged to introduce the first and second input beams into the display panel so that they traverse the display panel in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume (110) where the first and second input beams mix to generate a product beam having the sum frequency; a memory storing an eyesight prescription associated with an individual which comprises left eye and right eye correction values; and a controller (13) operable responsive to the input of the image information to form the images on the display panel by amplitude modulation of at least one of the first and second input beams and by phase modulation of at least one of the first and second input beams to set each of amplitude, wavevector and wavefront radius of curvature of the product beam, wherein the controller is operable to perform the phase modulation taking account of the correction values so as to display the images such that the eyesight prescription is incorporated.

33. The display apparatus of claim 32, wherein the eyesight prescription includes left and right eye correction values for at least one of: long-sightedness, short-sightedness and astigmatism.

34. The display apparatus of claim 32 or 33, wherein the eyesight prescription includes separate left and right eye correction values for near vision and for far vision.

35. The display apparatus of claim 32, 33 or 34, further comprising: an amplitude modulator (120) arranged in the first input beam and comprising an array of pixels, the pixels being controllable individually to amplitude modulate the first input beam on a pixel-by-pixel basis, and a phase modulator (122) arranged to phase modulate the second input beam, wherein the correction values are incorporated in the second input beam by setting corresponding values for the wavefront radius of curvature across the second input beam.

36. The display apparatus of any one of claims 32 to 35, further comprising: user identification software operable to identify the individual using the display apparatus; and prescription look-up software to access the eyesight prescription associated with the individual who has been identified and store their eyesight prescription in the memory.

37. The display apparatus of any one of the claims 32 to 36, wherein the display apparatus is incorporated in one of: augmented reality spectacles, virtual reality goggles, a smartphone, a tablet computer, a laptop computer, a computer monitor and a television set.

38. The display apparatus of any one of claims 32 to 37, further comprising: a computer program element configured to retrieve an eyesight prescription for an individual via a data communication link and store it in said memory.

39. The display apparatus of any one of claims 32 to 38, further comprising: a computer program element configured to perform a diagnostic test to generate an eyesight prescription with said correction values and store it in said memory.

40. A wearable headset comprising: a first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth; a second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth; a display panel which is transparent across the visible range, the display panel containing a nonlinear optical, NLO, material that has a phase matched region that is phase matched to first and second selected combinations of first and second input beam frequencies, and to a visible product beam frequency equal to the sum of the selected combinations; input beam routing components (222, 240) arranged to introduce the first and second input beams into the display panel to traverse the display panel in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume where the first and second input beams mix to generate a product beam having the sum frequency; and a controller (13) operable responsive to input of image information to render synthetic scene images on the display panel by amplitude modulating at least one of the first and second input beams and by phase modulating at least one of the first and second input beams to set each of amplitude, wavevector and wavefront radius of curvature of the product beam; and the headset having access to computing resource configured to perform a medical intervention, the medical intervention being one of: a test that is diagnostic of eyesight of the wearer; a treatment for an eyesight condition or defect of the wearer; and a correction for a wearer’s eyesight defect, wherein the test, treatment or correction comprises the computing resource serving synthetic scene image data to the controller for rendering said synthetic scene images on the display panel.41 . The headset of claim 40, further comprising left eye and right eye inwardly facing cameras to provide eye tracking image information.

42. The headset of claim 41 , wherein the computing resource as part of the medical intervention is configured to: process the eye tracking image information to determine eye tracking of the left and right eyes.

43. The headset of claim 42, wherein the medical intervention is a test comprising one of:a convergence test which uses eye tracking to measure how vergence angle changes as the object distance of synthetic scene images served to the display panel is varied; a saccades test which uses eye tracking to measure how a wearer’s gaze changes as synthetic scene elements appear and disappear at different locations on the display panel; a smooth pursuits test which uses eye tracking to measure how a wearer’s gaze follows a synthetic scene element as it is moved over the display panel; a speed of response test which uses eye tracking to measure how quickly a wearer’s eyes respond to a change in synthetic scene images being generated on the display panel; a colour blindness test which serves a sequence of differently coloured test images to the display and uses eye tracking to detect whether a response occurs each time the display transitions from one test image to another; a field-of-view test which displays in time sequence a series of synthetic scene elements to a variety of different locations in the display panel and uses eye tracking to detect whether a response occurs to each synthetic scene element.

44. The headset of claim 43, wherein the field-of-view test involves displaying the synthetic scene elements at different apparent distances to test at different eye focal lengths.

45. The headset of claim 43 or 44, wherein the test involves displaying the synthetic scene elements with different background brightnesses created in the synthetic scene to test under different lighting conditions.

46. The headset of claim 41 , wherein the medical intervention is a treatment for esotropia, wherein: the display apparatus has a memory storing an eyesight prescription with left eye and right eye correction values for long-sightedness and optionally also astigmatism; the controller is operable to perform the phase modulation taking account of the correction values so as to display the images such that the eyesight prescription is incorporated in synthetic scene images served to the wearer; eye tracking is used to detect when vergence misalignment occurs; and corrective synthetic scene images are served in response to detection of vergence misalignment, said corrective synthetic scene images being configured to induce the wearer to bring their vergence back into alignment.

47. The headset of claim 46, wherein said corrective synthetic scene images contain an entertainment animation to stimulate the wearer to watch.

48. The headset of claim 46 or 47, wherein said corrective synthetic scene images comprise a time sequence of images with an apparent distance that gradually reduces to induce a gradual increase in vergence angle.

49. The headset of claim 42, wherein the medical intervention is: a smooth pursuits treatment which uses eye tracking to measure how a wearer’s gaze follows a synthetic scene element as it is moved over the display panel.

50. The headset of any one of claims 40 to 49, further comprising an outwardly facing camera to capture images of the natural scene in a visible frequency range.51 . The headset of claim 50, wherein the computing resource as part of the medical intervention is configured to: process the visible natural scene images to generate synthetic scene images that reproduce at least part of the visible natural scene images, and wherein the controller is operable to superpose said at least part of the visible natural scene images onto the natural scene as viewed by the wearer.

52. The headset of claim 51 , wherein the medical intervention is a correction for colour blindness, wherein the synthetic scene images reproduce a colour-filtered version of the visible natural scene images, which, when superposed onto the natural scene, provide a combined synthetic and natural scene which is colour balanced for the wearer.

53. The headset of claim 52, wherein the medical intervention is a correction for poor vision in low-light conditions, wherein the synthetic scene images reproduce the visible natural scene images so that, when they are superposed onto the natural scene, provide a combined synthetic and natural scene which is an enhanced brightness version of the natural scene.

54. The headset of claim 51 , wherein the medical intervention is a correction for nystagmus, wherein: eye tracking is used to detect when rapid eye motion associated with nystagmus occurs; and in response the controller displays the visible natural scene images to the wearer as the synthetic scene images by varying the position on the display panel where the synthetic scene images are rendered to follow the rapid eye motion and thereby present a version of the natural scene image which appears stable to the wearer.

55. The headset of claim 54, wherein the display panel further comprises an actuatable dimming layer, and wherein, in response to detection of the rapid eye motion, the controller actuates the dimming layer to dim the natural scene as viewable by the wearer through the display panel.

56. The headset of claim 50 or 51 , further comprising an outwardly facing thermal imaging camera to capture images of the natural scene in an infrared frequency range.

57. The headset of claim 56, wherein the computing resource as part of the medical intervention is configured to: process the infrared natural scene images to generate synthetic scene images that reproduce at least part of the infrared natural scene images, and wherein the controller is operable to superpose a colourised version of the at least part of the infrared natural scene images onto the natural scene as viewed by the wearer.

58. A wearable headset (10) comprising: a first beam source for providing a first input beam of a first frequency and bandwidth; a second beam source for providing a second input beam of a second frequency and bandwidth; a transparent panel (88) containing a nonlinear optical, NLO, material (100) that is phase matched to first and second selected combinations of the frequencies of the first and second input beams, and to a frequency equal to the sum or difference of the selected combinations (82); input beam routing components arranged to introduce the first and second input beams into the transparent panel so that they traverse the transparent panel in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume where the first and second input beams mix to generate a product beam having the sum or difference frequency and propagating in an outward direction away from the wearer of the headset; and a controller (13) operable to perform amplitude and phase modulation on the first and second input beams to set each of amplitude, wavevector and wavefront radius of curvature of the product beam, so as to control the product beam outward direction.

59. The headset of claim 58, wherein the controller (13) is further operable to control the focal distance of the product beam through said amplitude and phase modulation of the first and second input beams.

60. The headset of claim 58 or 59, wherein the controller (13) is further operable to control the angular spread of the product beam from the transparent panel through said amplitude and phase modulation of the first and second input beams.61 . The headset of claim 58, 59 or 60, wherein the transparent panel is arranged in front of a wearer's eyes at a vertex distance of less than one of: 30 mm, 40 mm, 50 mm, 60 mm, 70 mm, 80 mm, 90 mm and 100 mm.

62. The headset of any one of claims 58 to 61 , further comprising: an inwardly facing image acquisition device to capture image of at least one eye of the wearer, wherein the headset has access to computing resource configured to process the captured images to determine the wearer’s gaze direction, the controller being operable, responsive to determination of the wearer’s gaze direction, to align the product beam outward direction with the gaze direction.

63. The headset of claim 62, wherein the controller being further operable, responsive to determination of the wearer’s gaze direction, to set the location of the intersection volume(110) where the product beam is generated so that the optical axis of the product beam intersects the eye’s pupil.

64. The headset of any one of claims 58 to 63, further comprising: an outwardly facing scene sensor to acquire directional sensor data from the natural scene, wherein the headset has access to computing resource configured to process the directional sensor data to determine a sensed direction, the controller being operable, responsive to determination of the sensed direction, to align the product beam outward direction with the sensed direction.

65. The headset of any one of claims 58 to 64, wherein the controller is operable to receive product beam control information from an external source and to control the product beam outward direction responsive to the control information.

66. The headset of any one of claims 58 to 65, further comprising: a third beam source for providing a third input beam (70) of a third frequency and bandwidth; and a fourth beam source for providing a fourth input beam (76) of a fourth frequency and bandwidth, wherein the transparent panel comprises a further phase matched region of NLO material that is phase matched to third and fourth selected combinations of third and fourth input beam frequencies, and to a display product beam frequency equal to the sum of the selected combinations, and the headset further comprises further input beam routing components (222, 240) which introduce the third and fourth input beams into the transparent panel to traverse the transparent panel in third and fourth paths that cross each other in the further phase matched region of NLO material in a noncollinear geometry to define a further intersection volume where the third and fourth input beams mix to generate a display product beam having the sum frequency and propagating in an inward direction into the eye of the wearer of the headset, wherein the controller is operable responsive to the input of image information to form the synthetic scene images by amplitude modulating at least one of the third and fourth input beams and by phase modulating at least one of the third and fourth input beams to set each of amplitude, wavevector and wavefront radius of curvature of the display product beam.

67. A display panel containing nonlinear optical, NLO, material capable of generating a visible beam for image formation by sum frequency generation from a pair of near-infrared beams that cross in the NLO material in a non-colinear configuration, the display panel comprising: a succession of alternating first and second NLO material layers, wherein the first and second NLO material layers have respective values of a second order nonlinear coefficient in the direction perpendicular to the layers which differ from one another to form a grating having a period selected to phase match to the frequencies of the near-infrared beams, or to selected combinations thereof, and to the visible beam, wherein the first and second NLO material layers are formed of a non- centrosymmetric organic material that is transparent at the wavelengths of the visible beam and the infrared beams.

68. The display panel of claim 67, wherein the organic material is one of a birefringent material for true phase matching, a polar material for quasi-phase matching and a ferroelectric material for quasi-phase matching.

69. The display panel of claim 67 or 68, wherein the first and second NLO material layers are poled in opposite directions.

70. The display panel of claim 69, wherein the opposite poling directions are aligned: in the planes of the first and second NLO material layers; perpendicular to the planes of the first and second NLO material layers; at an acute angle between an in-plane alignment and a perpendicular alignment.71 . The display panel of claim 67 or 68, wherein the first NLO material layers are poled in one direction and the second NLO material layers are unpoled.

72. The display panel of any one of claims 67 to 71 , wherein the first and second NLO material layers are made of the same organic material.

73. The display panel of any one of claims 67 to 71 , wherein the first and second NLO material layers are made respectively of first and second organic materials that are different.

74. The display panel of any one of claims 67 to 73, further comprising third material layers interposed between the first and second NLO material layers, the third material layers having an electrical conductivity which is at least one order of magnitude greater than that of both the first and second NLO material layers so as to form electrode layers for generating an electric field across the first and second NLO material layers.

75. The display panel of claim 74, wherein the electrode layers are provided with respective contacts via which a voltage can be externally applied to bias the NLO material layers and wherein the organic material of the NLO material layers has a third order nonlinearity that under said bias provides a second order nonlinearity.

76. The display panel of any one of claims 67 to 75, further comprising respective ones of said gratings for generating each of red, green and blue beams, each said grating having a period selected to phase match to the wavelengths of the pair of the near-infrared beams that generate the red, green and blue beams and of the red, green and blue beams that they respectively generate.

77. The display panel of claim 76, wherein the organic material for the NLO material layers is transparent in a visible range which spans at least between wavelengths of 450 nm and 630 nm, and is also transparent in a near infrared range which spans at least between wavelengths of 780 nm and 1300 nm.

78. The display panel of claim 77, wherein the visible range and the near-infrared transparency range for a continuous wavelength band of transparency.

79. The display panel of claim 77 or 78, wherein the near-infrared transparency range extends at least up to a wavelength of one of: 1350 nm, 1400 nm, 1450 nm, 1500 nm, 1550 nm and 1600 nm.

80. The display panel of any one of claims 67 to 79, wherein the organic material comprises a ring molecule selected from the group containing pyridine, aniline and benzene to which a donor and / or acceptor molecule is attached.81 . The display panel of claim 80, wherein the acceptor is selected from the group containing NO, NO2, CHO and ON.

82. The display panel of claim 80 or 81 , wherein the donor is selected from the group containing N(CH3)2, NH2, OCH3 and OH.

83. The display panel of claim 80, 81 or 82, wherein the pyridine, aniline and benzene are respectively nitro-pyridine, nitro-aniline and nitro-benzene.

84. The display panel of claim 80, wherein the organic material is one of:DCANP 2-docosylamino-5-nitropyridineAPDA 8-(4'-acetylphenyl)-1 , 4-dioxa-8-azaspiro[4.5]decaneDMNP organic 3, 5-dimethyl-1- (4-nitrophenyl)pyrazoleMBANP (-)2-(a-methylbenzylamino)-5-nitropyridine.

85. The display panel of any of claims 79 to 83, wherein the organic material is an at least partially deuterated or fluorinated analogue of its normal molecular form in which at least one hydrogen atom is substituted with a deuterium or a fluorine atom.

86. The display panel of any one of claims 67 to 85, wherein the NLO material layers are one of: Langmuir Blodgett films, liquid crystal layers, spin-coated layers, and vapour deposited layers.

87. A method of manufacturing a display panel according to any one of claims 67 to 86, the method comprising: providing a substrate; and forming a succession of alternating first and second NLO material layers such that the first and second NLO material layers have respective values of a second order nonlinear coefficient in the direction perpendicular to the layers which differ from one another to form a grating having a period selected to phase match to the frequencies of the near-infrared beams, or to selected combinations thereof, and to the visible beam, wherein the first and second NLO material layers are formed of a non-centrosymmetric organic material that is transparent at the wavelengths of the visible beam and the infrared beams.

88. The method of claim 87, wherein an electric field is applied to the first NLO material layers to pole those layers.

89. The method of claim 87, wherein first and second electric fields are applied respectively to the first and second NLO material layers, the first and second electric fields being different in directionality and / or magnitude to pole the first and second NLO material layers in opposite directions.

90. The method of claim 88 or 89, further comprising: forming electrode layers in between each of the first and second NLO material layers; applying said electric field via contacts made to the electrode layers.91 . The method of any of claims 87 to 90, wherein the display panel after formation of the NLO material layers is thinned and laterally extended by rolling the display panel through a pair of rollers.

92. The method of any of claims 87 to 91 , wherein the NLO material layers are one of: Langmuir Blodgett films, liquid crystal cells, spin-coated layers and vapour deposited layers.

93. A spin-coating method of manufacturing a display panel according to any one of claims 67 to 86, the method comprising: arranging a substrate on a rotatable mount; providing a liquid dispenser containing a liquid in which organic material for forming the NLO material layers is dispersed in a solvent; forming an NLO material layer of a first type by: dispensing an amount of the liquid onto the substrate surface; spinning the substrate on the rotatable mount to distribute the liquid by centrifugal force and create a layer of the liquid on the substrate; fixing the organic material while applying an electric field to pole the organic material with a first domain orientation as the organic material is fixed and the solvent evaporates; then in preparation for forming a subsequent NLO material layer of a second type, after fixation and solvent evaporation: adjusting the applied electric field to perform one of: switching its polarity, changing its magnitude and switching it off; followed by forming an NLO material layer of the second type by repeating the above, and then repeating to build up a succession of alternating NLO material layers of the first and second type.

94. A display panel comprising: a screen piece; a display piece set in an area portion of the screen piece, the display piece containing a nonlinear optical, NLO, material that has a phase matched region which is phase matched to first and second selected combinations of first and second input beam frequencies, and to a visible product beam frequency equal to the sum of the selected combinations.

95. The display panel of claim 94, wherein the NLO material is one of a birefringent material for true phase matching, a polar material for quasi-phase matching and a ferroelectric material for quasi-phase matching.

96. The display panel of claim 94 or 95, wherein the NLO material is arranged in a succession of alternating first and second NLO material layers, wherein the first and second NLO material layers have respective values of a second order nonlinear coefficient in the direction perpendicular to the layers which differ from one another to form the phase matched region with a grating having a period selected to provide said phase matching.

97. The display panel of any one of claims 94 to 96, wherein the display piece comprises a middle piece, which is made of the NLO material and hosts the phase matched region, as well as upper and lower pieces made of said NLO material that sandwich the middle piece.

98. The display panel of claim 97, wherein the middle piece has a thickness at least in the phase matched region of less than or equal to one of: 2.0 mm, 1 .5 mm, 1 .0 mm and 500 micrometres.

99. The display panel of claim 97 or 98, wherein the NLO material is periodically poled in the phase matched region whereas the NLO material in the remainder of the middle piece, and of the upper and lower pieces, is single domain.

100. The display panel of claim 97, 98 or 99, wherein the middle piece is an assembly of multiple pieces arranged extending laterally in the sandwich.

101. The display panel of any one of claims 94 to 100, wherein the screen piece forms a lens or visor which is transparent across the visible range, wherein the screen piece is shaped and sized to cover an eye or a pair of eyes of a wearer at a vertex distance of less than one of 30, 40, 50, 60, 70, 80, 90 and 100 mm, and wherein the display piece is transparent across the visible range.

102. The display panel of claim 101 , wherein said area portion is positioned away from a wearer’s line of sight for normal forward vision.

103. The display panel of claim 101 or 102, wherein the area portion is positioned in the screen piece above or below a wearer’s line of sight for normal forward vision.

104. The display panel of claim 103, wherein the area portion is positioned to lie outside a vertical arc of one of plus / minus 5, 10, 15 and 20 degrees of the line of sight for normal forward vision, for example, in an angular range of the upper or lower peripheral vision for normal forward vision which is greater than plus / minus 30 degrees of the line of sight for normal forward vision.

105. The display panel of any one of claims 101 to 104, wherein the area portion is positioned in the screen piece to the left or right of a wearer’s line of sight for normal forward vision.

106. The display panel of claim 105, wherein the area portion is positioned to lie outside a horizontal arc of one of plus / minus 5, 10, 15 and 20 degrees of the line of sight for normal forward vision.

107. A display apparatus comprising: a first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth; a second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth; a display panel according to any one of claims 94 to 106, wherein interfaces are formed between the display piece and the screen piece in which it is set; and input beam routing components (222, 240) arranged to introduce the first and second input beams into the display piece so that they enter into the display piece and traverse the display piece in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume (110) where the first and second input beams mix to generate a product beam having the sum frequency.

108. The display apparatus of claim 107, wherein the input beam routing components are configured such that the first and second input beams enter into the display piece through one or more of those interfaces.

109. The display apparatus of claim 108, wherein the interfaces through which the first and second input beams enter into the display piece are provided with anti-reflection coatings that take account of one or more of: the first and second input beam frequencies; an angle between the first and second input beams and the interfaces; and the refractive indices of the screen piece and the display piece at the interfaces.

110. The display apparatus of any one of claims 107 to 109, wherein over at least a portion of the interfaces there is provided a roughened surface on one or both of the screen piece and the display piece to cause visible light scatter and inhibit glancing angle specular reflection.

111. The display apparatus of any one of claims 107 to 110, wherein over at least a portion of the interfaces there is provided a visible light absorbing material to absorb light that would otherwise refract or reflect from the interfaces.

112. The display apparatus of any one of claims 107 to 111 , further comprising: a controller (13) operable responsive to input of image information to render synthetic scene images on the display panel by amplitude modulating at least one of the first and second input beams and by phase modulating at least one of the first and second input beams to set each of amplitude, wavevector and wavefront radius of curvature of the product beam.

113. A window panel comprising: a display panel which is transparent across the visible range and which is set in an area portion of the window panel, the display panel containing a nonlinear optical, NLO, material that has a phase matched region which is phase matched to first and second selected combinations of first and second input beam frequencies of non-collinearly incident first and second input beams, and to a visible product beam frequency equal to the sum of the selected combinations.

114. The window panel of claim 113, wherein the NLO material is one of a birefringent material for true phase matching, a polar material for quasi-phase matching and a ferroelectric material for quasi-phase matching.

115. The window panel of claim 113 or 114, wherein the NLO material is arranged in a succession of alternating first and second NLO material layers, wherein the first and second NLO material layers have respective values of a second order nonlinear coefficient in the direction perpendicular to the layers which differ from one another to form the phase matched region with a grating having a period selected to provide said phase matching.

116. The window panel of any one of claims 113 to 115, wherein the display panel is one of flat and curved.

117. The window panel of any one of claims 113 to 116, having a minimum width and / or height dimension of one of: 30 cm, 40 cm, 50 cm, 60 cm, 70 cm, 80 cm, 90 cm and 100 cm.

118. The window panel of any one of claims 113 to 117, wherein interfaces are formed between the display panel and the window panel in which it is set, and wherein the input beam routing components are configured such that the first and second input beams enter into the display panel through one or more of those interfaces.

119. The window panel of claim 118, wherein the interfaces through which the first and second input beams enter into the display panel are provided with anti-reflection coatings that take account of one or more of: the first and second input beam frequencies; an angle between the first and second input beams and the interfaces; and the refractive indices of the lens or visor piece and the display panel at the interfaces.

120. The window panel of claim 118 or 119, wherein over at least a portion of the interfaces there is provided a roughened surface on one or both of the display panel and the lens or visor piece to cause visible light scatter and inhibit glancing angle specular reflection.121 . The window panel of any one of claims 118 to 120, wherein over at least a portion of the interfaces there is provided a visible light absorbing material to absorb light that would otherwise refract or reflect from the interfaces.

122. A display apparatus comprising : a window panel according to any one of claims 113 to 121 ; a first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth; a second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth; and input beam routing components (222, 240) arranged to introduce the first and second input beams into the window panel so that they enter into the display panel and traverse the display panel in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume (110) where the first and second input beams mix to generate a product beam having the sum frequency.

123. The display apparatus of claim 122, further comprising: a controller (13) operable responsive to input of image information to render images on the display panel by amplitude modulating at least one of the first and second input beams and by phase modulating at least one of the first and second input beams to set each of amplitude, wavevector and wavefront radius of curvature of the product beam.

124. The display apparatus of claim 123, wherein the window panel has a proximal surface facing towards a viewing region and a distal surface facing away from the viewing position.

125. The display apparatus of claim 124, wherein the controller is operable to render the images in an image plane located at an offset distance from the distal surface.

126. The display apparatus of claim 124 or 125, further comprising : an image acquisition device to capture images of the left and right eyes of a person in the viewing region, and wherein the controller varies its setting of wavevector responsive to the images captured by the image acquisition device.

127. The display apparatus of claim 126, wherein the controller varies its setting of wavefront radius of curvature responsive to the images captured by the image acquisition device.

128. A light field generating apparatus comprising: a first light field source for providing a first input light field of a first frequency; a second light field source for providing a second input light field of a second frequency; a first phase modulator arranged to phase modulate the first input light field; a controller operable to control the first phase modulator; a piece of nonlinear optic, NLO, material containing a region that is phase matched to first and second selected combinations of the frequencies of the first and second input light fields, and to a frequency equal to the sum or difference of the selected combinations, wherein the first and second input light fields are directed to cross each other in the phase- matched region in a noncollinear geometry where the first and second input light fields mix to generate a product light field having the sum or difference frequency.

129. The light field generating apparatus of claim 128, further comprising: a second phase modulator arranged to phase modulate the second input light field, wherein the controller is further operable to control the second phase modulator.

130. The light field generating apparatus of claim 128 or 129, further comprising: wherein the controller is further operable to control the first and second light field sources.

131. The light field generating apparatus of claim 128, 129 or 130, further comprising: a device arranged to measure a characteristic of the product light field, wherein the controller is connected to receive the measured characteristic and is operable to vary control of at least the first phase modulator in a feedback control loop, so as to bring the characteristic of the product light field into closer conformity with a desired characteristic.

132. The light field generating apparatus of claim 131 , wherein the controller is operable to vary control of both the first and second phase modulators in the feedback control loop, so as to bring the characteristic of the product light field into closer conformity with a desired characteristic.

133. The light field generating apparatus of claim 131 or 132, wherein the characteristic is one or more of: product light field intensity profile; product light field phase profile; a light field quality parameter; and product light field power.

134. The light field generating apparatus of any one of claims 128 to 133, wherein the first and second input light fields for a light field pair and a plurality of such pairs are provided, all pairs exploiting the same sum or difference frequency mixing, wherein the piece of NLO material is rod shaped with a regular polygonal cross-section having an even number of side surfaces, wherein the input light fields of each pair are incident on opposed ones of the side surfaces and mix in the manner of the first-mentioned light field pair to generate a plurality of product light fields at the same frequency each lying in a common axis and each propagating in the same direction along that common axis.

135. A light field generating system, comprising a series arrangement of multiple stages of at least two of the light field generating apparatus according to any one of claims 128 to 134, wherein the series arrangement is such that the product light field of a preceding stage is incident on the NLO material of the subsequent stage coaxially with the product light field generated in the next stage, wherein the NLO material of the or each subsequent stage is transparent to the product light field frequency of the or each preceding stage.

136. The light field generating system of claim 135, wherein the product light field from the final stage has a frequency higher than both of the input light fields of the first stage.

137. The light field generating system of claim 135, wherein the product light field from the final stage has a frequency equal to the product light field of the first stage.

138. A contact lens having a proximal surface to be placed in contact with the eye and a distal surface to form an interface with the air, the contact lens incorporating a display panel containing a nonlinear optical, NLO, material that has a phase matched region which is phase matched to first and second selected combinations of non-collinear first and second input beam frequencies, and to a visible product beam frequency equal to the sum of the selected combinations.

139. The contact lens of claim 138, wherein the NLO material is one of a birefringent material for true phase matching, a polar material for quasi-phase matching and a ferroelectric material for quasi-phase matching.

140. The contact lens of claim 138 or 139, wherein the NLO material is arranged in a succession of alternating first and second NLO material layers, wherein the first and second NLO material layers have respective values of a second order nonlinear coefficient in the direction perpendicular to the layers which differ from one another to form the phase matched region with a grating having a period selected to provide said phase matching.141 . The contact lens of any one of claims 138 to 140, having a diameter in the range 13 mm to 15 mm.

142. The contact lens of any one of claims 138 to 141 , further comprising: at least one input beam port arranged on the distal surface for coupling in first and second input beams at the first and second input beam frequencies into the NLO material so that the first and second input beams traverse the display panel in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume (110) where the first and second input beams mix to generate a product beam having the sum frequency.

143. The contact lens of claim 142, wherein the at least one input beam port comprises one input beam port through which both of the first and second input beams are coupled in.

144. The contact lens of claim 142, wherein the at least one input beam port comprises first and second input beam ports through which the first and second input beams are coupled in respectively.

145. The contact lens of claim 142, 143 or 144, wherein the at least one input beam port comprises a flat surface through which at least one of the first and second input beams is refractively coupled into the first and second paths.

146. The contact lens of claim 142, 143 or 144, further comprising at least a first diffraction grating with a period matched to one or both of the first and second input beam frequencies that is disposed in the at least one input beam port so as to redirect at least one of the first and second input beams into the first and second paths.

147. A pair of contact lenses according to any one of claims 138 to 146.

148. A display apparatus (12) for displaying synthetic scene images responsive to input of image information, the display apparatus comprising: a contact lens according to any one of claims 142 to 146; and a wearable headset comprising: a first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth; a second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth; input beam routing components (222, 240) arranged to direct the first and second input beams to the at least one input beam port; and a controller (13) operable responsive to the input of the image information to form the synthetic scene images by amplitude modulating at least one of the first and second input beams and by phase modulating at least one of the first and second input beams to set each of amplitude, wavevector and wavefront radius of curvature of the product beam.

149. The display apparatus of claim 148, wherein the wearable headset further comprises at least one inwardly facing image acquisition device to capture images of the contact lens arranged on the wearer’s eye, wherein the display apparatus has access to computing resource configured to process the captured images to track the location of the at least one input beam port, and the controller is operable to direct the first and second input beams onto the at least one input beam port responsive to the location of the at least one input beam port as determined by the computing resource.

150. The display apparatus of claim 148 or 149, further comprising an additional contact lens according to any one of claims 5 to 9 to form a pair with the first-mentioned contact lens, wherein the headset is further adapted to form synthetic scene images in a coordinated manner on the contact lens pair for the left and right eye of a wearer.

151. A display apparatus (12) for displaying synthetic scene images responsive to input of image information, the display apparatus comprising: a first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth; a second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth; a display panel (88) containing nonlinear optical, NLO, material (100) that is phase matched to first and second selected combinations of the frequencies of the first and second input beams, and to a frequency equal to the sum of the selected combinations; input beam routing components (222, 240) arranged to introduce the first and second input beams into the display panel so that they traverse the display panel in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume (1 10) where the first and second input beams mix to generate a product beam having the sum frequency; and a controller (400) operable responsive to the input of the image information to form the synthetic scene images by amplitude modulating at least one of the first and second input beams and by phase modulating at least one of the first and second input beams to set each of amplitude, wavevector and wavefront radius of curvature of the product beam, wherein the display panel comprises a notch filter (147) to filter out the sum frequency, the notch filter being arranged in front of the NLO material layer to stop emission of scattered light from the product beam passing out of the display panel in an outward direction.

152. The display apparatus of claim 151 , wherein the sum frequency is in the visible range, and the first and second frequencies are in the infrared range.

153. The display apparatus of claim 152, wherein the display panel, including the NLO material, is transparent across the visible range.

154. The display apparatus of claim 153, wherein the image information includes images of the natural scene and the controller is configured to process the natural scene images so that the synthetic scene images reproduce at least part of the natural scene images and superpose said at least part onto the natural scene thereby to enhance said at least part of the natural scene as viewed through the display panel.

155. The display apparatus of claim 154, wherein the controller is configured to process the natural scene images to identify one or more elements in the natural scene images and to form the synthetic scene images to include said one or more natural scene elements andto superpose them onto their corresponding natural scene elements thereby to enhance those natural scene elements as viewed through the display panel.

156. The display apparatus of claim 154, wherein the controller is configured to form the synthetic scene images to include a reproduction of the natural scene and to superpose the reproduction onto the natural scene thereby to increase perceived brightness of the natural scene as viewed through the display panel.

157. The display apparatus of any one of claims 151 to 156, wherein the display panel comprises a dimming layer (140) that is controllable to provide a variable amount of attenuation to visible light, the dimming layer being arranged in front of the NLO material and the controller being further operable to adjust the amount of attenuation provided by the dimming layer.

158. The display apparatus of claim 157, further comprising an ambient light sensor (55) arranged and configured to deliver an ambient light signal to the controller indicative of ambient luminance, wherein the controller adjusts the amount of attenuation provided by the dimming layer responsive to the ambient light signal.

159. The display apparatus of claim 157 or 158, wherein the dimming layer is selected from the group of: a photochromic material layer, an electrochromic material layer and a liquid crystal layer.

160. The display apparatus of any one of claims 151 to 159, wherein the display panel comprises a front filter layer (130) arranged in front of the NLO material layer, the front filter layer being opaque to the first and second frequencies and transparent to the sum frequency, to block emission of light from the first and second input beams out of the display panel in an outward direction.161 . The display apparatus of any one of claims 151 to 160, wherein the display panel comprises a light blocking layer (134) comprising an array of pixels which are individually addressable by electric control lines in order to switch the pixels between a first state which is opaque to visible frequencies and a second state which is transmissive to visible frequencies, so that natural scene light can be blended out of selected areas of the light blocking layer.

162. The display apparatus of any any one of claims 151 to 161 , wherein the display apparatus is for displaying a synthetic scene image in colour and comprises a plurality of beam sources which includes the first and second input beam sources, wherein the first and second input beams form one of three input beam pairs generated by the plurality of beamsources, the first to third input beam pairs combining to generate first to third product beams with first to third sum frequencies which provide first to third primary colours.

163. The display apparatus of claim 162, wherein the notch filter is configured with first to third stop bands to filter out each of the first to third product beam frequencies.

164. A display apparatus according to any one of claims 151 to 163 incorporated in one of: a wearable headset (10) that has the display panel arranged in front of a wearer's eyes; a contact lens or a pair of contact lenses that each have the display panel integrated therein; a window panel of a structure through which a viewer observes a natural scene image.

165. A display apparatus (12) for displaying synthetic scene images responsive to input of image information which includes images of the natural scene, the display apparatus comprising: a first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth, the first frequency lying in the infrared range; a second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth, the second frequency lying in the infrared range; a display panel (88) containing nonlinear optical, NLO, material (100) that is transparent across the visible range and is phase matched to the first and second input beams and to a product beam (82) of a sum frequency that is equal to the sum of the first and second frequencies and lies in the visible range to first and second selected combinations of the frequencies of the first and second input beams, and to a frequency equal to the sum of the selected combinations; input beam routing components (222, 240) arranged to introduce the first and second input beams into the display panel so that they traverse the display panel in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume (110) where the first and second input beams mix to generate a product beam having the sum frequency; and a controller (400) operable responsive to the input of the image information to form the synthetic scene images by amplitude modulating at least one of the first and second input beams and by phase modulating at least one of the first and second input beams to set each of amplitude, wavevector and wavefront radius of curvature of the product beam, wherein the controller is configured to process the natural scene images so that the synthetic scene images reproduce at least part of the natural scene images and superposesaid at least part onto the natural scene thereby to enhance said at least part of the natural scene as viewed through the display panel.

166. The display apparatus of claim 165, wherein the controller is configured to process the natural scene images to identify one or more elements in the natural scene images and to form the synthetic scene images to include said one or more natural scene elements and to superpose them onto their corresponding natural scene elements thereby to enhance those natural scene elements as viewed through the display panel.

167. The display apparatus of claim 165, wherein the controller is configured to form the synthetic scene images to include a reproduction of the natural scene and to superpose the reproduction onto the natural scene thereby to increase perceived brightness of the natural scene as viewed through the display panel.

168. A display apparatus according to any one of claims 165 to 167 incorporated in one of: a wearable headset (10) that has the display panel arranged in front of a wearer's eyes; a contact lens or a pair of contact lenses that each have the display panel integrated therein; a window panel of a structure through which a viewer observes a natural scene image.

169. A video conferencing system for supporting a video conference at its location taking place between its local location and at least one remote location and involving at least one participant at each location, the video conferencing system comprising: at least one image capture device configured to capture image data including the or each local participant; a transceiver for transmitting and receiving video conferencing data between the locations via a data communication link; and a display apparatus comprising: a first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth; a second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth; a display panel (88) containing nonlinear optical, NLO, material (100) that is phase matched to first and second selected combinations of the frequencies of the first and second input beams, and to a frequency equal to the sum of the selected combinations (82); input beam routing components (222, 240) arranged to introduce the first and second input beams into the display panel so that they traverse the display panel in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume (110) where the first and second input beams mix to generate a product beam having the sum frequency; and a display controller (13) operable responsive to the video conferencing data to form images on the display panel by amplitude modulating at least one of the first and second input beams and by phase modulating at least one of the first and second input beams to set each of amplitude, wavevector and wavefront radius of curvature of the product beam.

170. The system of claim 169, having access to computing resource configured to: generate a three-dimensional, 3D, environment model for the video conference.171 . The system of claim 170, having access to computing resource configured to: process the image data to determine the position(s) of the local participant(s) relative to the display panel; and assign a position in the 3D environment model for the or each local participant; and assign a position in the 3D environment model for the or each remote participant, wherein the display controller is configured to form images of remote participants on the display panel for the or each local participant which have a perspective determined from the local participant position for whom the images are being formed to the or each remote participant position to be contained in the images being formed.

172. The system of claim 171 , wherein the display controller controls the display apparatus to form images specific to the or each local participant by directing different images to each said local participant position via wavevector control.

173. The system of claim 171 , wherein the display controller controls the display apparatus to form images specific to each eye of each local participant by directing different images to each eye of the or each said local participant position via wavevector control.

174. The system of claim 171 , 172 or 173, having access to computing resource configured to: process the image data to generate a 3D local environment model; and incorporate the 3D local environment model into the 3D environment model.

175. The system of claim 171 , 172 or 173, wherein the 3D environment model is virtual.

176. The system of any one claims 169 to 175, having access to computing resource configured to: process the image data to generate a 3D participant model of each local participant; and receive a 3D participant model for the or each remote participant.

177. The system of claim 176, wherein the 3D participant model includes the head.

178. The system of claim 177, having access to computing resource configured to: estimate a gaze direction of the or each local participant from their 3D participant model, and wherein the display controller is configured to form images on the display panel for the or each local participant responsive to their gaze direction.

179. The system of any one of claims 169 to 178, further comprising at least one additional data capture device configured to capture additional data that is combinable with the image data.

180. A computing resource for rendering images for a video conference taking place between at least two video conferencing systems according to any one of claims 169 to 179, the computing resource being configured to: host a three-dimensional, 3D, environment model for the video conference; store a participant position to associate each participant with a particular placement in the 3D environment model;store a 3D participant model for each participant; and render images for each participant of one or more of the remote participants, said images being rendered with a perspective determined from the participant position of the participant for whom the images are being rendered to the or each remote participant position to be contained in the images being rendered.181 . The computing resource of claim 180, wherein said rendering takes account of a gaze direction determined from the 3D participant model in the 3D environment model.

182. The computing resource of claim 180, wherein said 3D environment model incorporates for the at least one remote participant a 3D local environment model of the remote participant’s location, and wherein said rendering of said at least one remote participant shows them in their 3D local environment model.

183. The computing resource of any one of claims 180 to 182, wherein said rendering generates different images for the left and right eyes of the participant for whom the images are being rendered taking account of the difference in perspective between that participant’s left eye and right eye.

184. The computing resource of any one of claims 180 to 183, wherein said rendering computes for each image values of amplitude, wavevector and wavefront radius of curvature for a display controller associated with a display apparatus at each video conferencing location.

185. A display apparatus (12) for displaying images responsive to input of image frames, the display apparatus comprising: a first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth; a second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth; a display panel (88) containing nonlinear optical, NLO, material (100) that is phase matched to first and second selected combinations of the frequencies of the first and second input beams, and to a frequency equal to the sum of the selected combinations (82); input beam routing components (222, 240) arranged to introduce the first and second input beams into the display panel so that they traverse the display panel in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume (110) where the first and second input beams mix to generate a product beam having the sum frequency; an amplitude modulator (120) arranged in the first input beam and comprising an array of pixels, the pixels being controllable individually to amplitude modulate the first input beam on a pixel-by-pixel basis; a phase modulator (122) arranged to phase modulate the second input beam; and a controller (13) operable to form the images by controlling the amplitude modulator (122) and the phase modulator (122) to amplitude and phase modulate the first and second input beams to set each of amplitude, wavevector and wavefront radius of curvature of the product beam, wherein the controller (13) controls the amplitude modulator (122) on a pixel- by-pixel basis to imprint a succession of the image frames onto the first input beam and the controller (13) controls the phase modulator (122) to set the wavefront radius of curvature of the second input beam, thereby to set the wavefront radius of curvature of the product beam.

186. The display apparatus of claim 185, further comprising: a first focusing arrangement (240) placed in the first input beam to form an object in the plane of the display panel.

187. The display apparatus of claim 185, further comprising: a first focusing arrangement (240) placed in the first input beam to form a virtual object at infinity.

188. The display apparatus of claim 185, further comprising: a first focusing arrangement (240) placed in the first input beam to form a virtual object in a plane lying at a finite offset from the display panel.

189. The display apparatus of claim 185, further comprising:a first focusing arrangement (240) placed in the first input beam, the first focusing arrangement having a variable focal length which is variable under control of the controller (13).

190. The display apparatus of claim 189, wherein the focal length of the first focusing arrangement (240) is switchable by the controller between at least two of: forming an object in the plane of the display panel; forming a virtual object at infinity; and forming a virtual object in a plane lying at a finite offset from the display panel.

191. The display apparatus of claim 189 or 190, wherein the controller (13) is configured to set the focal length of the first focusing arrangement (240) to set the range of angles of light of the first input beam that form the image frame and thereby define apparent size of the image frame.

192. The display apparatus of any one of claims 185 to 191 , wherein the phase modulator (122) comprises a second focusing arrangement (928) placed in the second input beam (76), the second focusing arrangement having a focal length that is variable under control of the controller (13) to provide whole beam variation of the wavefront radius of curvature of the second input beam.

193. The display apparatus of claim 192, wherein the focal length of the second focusing arrangement (928) is variable with a response time of less than or equal to 40 milliseconds.

194. The display apparatus of claim 192 or 193, further comprising a third focusing arrangement placed in the second input beam, the third focusing arrangement consisting of a pair of lenses or mirrors of equal focal length arranged at a distance from each other of twice their focal lengths.

195. The display apparatus of any one of claims 185 to 194, wherein the phase modulator (122) comprises a spatial light modulator that is configurable under control of the controller to provide spatial phase modulation across the second input beam, thereby to provide areaspecific variation of the wavefront radius of curvature of the second input beam.

196. The display apparatus of any one of claims 185 to 195, wherein the controller is operable to control the phase modulator (122) to set the product beam’s wavefront radius of curvature to a constant value for all pixels of any given image frame.

197. The display apparatus of claim 196, wherein the constant value is set so that the object plane for a viewer lies at or closely adjacent the physical plane of the display apparatus.

198. The display apparatus of any one of claims 185 to 197, wherein the controller is operable to control the phase modulator (122) to set the product beam’s wavefront radius of curvature to different values for different areas of an image frame to provide the illusion of different apparent depths for different areas of the image frame.

199. The display apparatus of any one of claims 185 to 198, wherein the display apparatus is viewable from a range of viewing angles and further comprises a forward-facing camera having a field of view directed to capture images within the range of viewing angles, wherein the controller is configured to process the camera images to identify a viewer of the display and locate their head position.

200. The display apparatus of claim 199, wherein the controller is further configured to narrow the range of viewing angles of image frames being displayed through wavevector control such that product beam light is directed to the head position.201 . The display apparatus of claim 200, wherein the wavevector control is performed by controlling the phase modulator on a frame-by-frame basis.

202. The display apparatus of claim 199, 200 or 201 , wherein the controller is further configured to process the camera images to identify left and right eye locations of the viewer, and to serve different images of each image frame to the left and right eyes to provide a three-dimensional effect.

203. A display apparatus (12) for displaying images responsive to input of image frames, the display apparatus comprising: a first beam source (16) for providing a first input beam (70) of a first frequency and bandwidth; a second beam source (18) for providing a second input beam (76) of a second frequency and bandwidth; a display panel (88) containing nonlinear optical, NLO, material (100) that is phase matched to first and second selected combinations of the frequencies of the first and second input beams, and to a frequency equal to the sum of the selected combinations (82); input beam routing components (222, 240) arranged to introduce the first and second input beams into the display panel so that they traverse the display panel in first and second paths that cross each other in the NLO material in a noncollinear geometry to define an intersection volume (110) where the first and second input beams mix to generate a product beam having the sum frequency; an amplitude modulator (120) arranged in the first input beam; a phase modulator (122) arranged in the second input beam; anda first lens or mirror and a second lens or mirror that together form a lens pair and are arranged between the phase modulator and the NLO material, wherein, along the optical axis of the second input beam, the phase modulator, the lens pair and the NLO material are in a 4f imaging system configuration with the phase modulator being positioned one focal length from the first lens or mirror, the first and second mirrors being positioned two focal lengths apart, and the NLO material being positioned one focal length from the second lens or mirror.

204. The display apparatus of claim 203, further comprising: a controller (13) operable to form the images by controlling the amplitude modulator (122) and the phase modulator (122) to amplitude and phase modulate the first and second input beams to set each of amplitude, wavevector and wavefront radius of curvature of the product beam, wherein the controller (13) controls the phase modulator (122) to set the wavefront radius of curvature of the second input beam, thereby to set the wavefront radius of curvature of the product beam.

Images