AUGMENTED REALITY (AR) GLASSES AND METHOD FOR INCORPORATED VIRTUAL IMAGES INTO AN IMAGE VISIBLE TO A WEARER OF THE GLASSES THROUGH AT LEAST ONE LENS

Description

[1] Area of application

[0001] The invention relates to AR (Augmented Reality) glasses that are transparent like normal glasses, but can additionally overlay virtual objects or images in high resolution onto the real image visible through the lenses. Specifically, the invention relates to AR glasses that project a resolution-dynamic light field into the pupil of the wearer by placing a plurality of fiber optic ends directly in front of the pupil of the wearer. The fiber optic leads are guided through the lens invisibly from the outside by an optical camouflage device and can be distributed via optical switches. SLM processors and transmission matrix-transformed images are coupled into the fiber optics behind the wearer's ear or head so that the fiber optic ends emit a perfect partial image in front of the pupil. [2]State of the art

[0002] There are various systems known for the functional design of AR glasses. Roughly speaking, there's the older approach of reflecting a small display through the lens or within a prism. This approach makes the frame for the display excessively large or the visible area quite small, and is not discussed further here (e.g., Google Glass or Meta 2).

[0003] Another newer approach is the use of flat waveguides, which allow the image to emerge from a disc through diffraction gratings. However, this technology only offers a very limited viewing angle and poor visibility (e.g., Hololens or Magic Leap). Some newer, as yet unimplemented approaches generate a light field from several grid-like sections of the overall image. These partial images are projected (as in some patented approaches by Magic Leap) from the side into the lens and then into the eye through a grid of fast-switching mirror prisms (optical gates) or through diffraction gratings (waveguides with embedded diffraction gratings). Magic Leap has also proposed using an oscillating single-mode optical fiber as a particularly small image projector.

[0004] Another system (UNC's Pinlight Display) uses an SLM (Spatial Light Modulator) to direct a wavefront onto a grid of roughened dots on the lens. When viewed up close, the roughened dots appear like the screen of a light field, but are illuminated from outside.

[0005] There are known 3D monitors that use eye tracking to project a computer-generated hologram into the viewer's eyes. SLMs also create partial holograms, which are then integrated into a complete hologram.

[0006] US 2014 / 003762 A1, considered the closest prior art, discloses an optical device comprising a two-dimensional waveguide array composed of multiple waveguides arranged in rows and columns, which can be used in augmented reality glasses. The 2D waveguide array can be controlled either in series or in parallel, with the latter involving multiple multimode optical fibers connecting an RGB light source or an intensity modulator to the waveguide array. 2D images can be angle-encoded using a scanning projector system or by coupling a 2D microprojector to a pinhole.

[0007] US 2015 / 015879 A1 discloses the generation of an image from a pixel matrix using a single fiber optic projector with a multimode fiber optic cable, but does not suggest the production of an integral image from a grid of individual projections arranged side by side and one above the other, nor does it propose use in augmented reality glasses.

[0008] US 2011 / 227487 A1 discloses a display device in which light is transported via a planar waveguide and then coupled out of the waveguide pixel by pixel, but no light modulation takes place at the input end of the waveguide. [3]Disadvantages of the state of the art

[0009] The designs presented so far are far from looking like standard glasses with the smallest possible frames. However, market experience with Google Glass clearly confirms that even the slightest anomaly compared to standard glasses poses a problem with market acceptance if the glasses are intended for daily use. Individual facial expressions and eye contact are often too socially sensitive to be distorted by futuristic technology. Furthermore, the glasses have poor pixel resolution, limited transparency, do not focus like real objects, offer only a limited field of view, and shine a lot of light into the eyes. [4] Advantages and object of the invention

[0010] The advantages and objectives of the invention are to model a highly minimalistic pair of frameless glasses (1) with a nose bridge and side arms attached directly to the lenses and to integrate a nearly invisible optical system (10) into it, which projects a highly resolved, integral light field into the eyes (at least 20K) (48), invisible from the outside. The extent of the light field is limited only by the field of view of the lenses, so that with a suitable design, a field of view of 220 degrees is also possible. The lenses have an LCD darkening layer at the front. In addition, at least two video cameras are installed on the eyes and two video cameras in the field of view (53).

[0011] The lenses can also be made as normal optician lenses.

[0012] Any technology that adds a little extra bulk can be housed in a portable unit (3) or at the end of the ear hook behind the ear and head.

[0013] The image of the projected light field can, if necessary, provide the adjusted focus for each viewed object to replicate the eye-vergence-focus relationship, while displaying the focused objects at the highest possible retinal resolution (47) without having to turn the head. The patent demonstrates that building "mainstream" AR glasses is absolutely feasible. However, this also requires a paradigm shift toward "open source" operating systems that address AR privacy nightmares with blockchain constructions of absolute trust, making AR glasses the ultimate blockchain application.

[0014] The invention is defined by the claims. [5]Research keywords

[0015] Pinhole Projector, Integral Imaging, Optical Cloaking Device, Multi Mode Fiber Endoscope, Computer Generated Holography CGH, Foveated Rendering, Vergence Accommodation Conflict, Time Domain Imaging, Amplitude and Phase Complex Modulating SLM, Magic Leap, Hololens, SLM, DMD, DLP, FLCOS, Augmented Reality AR, Mixed Reality MR, Matrix Optics, Specles Reduction, Piezo FTIR, Electochromic Mirror, louvre glasses, [6]Sketches List

[0016] The invention will be explained in more detail below with reference to several embodiments shown in the drawings. Fig. 1 AR glasses overview with MicroElectroMechanical Systems (MEMS) technology or switching mirror Fig. 2 Conceptual representation of the overall system Fig. 3 Conceptual representation of multimode fiber optic projector Fig. 4 AR glasses resolution dynamic integral image Fig. 5 Gradient optics camouflage channels in the lens Fig. 6Projection planes for focus and ambience in the lens Fig. 7 Conceptual representation of a two-stage projector Fig. 8 Conceptual representation of a single-stage projector Fig. 9 AR glasses calibration machine Fig. 10 Conceptual representation of transmission pattern measurement [7]Detailed description

[0017] As shown in the drawing, for the AR glasses, at least 300 fiber optic projection points (48), each with a low-resolution partial image (51), are fed into a hologram projector (3) placed at the back of the head, consisting of several SLM light processors. The supply fiber optics are positioned as desired for projection on the glasses through the temples (9) and the rim (62). This results in various approaches (34) for how the projections can be placed directly in front of the eye in a grid, thus projecting a multitude of partial images or holograms (33) directly into the pupils. The line variant (2) places the projection points on the rim of the glasses and uses quickly switchable mirror surfaces (19) within the lens. The preferred point variant (1) described here uses gradient camouflage optics (12, 69).The glass fibers in the lens are routed through an optical camouflage tube (12), so that the fibers, or the MEMS switch systems (10, 70) attached to them, are invisible from all sides (66). The light guides can thus be projected into the eye in a grid directly in front of the eyes in the lens (59).

[0018] From the wearer's perspective, the individual projections merge into a high-resolution overall image (48) that fills the field of vision. The 3D impression is created by the stereoscopic separation of the projectors by the right and left lenses. Nevertheless, a holographic image can be delivered to each eye, on which the eye can focus seamlessly from far to very close. Only in this way, in conjunction with the dynamic calculation of the 3D images according to head movements (gyro sensors) and the display of the corresponding blackout LCD mask (31, 67), is a perfect holographic image blended into reality. Due to the very small fiber optic projection points (43), they even deliver an image with a particularly high depth of field, so that the projector only needs to adjust the focus for very close objects. Furthermore, the lens is in no way more opaque than normal glasses (66).Conversely, some fibers are also used in parallel or exclusively as cameras (25, 53). For each fiber, a computer transforms the image section by the individual transmission matrix (23, 36), which was previously measured during calibration (87) for each fiber (92) and each point (102). An SLM (Spatial Light Modulator) (24) generates a light front (39) from this, which couples not only the precise luminous intensity distribution but also the phase shift distribution into the fiber (41). Only through this light front conversion, including phase distribution, can a single multimode fiber (42) be used as a light channel and projector for one primary color at a time (44). Various methods have been described for projecting the images, which use only one SLM (83) or several SLMs (76, 77) in series to save computational effort (78, 79).One method consistently relies on computer-generated holograms and effective holographic grayscale generation (the holo-projector). A second method consistently relies on the computational application of the various matrices for transmission and focusing (matrix projector), and requires specially optimized SLMs with special illumination to generate the phase shifts. AR applications typically only depict a few objects within the wide field of view (46). The projector dynamically controls only the fiber optic projectors that also contain objects. Through eye tracking (53), only the currently focused projectors (47) are addressed in higher resolution. Thus, most projection points are only supplied with a low resolution (45), because the human perspective does not perceive the environment in high resolution anyway.Due to the strong eyeball movements (71), at least two parallel integral imaging grid planes are used for focus and ambient projections. These planes allow for different pixel densities through different projection angles (74, 75) and are precisely cropped to the pupil to avoid distracting eye glare. Only through this massive reduction in resolution and workload, and the extensive use of pre-calculated modular diffraction patterns or image patterns, is it possible to realize the imaging process today with the fastest SLM light processors and parallel graphics processors. The disruptive speckle effect is filtered out through a whole range of techniques, primarily through slightly incoherent laser light during projection.

[0019] MEMS fiber optic switches (10) in the camouflage tubes reduce the number of fiber optic strands (8), thereby increasing the thickness and resolution of the fibers. Technical optimizations are being made to the SLM chips, enabling more optimal hologram or matrix projection for the AR glasses. By illuminating the SLM chips with several different light sources, or entire illumination holograms, and using DMD mirrors that can be tilted in multiple directions, the projections for each coupling point can be greatly optimized. This allows grayscale to be generated within a single imaging cycle for the extremely fast SLM light processors. The 24,000 to 32,000 cycles per second of the DMD chips can then build up the complete image sequentially for each individual projection of the integral light field per eye (e.g. 150), and for each RGB color component, and at a frame rate of 60 Hz.In practice, however, only a much smaller image area is used for normal AR applications.

[0020] For further details and variants of the technology, we refer to German patent application No. 10 2017 003 336.6, the content of which is incorporated into the present patent application by reference. [8]PV Point variant (11)

[0021] This variant takes its name from the tiny fiber optic tips, which protrude just far enough from the optical camouflage to cast only a vague shadow. If the fiber optic tip is moved completely into the camouflage area, nothing at all can be seen. The appeal of this variant is the perceived magical effect of the image appearing in the glass, literally out of nowhere. Technically, this variant is particularly appealing because it allows for full utilization of the possibilities and degrees of freedom offered. Stealth effect:The technical basis here is the stealth effect (66), which allows a thin cavity (64) to be rendered optically invisible in a dense optical medium (68). This is achieved by means of ring-shaped, steeply decreasing refractive index layers (69) around the cavity. The surrounding optical medium must have a relatively high refractive index (at least 1.7), and the final layer at the camouflaged area must have only a minimal refractive index (maximum 1.5). The light rays are guided perfectly around the cavity (66). Because humans always interpret light as being straight, the cavity is not perceived.

[0022] A perfect camouflage body is a corresponding tube (12), but even more complex spaces such as an oval are perfectly camouflaged from all sides.

[0023] Particularly many glass fibers:For the point variant, a large number of glass fibers are wrapped around the rim of the glasses and distributed into a grid of projection points ending directly in the lens (11). To access the interior of the lens, several camouflage tubes are guided side by side through the lens (12). The glass fibers are threaded into the glass tubes and bend within the camouflage tube with a radius of approximately 1-2 mm (64) before being directed through a hole in the tube directly toward the pupil (60). The glass fibers have an optic at the end which, in combination with the properties of the camouflage tube, emits a projection cone toward the pupil. This system allows the fiber projections to be directed precisely in the desired direction (59).

[0024] The dense grid of projectors is combined into a coherent image using the principle of near-field projection (48). The technique of projection cropping (50, 51) then transforms the overlapping images into a perfectly composite image. Curved tube and rim: The camouflage tubes are bent to follow the curve of the lens. To camouflage a cable duct around the lens, the same technique is used with a camouflage half-shell (61).

[0025] The brackets can be attached by drilling a hole between two tubes. More projection levels:An important requirement for a design variant is the ability to accommodate multiple projection planes. This is necessary due to the strong eye movements involved in depicting the ambience and is described in a further paragraph below. A second and third grid can be inserted in parallel, providing different projection angles. The projection unit then controls which plane is used depending on the eye position.

[0026] Use of MEMS technology: In advanced implementation using MEMS technology, a complex rod-shaped microsystem (10,70) is housed in the camouflage tubes, which is manufactured using MEMS, wafer-level optics and nano-optics printing methods, and which is composed of light guides (14) with deflection mirror switches and projection optics (16) as well as control electronics (13). [9]SV stroke variant (2)

[0027] This variant takes its name from the fine grooves between individual glass bodies, which, however, are not visible in practice due to the bonding.

[0028] Switchable mirror: The technical basis here is a method that makes it possible to make the layer between two glass bodies reflective, or at least partially reflective, at high frequencies (19). A fiber optic projector (18) positioned on the edge of the glasses can then direct its projection onto this mirror surface (21) toward the pupil. By connecting several such switching mirrors and appropriate circuitry, a single projector can generate multiple images consecutively and from different directions. A mirror layer extends across the entire width of the glass, allowing multiple projectors to project images for a Y coordinate in parallel. This creates a light field grid of individual images in front of the eyes. Possible switching mirror methods are: SV6 Frustrated Total Internal Reflection FTIR: A total internal reflection, or just a frustrated TIR, is generated by expanding a gap using a PIEZO element, which forms the gap by expansion or vibration. SV8 Electrochromic Mirror: The system of electrochromic mirrors is used as a mirror.

[10] Projector mounting options

[0029] PA1: Depending on the degree of miniaturization, the computing unit can be designed directly on the glasses (3) or as a separate unit. The connection can then be wireless or wired.

[0030] PA2: The projector and computing unit can be placed in one unit behind the head.

[0031] PA3:The projector and / or processing unit can be placed in a single unit in a pocket or belt on the body. The processing unit alone can also be carried wirelessly in a handbag or simply within the user's vicinity, within a range of up to 10 meters.

[11] Variations of light field projection clipping

[0032] The integral light field creates circular projections (50) that overlap. The overlaps must be clipped during image rendering (51).

[0033] PB1: Eye tracking allows us to precisely capture the direction of gaze and calculate the clipping frames.

[0034] PB2: In addition, the pupil width is recorded by the eye-tracking camera (53) and the clipping frames are adjusted.

[0035] PB3: Smooth transitions: The edges of the clipping are cut slightly, seamlessly and softly, so that the projections fit together with less tolerance. [ 12] Cover mask functions

[0036] The AR glasses are equipped with a high-resolution dimming LCD display (67) at the front, which covers the entire field of vision through the lenses. Background coverage for non-transparent virtual objects: The main function of dimming is to make displayed virtual objects opaque when needed, thus masking reality. For this purpose, a corresponding mask is calculated for each virtual object.

[13] Environmental detection and object detection

[0037] Local AI servers are used for pattern recognition. The environment is scanned with the environmental cameras (53), and all objects are assigned to known logical objects as closely as possible using neural networks (deep learning). A digitally interpreted world is created in the computer, which the user interface can access. The user interface can then function in an object-related manner.

[14] Variants of camouflage optics

[0038] The camouflage optics can be incorporated into a dense glass medium in various ways. The shape doesn't have to be a tube, but can also take on more complex forms while still providing consistent camouflage, such as the following: TO1 Round shape: The camouflage channel has a perfect cylinder shape (12). TO2 Oval shape: The camouflage channel has a perfect oval shape. TO3 shell shape: At the edge of the lens, the camouflage optics have a bowl shape (61) that can be filled from the edge. TO4 With bend: In addition to their uniform shape, the shapes are curved to varying degrees or evenly. TO5 Whole grids: The shapes are formed into nets or grids with corresponding branches or intersections. TO6: Curves at junctions: The sharp transitions at junctions are softly rounded.

[0039] The camouflage function is achieved by the uniform redirection of light, which is created by a multitude of different layers (69) with decreasing optical density around a cavity. These gradients can be designed in various ways: GR1 Different refractive indices: The layers have different and coordinated refractive indices. GR2 Various dispersion values: The layers have different and coordinated dispersion values to increase the optical consistency of the camouflage. GR3 Very thick or thin layers: The layers are designed to be quite thin or thick. GR4 Unevenly sloping: The layers do not decrease uniformly, but exponentially or at variable rates in their density (refractive index) or thickness. GR5 steps or stepless: The layers are arranged in steps relative to one another, or the layers flow seamlessly into one another in terms of their density.

[15] Variations of the camouflage piercing

[0040] To project from within the camouflage, it may be necessary to pierce the cloak (piercing) (69). This can be done in several ways: TP1 drilling: A hole (69) is drilled at the appropriate locations and angles. TP2 drilling depth: The correct optical behavior is set by the depth of the hole. TP3 Drilling shape: The behavior is changed by the shape of the hole. The shape can be a sphere, cone, slot, wedge, or other shape. TP4 Filling the borehole: The borehole is filled or sealed with another optical medium. TP5 Inner bore: The holes are drilled at an intermediate stage in the buildup of the layers, sealed, and the buildup continues. Thus, the drilling only works from the inside.

[16] Camouflage optics layer structure and piercing pressure

[0041] To make the camouflage optics suitable for mass production, an entire grid with a frame is built up layer by layer. A UV-curing polymer layer or another material with a corresponding refractive index is applied. To create an optical piercing at the projection points of the camouflage tubes, a different polymer with a different refractive index is sprayed onto these areas using a "bubblejet"-like printing technique. This polymer displaces the previously applied layer at this point. The layer is then UV-cured, and the next layer is applied. The sequence of steps and the printing material can vary.

[0042] PD1 print before layer application: The pierced areas are printed before the entire coating is applied. The printing material can also contain an anti-flow agent to prevent the coating from shrinking in these areas.

[0043] PD2 print after layer application:It is printed after the total layer application.

[0044] PD3 Only doping agent printed: Only a substance that changes the refractive index is injected.

[0045] PD4 pressure before and after visual application: A combination of pre- and post-printing is used, which may also include intermediate curing of the layer.

[0046] PD5 Further correction prints: Additional areas on the layers are printed to make optical corrections that could otherwise be disrupted by the piercing prints.

[17] Variants of camouflage inlays

[0047] The interior of the camouflage tubes can be designed differently depending on the current state of technology. The main task is to ensure the correct positioning and bending of the projection in a way that protects the light guides. Switching and branching can also be accommodated in higher technology versions. Projection and recording systems can also be installed.

[0048] TI1 Structure:A more complex support structure is manufactured in which the glass fibers are guided and other optical elements are attached.

[0049] TI2 optic inlay: A narrow microchip is manufactured with wafer-level optics-generated waveguides, deflection optics, and lenses, which generates precisely the desired projection point and provides the lines at the edge of the glasses. A narrow IC with MEMS switches and optics is then housed at the edge of the glasses, which distributes the fiber optic signal to these lines.

[0050] TI3 MEMS board (70): It uses an extremely narrow and long microchip that integrates not only electrical circuits but also numerous MEMS-DMD micromirrors as optical switches, as well as optical conductors and lenses. The chip can be mass-produced using IC, MEMS (13), and wafer-level optics (15) technology and refined using nano-polymer optics printers.

[18] Description and variants of the projection planes

[0051] The light field projection consists of a grid of fiber optic projectors that are mounted in the lens relatively close to the eye. The pupil in the eyeball (58) is located relatively close to the lens and therefore shifts relatively strongly due to the rotation of the relatively large eyeball (71). This strong eye movement occurs right / left as well as up and down. The position of the target is therefore strongly dependent on the direction of gaze. This is not a problem for a conventional display because a normal screen images each pixel with a high viewing angle. The light from a screen, even when reflected by a mirror surface on the lens, shines relatively brightly in all directions and illuminates the entire eye area.

[0052] Light field fundamentally different:The individual projections of a light field, on the other hand, project an entire image from a single point toward the pupil, like a film projector. This is a fundamentally different principle for how the image is projected into the eyes. The projection angle (74) is limited.

[0053] Depending on its thickness, the fiber optic cable only transports a limited resolution image, which is then brought to the desired exit angle of 20° to 40° (74.75) at the end of the fiber.

[0054] These projection points only become a round image due to the relatively wide pupil in the human eye. The exit angle of the projection determines the resolution density of the image.

[0055] To obtain the highest possible resolution image in the currently focused area, the projectors must provide a particularly narrow-angle projection in the light field in this direction (59). To obtain a low-resolution image of the surroundings in the out-of-focus area, the projectors can provide a wide-angle projection in the light field in this direction (60), so that the projections can be used for the widest possible viewing angle of the pupil.

[0056] Because of these sometimes small projection angles, the light field must be precisely adjusted to the eye movements of the eyeball.

[0057] These circumstances also justify the need to use at least two complete projection field planes in parallel: one plane for the focused areas (47), and one for the image of the remaining unfocused ambience (46).

[0058] Thanks to the MEMS optical switches located directly at the projection points and optimized light guides using wafer-level optics, there are no limits to the number of projection points used for different resolutions. The number of fiber optic cables required is drastically reduced, making the bundle very thin. The individual fibers, however, can be much thicker and achieve higher resolutions.

[19] Variants of field of view resolution reduction

[0059] The theoretical pixel resolution for high-resolution AR glasses that cover the entire field of view easily reaches 20 to 40 megapixels. Fortunately, however, in practical use, an AR hologram projection is fundamentally different from a conventional image projector or monitor.

[0060] AR 1 Only existing objects: In augmented reality, the entire field of view is almost never filled with virtual objects. Usually, only isolated objects (46) are superimposed into reality.

[0061] AR 2 Focused area only HQ: A second aspect of practical AR is the fact that a person can only look at one point at a time and only perceives the details there in high resolution and with focus (47). The rest of the environment is perceived at a much lower resolution (45). An AR hologram projector always knows exactly where and to what depth the viewer is looking thanks to eye tracking. With this information, the projector only needs to project a very small portion at the highest resolution. The remaining projectors can simply use lower resolution with far fewer frame cycles. This again results in enormous frame rate savings. The types of resolution reduction can be implemented for the ambience in the form of lower frame rates, pixel resolution, color depth, and focusability.

[0062] AR 3 Lower frame rate in the focus area:A third aspect is that the repetition rate for focused, relatively still objects needs to be much lower than for fast-moving objects. The human eye only detects flicker in the periphery and is quite slow in the focused area (47).

[0063] Technically, the refresh rate in the focused area can be reduced and the resolution can be slightly reduced during fast movement.

[0064] AR 4 Focus adjustment only in the focus area: The possibility of actually focusing on the objects was only possible in the focus area.

[20] Matrix holo and hybrid projectors (35)

[0065] A1 Matrix Projector:The transformation matrix TM is not applied optically by a transformation hologram, but purely computationally in the computer. The SLM is used in an orthodox manner to position this image in front of the fiber optic ends. This avoids the complexity of computer-generated holograms and requires only an optimized SLM with illumination system (83).

[0066] B1 Hybrid Projector: A standard pixel grid from an orthodox SLM is used as the image. However, for the application of the fiber optic transformation matrix, a second SLM surface is used as a CGH, thus projecting a real hologram in front of the eyes (76, 77). This is the simplest implementation that can be achieved because only minimal additional computing power and know-how are required for CGH (Computer Generated Holography) (78, 79).

[0067] B2 Hybrid Projector:The procedure is the same as in B1, but the eye focusing of the objects is still used. The second SLM not only transforms the TM for the fiber optics, but also transforms the image depth of the individual pixels. The information for this is taken from the Z-buffer of the upstream calculated 2D image.

[0068] C1 Hologram Projector:While in the B variant, an image is calculated directly from pixels using traditional OpenGL methods, the image calculations of the C variant can generate the hologram data, i.e., the interference patterns, directly from the models, which are then transferred to the SLM. Only after the SLM is irradiated with the appropriately modulated RGB color component lasers does the coupling light front emerge. A single SLM (82) can be used to perform imaging and fiber optic transformation in a single step. However, this requires either multiplying the transformations or calculating the hologram (84) at high frequency.

[21] Light sources and speckle reduction

[0069] E1 Reduced coherence: The laser light source is modulated in one or more different aspects to a tolerable extent for the holographic process. 1. The light wavelength is modulated into a correspondingly limited, mixed spectrum. 2. The phase shifts are evenly mixed. (Moving diffusers or multimode fibers) 3. The angle of the beams to each other (the parallelism) is mixed within a correspondingly limited range.

[0070] These modulations can be achieved by moving diffusers, piezo benders, acousto-optical modulators or passive transmission through multimode optical fibers.

[0071] E3 Fiber optic pass: The system-specific passage of the wavefront through a multimode optical fiber up to the pupil causes a coherence phase mixing which strongly suppresses speckle formation.

[0072] E4 Constant hologram variation:If a diffraction pattern generation method is used for the image content that allows for multiple different ways to produce the same result, the image generation can continuously use different pattern variants. A random generator can select the next variant. The speckle patterns would only appear in one cycle, while the desired images would remain in each cycle.

[0073] E5 Piezo displacement: The coherence is reduced by piezo-vibrating mirrors or diffusers.

[22] Splitting field, description and variants

[0074] The AR glasses are based on a dense grid of fiber optic projection points placed close to the eyes. The larger the active field of view (maximum 220 degrees horizontally) and the higher the required virtual point resolution, the more fiber optic ends are used. However, depending on the number of SLM projectors used and the holographic parallel use of the projectors, only a few of these fiber optics (2-32) are used in parallel. These fiber optics must therefore be selected at a high frequency. The abstract basis for this is the optical switch, and ultimately a mix of different switch shapes is used.

[0075] Coupling field as first switch stage:In AR glasses, the coupling field (28) not only couples the images into the fiber ends, but also performs the function of selecting the fiber ends. This selection is already taken into account in the image calculation during hologram projection, making it highly effective yet deeply integrated into the calculation methods. However, the coupling field (28) is limited in the number of selectable fibers due to the need for the largest possible optics per fiber.

[0076] Second switch stage: The fiber optics from the coupling field can be branched in various ways on their way to the projection exit points to achieve the final number of required projections. This can be achieved using individual switches or a tree structure. However, switches with multiple branches can also be used.

[0077] Active crossover:Active switches that completely divert the light flux are best. However, some technologies only achieve just under 100%, meaning the residual light is either tolerable or must be blocked by an additional barrier layer.

[0078] MEMS technology: With MEMS fabrication technology, the design can be optimized to such an extent that the number of fibers installed can be significantly reduced. This is achieved by placing the switches just before the exit points.

[0079] Switch and angle: Due to the particularly small design of MEMS or micro-optics, a combination of active switching and 90-degree deflection can be implemented directly in the camouflage tube (16). This kills two birds with one stone, because the 90-degree deflection is also a critical point. The massive savings in fiber optic cables allow the fibers to be much thicker, thus transmitting much higher-resolution images.

[0080] Combination of ambience and focus fibers: If the fiber optics are combined in a solid configuration, it is also possible to combine the fiber optics for focus and ambient projectors. These projector types differ only in the input and output optics. The more complex input optics for focus projectors can also be used for simpler ambient projectors.

[0081] Fiber merging: To combine Focus and Ambiente projectors, the two fiber optic cables can also merge into one.

[0082] Switch techniques: Various physical methods can be used to set the switch: WT1 Piezo Actuators: Optics are physically shifted relative to each other. This allows entire fields to be moved extensively using piezo actuators. WT2 FLCD: The selection can be made by simply opening and closing FLCD shutters (27). WT3 FTIR:Adjustable frustrated or full internal reflection FTIR. WT4 Electrochromic Mirror: Switching with electrochromic layers. WT5 MEMS: Micromechanical turning, lowering or sliding of mirrors or other elements. WT6 DMD: Tilting DMD mirrors (16) as a whole field or as a large mirror. Switch trees: By combining several switch systems, many branches can be realized.

[0083] WB1 Cascading switch: A switch type can be connected multiple times in series to create multiple deviations on a single line (10). This is particularly effective if the switch type can efficiently route signals at one end of the switch.

[0084] WB2 Tree branching: All switch ends are further branched.

[23] Special switch shapes in the camouflage tube inlay

[0085] The AR glasses can cascade the final switches onto the camouflage tube inlays, which are manufactured using MEMS and / or wafer-level optics technology (10). The projection optics can be further refined, for example, using a UV-cured nano-gradient polymer printer. The switch can also perform a rotation of approximately 90 degrees. The following design variants are advantageous for this: DMD mirror with decoupling and recoupling optics:

[0086] The switch function is performed by a large array of DMD mirrors. The mirrors can be tilted into various positions.

[0087] A single mirror can have a focusing bowl shape, or the mirror array can also perform a desired focusing.

[0088] Depending on the tilt direction, the focus may be different.

[0089] The output can be used as the final projection directly through focusing or via an additional lens. The mirrors do not need to be controlled individually, but only collectively.

[0090] The wavefront is then redirected into a coupling optic.

[0091] By expanding the system and using a larger waveguide, the wavefront can be trimmed to a less steep angle. The wavefront can be guided to the DMDs by a steep total reflection mirror. The outcoupled wavefront, however, can pass through the mirror due to the steeper angle.

[24] Refined fiberglass tip

[0092] The ends of the glass fibers can be coated with a fine lens optic using nano-optical polymer printers. These printers can also print gradient optics and expand and shape the fiber aperture in such a way that the complex light wave flow is not disrupted, thus increasing the resolution of the overall system, which is limited by the aperture size. In this case, there is no need to worry about geometry preservation, since the geometry is reproduced by the transmission matrix.

[25] Hologram-optimized FLCOS and DMD systems

[0093] The well-known SLM systems were primarily developed for conventional video projection. They can also be used effectively for CGH applications.

[0094] Nevertheless, for AR glasses, the SLM system should be opted for CGH application and fiber optic coupling, since only a single perspective of the wavefront is used and the rest, unlike with normal projectors, is irrelevant. KF2 complex modulating FLCOS SLM

[0095] To achieve independently adjustable amplitude and phase modulation, two thick FLCD layers are used, each capable of rotating the polarization in opposite directions. Pure phase modulation can thus be achieved by equally activating both layers. The polarization of the light is first rotated in one direction and then reversed exactly. The polarization remains the same in the end. Only the refractive index of the two layers, and thus the phase shift, changes depending on the activation strength.

[0096] Different activation strengths of the two layers also generate a polarization rotation and thus an amplitude modulation. KD2 Various light sources

[0097] By using different light sources from different directions, many different amplitude strengths can be generated by combining the light sources accordingly.

[0098] The different light sources can also offer different phase shifts to generate coupling patterns with different phases. KD3 Binary Phase Modulating MEMS-SLM

[0099] A simplified variation allows the micromirror to switch between only two phase shift positions, such as 0 and π of the used wavelength. As a third state, the mirror can be completely tilted to additionally function as a binary amplitude modulator.

[0100] CM1: Multiple direction tilting micromirrors: The mirrors can tilt not only in two directions but in more directions to capture even more different phases or amplitudes from the illumination light front.

Claims

1. Augmented reality (AR) spectacles for mixing virtual images into an image that is visible for a wearer of the spectacles through at least one spectacle glass, comprising a plurality of optical fiber projectors (59, 18) made of multimode optical fibers (42, 17) which project a virtual image with a large depth of field into at least one eye of the spectacle wearer, wherein the virtual image is an integral image of a grid of individual projections (48) generated by the optical fiber projectors (59, 18) arranged besides one another and above one another, which, when viewed from close proximity, for the spectacle wearer merge into an overall projection, wherein each of the individual projections (48) is a section (37) of the virtual image, wherein at an entry end of each optical fiber (42, 17) a computer-controlled spatial light modulator (SLM) (24) or another imaging system is mounted, for which a computer calculates a complex pixel pattern, which is determined by the transformation of the original image through a transmission matrix (36) of the optical fiber (42, 17), wherein an optics couples (41) a wave front supplied by the spatial light modulator (SLM) as a hologram or as a complex projection with amplitudes and relative phase shifts (39) into the optical fiber (42), and wherein in a point variant, for generating the individual projections, a large number of multimode optical fibers is routed around the spectacle rim and is distributed to a grid of projection points arranged directly next to one another and above one another in the spectacle glass through camouflage tubes, wherein the optical fibers (42) have an optical system at their end and thus the optical fibers form projectors and thus in combination with the properties of the camouflage tubes, the wave front coupled into the individual multimode optical fiber is emitted in the direction of the pupil in a projection cone as a light signal, or wherein in a line variant, a plurality of multimode optical fibers (17) having respective optical fiber projector ends (18) are directed to the spectacle rim for generating the individual projections (48) and the individual wave front coupled into the multimode optical fiber (17) exits the respective optical fiber projector end (18) as a light signal and propagates in the spectacle glass, until the light signal is directed onto the eye from the spectacle glass through one of the fast-switching mirror surfaces (19), which are switched to be reflective, from the spectacle glass into the eye.

2. Augmented reality (AR) spectacles according to claim 1, characterized in that for the point variant, the optical fiber projectors (42) or adjoining extending micro-optical systems are routed (64) through the camouflage channels in the spectacle glass directly to an optimum position for the integral image grid, wherein a gradient optics deflects (66) light that passes through the spectacle glass around the channels and makes the channels invisible from the outside, wherein the gradient of the refractive index of an optical medium of the gradient optics decreases (69) in the direction of the channels from the surrounding high index to a lower index.

3. Augmented reality (AR) spectacles according to claim 1, characterized in that for the line variant, a plurality of optical fiber projectors (18) are positioned around a rim of a spectacle glass and are directed from there into the spectacle glass, and in that the spectacle glass is provided with a plurality of rapidly switchable mirror surfaces (19) located besides one another or above one another, which, when they are switched to be reflective, deflect (21) the projection and direct it out of the spectacle glass onto the eye.

4. Augmented reality (AR) spectacles according to claim 1, characterized in that for the point variant, entire micro-optical systems (10) are hidden in the camouflage channels, which additionally integrate IC switching logics, and / or MEMS micro-mechanics or DMD mirror fields and / or 3D-printed gradient micro-optics (70, 15).

5. Augmented reality (AR) spectacles according to any one of the preceding claims, characterized in that projection points of the integral image are positioned in at least two grid planes (47, 46) in parallel, wherein different exit optics, different exit angles and thereby different resolutions are provided for each plane.

6. Augmented reality (AR) spectacles according to any one of the preceding claims, characterized in that the overlaps of the individual projections arranged besides one another and above one another can be trimmed in a resolution-efficient manner matching each other (51) by choosing the distances of the integral projection points from one another (11) in relation to an optimum or minimum pupil size of the spectacle wearer (58), and / or in that the exit optics of the projection points are formed in such a way, that the projection generates an effective useful square shape, and in that at least one eye camera (53) is provided, which measures the size of the pupils.

7. Augmented reality (AR) spectacles according to any one of the preceding claims, characterized in that the SLM (24) is mounted outside the facial field in a hologram projector (3) that is positioned at the back of the head, and in that the SLM (24) is illuminated by at least one light source (22), and in that a light front generated by the SLM (24) is incident onto a further SLM surface (76) or directly onto a coupling-in optics (26), and in that at least one optical fiber with a preceding coupling-in optics is placed in a coupling-in field (28), and in that the fiber is guided through the eyeglass side arm (9) or on another path to the spectacle glass, where the projection exit point, which may have further optics, or light guide extensions with optical switches and micro-optics, or whole chip systems with IC logic (10) and MEMS elements in the form of switches and micro-optics, is positioned in at least one point (18).

8. Augmented reality (AR) spectacles according to any one of the preceding claims, characterized by DMD chip SLM optimized for AR optical fiber projection for the utilization of multiple or complex light sources, which can tilt or adjust a plurality of micro-mirrors on the DMD chip in additional directions and positions respectively, and / or has various additional mirror or diffraction structures on the individual mirrors, in order to use different light sources from different directions for the projection.

9. Augmented reality (AR) spectacles according to any one of the preceding claims, characterized by a FLCOS (ferroelectric liquid crystal on silicon) SLM optimized for AR optical fiber projection with mirrors located thereon, which comprise a plurality of independently adjustable FLCD (ferroelectric liquid crystal display) layers in order to independently set grey values and phase shifts.

10. Augmented reality (AR) spectacles according to any one of the preceding claims, characterized by optical fibers for the transmission of light and / or vibrating piezo-electrically moved mirrors or diffusers for reducing the coherence of the light source.

11. Augmented reality (AR) spectacles according to any one of the claims 7 to 10, characterized by coupling-in field (28) having a plurality of coupling-in optics as the first stage, that is used like an optical fiber multi-switch, wherein the in the holographic projector (3) the holograms themselves control the respective coupling-in optic, whereas the optical fibers that are not in use can be closed by preceding FLCD diaphragms (27) or other diaphragms.

12. Augmented reality (AR) spectacles according to claim 11, characterized in that behind the coupling-in field (28) active optical fiber single switches or successively switching switch systems enable the control of the projection points (10), which are switched by MEMS elements, or by means of DMD mirrors or electrochromic mirror or piezo actuators.

13. Augmented reality (AR) spectacles according to any one of the preceding claims, characterized in that a gradient index fiber enlargement or constriction is mounted on the optical fiber ends for the entry and exit of the wave fronts, which makes it possible to generate a higher resolution or better point light sources.

14. Method for mixing virtual images in augmented reality (AR) spectacles into an image visible for a wearer of the spectacles through at least one spectacle glass, wherein by a plurality of optical fiber projectors (59, 18) made of multimode optical fibers (42) a virtual image having a large depth of field is projected into at least one eye of the spectacle wearer, wherein the virtual image is an integral image of a grid of individual projections (48) arranged besides one another and above one another, generated by the optical fiber projectors (59, 18) which, when viewed from close proximity for the spectacle wearer merge into an overall projection, wherein each of the individual projections (48) is a section (37) of the virtual image, wherein for a computer-controlled spatial light modulator (SLM) (24) or another imaging system that is mounted at an entry end of each optical fiber (42) a complex pixel pattern is calculated by a computer, which is determined by the transformation of the original image through a transmission matrix (36) of the optical fiber, wherein a wave front supplied by the spatial light modulator (SLM) (24) as a hologram or as a complex projection with amplitudes and relative phase shifts (39) is coupled into (41) the optical fiber (42) by means of an optics (40), and wherein in a point variant, for generating the individual projections, a large number of multimode optical fibers (42) is routed around the spectacle rim and is distributed to a grid of projection points arranged directly next to one another and above one another in the spectacle glass through camouflage tubes, wherein the optical fibers have an optical system at their end and thus the optical fibers form projectors and thus in combination with the properties of the camouflage tubes, the wave front coupled into the individual multimode optical fiber (42)is emitted in the direction of the pupil in a projection cone as a light signal, or wherein in a line variant, a plurality of multimode optical fibers (17) having respective optical fiber projector ends (18) are directed to the spectacle rim for generating the individual projections (48) and the individual wave front coupled into the multimode optical fiber (17) exits the respective optical fiber projector end (18) as a light signal and propagates in the spectacle glass, until the light signal is directed onto the eye from the spectacle glass through one of the fast-switching mirror surfaces, which are switched to be reflective, from the spectacle glass into the eye.

15. Method for mixing virtual images according to claim 14, characterized in that in a first calibration process (87) for all of the routed multi-mode optical fibers (92) the light-scrambling property (96) of the fiber is measured separately for all color components and all possible transmission paths (102) and from that, mathematical transformation matrices (98) are statistically calculated, or the measurements are stored in other data forms and are further optimized for optimum use, in that in the normal projection process, the images or diffraction patterns required for an SLM are assembled and calculated by a computer from the image to be displayed itself and from the stored transformation matrix or from the other stored and optimized transformation information (84), in that the wave front (39) originating therefrom is generated by an SLM in a fast sequence in a time domain in the different color components and for the required projection points and is coupled into the respective optical fiber, and in that possibly further switches (10) or mirror surfaces (19) at the spectacle glass are set accordingly in order to direct the light to the desired exit points.

16. Method for mixing virtual images according to any one of claims 14 to 15, characterized in that for the point variant, the optical fiber projectors (42) or adjoining extending light channels or optical systems are routed (64) through the camouflage channels in the spectacle glass directly to an optimum position for the integral image grid, wherein a gradient optics deflects (66) light passing through the spectacle glass around the channels and makes the channels invisible from the outside, wherein the gradient of the refractive index of the optical medium (69) is constructed from a low index in the interior to a higher index towards the outside, up to the index of the surrounding glass, in layers or without transition.

17. Method for mixing virtual images according to any one of claims 14 to 16, characterized in that projection points of the integral image in at least two grid planes (47, 46) are controlled in parallel (74, 75), in that an area focused by the spectacle wearer is captured by an eye camera (53), and in that it is determined which high-resolution projection points and which low-resolution projection points have to be addressed, wherein different resolutions and color depths can be rendered for each plane.

18. Method for mixing virtual images according to any one of claims 14 to 17, characterized in that the overlaps of the individual integral-image projections are trimmed to match each other (51) by determining the pupil size and / or the ambient light intensity by means of the eye camera and sensors and are incorporated into the calculation of the clipping frames, and in that during the image rendering the protruding image areas are rendered in a hard manner, or not at all with a soft transition.

19. Method for mixing virtual images according to any one of claims 14 to 18, characterized in that the generation of the images is primarily realized by holographic methods, that is to say by means of computer-calculated complex diffraction patterns, which are displayed on the SLM, illuminated with laser light or other light and thus generate a complex wave front, which adjust both amplitudes and phase shift, wherein by illuminating the SLM with multiple light sources or more complex light fronts and by the use of optimized SLM constructions, the quality and grey scale depth of the holograms for the optical fiber projector can be enhanced.

20. Method for mixing virtual images according to any one of claims 14 to 19, characterized in that the generated images are implemented primarily by means of simple pixel graphics, that is to say by direct transmission of the calculated complex amplitude and phase pattern, however by using optimized SLM constructions and more complex illuminations, with which then likewise amplitude and phase shift-differentiating projections are generated.

21. Method for mixing virtual images according to any one of claims 14 to 20, characterized in that when using an optical fiber projector optimized DMD SLM, together with a respective multiple or complex light source suitable for this purpose, the calculated SLM pattern assembles an optimized mixture of the given light sources, in order to generate an optimum amplitude and phase shift pattern at the coupling-in point.

22. Method for mixing virtual images according to any one of claims 14 to 21, characterized in that for a low-resolution optical fiber projection 3 SLM cycles for the 3 color components are sufficient, since the grey values per cycle are determined by the hologram, the multiple light sources, or by the transit pattern (96) per pixel, as a result of which it is possible with a fast SLM to handle the large number of ambient projections.