Method for calibrating a replication limit of a beamlet replication-based virtual retinal display, beamlet replication-based virtual retinal display and augmented reality glasses

The method for calibrating beamlet replication-based virtual retinal displays using a pupil tracking device simplifies the complex calibration of AR glasses, enabling user-friendly, cost-effective, and precise alignment, addressing the challenges of existing AR glass calibration processes.

WO2024231087A1PCT designated stage expired Publication Date: 2024-11-14ROBERT BOSCH GMBH
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2024/060940
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-08
Filing Date
2024-04-22
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

The calibration of augmented reality (AR) glasses based on the free space combiner principle is complex, requiring precise measurement and alignment, especially for beamlet replication systems, which is difficult to achieve without professional expertise and results in high production costs and user dissatisfaction.

Method used

A method for calibrating the replication limit of a beamlet replication-based virtual retinal display using a pupil tracking device, allowing users to self-calibrate by determining the first pupil position, projecting beamlets and their replicas into a partial area of the eyebox, and adjusting image content and distortion settings to achieve optimal overlap and alignment.

Benefits of technology

This method simplifies the calibration process, enabling cost-effective, user-friendly, and time-saving calibration that can be performed by the user, ensuring precise alignment and high user satisfaction, with minimal need for professional assistance and allowing for recalibration as needed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2024060940_14112024_PF_FP_ABST
    Figure EP2024060940_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The invention relates to a method for calibrating a replication limit (30) of a beamlet replication-based virtual retinal display (12) having a pupil tracking device (20), wherein, in at least one method step (38), at least a first pupil position (40) of a user's eye (10) of a user of the virtual retinal display (12) is determined, wherein, in at least one further method step (42), at least one beamlet (24) of the virtual retinal display (12) and at least one beamlet replica (26) of the beamlet (24) are repeatedly and alternately projected into a partial region (44) of an eyebox (28) of the virtual retinal display (12) corresponding to the first pupil position (40) of the user's eye (10), and wherein, in at least one calibration step (46), image content of the beamlet (24) projected at a first time point into the partial region (44) of the eyebox (28) and image content of the beamlet replica (26) projected at a second time point into the partial region (44) of the eyebox (28) are shifted relative to one another by changing the settings of the virtual retinal display (12) and / or of the augmented reality glasses (14) comprising the virtual retinal display (12), and / or wherein, in the calibration step (46), warping and / or optical distortion of the image content of the beamlet (24) projected at the first time point into the partial region (44) of the eyebox (28) and warping and / or optical distortion of the image content of the beamlet replica (26) projected at the second time point into the partial region (44) of the eyebox (28) are adapted to one another or with respect to one another by changing the settings of the virtual retinal display (12) and / or of the augmented reality glasses (14) comprising the virtual retinal display (12).
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Description

[0002] Method for calibrating a replication limit of a beamlet replication-based virtual retinal display, beamlet replication-based virtual retinal display and data glasses

[0003] State of the art

[0004] Augmented Reality (AR) glasses, in which both the real environment and virtually superimposed image content are displayed simultaneously, are well known. Depending on the technology, various combiners are used, which transmit light from the environment to the eye and overlay this with the light from artificially generated image content. In AR glasses based on the free space combiner principle, the lens (or an additional offset or integrated optical element) overlays the artificial image content with the environment by serving as a reflective or diffractive surface for the artificial image content. The overlay can arise from reflection at a refractive index transition or from diffractive structures (such as holograms). It is known that a holographic optical element (HOE) located in the lens is used as a free space combiner.Furthermore, it is state of the art to use laser light to project the artificial image content. The beam generated by the laser is designed so that it has a small beam diameter at the entrance pupil of a user's eye of the AR glasses. This small circle of confusion in the pupil plane is also called a "beamlet." A beamlet is therefore a small area, preferably in the pupil of the user's eye, through which the individual rays of all pixels pass. A beamlet contains all the image information of the image content. If the pupil of the user's eye moves due to eye movement of the user, the user only sees non-vignetted artificial image content as long as the beamlet is still within an eyebox of the AR glasses-user system.Furthermore, it is known that a hologram can contain several different diffractive combiner functions, so that within a hologram layer, instead of one beamlet, several beamlets can be generated simultaneously. These beamlets are then spatially separated copies of each other (so-called beamlet replicas). In this case, both beamlets are active simultaneously and move synchronously.

[0005] Calibrating AR glasses according to the explained free space combiner principle is usually very complex, especially since, for example, for each built sample, it is necessary to precisely measure and calibrate which mirror positions of a MEMS mirror system projecting the artificial image content correlate with pixels in the displayed artificial image content. This calibration effort is multiplied by the need to do this for all laser colors of a laser projection system of the AR glasses, e.g., for red, green, and blue, as well as for all beamlets and beamlet replicas output by the AR glasses. In particular, calibration at a boundary of a beamlet replication area must achieve a handover between the beamlets, if possible without changing the image content between the beamlet and the beamlet replica. The tolerance for this is usually only approximately 0.2 px and is therefore very complex to achieve.In addition, due to the circumstances explained above, the known systems require very precise alignment of the AR glasses to the user's eye distance, which can usually only be done by a trained specialist.

[0006] Disclosure of the invention

[0007] A method is proposed for calibrating, in particular at least a field calibration, a replication limit of a virtual retinal display (retinal scan display) based on beamlet replication with a pupil tracking device, wherein in at least one method step at least a first pupil position of a user eye of a user of the virtual retinal display, in particular a partial area of ​​an eyebox of a system comprising a virtual retinal display and a user, is determined, wherein in at least one further method step at least one beamlet of the virtual retinal display and at least one beamlet replica of the beamlet are repeatedly projected alternately into a partial area of ​​the eyebox of the virtual retinal display corresponding to the first pupil position of the user eye,wherein in particular the partial area of ​​the eyebox in the first pupil position overlaps at least with a part of the replication limit of the virtual retinal display to be calibrated, and wherein in at least one calibration step an image content of the beamlet projected at a first point in time into the partial area of ​​the eyebox and an image content of the beamlet replica projected at a second point in time into the partial area of ​​the eyebox are calibrated by a setting change of the virtual retinal display and / or data glasses having the virtual retinal display, in particular at least in one eyebox plane of the virtual retinal display,are shifted relative to each other and / or wherein, in the calibration step, a distortion and / or optical distortion of the image content of the beamlet projected into the partial area of ​​the eyebox at the first time and a distortion and / or optical distortion of the image content of the beamlet replica projected into the partial area of ​​the eyebox at the second time are adapted to each other or to each other by changing the setting of the virtual retinal display and / or the data glasses having the virtual retinal display.

[0008] The inventive design of the method advantageously makes it possible to meet the high calibration requirements for calibrating the replication limit in a particularly simple manner. Advantageously, this calibration can be carried out by the user themselves. Advantageously, cost-effective calibration is enabled. Advantageously, high user satisfaction can be achieved, in particular because the calibration can be carried out by the user themselves in a cost-effective and time-saving manner (no visit from a specialist is necessary). Advantageously, user-specific "perfect" calibration is enabled. Furthermore, recalibration can advantageously be achieved, e.g., when the device is passed on to another user or when the virtual retinal display is adjusted, e.g., due to aging effects. Advantageously, complex, precise "end-of-line" calibration during production can be dispensed with.Advantageously, a user-specific calibration can also be enabled. For example, if the user prefers to use a laser color, its robustness and resolution can be optimized. Or if the user only needs limited image content (often sees the same images), this image content can be specifically optimized using the suggested calibration.

[0009] “Data glasses” should be understood in particular to mean a wearable (head-mounted display) by means of which information can be added to a user’s field of vision. Data glasses preferably enable augmented reality and / or mixed reality applications. Data glasses are also commonly referred to as smart glasses. In particular, the data glasses have a virtual retinal display (also called a retinal scan display or light field display), which is particularly familiar to those skilled in the art. The virtual retinal display is particularly designed to sequentially scan an image content by deflecting at least one light beam, in particular a laser beam, from at least one time-modulated light source, such as one or more laser diodes of a laser projector, and to project it directly onto the retina of the user’s eye using optical elements.The image source is designed in particular as an electronic image source, for example as a graphics output, in particular an (integrated) graphics card, of a computer or processor or the like. The image data is designed in particular as color image data, e.g. RGB image data. In particular, the image data can be designed as still or moving images, e.g. videos. In particular, the laser projector unit is provided to generate the image data and output it via a visible (RGB) laser beam. In particular, the laser projector unit has RGB laser diodes which generate the visible laser beam. In particular, the laser projector unit has an infrared laser diode which generates the infrared laser beam. Preferably, the visible laser beams and the infrared laser beam are combined to form a common laser beam bundle.The infrared laser diode can be integrated into a laser diode system with the RGB laser diodes, or the infrared laser beam can be coupled via optical elements into the visible laser beam generated by a separate laser diode system. The data glasses can also comprise a pupil tracking device (eye tracking system). The pupil tracking device can be implemented in a manner known to those skilled in the art, e.g., via a camera observing the user's eyes, via an evaluation of a back reflection of an infrared laser signal, e.g., based on the so-called dark pupil effect or the so-called bright pupil effect. In particular, during the calibration of the replication boundary, the transfer of image content artificially generated by the virtual retinal display and output to the user between two beamlets, in particular between the beamlet and the beamlet replica, is optimized.During the transfer of the artificially generated image content calibrated using the method, the offset of the image content before and after the transfer should be less than 0.5 pixels, preferably less than 0.3 pixels, and most preferably less than 0.2 pixels. Preferably, the offset of the image content before and after the transfer should be so small that the user can no longer perceive it or can only perceive it with concentrated observation. In particular, during the calibration of the replication boundary, the offset of the image content of the beamlets on both sides of the replication boundary is adjusted to one another. It is conceivable that the replication boundary is a sharp line in space. In this case, there is no area in which both beamlets to be calibrated relative to one another can be located, at least at different times. Alternatively, it is also conceivable that the replication boundary is not a sharp line, but an overlapping area.In this case, the beamlets to be calibrated relative to one another can occupy an identical location, at least alternately (not simultaneously). Preferably, the replication boundary (horizontal or vertical) runs approximately in the center of the eyebox and / or the partial area of ​​the eyebox. In particular, the replication boundary (horizontal or vertical) runs approximately in the center of a pupil surface area determined by the pupil tracking device. Preferably, a "field calibration" should be understood to mean a calibration that can be performed by the user themselves, preferably without the need for complicated tools. Preferably, the field calibration can be performed at many different locations, preferably at almost any location, for example, at the user's home, in the office, in nature, in a vehicle interior, etc.

[0010] In particular, the beamlet replication-based virtual retinal display has at least one diffractive optical element (DOE), preferably a HOE, which is at least configured to replicate an output of a laser projection unit of the virtual retinal display to form the beamlet replica. In particular, the first pupil position of the user's eye corresponds to a specific viewing direction of the user wearing the virtual retinal display, preferably predetermined by a calibration procedure. Preferably, the user is instructed not to move their eyes during calibration, in particular to keep them in the first pupil position. The beamlet is in particular a beam of rays that preferably encompasses the entire artificially generated image content. The beamlet is in particular intended to enter the user's eye and preferably impinge on a retina of the user's eye.In a repeatedly alternating projection of two beamlets to a specific location, the two beamlets are preferably directed such that they repeatedly hit the location immediately one after the other. In particular, the partial area of ​​the eyebox corresponds to an area within the virtual retinal display in which the user's eye, in particular a pupil of the user's eye, must be located, so that at a first point in time the beamlet and at a second point in time the beamlet replica completely hits the pupil of the user's eye and thus enters the user's eye / is perceptible to the user. In particular, the partial area of ​​the eyebox is determined by means of the pupil tracking device based on the first pupil position of the user's eye.In particular, the eyebox sub-area is formed by an area in a pupil plane of the virtual retinal display, in which a beamlet can currently be located, so that the image content output by the virtual retinal display can reach the user's eye without vignetting at the respective pupil position. In particular, the beamlet jumps out of the respective eyebox sub-area when the beamlet jumps into the same eyebox sub-area, and vice versa.

[0011] In particular, the setting change of the virtual retinal display, which adjusts the displacement of the image content from the beamlets relative to one another and / or the distortion and / or the optical distortion of the image content in the beamlets relative to one another, can be carried out automatically, in particular controlled by software, or manually, in particular mechanically and / or optically, e.g., via an adjustable optical system / lens and / or mirror system. If carried out via software, this can take place internally or externally to the data glasses, e.g., in a cloud or on a mobile phone coupled to the data glasses. For example, for calibration, in particular for displacement / adjustment, a transformation between a projection mirror position of a MEMS system of the virtual retinal display (C.Rasterizer Image) and a displayed video image could be adjusted.In particular, the described method is intended for use in a virtual retinal display of data glasses / AR glasses. Alternatively, the described method can also be intended for use in binoculars, a telescope, a microscope, a periscope, other data glasses, such as virtual reality glasses, cycling goggles, diving goggles, motorcycle helmet visors, safety goggles, etc., and / or devices for ophthalmic examinations, each with virtual retinal displays. “Intended” and / or “configured” should be understood in particular to mean specially programmed, designed and / or equipped. The fact that an object is intended and / or configured for a specific function should be understood in particular to mean that the object fulfills and / or executes this specific function in at least one application and / or operating state.

[0012] It is further proposed that the virtual retinal display outputs at least one first calibration marker during the projection of the beamlet into the eyebox, and that the virtual retinal display outputs at least one second calibration marker, in particular one different from the first calibration marker, during the projection of the beamlet replica into the eyebox, wherein in the calibration step, at least the two calibration markers are shifted relative to one another by a setting change of the virtual retinal display and / or the data glasses, in particular by shifting the image contents and / or adjusting the distortions and / or the optical distortion of the image contents within the partial area of ​​the eyebox, preferably brought into a relative position corresponding to an optimal calibration, preferably into an at least partial overlap with one another.This advantageously enables simple and / or intuitive calibration. Advantageously, a high level of calibration accuracy can be achieved. The different calibration markers preferably have different geometric shapes. The different geometric shapes of the calibration markers preferably correspond to one another, so that an optimal relative position of the calibration markers to one another can advantageously be intuitively recognized by the user. In particular, the virtual retinal display changes the output of the calibration marker synchronously with the alternating projection of the beamlet and beamlet replica into the partial area of ​​the eyebox. The relative position of the calibration markers corresponding to an optimal calibration can be an exact or partial overlap, or else a different relationship between the calibration markers to one another.

[0013] It is also proposed that, in at least one further method step, at least the beamlet and at least the beamlet replica of the beamlet are repeatedly and alternately projected into a further partial area of ​​the eyebox of the virtual retinal display, which corresponds to a second pupil position of the user's eye that is different from the first pupil position, wherein, in particular, the further partial area of ​​the eyebox in the second pupil position overlaps at least with a further part of the replication boundary of the virtual retinal display to be calibrated. This advantageously enables particularly precise calibration. Advantageously, this allows a calibration valid for a multitude of user viewing directions to be obtained. The method is preferably repeated for additional pupil positions at the replication boundary.For example, the process is repeated at at least four, preferably at least five, and preferably at least ten positions at the replication boundary. It is also conceivable that a virtual retinal display generates more than one replication, resulting in more than one replication boundary (e.g., one vertical and one horizontal for three beamlet replicas). In this case, the process can be repeated at multiple positions at each of the replication boundaries.

[0014] If, in at least one further calibration step, in which the user's eye is preferably in the second pupil position, at least the two calibration markers are shifted relative to one another by a further setting change of the virtual retinal display and / or the data glasses, in particular by shifting the image content and / or adjusting the distortions and / or the optical distortion of the image content within the further partial area of ​​the eyebox, a particularly precise calibration can advantageously be enabled, which is valid in particular for a multitude of user viewing directions. In particular, the further calibration step proceeds at least substantially identically to the calibration step, with the important difference that the user's eye assumes the second pupil position instead of the first pupil position.For all other pupil positions calibrated in the procedure, comparable additional calibration steps are preferably carried out.

[0015] Furthermore, it is proposed that the repeatedly alternating projection of the beamlet and the beamlet replica into the respective sub-area of ​​the eyebox be generated by alternately pushing the respective beamlets into and out of the respective sub-area of ​​the eyebox, in particular by means of a MEMS mirror system of a (laser) projector unit of the virtual retinal display. This advantageously allows for a simple and / or reliable calibration of the replication limit.

[0016] Furthermore, it is proposed that, during the adjustment of the virtual retinal display and / or the data glasses performed in the calibration step, at least one adjustment of a transformation between a projection mirror position of a MEMS mirror system of a laser projector unit of the virtual retinal display and a video image of the virtual retinal display output via the beamlets be performed. This advantageously allows for a simple and / or reliable calibration of the replication limit.

[0017] It is further proposed that, in at least one further calibration step following the setting change of the virtual retinal display and / or the data glasses, the image sharpness of the virtual retinal display is readjusted. This advantageously allows optimal calibration to be achieved. Preferably, image sharpness can also be calibrated and / or maintained high even after recalibration. For example, to adapt and / or calibrate the image sharpness of the artificially generated image content, a focal length of variable lenses of an optical imaging system of the virtual retinal display can be adjustable, in particular manually by the user or automatically. For example, after a displacement or distortion / optical distortion of the calibration markers, a partial area of ​​a calibration marker, e.g., an edge of a calibration marker, can be brought into focus manually or automatically, e.g.by adjusting the focal length of the variable lenses. Alternatively, the user can be shown at least one test image / pattern different from the calibration markers, based on whose sharpness impression the focal length of the variable lens is then calibrated.

[0018] In this context, it is additionally proposed that, in the further calibration step, at least one of the calibration markers be used in addition to adjusting the image sharpness of the virtual retinal display, in particular by adjusting a focal length of a variable lens of the virtual retinal display such that a contrast and / or an edge region of at least the calibration marker is optimized. This advantageously allows for optimal individual calibration of the virtual retinal display.

[0019] It is also proposed that a frame rate of the alternating projection be selected such that the user can perceive the beamlet and the beamlet replica simultaneously. This can advantageously achieve a high level of user comfort during calibration. Advantageously, a smooth calibration display can be enabled, which in particular allows particularly precise, simple and / or fast calibration. Preferably, the frame rate is at least 16 frames per second or more. This exceeds the speed resolution of the human eye and advantageously creates a fluid image impression. Alternatively, however, a frame rate below 16 frames per second, e.g. only one frame per second or only two frames per second, is also conceivable.

[0020] Furthermore, a beamlet replication-based virtual retinal display with a calibratable, in particular field-calibratable, replication limit is proposed, with a pupil tracking device for determining a first pupil position of a pupil of a user eye of a user of the virtual retinal display, with at least one optical system comprising at least one laser projector unit with a MEMS mirror system for repeatedly alternating projection of at least one beamlet and at least one beamlet replica in a partial area of ​​the eyebox corresponding to the first pupil position of the user eye, wherein in particular the partial area of ​​the eyebox in the first pupil position overlaps at least with a part of the calibratable replication limit of the virtual retinal display, and with at least one adjustment unit,which at least leads to a shift of an image content projected at a first point in time via the beamlet into the partial area of ​​the eyebox and an image content projected at a second point in time via the beamlet replica into the partial area of ​​the eyebox by a mechanical and / or electronic adjustment process of the virtual retinal display and / or data glasses having the virtual retinal display, in particular at least in one eyebox plane of the virtual retinal display,relative to each other and / or which at least lead to a distortion and / or optical distortion of the image content projected at the first time via the beamlet into the partial area of ​​the eyebox and of the image content projected at the second time via the beamlet replica into the partial area of ​​the eyebox by a mechanical and / or electronic adjustment process of the virtual retinal display and / or the data glasses comprising the virtual retinal display. This advantageously provides a particularly user-friendly virtual retinal display and / or data glasses. A virtual retinal display and / or data glasses that can be calibrated easily, cost-effectively, and / or time-savingly can be provided. A particularly high precision of the output of the virtual retinal display and / or the data glasses, in particular via a particularly large eyebox, can be advantageously provided.

[0021] In addition, data glasses, in particular AR (Augmented Reality) glasses, with the beamlet replication-based virtual retinal display, which provides the same advantages for data glasses, are proposed.

[0022] The method according to the invention, the beamlet replication-based virtual retinal display according to the invention, and the data glasses according to the invention are not intended to be limited to the application and embodiment described above. In particular, the method according to the invention, the beamlet replication-based virtual retinal display according to the invention, and the data glasses according to the invention can have a number of individual elements, components, units, and method steps that differs from the number stated herein in order to fulfill a functionality described herein. Furthermore, in the value ranges specified in this disclosure, values ​​within the stated limits are also to be considered disclosed and can be used as desired. Drawing

[0023] Further advantages will become apparent from the following description of the drawings. The drawings illustrate an exemplary embodiment of the invention. The drawings, the description, and the claims contain numerous features in combination. Those skilled in the art will also expediently consider the features individually and combine them into useful further combinations.

[0024] They show:

[0025] Fig. 1 is a schematic representation of data glasses with a virtual retinal display,

[0026] Fig. 2 schematically shows a top view of an exemplary eyebox in an eyebox plane of the virtual retinal display with a beamlet and a beamlet replica,

[0027] Fig. 3 shows a further schematic plan view of the exemplary eyebox with the beamlet and the beamlet replica during a calibration of a replication boundary of the eyebox,

[0028] Fig. 4 is a schematic flow diagram of a method for calibrating the replication limit of the virtual retinal display,

[0029] Fig. 5 shows an exemplary image content of a calibration display of the virtual retinal display in an insufficiently calibrated initial situation and

[0030] Fig. 6 shows an exemplary image content of the calibration display of the virtual retinal display in a calibrated state.

[0031] Description of the embodiment

[0032] Fig. 1 shows a schematic representation of data glasses 14. Alternatively, an AR display device other than data glasses 14 would also be conceivable. The data glasses 14 have a virtual retinal display (retinal scanning display) 12. The virtual retinal display 12 is designed as a beamlet replication-based virtual retinal display 12. The data glasses 14 comprise a spectacle frame 62. The data glasses 14 comprise spectacle lenses 64. Fig. 1 shows an example of a user's eye 10. The user's eye 10 has a pupil 60. The user's eye 10 has a retina 58. The virtual retinal display 12 has a laser projector unit 16. The laser projector unit 16 has an optical system (not shown in detail) which comprises a MEMS mirror system. The laser projector unit 16 is configured at least to generate scanned laser beams. The laser projector unit 16 outputs a scanned laser beam 18.The scanned laser beam 18 generates an image display / image content of the virtual retinal display 12. The laser projector unit 16 is at least partially integrated into the spectacle frame 62. The spectacle lens 64 forms part of the virtual retinal display 12 by comprising part of an optical system of the virtual retinal display 12, e.g., an integrated holographic optical element (HOE) 32. The HOE 32 forms a deflection element, which is provided to redirect the laser beam 18 output by the laser projector unit 16 to the user's eye 10. The HOE 32 is configured to duplicate the image display / image content output by the laser projector unit 16. Alternatively or additionally, a segment lens or other optical duplication elements introduced into the optical path of the laser beam 18 could also be configured to duplicate the image display / image content.The laser beam 18 is focused onto the user's eye 10 by the optical system of the virtual retinal display 12, in particular by the HOE 32. The laser beam 18 is focused into a pupil plane / eyebox plane 22 of the virtual retinal display 12 by the optical system of the virtual retinal display 12, in particular by the HOE 32. In the pupil plane / eyebox plane 22, the laser beam 18 has a smallest beam diameter. A circle of confusion of the focused laser beam 18 in the pupil plane / eyebox plane 22 forms a beamlet 24, 26. By multiplying the image display / image content output by the laser projector unit 16 in the optical system of the virtual retinal display 12, a beamlet 24 and a beamlet replica 26 are formed in the pupil plane / eyebox plane 22. The beamlet 24 has a diameter which is smaller than a minimum diameter of the pupil 60 of the user's eye 10.For a vignetting-free representation of the image display / image content, all individual beams contained in the beamlet 24, 26 of all pixels of the scanned laser beam bundle 18 pass through the pupil 60 of the user's eye 10 and are imaged onto the retina 58 of the user's eye 10. A movement of the user's eye 10, e.g., when the viewing direction changes, can cause the beamlet 24 to slide outside the pupil 60. To maintain the user's perception of the image display / image content, another beamlet 26, e.g., the beamlet replica 26, must overlap with the pupil 60 in the new eye position of the user's eye 10 after the eye movement. An area within which the pupil 60 of the user's eye 10 can move so that a complete image display / image content remains visible is called the eyebox 28.

[0033] The data glasses 14 / the virtual retinal display 12 have a pupil tracking device 20 for monitoring an eye position of the user's eye 10 / a pupil position of the pupil 60 of the user's eye 10 in the virtual retinal display 12. The pupil tracking device 20 is configured to determine a first pupil position 40 of the pupil 60 of the user's eye 10 of the user of the virtual retinal display 12. The pupil tracking device 20 is configured to determine further pupil positions 56 of the pupil 60 of the user's eye 10 of the user of the virtual retinal display 12. The pupil tracking device 20 has, for example, a sensor unit 34. The sensor unit 34 can be designed as a laser feedback interferometry (LFI) sensor. The sensor unit 34 can be designed as an eye tracking camera.The sensor unit 34 can be configured to detect back reflections of an infrared laser beam (not shown) from the user's eye 10. The virtual retinal display 12 has a computer unit 36. The computer unit 36 ​​is configured at least to evaluate the detected back reflections from the user's eye 10. The computer unit 36 ​​is configured at least to track and / or monitor an eye position of the user's eye 10 and / or a pupil position of the pupil 60 from the detected back reflections from the user's eye 10.

[0034] The data glasses 14 / the virtual retinal display 12 has an adjustment unit 70. The adjustment unit 70 is configured to shift an image content projected at a first point in time via the beamlet 24 into a partial area 44 of the eyebox 28 (cf. Fig. 3) and an image content projected at a second point in time via the beamlet replica 26 into the partial area 44 of the eyebox 28 relative to one another in the eyebox plane 22 of the virtual retinal display 12 by means of a mechanical and / or electronic adjustment process. The adjustment unit 70 is configured to distort and / or optically distort the image content projected at the first point in time via the beamlet 24 into the partial area 44 of the eyebox 28 and the image content projected at the second point in time via the beamlet replica 26 into the partial area 44 of the eyebox 28 by means of a (further) mechanical and / or electronic adjustment process.

[0035] Figure 2 schematically shows a top view of an exemplary eyebox 28 of the virtual retinal display 12. The eyebox 28 shown in Fig. 2 corresponds to the area in the eyebox plane 22 in which either the beamlet 24 or the beamlet replica 26 can enter the user's eye 10 through the pupil 60 at different pupil positions 40, 56. By replicating the beamlet 24, the total size of the eyebox 28 was doubled. A first half 72 of the eyebox 28 indicates the area of ​​the eyebox 28 that can be reached by the beamlet 24. A second half 74 of the eyebox 28 indicates the area of ​​the eyebox 28 that can be reached by the beamlet replica 26. The eyebox 28 has a replication boundary 30. The two halves 72, 74 are spatially separated from each other by the replication boundary 30. The replication boundary 30 is calibratable. The replication boundary 30 is field-calibratable.When beamlet 24 is moved, beamlet replica 26 moves analogously. Both beamlets 24, 26 are active simultaneously and move synchronously (see arrows 76, 78 in Fig. 2). If pupil 60 now moves from the first half 72 of eyebox 28 across replication boundary 30 into the second half 74 of eyebox 28, a type of transfer of the image display / image content takes place. In the example shown, beamlet 24 must jump to the left edge of first half 72, so that beamlet replica 26 also jumps to the left edge of second half 74, where pupil 60, which crosses replication boundary 30, is located. The laser projector unit 16 is capable of generating this jump of beamlets 24, 26 using the MEMS mirror system.The laser projector unit 16 is also configured, by means of the MEMS mirror system, for repeatedly alternating projection of the beamlet 24 and the beamlet replica 26 into the partial area 44 of the eyebox 28. Figure 3 shows this situation as an example using the eyebox 28 and the beamlets 24, 26 moving therein. Fig. 3 also shows the partial area 44 of the eyebox 28 in the exemplary first pupil position 40. To perform a calibration of the replication boundary 30, the partial area 44 of the eyebox 28 overlaps with a part of the replication boundary 30 to be calibrated. To calibrate the replication boundary 30, the beamlet 24 and the beamlet replica 26 are displayed virtually simultaneously, with a transfer of the type described above taking place for each frame of the virtual retinal display 12. In Fig. 3, this is shown as an example for the first pupil position 40. In addition, in Fig.3, a second pupil position 56 is indicated, which corresponds to a different viewing direction of the user's eye 10 to be calibrated. In the second pupil position 56, an analogous process with a "frame-by-frame" transfer is then performed following the first pupil position 40. This is repeated for a number of additional pupil positions along the replication boundary 30, in particular until the entire replication boundary 30 of the eyebox 28 is covered by calibrations.

[0036] Figure 4 shows a schematic flow diagram of a method for calibrating, in particular field calibration, the replication limit 30 of the virtual retinal display 12 based on beamlet replication. In at least one method step 38, a first pupil position 40 of the user's eye 10 is determined. In at least one further method step 42, the beamlet 24 and the beamlet replica 26 are repeatedly projected alternately into the partial area 44 of the eyebox 28 of the virtual retinal display 12 corresponding to the determined first pupil position 40 of the user's eye 10. A frame rate of the alternating projection is selected such that the user can perceive the beamlet 24 and the beamlet replica 26 simultaneously.The repeatedly alternating projection of the beamlet 24 and the beamlet replica 26 into the partial area 44 of the eyebox 28 by alternately pushing the respective beamlets 24, 26 into and out of the partial area 44 of the eyebox 28 is generated by the laser projector unit 16, in particular by the MEMS mirror system of the laser projector unit 16. The virtual retinal display 12 has a light engine. The light engine can be part of the computer unit 36 ​​or be formed separately therefrom. The virtual retinal display 12 outputs at least one first calibration marker 48 (see Figures 5 and 6), in particular by means of the light engine, during the projection of the beamlet 24 into the eyebox 28. The virtual retinal display 12 outputs at least a second calibration marker 50, in particular by means of the light engine during the projection of the beamlet replica 26 into the eyebox 28 (cf. Figures 5 and 6).The second calibration marker 50 is configured differently from the first calibration marker 48. In the example shown in Figures 5 and 6, the first calibration marker 48 is configured as at least one circle. In the example shown in Figures 5 and 6, the second calibration marker 50 is configured as at least one cross.

[0037] Fig. 5 shows an example initial situation of a calibration display of the method. Due to a relative calibration error and / or distortion error, the calibration markers 48, 50 do not lie on top of one another. The task of the user of the virtual retinal display 12 during calibration is now to align the calibration markers 48, 50 with one another, e.g. to slide them over one another, until a situation as shown by way of example in Fig. 6 is achieved, for example by means of a control via the computer unit 36 ​​or via an external device such as a smartphone or the like. This provides the virtual retinal display 12 / the data glasses 14 with a correction of its original (Fig. 4) calibration. This calibration can be repeated as often as desired and for as many users as desired.

[0038] In a calibration step 46, an image content of the beamlet 24 projected at a first point in time into the partial area 44 of the eyebox 28 (in the example shown in Figures 5 and 6, the first calibration marker 48) and an image content of the beamlet replica 26 projected at a second point in time into the partial area 44 of the eyebox 28 (in the example shown in Figures 5 and 6, the second calibration marker 50) are shifted relative to one another by a setting change of the virtual retinal display 12 and / or the data glasses 14 in the eyebox plane 22 of the virtual retinal display 12.Alternatively or additionally, in the calibration step 46, the two calibration markers 48, 50 could be shifted relative to one another and / or adjusted to one another or to each other by an alternative or additional setting change of the virtual retinal display 12 and / or the data glasses 14, such as adjusting the distortions of the image content and / or an optical distortion of the image content within the partial area 44 of the eyebox 28. The shift, distortion, and / or distortion continues until the calibration markers 48, 50 assume a relative position corresponding to an optimal calibration.During the setting change of the virtual retinal display 12 and / or the data glasses 14 performed in calibration step 46, an adjustment of a transformation between a projection mirror position of the MEMS mirror system of the laser projector unit 16 and a video image of the virtual retinal display 12 output via the beamlets 24, 26 is performed. In at least one further calibration step 68, following the setting change of the virtual retinal display 12 and / or the data glasses 14, an image sharpness of the virtual retinal display 12 is readjusted. This can be done, for example, by adjusting a focal length of a variable lens of the optical system of the laser projector unit 16. For the readjustment, for example, a structure of one or both calibration markers 48, 50 can be brought into focus by changing the setting on the laser projector unit 16, e.g., via the computer unit 36.In this case, in the further calibration step 68, at least one of the calibration markers 48, 50 is additionally used to adjust the image sharpness of the virtual retinal display 12.

[0039] This calibration method / scheme can be repeated in further method steps 52 and calibration steps 66, which in particular also have the previously described pattern, but are performed at second and further pupil positions 56 of the user's eye 10 along the replication boundary 30. Before each iteration, a new pupil position 56 of the user's eye 10 is determined in at least one method step 80. At the end, if the user agrees to the calibration, the new calibration values ​​are set in the virtual retinal display 12 / in the data glasses 14 in at least one further method step 82.

[0040] Each time, the light engine projects the beamlets 24, 26 into the respective partial area 44, 54 of the eyebox 28, which in each case overlaps with the replication boundary 30. The user's eye 10 then perceives the calibration markers 48, 50. The user then controls the setting unit 70 (e.g., the computer unit 36 ​​or a mobile device connected to the data glasses 14, such as a smartphone) such that the image content of the beamlets 24, 26 is changed. The light engine now projects slightly adjusted calibration markers 48, 50 according to the user input. This process continues until the user indicates that there is an optimal match between the calibration markers 48, 50, e.g., the + and 0 symbols. The process then starts again at a different pupil position 40, 56 and thus also at different beamlet positions until enough positions along the replication limit 30 have been passed through and the replication limit 30 is calibrated.

Claims

Claims 1. A method for calibrating, in particular at least a field calibration, a replication limit (30) of a virtual retinal display (retinal scan display, 12) based on beamlet replication with a pupil tracking device (20), wherein in at least one method step (38) at least a first pupil position (40) of a user eye (10) of a user of the virtual retinal display (12) is determined, wherein in at least one further method step (42) at least one beamlet (24) of the virtual retinal display (12) and at least one beamlet replica (26) of the beamlet (24) are repeatedly projected alternately into a partial area (44) of an eyebox (28) of the virtual retinal display (12) corresponding to the first pupil position (40) of the user eye (10),wherein, in particular, the partial area (44) of the eyebox (28) in the first pupil position (40) overlaps at least with a part of the replication limit (30) of the virtual retinal display (12) to be calibrated, and wherein, in at least one calibration step (46), an image content of the beamlet (24) projected at a first point in time into the partial area (44) of the eyebox (28) and an image content of the beamlet replica (26) projected at a second point in time into the partial area (44) of the eyebox (28) are calibrated by a setting change of the virtual retinal display (12) and / or of a pair of data glasses (14) having the virtual retinal display (12), in particular at least in one eyebox plane (22) of the virtual retinal display (12),are displaced relative to one another and / or wherein in the calibration step (46) a distortion and / or optical distortion of the image content of the beamlet (24) projected at the first time into the partial area (44) of the eyebox (28) and a distortion and / or optical distortion of the image content of the beamlet replica (26) projected at the second time into the partial area (44) of the eyebox (28) are compensated by a setting change of the virtual retinal display, (12) and / or the data glasses (14) having the virtual retinal display (12) are adapted to each other or to each other.

2. The method according to claim 1, characterized in that the virtual retinal display (12) outputs at least one first calibration marker (48) during the projection of the beamlet (24) into the eyebox (28), wherein the virtual retinal display (12) outputs at least one second calibration marker (50), in particular different from the first calibration marker (48), during the projection of the beamlet replica (26) into the eyebox (28), and wherein in the calibration step (46), at least the two calibration markers (48, 50) are shifted relative to one another by a setting change of the virtual retinal display (12) and / or the data glasses (14), in particular by shifting the image contents and / or adjusting the distortions and / or the optical distortion of the image contents within the partial area (44) of the eyebox (28), preferably into a relative position corresponding to an optimal calibration,preferably in at least partial overlap with each other.

3. Method according to claim 1 or 2, characterized in that in at least one further method step (52) at least the beamlet (24) and at least the beamlet replica (26) of the beamlet (24) are repeatedly and alternately projected into a further partial area (54) of the eyebox (28) of the virtual retinal display (12), which corresponds to a second pupil position (56) of the user's eye (10) different from the first pupil position (40), wherein in particular the further partial area (54) of the eyebox (28) in the second pupil position (56) overlaps at least with a further part of the replication limit (30) of the virtual retinal display (12) to be calibrated.

4. Method according to claims 2 and 3, characterized in that in at least one further calibration step (66) at least the two calibration markers (48, 50) are adjusted by a further setting change of the virtual retinal display (12) and / or the data glasses (14), in particular by shifting the image contents and / or adjusting the distortions and / or the optical distortion of the image contents within the further partial area (54) of the eyebox (28), relative to one another.

5. Method according to one of the preceding claims, characterized in that the repeatedly alternating projection of the beamlet (24) and the beamlet replica (26) into the partial area (44) / the partial areas (44, 54) of the eyebox (28) is generated by alternately pushing the respective beamlets (24, 26) in and out of the partial area (44) / the partial areas (44, 54) of the eyebox (28), in particular by means of a MEMS mirror system of a laser projector unit (16) of the virtual retinal display (12).

6. Method according to one of the preceding claims, characterized in that in the setting change of the virtual retinal display (12) and / or the data glasses (14) carried out in the calibration step (46, 66), at least one adaptation of a transformation between a projection mirror position of a MEMS mirror system of a laser projector unit (16) of the virtual retinal display (12) and a video image of the virtual retinal display (12) output via the beamlets (24, 26) is carried out.

7. Method according to one of the preceding claims, characterized in that in at least one further calibration step (68) following the change in the setting of the virtual retinal display (12) and / or the data glasses (14), an image sharpness of the virtual retinal display (12) is readjusted.

8. Method at least according to claims 2 and 7, characterized in that in the further calibration step (68) at least one of the calibration markers (48, 50) is used in addition to an adjustment of the image sharpness of the virtual retinal display (12), in particular by adjusting a focal length of a variable lens of the virtual retinal display (12) is adjusted such that a contrast and / or an edge region of at least the calibration marker (48, 50) is optimized.

9. Method according to one of the preceding claims, characterized in that an image frequency of the alternating projection is selected such that the user can perceive the beamlet (24) and the beamlet replica (26) simultaneously.

10. A beamlet replication-based virtual retinal display (12) with a calibratable, in particular field-calibratable, replication limit (30), which is particularly configured to carry out a method according to one of claims 1 to 9, with a pupil tracking device (20) for determining a first pupil position (40) of a pupil (60) of a user eye (10) of a user of the virtual retinal display (12), with at least one optical system comprising at least one laser projector unit (16) with a MEMS mirror system, at least for a repeatedly alternating projection of at least one beamlet (24) and at least one beamlet replica (26) into a partial region (44) of the eyebox (28) corresponding to the first pupil position (40) of the user eye (10),wherein, in particular, the partial area (44) of the eyebox (28) in the first pupil position (40) overlaps at least with a part of the calibratable replication limit (30) of the virtual retinal display (12), and with at least one adjustment unit (70) which is designed to shift at least an image content projected at a first point in time via the beamlet (24) into the partial area (44) of the eyebox (28) and an image content projected at a second point in time via the beamlet replica (26) into the partial area (44) of the eyebox (28) by a mechanical and / or electronic adjustment process of the virtual retinal display (12) and / or a pair of data glasses (14) having the virtual retinal display (12), in particular at least in one eyebox plane (22) of the virtual retinal display (12),relative to each other and / or which at least lead to a distortion and / or an optical distortion of the image content projected at the first time via the beamlet (24) into the partial area (44) of the eyebox (28) and of the image content projected at the second time via the beamlet replica (26) into the partial area, (44) of the eyebox (28) projected image content by a mechanical and / or electronic adjustment process of the virtual retinal display (12) and / or the data glasses (14) having the virtual retinal display (12).

11. Data glasses (14), in particular AR (Augmented Reality) glasses, with a beamlet replication-based virtual retinal display (12) according to claim 10.

Citation Information

Patent Citations

  • Optical system for a virtual retinal display and method for projecting image content onto a retina

    DE102021200893A1

  • Systems, devices, and methods for eyebox expansion in wearable heads-up displays

    EP3259633A1

  • Method and apparatus for head worn display with multiple exit pupils

    US20170293147A1

  • Optical system for a virtual retinal scan display, data glasses and method for projecting image contents onto the retina of a user

    US20220400236A1