Compensation for deformations in head mounted display systems

The calibration process for head-mounted displays addresses deformation-induced misalignment by adjusting image transformations in real-time, ensuring clear and comfortable virtual content display in augmented and virtual reality systems.

JP7578698B2Active Publication Date: 2024-11-06MAGIC LEAP INC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2022542949
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-06-25
Filing Date
2020-12-21
Publication Date
2024-11-06
Estimated Expiration
2040-12-21

AI Technical Summary

Technical Problem

Head-mounted display systems, particularly in virtual or augmented reality, are susceptible to deformation due to their lightweight and flexible design, leading to misalignment of virtual content with the real-world view, causing discomfort and distorted images.

Method used

A calibration process is implemented to determine image transformations for each eyepiece, adjusting for deformation by using imaging sensors to capture and correct misalignments in real-time, ensuring proper display of virtual content.

Benefits of technology

The calibration process effectively reduces user discomfort by maintaining accurate alignment of virtual content with the real-world view, even when the display system is deformed, providing a comfortable and clear viewing experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007578698000001
    Figure 0007578698000001
  • Figure 0007578698000002
    Figure 0007578698000002
  • Figure 0007578698000003
    Figure 0007578698000003
Patent Text Reader

Abstract

Described systems and methods can include approaches for calibrating a head-mounted display for an enhanced viewing experience. Some methods include receiving data for a first target image associated with an undeformed state of a first eyepiece of a head-mounted display device, receiving data for a first captured image associated with a deformed state of the first eyepiece of the head-mounted display device, determining a first transformation that maps the first captured image to an image, and applying the first transformation to a subsequent image for viewing on the first eyepiece of the head-mounted display device.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (Technical field) This specification relates generally to image processing and display alignment calibration for head mounted displays. [Background technology]

[0002] (background) As head-mounted display systems become lighter, thinner, and more flexible to facilitate transport, comfort, and aesthetics, these wearable devices are also more susceptible to deformation. If the system is not structurally stable, the display will move, deform, and become misaligned. These deformations introduce distortions and other errors into the virtual binocular image. When this happens, a person perceiving the image on the display may feel confused or uncomfortable. This results in a poor viewing experience. Summary of the Invention [Means for solving the problem]

[0003] (overview) An innovative aspect of the subject matter described herein relates to calibration of a head-mounted display device used within a virtual or augmented reality (VAR) system. In particular, a VAR system can be used to display virtual content to augment a physical reality view. When one or more display-related components of the VAR system are deformed or do not operate as desired, calibration may be required to ensure that the virtual content is displayed properly.

[0004] In some cases, transparent displays are used, which allow virtual content in the form of images to be shown superimposed with a view of the real-world environment. If one display is deformed, the virtual content is moved relative to the user's eyes in response to the deformation. If not taken into account, the user's perception of the image relative to the real-world environment will be incorrect. This may be perceived by the person as two separate images or double vision (i.e., one image per eye). Instead of seeing the intended single image, the user sees the two images separately with a space between them. This may result in an unpleasant viewing experience.

[0005] Research shows that binocular misalignment can cause physiological strain on the human visual system, and that humans are sensitive to binocular rotational misalignment of virtual images about the pitch, roll, and yaw axes of up to 2 arc-minutes. For example, in some cases, a 2-arc-minute pitch of a one-eye display relative to the other eye display is enough to cause discomfort. Furthermore, deformations in one display can cause the individual color channels (e.g., red, green, and blue channels) of the display to be shifted relative to each other.

[0006] The systems and methods described herein operate to reduce or prevent this uncomfortable viewing experience. This is accomplished by using a calibration process to determine an image transformation for the left eye display and an image transformation for the right eye display. The calibration process is configured to display images, one for each eye, on each of the two eye displays (also referred to as eyepieces), observe how the images of each eye display change in response to deformations of each display, and determine a transformation (e.g., a mapping or look-up table (LUT)) for each display. The transformation associated with the left eye display is then applied to each subsequent image to be shown on the left eye display, and similarly, the transformation associated with the right eye display is then applied to each subsequent image to be shown on the right eye display. The calibration process is then repeated as needed (e.g., when a deformation is detected), when triggered (e.g., by a blink), or periodically (e.g., every second). In some cases, the transformations for each eye display are determined together and / or are interdependent. In some cases, however, a transformation is determined and applied to a display associated with one eye.

[0007] The calibration technique is advantageous because it allows each display to be recalibrated independently and does not require any input by a human operator. Another advantage is that it allows the physical hardware to be lightweight and compact. For example, a head mounted display can be implemented in a small form factor such as glasses. This allows the device to tolerate deformations and yet correct any issues with the display caused by the deformations in near real time (e.g., within 100 milliseconds).

[0008] Other implementations of the present aspects include corresponding systems, apparatus, and computer programs recorded on computer storage devices, each configured to perform the operations of the present methods.

[0009] The details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. The present invention provides, for example: (Item 1) 1. A computer-implemented method, comprising: receiving data of a first target image associated with a non-deformed state of a first eyepiece of a head mounted display device, the first eyepiece comprising a first projector optically coupled to the first eyepiece; receiving data of a first captured image associated with a deformation state of the first eyepiece of the head mounted display device, the first captured image representing a transformation state of the first target image and received by a first imaging sensor optically coupled to the first projector of the first eyepiece, the first imaging sensor being part of the head mounted display device; determining a first transformation that maps the first captured image to the first target image; applying the first transformation to a subsequent image for viewing on the first eyepiece of the head mounted display device; 23. A computer-implemented method comprising: (Item 2) 2. The computer-implemented method of claim 1, further comprising transmitting the converted subsequent image to the first projector optically coupled to the first eyepiece. (Item 3) 2. The computer-implemented method of claim 1, further comprising transmitting the data of the first target image to the first projector. (Item 4) Item 1, the computer-implemented method of item 1 further comprising: a first imaging sensor optically coupled to the first projector capturing the data of the first target image. (Item 5) receiving a temperature measurement of the first imaging sensor; taking into account the received temperature measurements in the first conversion; 2. The computer-implemented method of claim 1, further comprising: (Item 6) Item 14. The computer-implemented method of item 1, further comprising receiving data from one or more sensors representing a state of a real-world environment external to the head-mounted display device. (Item 7) 2. The computer-implemented method of claim 1, wherein the first transformation aligns features of a real-world environment with corresponding features from the first target image. (Item 8) 2. The computer-implemented method of claim 1, wherein the first transformation aligns each individual color channel of the first captured image with each individual color channel of the first target image. (Item 9) 2. The computer-implemented method of claim 1, further comprising receiving a trigger signal, and wherein determining the first transformation is in response to receiving the trigger signal. (Item 10) 2. The computer-implemented method of claim 1, wherein the first eyepiece of the head mounted display device is optically transparent. (Item 11) 2. The computer-implemented method of claim 1, wherein the first transformation takes into account a relative position of the first image sensor with respect to the first eyepiece. (Item 12) receiving data of a second target image associated with a non-deformed state of a second eyepiece of the head mounted display device, the second eyepiece comprising a second projector optically coupled to the second eyepiece; receiving data of a second captured image associated with a deformation state of the second eyepiece, the second captured image representing a transformation state of the second target image and received by a second imaging sensor optically coupled to the second projector of the second eyepiece, the second imaging sensor being part of the head mounted display device; determining a second transformation that maps the second captured image to the second target image; applying the second transform to a subsequent image for viewing on the second eyepiece of the head mounted display device; and 2. The computer-implemented method of claim 1, further comprising: (Item 13) Item 13. The computer-implemented method of item 12, wherein the first transformation depends on the second transformation. (Item 14) Item 13. The computer-implemented method of item 12, wherein the second transformation aligns a subsequent image for viewing through the second eyepiece with a subsequent image for viewing through the first eyepiece. (Item 15) Item 13. The computer-implemented method of item 12, wherein the first imaging sensor is rigidly connected to the second imaging sensor. (Item 16) Item 13. The computer-implemented method of item 12, wherein the first eyepiece lens and the second eyepiece lens are deformable relative to the first imaging sensor and the second imaging sensor. (Item 17) Item 13. The computer-implemented method of item 12, wherein the second transformation takes into account a relative position of the second image sensor with respect to the second eyepiece. (Item 18) receiving a temperature measurement of the second imaging sensor; taking into account the received temperature measurements in the second conversion; Item 13. The computer-implemented method of item 12, further comprising: (Item 19) A head mounted display device, comprising: A wearable frame and a first eyepiece resiliently mounted to the wearable frame, the first eyepiece comprising a first projector rigidly and optically coupled to the first eyepiece, the first projector configured to emit light into the first eyepiece; a first imaging sensor resiliently mounted to the wearable frame and configured to capture light emitted from the first eyepiece; a processor communicatively coupled to the first projector and the first imaging sensor, transmitting data of a first target image to the first projector; receiving first captured image data from the first imaging sensor, the first captured image data representing a transformed state of the first target image; determining a first transformation that maps the first captured image to the first target image; applying the first transformation to a subsequent image for viewing on the first eyepiece of the head mounted display device; a processor configured to: A head-mounted display device comprising: (Item 20) 20. The head-mounted display device of item 19, further comprising one or more sensors associated with and mounted on the wearable frame, the one or more sensors configured to receive data regarding conditions of a real-world environment external to the head-mounted display device. (Item 21) 20. A head-mounted display device as described in item 19, wherein the first transformation aligns features of a real-world environment with corresponding features from the first target image. (Item 22) 20. The head-mounted display device of item 19, further comprising a trigger sensor configured to transmit a signal to the processor to initiate the process performed by the processor. (Item 23) 20. A head-mounted display device as described in item 19, wherein the first eyepiece lens is optically transparent. (Item 24) 20. The head-mounted display device of item 19, further comprising a temperature sensor configured to measure a temperature of the first imaging sensor, and the first conversion is based, at least in part, on the measured temperature of the first imaging sensor. (Item 25) a second eyepiece resiliently mounted to the wearable frame, the second eyepiece comprising a second projector rigidly and optically coupled to the second eyepiece, the second projector configured to emit light into the second eyepiece; a second imaging sensor resiliently mounted to the wearable frame and configured to capture light emitted from the second eyepiece; and The processor further comprises: transmitting data of a second target image to the second projector; receiving second captured image data from the second imaging sensor representing a transformed state of the second target image; determining a second transformation that maps the second captured image to the second target image; applying the second transform to a subsequent image for viewing on the second eyepiece of the head mounted display device; and 20. The head-mounted display device of item 19, further configured to: (Item 26) 26. A head-mounted display device as described in item 25, wherein the second transformation aligns features of a real-world environment with corresponding features from the second target image. (Item 27) 26. A head-mounted display device as described in item 25, wherein the second transformation is dependent on the first transformation. (Item 28) A head mounted display device, comprising: A wearable frame and two or more eyepieces resiliently mounted to the wearable frame, each of the two or more eyepieces optically coupled to a respective projector; a rigid sensing element resiliently mounted to the wearable frame and configured to measure a geometric difference between the two or more ocular lenses; A head-mounted display device comprising: (Item 29) and a processor communicatively coupled to each projector and to the rigid sensing element, the processor comprising: receiving data from the rigid sensing element representative of a geometric difference between the two or more ocular lenses; determining an image transformation for each of the two or more eyepieces, the image transformation when applied to subsequent images projected by each projector to generate aligned binocular representations of the subsequent images; 29. The head-mounted display device of item 28, configured to perform the following: (Item 30) 29. A head-mounted display device as described in item 28, wherein the rigid sensing element is an imaging sensor. (Item 31) 29. A head mounted display device as described in item 28, wherein the rigid sensing element is a position sensing diode. (Item 32) 29. A head-mounted display device as described in item 28, wherein the rigid sensing element is a LiDAR sensor. [Brief description of the drawings]

[0010] BRIEF DESCRIPTION OF THE DRAWINGS [Figure 1] 1A-1B depict a wearable frame for a head-mounted display in an undeformed state.

[0011] [Diagram 2]2A-2B depict a wearable frame for a head mounted display in a deformed state.

[0012] [Diagram 3] 3A-3B illustrate a head mounted display calibration process.

[0013] [Figure 4] FIG. 4 is a flow chart of a calibration process for a head mounted display.

[0014] [Diagram 5] FIG. 5 is a system diagram of a head mounted display.

[0015] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] (Detailed Description) 1A and 1B illustrate a head-mounted display device 100 of a VAR system in an undeformed or ideal state. Fig. 1A shows a top view of the head-mounted display device 100 with aligned left and right eyepieces (or displays) 70L, 70R. Virtual content may be presented to and perceived by the left and right eyes through the pair of eyepieces 70L, 70R, respectively, as part of a virtual image generation system.

[0017] 1B illustrates the left and right monocular virtual content 72L, 72R through the two eyepieces 70L, 70R to the user's eyes as binocularly aligned virtual content 74. FIG. 1B shows the head mounted display device 100 in an undeformed state. When the two eyepieces 70L, 70R are in an undeformed state, the monocular virtual content 72L, 72R combine to produce the appropriate binocularly aligned virtual content 74 as shown. This is represented by perfectly overlapping left and right monocular virtual content 72L, 72R within the virtual content 74.

[0018] The VAR system may be operated as an augmented or mixed reality system that may provide images of virtual objects intermingled with physical objects in the user's field of view, such that the virtual object(s) appear as if they were in the user's physical environment. It may be desirable to spatially position the various virtual objects relative to the respective physical objects in the user's field of view. The projection assemblies (also referred to as projectors) 108L, 108R of the head mounted display device 100 project the virtual objects onto the eyepieces 70L, 70R for display. The virtual objects may be referred to as virtual tags, tags, or callouts, and may be implemented in various suitable forms.

[0019] Examples of virtual objects may include, but are not limited to, a virtual text object, a virtual numeric object, a virtual alphanumeric object, a virtual tag object, a virtual field object, a virtual chart object, a virtual map object, a virtual measurement object, or a virtual visual representation of a physical object. For example, the VAR system may determine when a user views an empty chair in a room and project an image representing a person sitting in the chair onto each eyepiece 70L, 70R associated with each eye of the head mounted display device 100 such that the user perceives that the virtual person is sitting in an actual chair in the room.

[0020] As shown in Figures 1A and 1B, the two eyepieces 70L, 70R are aligned with each other in an ideal or non-deformed manner. In other words, the alignment of the two eyepieces 70L, 70R has not changed since the head mounted display device 100 was manufactured. For example, when viewing a person in a chair as described above, each of the two eyepieces 70L, 70R displays monocular virtual content 72L, 72R of the person sitting in the chair. The user perceives this combination of virtual content as virtual content 74.

[0021] This is important for 3D perception because the user's brain may not properly associate a single monocular image with 3D depth information. However, the user's brain may properly associate two monocular images that are properly aligned with 3D depth information to give the illusion that the user is viewing a 3D object located at a distance from the user. For example, this may give the illusion that a person is sitting in a chair at a distance from the user, even though the image of the person is on the eyepieces 70L, 70R, i.e., less than an inch from the user's eyes.

[0022] The head mounted display device 100 includes a wearable frame 102 that mounts on a user's head during use. The wearable frame 102 includes left and right temples 302L, 302R that are positionable over the user's left and right ears, respectively. A nose pad 306 is provided to allow the wearable frame 102 to rest comfortably against the user's nose. The wearable frame 102 is preferably made from injection molded plastic to remain lightweight.

[0023] The wearable frame 102 includes two cantilever arm portions 312 extending away from the bridge 304. The cantilever arm portions 312 provide a resilient mounting for the display subsystem 104, which is intended to be positioned above a user's nose and in front of the eye, similar to the position of the lenses in eyeglasses. The cantilever arm portions 312 connect to left and right cantilever arms 310L, 310R. Each left and right cantilever arm 310L, 310R includes an attachment arm portion 314 extending from the respective cantilever arm portion 312 in a plane parallel to the plane of the end user's eye.

[0024] The left and right eyepieces 70L, 70R are respectively affixed to the attachment arm portion 314, and the left and right projection subassemblies 108L, 108R are respectively affixed to the outer ends of the attachment arm portion 314. This facilitates the introduction of light beams into the left and right eyepieces 70L, 70R, respectively, such that light rays exit the left and right eyepieces 70L, 70R, respectively, to display left and right monocular images as binocular images to a user wearing the head mounted display device 100.

[0025] The left and right eyepieces 70L, 70R effectively act as display interfaces when image data is projected onto them. In some implementations, the displays of the left and right eyepieces (or displays) 70L, 70R may be "optical see-through" displays through which the user can view light from real objects directly through a transparent (or translucent) element. In this case, the left and right eyepieces 70L, 70R may be fully transparent or partially transparent so that each eyepiece may superimpose light from the projection subsystem 108L, 108R over the user's view of the real world.

[0026] The display subsystem 104 is configured to present light-based radiation patterns to each eye of a user that are intended to be comfortably perceived as an augmentation to physical reality with high quality 2D or 3D image content.

[0027] The left and right projection subsystems 108L, 108R may project left and right monocular images onto the left and right eyepieces 70L, 70R, respectively. The eyepieces 70L, 70R may be placed in front of the user's eyes so that the monocular images are viewed as binocular images. In some cases, the eyepieces 70L, 70R are less than an inch from the user's eyes. Additionally, the eyepieces 70L, 70R may be placed within the user's field of view between the user's eyes and the surrounding environment so that direct light from the surrounding environment travels through the eyepieces 70L, 70R to the user's eyes.

[0028] The projection assemblies 108L, 108R may provide scanned light to the eyepieces 70L, 70R, respectively. In some implementations, the projection subsystems 108L, 108R may be implemented as fiber optic scanning based projection devices and the eyepieces 70L, 70R may be implemented as waveguide based displays into which the scanned light from the respective projection subsystems 108L, 108R is incident. The display subsystem 104 may output a series of frames at various frequencies obtained from a frame buffer. In some cases, the display subsystem 104 may output frames at a high frequency to provide the perception of a single coherent scene.

[0029] Each of the projection subsystems 108L, 108R may include a spatial light modulator ("SLM"), such as a liquid crystal on silicon ("LCoS") component or a microelectromechanical ("MEMs") scanning mirror. The left projection subsystem 108L may project light representing virtual content toward the left eyepiece 70L, which in turn in-couples and directs this light toward a diffractive optical element (DOE) configured to provide orthogonal pupil expansion (OPE) and / or exit pupil expansion (EPE) functionality. While most of the directed light may exit the eyepiece 70L as it traverses the DOE(s) (e.g., directed toward the user's left eye), some of this light may continue toward the out-coupled DOE 190L, where it is out-coupled from the eyepiece 70L as light (represented by light ray 203) and may be intercepted, at least in part, by the light-sensing assembly 122.

[0030] The right projection subsystem 108R, along with the right eyepiece 70R and its DOE(s) (e.g., output coupling element 190R, input coupling element (ICE), OPE, and EPE), may operate in a similar manner to the projection subsystem 108L. For example, the projection subsystem 108R, right eyepiece 70R, and its DOE(s) may present virtual content to a user's right eye and out-couple light representing the virtual content through the output coupling DOE 190R and direct it to the light-sensing assembly 122.

[0031] 1A, the light-sensing assembly 122 is located in the bridge 304 of the wearable frame 102. The light-sensing assembly 122 includes separate cameras (imaging sensors) for the left and right eyepieces. The cameras are configured to determine the virtual content to be displayed on the left and right eyepieces.

[0032] The light-sensing assembly 122 can be sufficiently rigid so that the camera associated with the left eyepiece has a fixed position and orientation relative to the position and orientation of the camera associated with the right eyepiece. The light-sensing assembly 122 is preferably made of a rigid material such as aluminum, titanium, or ceramic. The light-sensing assembly 122 is also referred to as a rigid sensing element. Both cameras of the light-sensing assembly 122 are configured to capture light rays 203 that represent images displayed on each respective eyepiece 70L, 70R. The light-sensing assembly 122 includes a temperature sensor for monitoring the temperature for each camera. The light-sensing assembly 122 is preferably resiliently mounted to the wearable frame 102 using an insulating material such as foam or rubber.

[0033] Although the light sensing assembly 122 is described as a camera, in some cases a position sensing diode or LiDAR sensor is used instead of or in addition to a camera. For example, a LiDAR sensor can project a dense point cloud onto each respective eyepiece 70L, 70R, which is reflected and analyzed by a processor to determine the relative position of each eyepiece 70L, 70R. Similarly, a position sensing diode can measure the position of the light beam 203, and this information can be analyzed by a processor to determine the relative position of each eyepiece 70L, 70R.

[0034] The head mounted display device 100 and / or the VAR system may also include one or more sensors mounted on the wearable frame 102 for detecting the position and movement of the user's head and / or the position and interocular distance of the user's eyes. Such sensor(s) may include an image capture device (such as a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope. For example, an eye blink sensor may indicate when a user blinks, and this information can be used to trigger a calibration process by the VAR system.

[0035] The ends of the left and right cantilever arms 310L, 310R that are away from the user's nose include cameras 103L, 103R, respectively. The left camera 103L and the right camera 103R are configured to capture images of the user's environment, for example objects in front of the user.

[0036] Although FIG. 1A illustrates the left and right eyepieces 70L, 70R away from the user and the light-sensing assembly 122 closer to the user, in some cases the left and right eyepieces 70L, 70R are closer to the user and the light-sensing assembly 122 is farther from the user.

[0037] 2A and 2B illustrate the head-mounted display device 100 of the VAR system in a deformed or non-ideal state. An eyepiece 70R is shown bent toward the user. When one or both eyepieces 70L, 70R are in a deformed state, the monocular virtual contents 72L, 72R combine to produce the binocularly misaligned virtual content 74 as shown in FIG. 2B. FIG. 2B illustrates this misalignment of the monocular virtual contents 72L, 72R in the virtual content 74 due to the pitch of the right eyepiece 70R. The misalignment is represented by the left and right monocular virtual contents 72L, 72R not perfectly overlapping in the virtual content 74. Such misalignment between the left and right eyepieces 70L, 70R may result in a perceived translational and / or rotational misalignment between the left and right virtual contents 72L, 72R.

[0038] The head mounted display device 100 may be deformed during use, for example, by movement of the head mounted display device 100 or by accidental contact with the user's hand or other objects in the room. In some cases, the head mounted display device 100 is deformed during transportation or even during initial assembly. As explained above, movement of one eyepiece 70L relative to the other eyepiece 70R by as little as two arc-minutes may be enough to cause discomfort. This is the case when the eyepieces 70L, 70R are translucent or opaque (e.g., as in a virtual reality system). Taking this misalignment into account improves the user's comfort.

[0039] The described systems and methods allow the display to be recalibrated to account for this deformation, which can be achieved by capturing the light intended for the left and right eyes and comparatively detecting and correcting the relative misalignment in the virtual images.

[0040] 1A and 2A illustrate light rays 203 being transmitted from the left and right out-coupling DOEs 190L, 190R. In the undeformed state of FIG. 1A, light rays 203 arrive at their respective cameras of the light-sensing assembly 122 at substantially the same time. This time-of-flight information depends on the relative position of the left eyepiece 70L to the right eyepiece 70R. For example, light arriving at the display earlier than expected may indicate that the eyepieces are bending away from the user and the out-coupling DOEs are closer to the light-sensing assembly 122. Conversely, light arriving at the display later than expected may indicate that the eyepieces are bending towards the user and the out-coupling DOEs are farther from the light-sensing assembly 122.

[0041] Additionally, images captured by light-sensing assembly 122 can be processed to determine information about the position of each respective eyepiece. Light received from left output-coupled DOE 190L is generated into a left captured image using a left camera of light-sensing assembly 122. Light received from right output-coupled DOE 190R is generated into a right captured image using a right camera of light-sensing assembly 122.

[0042] For example, by comparing the left captured image with the left target image representing the undeformed state, the VAR system can determine that the left eyepiece 70L is deformed. The VAR system can also determine a transformation to be applied to subsequent images to correct for this deformation. The transformation is then applied to the subsequent images for the left eyepiece and transmitted to the left projection subsystem 108L. The user regains a comfortable viewing experience.

[0043] Characteristics of the captured images may also indicate deformations. For example, a rotated image may indicate roll of the eyepiece, and a trapezoidal image may indicate yaw or pitch depending on the length of the sides of the trapezoid. A larger image may indicate that the eyepiece is farther from the light-sensing assembly 122, while a smaller image may indicate that the eyepiece is closer to the light-sensing assembly.

[0044] Figures 3A and 3B illustrate the transformation and calibration process associated with each eyepiece. The left diagram of Figure 3A illustrates the misaligned left and right monocular virtual content 72L, 72R from the pitching of the right eyepiece 70R, as shown in Figure 2B. Once transformed, the right monocular virtual content 72R perfectly overlays the left monocular virtual content 72L (as shown in Figure 3B), even though the right eyepiece 70R is still pitched. Figure 3B illustrates how the transformed image shown on the misaligned frame of Figure 1B has a proper binocular representation after the transformation process.

[0045] 4 is a flow chart of a calibration process 400. A calibration is determined for each eyepiece 70L, 70R. In some cases, the calibration is triggered (step 408), for example, by a user blinking or by the user requesting that a calibration be performed (e.g., through a user interface or via a setup mode). The calibration is performed fast enough that the user does not realize that a calibration process is occurring. In some cases, the calibration is completed within 100 milliseconds. In some cases, a separate trigger signal is received for each eyepiece.

[0046] In the context of the left (first) eyepiece, the VAR system receives data for a left target image associated with the undeformed state of the left eyepiece (step 410L). The left target image can be retrieved from a database of target images. The left target image may include geometric features distinguishable by the processor of the VAR system. For example, the left target image may be a checkerboard pattern of squares with a unique image in each square. The left target image may be monochrome or color. A color left target image can be used to show the alignment of color channels (e.g., red, green, and blue color channels) in an image. The left target image may be adjusted for the left eyepiece 70L and may be different from the right target image for the right eyepiece 70R. However, in some cases, the left and right target images are the same.

[0047] In some cases, data for the left target image is transmitted to the left projector for display on the left eyepiece (step 412L). The left camera of the light sensing assembly 122 captures light from the output-coupled DOE representing data for the left captured image (step 418L). Data for the left captured image associated with the deformation state is received from the left camera of the left eyepiece of the head mounted display device (step 420L).

[0048] In some cases, the calibration process 400 includes receiving a temperature measurement of the left imaging sensor (step 426L). In some cases, data representative of conditions of a real-world environment external to the head mounted display device is received from one or more sensors (step 428L).

[0049] A left transform is determined (step 430L) that maps the left captured image to the left target image. In some cases, the received temperature is taken into account in the left transform. In some cases, the left transform aligns features of the real world environment with corresponding features from the left target image. In some cases, the left transform takes into account the relative position of the left imaging sensor with respect to the left eyepiece. In some cases, the left transform is determined when a trigger signal associated with the left eyepiece is received, but the left transform can also be determined when a trigger signal associated with the right eyepiece and / or trigger signals associated with both the left and right eyepieces are received. In some cases, more than one trigger signal is received.

[0050] In one case, the processor determines that the virtual content is located within a top left region of the frame, identifies pixel locations and pixel values ​​associated with the detected content within the frame, and uses this information to determine the transformation. In one case, the direction of light entering the imaging sensor is used to determine the transformation.

[0051] For example, the transformation matches pixels in a display buffer. In this case, the video buffer maps each pixel to a virtual light space, and the transformation maps the pixel to a light source, resulting in a smooth transformation map. In some cases, it is preferable to have dozens of points (e.g., 15-30 points) horizontally and dozens of points (e.g., 15-30 points) vertically in the image. This allows the processor to determine where the pixel has moved in pixel space. For example, if the target image has a population of white pixels at location A, and that population of white pixels moves to location B, the transformation may simply be a pixel shift (inverse shift) of the entire image from B to A. As another example, if the target image has a population of white pixels of size A, and that population of white pixels expands to size B, the transformation may simply be a pixel scaling (inverse scaling) of the entire image from size B to size A. Other transformations may follow imaging processing techniques known in the art.

[0052] In some cases, more than one left transform is determined. For example, several captured images can be processed to determine several transforms. These can be averaged or filtered to increase the accuracy of the transformations.

[0053] The left transformation is applied to the subsequent image for viewing on the left eyepiece of the head mounted display device (step 440L). In some cases, the transformed subsequent image is transmitted to a left projector optically coupled to the left eyepiece for viewing on the left eyepiece (step 450L). In some cases, process 400 is repeated as desired or necessary.

[0054] In some cases, this process is repeated for the right (second) display. In such a case, data for a right target image associated with a non-deformed state of the right eyepiece of the head mounted display device is received (step 410R). In some cases, the right target image is identical to the left target image (step 410) or otherwise dependent on the left target image. For example, if the left target image is monochrome, the right target image may also be monochrome. These and other properties of the target images may be communicated to each other by the processor (step 410). For example, the left and right transformations may align each individual color channel of the respective captured image with each individual color channel of the respective target image.

[0055] In some cases, data of the right target image is transmitted to the right projector (step 412R). Data of the right captured image is captured by a right imaging sensor optically coupled to the right projector (step 418R). In some cases, data of the right captured image associated with a deformation state of the right eyepiece of the head mounted display device is received (step 420R). In some cases, a temperature measurement of the right imaging sensor is received (step 426).

[0056] A right transformation is determined (step 430R) that maps the right captured image to the right target image. In some cases, the received temperature is taken into account in the right transformation. In some cases, the right transformation aligns features of the real-world environment with corresponding features from the right target image. In some cases, the right transformation takes into account the relative position of the right imaging sensor with respect to the right eyepiece. In some cases, the right transformation is determined when a trigger signal associated with the right eyepiece is received, but the right transformation can also be determined when a trigger signal associated with the left eyepiece and / or trigger signals associated with both the left and right eyepieces are received. In some cases, more than one trigger signal is received.

[0057] In some cases, the right transformation depends on the left transformation (step 430). For example, in some cases, a binocular transformation algorithm determines a preferred transformation for both the left transformation and the right transformation. The binocular transformation is determined not only from the left and right displays, but also from one or more sensors on the head mounted display device 100 or the VAR system. The sensors are used to measure the real world around the user and properly align the superimposed visual content with the real world.

[0058] For example, consider the case where an object in the real world cannot be properly aligned with either of the eyepieces when the left eyepiece is shifted 3 pixels to the left. In this case, a left transformation may shift the images and a right transformation may shift the images so that both displays are aligned with each other and with the real world surroundings.

[0059] The right transformation is applied to the subsequent image for viewing on the right eyepiece of the head mounted display device (step 440R). In some cases, the transformed subsequent image is transmitted to a right projector optically coupled to the right eyepiece for viewing on the right eyepiece (step 450R).

[0060] 5 is a system diagram of a VAR system 500. The VAR system 500 includes a control subsystem with various software and hardware components, and includes the head-mounted display device 100 shown in FIGS. 1A and 2A. The VAR system 500 includes a computer processing unit (CPU) 502 for performing the processing tasks of the VAR system 500, and a graphics processing unit (GPU) 504. Left and right monocular calibration algorithms and left and right thermal models are stored in memory and run on the processor to aid in the calibration process.

[0061] In some cases, the left monocular calibration algorithm includes steps 410L-450L of the calibration process 400, and the right monocular calibration algorithm includes steps 410R-450R of the calibration process 400 (but this is not limited to this). Other steps of the calibration process 400 can be implemented in the left and right monocular calibration algorithm. The left and right display calibration servers also communicate with the processor. In addition, the binocular calibration algorithm, the online calibration, the collateral calibration server, and the left and right combined thermal models communicate with the processor for the calibration process. In some cases, the binocular calibration algorithm includes steps 410 and 430 of the calibration process 400 (but this is not limited to this). Other steps of the calibration process 400 can be implemented in the binocular calibration algorithm.

[0062] The VAR system 500 includes sensors for monitoring a real-world environment 510. These sensors 510 are shown to include a user orientation sensor and an angle sensing assembly, although the sensors described above are also included in the VAR system 500. The VAR system 500 also includes a three-dimensional (3D) database 508 for storing 3D scene data. The CPU 502 may control the overall operation of the VAR system 500, while the GPU 504 renders frames from the 3D data stored in the 3D database 508 (e.g., converts the 3D scene into a 2D image) and stores these frames in a frame buffer(s) 506.

[0063] Also included are left and right frame buffers 506L, 506R that transmit images to the left and right projection subsystems 108L, 108R, respectively. Specifically, images are sent to the projection system and displayed on the eyepieces of the head mounted display 100. Once captured by the left and right image sensors of the light sensing assembly 122, the captured images are transmitted back to the processor as part of a calibration process.

[0064] In general, the control subsystem may include various controllers, such as a microcontroller, microprocessor, CPU, digital signal processor, GPU, application specific integrated circuit (ASIC), programmable gate array (PGA), field PGA (FPGA), and / or programmable logic controller (PLU). The control subsystem may include and / or communicate with one or more processors, such as CPU 502 and GPU 504, that perform the operations described herein, for example, through execution of executable instructions. Although not shown, one or more integrated circuits may be used to control the loading and / or reading of one or more frames into and / or from frame buffer 118 and the operation of left and right projection subsystems 108L, 108R of display subsystem 104.

[0065] Specifically, the CPU 502 may receive and process data acquired by the light-sensing assembly 122. The CPU 502 may compare data derived from light incident on the light-sensing assembly 122 when the wearable frame 102 is in a non-deformed state with data derived for light incident on the light-sensing assembly 122 when the wearable frame 102 is in a deformed state to determine a relative deformation state of the left and right eyepieces 70L, 70R. In response to detecting a relative deformation state or misalignment of the virtual image, the VAR system 500 may perform one or more calibration procedures to compensate the virtual or displayed image depending on the deformation / misalignment.

[0066] The VAR system 500 and various techniques disclosed herein may be employed in applications other than augmented reality and virtual reality subsystems, and although certain implementations are described in the context of an augmented reality or virtual reality subsystem, the VAR system 500 is not limited to such subsystems.

[0067] In some cases, the processing aspects of the VAR system 500 (CPU 502 and GPU 504) are implemented in a device that is attached to the user's waist or in the user's pocket, while in some cases the processing aspects are implemented directly in the head mounted display device 100 shown in Figures 1A and 2A. In some cases the processing aspects are implemented in a nearby computer and communicated wired or wirelessly to the head mounted display device 100. In some cases at least some of the processing aspects are implemented on a remote server and communicated wirelessly using WiFi or the like to the head mounted display device 100 shown.

[0068] The calibration techniques may be performed by one or more processors (hereinafter simply referred to as processors), such as CPU 502 or GPU 504. Although the calibration process described above is described in the context of a single captured image, in some cases more than one captured image is processed and more than one transformation is determined for each eyepiece. In this case, the transformations may be averaged or filtered to determine the most accurate transformation.

[0069] In some implementations, the VAR system 500 may be calibrated with respect to the color of the virtual content being displayed to the user. For example, if only blue virtual content is being displayed, the processor may perform the calibration using a blue test pattern. If only red virtual content is being displayed, the processor may perform the calibration using a red test pattern. If only green virtual content is being displayed, the processor may perform the calibration using a green test pattern. If virtual content having a combination of red, blue, and green colors is being displayed, the processor may perform the calibration using a combination of red, blue, and green calibration frames.

[0070] Various characteristics of the calibration frames (such as intensity) may be configured to match or resemble characteristics of the representative virtual content frames. For example, if the intensity of the virtual content is determined to be above or equal to a minimum threshold level of intensity, the intensity of the calibration frames may be equal to the intensity of the corresponding virtual content. If the intensity of the virtual content is determined to be below a minimum threshold level of intensity, the intensity of the calibration frames may be set to the minimum threshold level of intensity.

[0071] In some implementations, image properties of the calibration frames, such as contrast ratio or brightness, may be configured to further reduce the perceptibility of the test frames. In some implementations, the calibration frames may be diluted by hiding the test images behind edges of the virtual content. The calibration frames may be further camouflaged by using textures and colors similar to the virtual content.

[0072] The described systems, methods, and techniques may be implemented in digital electronic circuitry, computer hardware, firmware, software, or combinations of these elements. Apparatuses implementing these techniques may include suitable input and output devices, a computer processor, and a computer program product tangibly embodied in a machine-readable storage device for execution by the programmable processor. Processes implementing these techniques may be performed by a programmable processor executing a program of instructions to perform a desired function by operating on input data and generating appropriate output. The techniques may be implemented using one or more computer programs or non-transitory computer-readable storage media that include instructions executable on a programmable system that includes at least one programmable processor coupled to receive data and instructions from and transmit data and instructions to a data storage system, at least one input device, and at least one output device.

[0073] Each computer program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language if desired; in either case, the language may be a compiled or interpreted language. Suitable processors include, by way of example, both general-purpose and special-purpose microprocessors. In general, the processor receives instructions and data from a read-only memory and / or a random-access memory. Suitable storage devices for tangibly embodying computer program instructions and data include all forms of non-volatile memory, by way of example, semiconductor memory devices such as erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), and flash memory devices, magnetic disks such as internal hard disks and removable disks, magneto-optical disks, and compact disk read-only memories (CD-ROMs). Any of the foregoing may be supplemented by, or incorporated in, specially designed ASICs (application-specific integrated circuits).

[0074] The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter that provides a machine-readable propagated signal, or a combination of one or more of them. The term "data processing apparatus" encompasses any apparatus, device, and machine for processing data, including, as examples, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, an apparatus may include code that creates an execution environment for the computer program (e.g., code that constitutes a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them). A propagated signal is an artificially generated signal (e.g., a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiver device).

[0075] A computer program, also known as a program, software, software application, script, plug-in, or code, may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form (including deployed as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment). A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data, either in a single file dedicated to the program, or in multiple associated files. A computer program may be executed on one computer or on multiple computers located in one facility or distributed across multiple facilities and interconnected by a communication network.

[0076] The processes and logic flows described herein may be implemented by one or more sensor programmable processors that execute one or more sensor computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry (e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit)), and an apparatus may be implemented as such.

[0077] Processors suitable for the execution of a computer program include, by way of example, both general purpose and special purpose microprocessors, and any one or more sensor processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random access memory or both.

[0078] The elements of a computer may include a processor for carrying out instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more sensor mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, or is operatively coupled to receive data from them, transfer data to them, or both. However, a computer need not have such devices. In addition, a computer may be embedded in another device (e.g., tablet computers, mobile phones, personal digital assistants (PDAs), mobile audio players, VAR systems, to name only a few). Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0079] Although the specification contains many details, these should not be construed as limitations on the scope of the disclosure or the claims, but rather as descriptions of features specific to specific embodiments. Certain features described in the specification in the context of separate embodiments may be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may be described above as acting in a combination and may even be claimed as such, one or more features from the claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be directed to a subcombination or a variation of the subcombination. For example, while a mapping operation is described as a series of separate operations, various operations may be divided into additional operations, combined into fewer operations, reordered in execution, or eliminated depending on the desired implementation.

[0080] Similarly, the separation of various system components in the embodiments described above should not be understood as requiring such separation in every embodiment, and it should be understood that the program components and systems described may generally be integrated together in a single software product or packaged into multiple software products. For example, while some operations are described as being performed by a processing server, one or more of the operations may be performed by a smart meter or other network component.

[0081] Terms used in this specification, and particularly in the appended claims (e.g., the body of the appended claims), are generally intended to be "open-ended" terms (e.g., the term "including" should be interpreted as "including, but not limited to," the term "having" should be interpreted as "having at least," the term "includes" should be interpreted as "including, but not limited to," etc.).

[0082] In addition, if a specific number of claim recitations to be introduced is intended, such intent is expressly stated in the claim, and in the absence of such a statement, such intent does not exist. For example, as an aid to understanding, the following appended claims may include the use of the introductory phrases "at least one" and "one or more" to introduce claim recitations. However, even when the same claim includes the introductory phrases "one or more" or "at least one" and an indefinite article such as "a" or "an", the use of such phrases should not be interpreted as implying that the introduction of a claim recitation with the indefinite article "a" or "an" limits any particular claim that includes such an introduced claim recitation to an embodiment that includes only one such recitation (e.g., "a" and / or "an" should be interpreted to mean "at least one" or "one or more"), and the same applies to the use of definite articles used to introduce claim recitations.

[0083] In addition, even when a particular number of claim recitations introduced is explicitly recited, one of ordinary skill in the art will recognize that such recitation should be interpreted to mean at least the recited number (e.g., the bare recitation of "two recitations" without other modifiers means at least two recitations, or two or more recitations). Furthermore, in those cases where a convention similar to "at least one of A, B, and C, etc." or "one or more of A, B, and C, etc." is used, such a structure is generally intended to include only A, only B, only C, both A and B, both A and C, both B and C, or both A, B, and C. The term "and / or" is also intended to be interpreted in this manner.

[0084] Use of the terms "first", "second", "third", etc., is not necessarily used herein to suggest a particular order or number of elements. Generally, the terms "first", "second", "third", etc., are used as generic identifiers to distinguish between different elements. Absent a statement indicating that the terms "first", "second", "third", etc., imply a particular order, these terms should not be understood to imply a particular order. Furthermore, absent a statement indicating that the terms "first", "second", "third", etc., imply a particular number of elements, these terms should not be understood to imply a particular number of elements. For example, a first widget may be described as having a first side and a second widget may be described as having a second side. Use of the term "second side" for a second widget may be to distinguish such side of the second widget from the "first side" of the first widget and does not imply that the second widget has two sides.

Claims

1. 1. A computer-implemented method comprising: transmitting data of a first target image associated with a non-deformed state of a first eyepiece of a head mounted display device to a first projector optically coupled to the first eyepiece; capturing the data of the first target image associated with the non-deformed state of the first eyepiece of the head mounted display device by a first imaging sensor optically coupled to the first projector, the first imaging sensor being part of the head mounted display device; receiving data of a first captured image associated with a deformation state of the first eyepiece of the head mounted display device, the first captured image representing a transformed state of the first target image and received by the first imaging sensor optically coupled to the first projector of the first eyepiece; determining a first transformation that maps the first captured image to the first target image; applying the first transformation to a subsequent image for viewing on the first eyepiece of the head mounted display device; 23. A computer-implemented method comprising:

2. The computer-implemented method of claim 1 , further comprising transmitting the transformed subsequent image to the first projector optically coupled to the first eyepiece.

3. The computer-implemented method of claim 1 , further comprising receiving a temperature measurement of the first imaging sensor.

4. 2. The computer-implemented method of claim 1, further comprising receiving data from one or more sensors representing a state of a real-world environment external to the head-mounted display device, and wherein the first transformation aligns features of the real-world environment with corresponding features from the first target image.

5. The computer-implemented method of claim 1 , wherein the first transformation aligns each individual color channel of the first captured image with each individual color channel of the first target image.

6. The computer-implemented method of claim 1 , further comprising receiving a trigger signal, and wherein determining the first transformation is responsive to receiving the trigger signal.

7. 2. The computer-implemented method of claim 1, wherein the first eyepiece of the head mounted display device is optically transparent, and the first transformation takes into account a relative position of the first imaging sensor with respect to the first eyepiece.

8. receiving data of a second target image associated with a non-deformed state of a second eyepiece of the head mounted display device, the second eyepiece comprising a second projector optically coupled to the second eyepiece; receiving data of a second captured image associated with a deformation state of the second eyepiece, the second captured image representing a transformed state of the second target image and received by a second imaging sensor optically coupled to the second projector of the second eyepiece, the second imaging sensor being part of the head mounted display device; determining a second transformation that maps the second captured image to the second target image; applying the second transform to a subsequent image for viewing on the second eyepiece of the head mounted display device; and The computer-implemented method of claim 1 , further comprising:

9. 9. The computer-implemented method of claim 8, wherein the first transformation depends on the second transformation, and the second transformation aligns a subsequent image for viewing through the second eyepiece with a subsequent image for viewing through the first eyepiece.

10. 9. The computer-implemented method of claim 8, wherein the first image sensor is rigidly connected to the second image sensor, and the first eyepiece and the second eyepiece are deformable relative to the first image sensor and the second image sensor.

11. The computer-implemented method of claim 8 , wherein the second transformation takes into account a relative position of the second image sensor with respect to the second eyepiece.

12. The computer-implemented method of claim 8 , further comprising receiving a temperature measurement of the second imaging sensor.

13. A head mounted display device, comprising: A wearable frame and a first eyepiece resiliently mounted to the wearable frame, the first eyepiece comprising a first projector rigidly and optically coupled to the first eyepiece, the first projector configured to emit light into the first eyepiece; a first imaging sensor resiliently mounted to the wearable frame and configured to capture light emitted from the first eyepiece; a processor communicatively coupled to the first projector and the first imaging sensor, transmitting data of a first target image associated with a non-deformed state of the first eyepiece of the head mounted display device to the first projector optically coupled to the first eyepiece; capturing the data of the first target image associated with the non-deformed state of the first eyepiece of the head mounted display device by the first imaging sensor optically coupled to the first projector, the first imaging sensor being part of the head mounted display device; receiving first captured image data from the first imaging sensor, the first captured image data representing a transformed state of the first target image; determining a first transformation that maps the first captured image to the captured data of the first target image; applying the first transformation to a subsequent image for viewing on the first eyepiece of the head mounted display device; a processor configured to: A head-mounted display device comprising:

14. The head-mounted display device of claim 13, further comprising one or more sensors associated with and mounted on the wearable frame, the one or more sensors configured to receive data regarding conditions of a real-world environment external to the head-mounted display device, and the first transformation aligns features of the real-world environment with corresponding features from the first target image.

15. a temperature sensor configured to measure a temperature of the first imaging sensor; a trigger sensor configured to transmit a signal to the processor to initiate the determination of the first conversion; 14. The head mounted display device of claim 13, further comprising: the first eyepiece lens being optically transparent.

16. a second eyepiece resiliently mounted to the wearable frame, the second eyepiece comprising a second projector rigidly and optically coupled to the second eyepiece, the second projector configured to emit light into the second eyepiece; a second imaging sensor resiliently mounted to the wearable frame and configured to capture light emitted from the second eyepiece; and The processor further comprises: transmitting data of a second target image to the second projector; receiving second captured image data from the second imaging sensor representing a transformed state of the second target image; determining a second transformation that maps the second captured image to the second target image; applying the second transform to a subsequent image for viewing on the second eyepiece of the head mounted display device; and The head mounted display device of claim 13 , further configured to:

17. 17. The head mounted display device of claim 16, wherein the second transformation aligns features of a real world environment with corresponding features from the second target image, the second transformation being dependent on the first transformation.

Citation Information

Patent Citations

  • Head mounted display, and image display method for the same

    JP2011002753A

  • Optical position adjusting method in head-mounted display

    JP2011221235A

  • Head mounted display device and computer program

    JP2017102696A

  • Display active alignment systems utilizing test patterns for calibrating signals in waveguide displays

    US20180074578A1