EYE CAMERA SYSTEM AND METHOD FOR DISPLAY SYSTEM CALIBRATION - Patent application

Ocular cameras mimicking human eye characteristics are used to calibrate AR/VR systems, improving the integration of virtual and real-world images for a more accurate and natural user experience.

JP7580174B2Active Publication Date: 2024-11-11MAGIC LEAP INC
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2022500895
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-07-12
Filing Date
2020-07-10
Publication Date
2024-11-11
Estimated Expiration
2040-07-10

AI Technical Summary

Technical Problem

Calibrating cameras in augmented and virtual reality glasses to accurately present real-world and virtual images is challenging due to the complexity of the human visual system, making it difficult to achieve a seamless integration of digital and real-world visuals.

Method used

Employing ocular cameras that mimic human eye geometry, optical performance, and motion, with adjustable focus and independently mounted on hexapods to simulate gaze and convergence, allowing for precise calibration based on pupil location and line of sight.

Benefits of technology

Enables more accurate calibration of AR/VR systems, resulting in a more seamless and natural integration of virtual and real-world images, enhancing the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007580174000001
    Figure 0007580174000001
  • Figure 0007580174000002
    Figure 0007580174000002
  • Figure 0007580174000003
    Figure 0007580174000003
Patent Text Reader

Abstract

Embodiments provide a camera design (e.g., an eye camera) that mimics the human eye in geometry, optical performance, and / or movement. The eye camera employs the same corneal and pupil geometry as the human eye, with an iris and pupil configured with multiple texture, color, or diameter options. The resolution of the eye camera is designed to match the acuity of typical 20 / 20 human vision, and the focus is adjustable from 0 to 4 diopters. The pair of eye cameras are independently mounted on two hexapods to simulate the gaze and convergence-divergence movements of the human eye. The perceived virtual and real worlds are calibrated and evaluated based on ocular conditions such as pupil location and gaze using the eye cameras.
Need to check novelty before this filing date? Find Prior Art

Description

[Background technology]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application is a nonprovisional application claiming the benefit of U.S. Patent Application No. 62 / 873,641, entitled "EYE BALL CAMERA SYSTEM AND METHODS FOR DISPLAY SYSTEM CALIBRATION," filed July 12, 2019, which is incorporated by reference in its entirety for all purposes.

[0002] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally produced images or portions thereof are presented to a user within a wearable device in a manner that appears or can be perceived as real. Virtual reality, or "VR", scenarios typically involve the presentation of digital or virtual image information without transparency to other actual real-world visual inputs, while augmented reality, or "AR", scenarios typically involve the presentation of digital or virtual image information as an extension to the visualization of the real world around the user.

[0003] The wearable device may include augmented and / or virtual reality glasses. A camera may be coupled to the glasses. In order to view the actual location of real-world objects on the augmented and / or virtual reality glasses, the camera needs to be calibrated. Calibrating the camera may include determining intrinsic and / or extrinsic parameters of the camera. The intrinsic parameters represent a projective transformation from 3-D camera coordinates to 2-D image coordinates. The intrinsic parameters include a focal length (f x, f y ), the principal point, and distortion coefficients. Extrinsic parameters, on the other hand, represent the transformation from the world coordinate system to the camera's coordinate system. Extrinsic parameters include the rotation matrix and the translation vector. Extrinsic parameters can help determine the location of the camera center and the heading direction of the camera in world coordinates.

[0004] Thus, camera calibration can estimate the parameters of the camera's lens and image sensor. The determined parameters can be used to correct lens distortion, measure the size of an object in world units, or determine the location of the camera within a scene in a 3-D scene reconstruction.

[0005] In conclusion, the human visual perception system is highly complex, making it difficult to produce VR, AR, or MR technologies that facilitate comfortable, natural-feeling, and rich presentation of virtual image elements among other virtual or real-world image elements. The systems and methods disclosed herein address various challenges associated with VR, AR, and MR technologies. Summary of the Invention [Means for solving the problem]

[0006] The embodiments generally relate to image display systems and methods for display system calibration. The embodiments provide a camera design (e.g., an eye camera) that mimics the human eye in geometry, optical performance, and / or motion. The eye camera adopts the same cornea and pupil geometry from the human eye, with an iris and pupil configured with multiple texture, color, or diameter options. The resolution of the eye camera is designed to match the acuity of typical 20 / 20 human vision, and the focus is adjustable from 0 to 4 diopters in some implementations. The pair of eye cameras are independently mounted on two hexapods and can simulate the gaze and vergence-divergence movements of the human eye. The perceived virtual and real world can therefore be calibrated and evaluated based on eye conditions such as pupil location and gaze using the eye cameras. The eye camera acts as a bridge, combining data from spatial computing like eye tracking, the 3D geometry of the digital world, display color accuracy / uniformity, and display optical quality (sharpness, contrast, etc.).

[0007] Various embodiments provide a system for applying corrections to a virtual image display system. The system includes a display device configured to display a virtual image, an image capture device positioned in front of the display device, and a processor coupled to the display device and the image capture device to receive image data from the image capture device. The image capture device has one or more properties similar to a human eye. The processor is programmed to: (a) receive a first image of a scene; (b) determine a first set of visual properties associated with the first image; (c) receive a second image of the scene captured with the image capture device; (d) determine a second set of visual properties associated with the second image; (e) identify a difference between the first set of visual properties and the second set of visual properties; (f) determine a correction to apply to the virtual image display system using the one or more properties of the image capture device to correct, at least in part, the difference; and (g) apply the corrections to the virtual image display system based on the one or more properties of the image capture device.

[0008] Various embodiments provide a method of applying corrections to a virtual image display system. The method includes receiving, by a processor, a first image of a scene from a real-world image capture device. The processor determines a first set of visual properties associated with the first image. The method further includes receiving, by the processor, a second image of the scene captured with the image capture device from the image capture device, the second image having one or more properties similar to a human eye. The image capture device is positioned in front of a display device. The method also includes determining, by the processor, a second set of visual properties associated with the second image, and identifying, by the processor, a difference between the first set of visual properties and the second set of visual properties. The processor determines a correction to apply to the virtual image display system using the one or more properties of the image capture device to correct for the difference, at least in part, and applies the correction to the virtual image display system based on the one or more properties of the image capture device.

[0009] Numerous advantages over conventional techniques are achieved by the methods of the present disclosure. For example, the embodiments provide a method and system that uses one or more cameras (e.g., eye cameras) that mimic the human eye to calibrate a real-world image capture device and / or an AR / VR display system. Properties of the eye camera, such as one or more of corneal position, corneal geometry, eye position, pupil size, pupil position, line of sight distance, or line of sight orientation, can be known and controlled. Thus, calibration can be performed using a set of known and controllable parameters. For example, the line of sight can be fixed on infinity and the data can be used to calibrate the device. Thus, the embodiments provide a more accurate calibration of the AR / VR display system, thus resulting in a more accurate and seamless AR / VR experience.

[0010] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Neither this summary nor the following detailed description purports to define or limit the scope of the inventive subject matter. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 illustrates an exemplary system for calibration of a display device according to some embodiments of the present disclosure.

[0012] [Diagram 2] 2A-2B illustrate an exemplary image capture element according to some embodiments of the present disclosure.

[0013] [Diagram 3] FIG. 3 is a simplified schematic diagram illustrating a camera rig, according to some embodiments of the present disclosure.

[0014] [Figure 4] FIG. 4 is an example block diagram illustrating a method for calibrating a virtual image display system according to some embodiments of the present disclosure.

[0015] [Diagram 5] FIG. 5 is an example block diagram illustrating various calibrations that may be performed in connection with a virtual image display system according to some embodiments of the present disclosure.

[0016] [Figure 6] FIG. 6 is an example block diagram illustrating eye tracking matching implemented in connection with a virtual image display system according to some embodiments of the present disclosure.

[0017] [Figure 7]FIG. 7 is a simplified flowchart illustrating a method for determining and applying corrections to a virtual image display system according to some embodiments of the present disclosure.

[0018] [Figure 8] FIG. 8 is a simplified flowchart illustrating a method for determining and applying corrections to a virtual image display system to adjust display color according to some embodiments of the present disclosure.

[0019] [Figure 9] FIG. 9 is a simplified flowchart illustrating a method for determining and applying corrections to a virtual image display system and adjusting geometric attributes according to some embodiments of the present disclosure.

[0020] [Figure 10] FIG. 10 is a simplified flowchart illustrating a method for determining and applying corrections to a virtual image display system and adjusting spatial positioning according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] The embodiments are directed to image display systems and methods for display system calibration. Spatial computing allows digital world content to be overlaid on real world content in a spatially interactive manner through a combination of digital light fields, sensing, and computing. Digital content presented by spatial computing techniques preferably works in conjunction with real world surroundings and, more importantly, with the human eye-brain system, which is the ultimate judge of the success of the system. As a result, in order to develop such a spatial computing system, it will be essential for the human eye-brain system to have a surrogate to calibrate and verify the performance of the spatial computing system.

[0022] The embodiments provide a camera design that mimics human physiology (e.g., the human eye) in at least one or more of the following aspects: geometry, optical performance, and / or motion. Specifically, the embodiments provide a camera (e.g., an eye camera) that not only adopts the same corneal and pupil geometry from the human eye, but also has an iris and pupil that can be configured with multiple texture, color, or diameter options. Additionally, the resolution of the eye camera is designed to match the acuity of typical 20 / 20 human vision, and the focus can be adjusted, for example, from 0 to 4 diopters, by a piezo motor or other suitable mechanical system.

[0023] According to various embodiments, a pair of eye cameras are independently mounted on two hexapods to simulate the gaze and convergence-divergence movements of the human eye. With the help of the eye cameras, both the perceived virtual and real world content can be calibrated and evaluated in deterministic and quantifiable eye conditions, such as pupil location and gaze. According to various embodiments, the eye cameras act as a bridge, combining data from spatial computing such as eye tracking, 3D geometry of the digital world, display color accuracy / uniformity, and display optical quality (sharpness, contrast, etc.) for a holistic view, which helps to blend the virtual and real worlds together seamlessly and effectively.

[0024] 1 illustrates an exemplary system 100 for calibrating a virtual image display system 110. The system 100 may include a virtual image display system 110, an image capture device 120, and a data processing module 140. The virtual image display system 110 may include a display device 112 configured to display a virtual image, a real-world image capture device 150 configured to capture and display images on the display device 112, and various mechanical and electronic modules and systems to support the functionality of the virtual image display system 110. The display device 112 may be coupled to a frame 130, which is wearable by a user, wearer, or viewer of the display system and configured to position the display device 112 directly in front of the wearer's eyes during use. The display device 112 may be a light field display. In some embodiments, a speaker may be coupled to frame 130 and positioned adjacent the user's ear canal (in some embodiments, another speaker, not shown, is positioned adjacent the user's other ear canal to provide stereo / adjustable sound control). Real world image capture device 150 may be coupled to display device 112 via wired leads or wireless connectivity. In some embodiments, real world image capture device 150 may be provided on frame 130. In other embodiments, real world image capture device 150 may be provided separately from frame 130 and configured to be worn by a user of virtual image display system 110 (e.g., on the head as a head camera, on the body as a belt, bracelet, or watch).

[0025] According to various embodiments, image capture device 120 may be positioned in front of display device 112. The system may also include a data processing module 140 that is operably coupled to virtual image display system 110 and image capture device 120, such as by wired leads or wireless connectivity. Data processing module 140 may include various input / output devices and may receive data from external modules.

[0026] Data processing module 140 may include one or more processors configured to analyze and process data and / or image information, such as image or video information captured by real-world image capture device 150. Image or video data may be stored locally within data processing module 140 and / or remotely in a remote data repository. In some embodiments, the remote data repository may comprise a digital data storage facility, which may be available through the Internet or other networking configurations in a "cloud" resource configuration. In some embodiments, all data is stored and all computations are performed within data processing module 140, allowing for fully autonomous use from the remote modules, while in other embodiments, data storage and / or computations are distributed. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0027] Display device 112 may include a first display 106 (e.g., a left display element) and a second display 108 (e.g., a right display element). Similarly, image capture device 120 may include a first image capture element 102 (e.g., a left image capture element) and a second image capture element 104 (e.g., a right image capture element). Image capture device 120 may be positioned in front of display device 112 such that the first image capture element 102 is positioned in front of the first display 106 and the second image capture element 104 is positioned in front of the second display 108.

[0028] According to various embodiments, the image capture device 120 may simulate a human eye. That is, the image capture device 120 may include properties similar to a human eye. For example, the first image capture element 102 and the second image capture element 104 may each include a corneal position, a corneal geometry, an eye position, a pupil size, a pupil position, a line of sight distance, a line of sight orientation, and a iris color. These properties may be configurable and controlled for each of the first image capture element 102 and the second image capture element 104 independently of each other. For example, the first image capture element 102 and the second image capture element 104 may include an eye camera.

[0029] 2A and 2B illustrate a first image capture element 102 according to some embodiments of the present disclosure. The second image capture element 104 is similar (or the same) as the first image capture element 102, and the description of the first image capture element 102 is also applicable to the second image capture element 104. The first image capture element 102 and / or the second image capture element 104 may include an artificial cornea 200, an artificial pupil 202, and an artificial iris 204 that simulates a human eye and is mounted on the end of a lens mount 206 that is coupled to a camera body 210. The artificial cornea 200, the artificial pupil 202, and the artificial iris 204 may be referred to as an artificial eye 220. The camera body 210 may include a light-tight box that houses the camera elements, such as a camera shutter and an image sensor.

[0030] Various properties of the artificial cornea 200, the artificial pupil 202, and the artificial iris 204 may be configured according to various embodiments. For example, the position and / or geometry of the artificial cornea 200, the position and size of the artificial pupil 202, and the position and / or color of the artificial iris 204 may be configured, customized, or otherwise adjusted according to various embodiments. Similarly, the position of the artificial eye 220 at the end of the lens mount 206 may also be configured. The embodiments allow for determining and adjusting the convergence-divergence movement (including convergence, i.e., the rotation of the eyes toward each other when viewing a closer object, and divergence, i.e., the rotation of the eyes away from each other when viewing a farther away object), the gaze distance, and / or the gaze orientation of the first image capture element 102 and / or the second image capture element 104. For example, the embodiments allow for fixing the gaze distance at infinity for a desired (predetermined) amount of time. Because the human eye is unable to maintain a constant line of sight at infinity, the first image capture element 102 and / or the second image capture element 104 may mimic the capabilities of the human eye and be constructed accordingly. Thus, the embodiments enable more accurate calibration of the virtual image display system 110 (i.e., the real-world image capture device 150 of the virtual image display system 110).

[0031] According to various embodiments, the properties of the first image capture element 102 can be configured and adjusted independently from the second image capture element 104. Similarly, the first image capture element 102 can be controlled to move independently from the second image capture element 104. This may be accomplished by mounting each of the first image capture element 102 and the second image capture element 104 on a separate individual hexapod, as illustrated in FIG.

[0032] 3 is a simplified schematic diagram illustrating a camera rig 300 according to some embodiments of the present disclosure. The camera rig 300 may include a first controllable mount 302 and a second controllable mount 304. The controllable mounts 302, 304 may be controlled to move independently of each other or simultaneously with each other. The controllable mounts 302, 304 may be controlled to mimic human eye muscles as well as the human head.

[0033] 3, the first image capture element 102 may be mounted on a first controllable mount 302, and the second image capture element 104 may be mounted on a second controllable mount 304. The camera rig 300 and the image capture elements 102, 104 may work together to mimic (e.g., simulate) a pair of human eyes moving within their sockets.

[0034] According to various embodiments, the controllable mounts 302, 304 may include one or more actuators 306 (e.g., linear actuators, prismatic joints, or the like) coupled to a bottom platform 308 and / or a top platform 310. In some embodiments, the controllable mounts 302, 304 may be in the form of a parallel manipulator including six linear actuators.

[0035] The controllable mounts 302, 304 may be controlled via a computing device 350 configured to communicate with the controllable mounts 302, 304 via a wired or wireless connection. In some embodiments, the computing device 350 may include or be coupled to the data processing module 140 to function in conjunction with the data processing module 140. According to various embodiments, the computing device 350 may send commands (e.g., in the form of signals) to and receive feedback (e.g., in the form of signals) from the controllable mounts 302, 304.

[0036] According to various embodiments, the camera rig 300 may be used to fine-tune the calibration of the virtual image display system during production of the virtual image display system. Additionally, in some embodiments, the camera rig 300 may be sized and dimensioned to be provided to a retail location where a user may bring in their virtual image display system 110 for calibration or to fine-tune the virtual image display system according to the user's particular eye characteristics (e.g., geometry, color, vision).

[0037] Embodiments allow for calibrating a virtual image display system (e.g., a real-world image capture device of an AR / VR system) using an image capture device (including one or more image capture elements). In some embodiments, the image capture device may be controlled or configured to mimic the human eye.

[0038] FIG. 4 shows an example block diagram illustrating a method for calibrating a virtual image display system according to some embodiments of the present disclosure. An image 450 of a scene, environment, or object in a real world 400 may be captured using a real-world image capture device (e.g., a world camera) 402. The captured image 450 may be sent to a rendering engine 410, which then processes the received image and transmits the processed image to a projector 412, which will then display it on a display device (e.g., an eyepiece of the virtual image display system) 414. The displayed image may correspond to a group of pixels 416 illuminated on the display device 414. However, the displayed image may not exactly match (e.g., align) with the real world 400 that the image is said to represent. For example, the contours of an object in the image may not align with the contours of the object in the real world. To correct this misalignment, the real-world image capture device 402 may need to be calibrated.

[0039] For example, images captured using the real-world image capture device 402 may be processed (e.g., refined) using data from an eye tracking camera 404 and one or more eye tracking light emitters (e.g., light emitting diodes (LEDs)) 406. The data from the eye tracking camera 404 and one or more eye tracking light emitters 406 may include additional information about where a user is looking to render an image at a specific depth or range. For example, eye vergence may be determined using the eye tracking camera 404 and one or more eye tracking light emitters 406, and calibration / accommodation for a virtual image display system may be determined using a vergence / accommodation model.

[0040] According to various embodiments, the eye tracking camera 404 detects the user's corneal position and, based on the corneal position, determines the eye aperture and calculates the user's gaze. That is, the eye tracking camera 404 estimates the user's eye position. Therefore, it would be desirable to know details of the user's eye geometry, position, and gaze in order to more accurately calibrate the virtual image display system.

[0041] An embodiment may use an image capture device 408 to simulate a human eye (or a pair of human eyes) to calibrate the virtual image display system. The image capture device 408 may be used in conjunction with an eye tracking camera 404. According to various embodiments, the image capture device 408 may have the physical properties of a human eye. For example, as described in connection with FIGS. 2A-2B, the image capture device 408 may have a geometry similar to that of a human eye, including an artificial cornea, an artificial pupil, and an artificial iris. Thus, the eye tracking camera 404 and one or more eye tracking light emitters 406, when used in conjunction with the image capture device 408, provide more accurate data for calibrating the virtual image display system.

[0042] Because the properties of the image capture device 408 are known, the eye tracking camera 404 and one or more eye tracking light emitters 406 use data regarding the properties (e.g., corneal position, corneal geometry, eye position, pupil size, pupil position, line of sight distance, and gaze orientation) instead of estimating or determining these properties. Corrections for the virtual image display system may be determined based on one or more of these properties of the image capture device 408 to more accurately match an image 450 received from the real-world image capture device 402 with an image 460 of the same scene, environment, or object in the real world 400 captured using the image capture device 408.

[0043] The rendering engine 410 may receive the image 460 from the image capture device 408 and compare the image 460 with the image 450 received from the real-world image capture device 402. If it is determined that the images 460 and 450 do not overlap to a predetermined degree or target, the rendering engine 410 may determine one or more corrections to be applied to the real-world image capture device 402 to more accurately capture an image of the scene, environment, or object in the real world 400. The virtual image display system (more specifically, the real-world image capture device 402 and / or the display device 414) may be calibrated until the image 450 captured by the real-world image capture device 402 matches the image 460 captured using the image capture device 408 to within a predetermined threshold.

[0044] FIG. 5 shows an exemplary block diagram illustrating various calibrations that may be performed in connection with a virtual image display system. As also described in connection with FIG. 1, the virtual image display system 500 may have a sensing component 502 (e.g., real-world image capture device 150) and a display component 504 (e.g., display device 112). The sensing component 502 may be calibrated using an eye-tracking camera calibration 506 and an eye-tracking LED calibration 508 to verify and refine eye tracking 514. The display component 504 can be calibrated using a geometry calibration 510 to verify and refine the geometry 516 of the displayed objects and a color calibration 512 to verify and refine the color 518 of the displayed objects. Overall, the display quality 520 of the virtual image display system may be verified and refined.

[0045] An image capture device simulating the human eye as described in connection with various embodiments may be used to combine and perform all the above described calibrations in parallel to improve eye tracking, geometry, color, and display quality for the virtual image display system.

[0046] 6 shows an example block diagram illustrating a detailed view of eye tracking matching. The eye tracking camera 604 and eye tracking LED 606 can estimate properties 610 of the image capture device 602, such as corneal position, corneal geometry, eye position, pupil size, pupil position, gaze distance, and gaze orientation. The estimated properties 610 may be provided to a computing device 614 (e.g., data processing module 140) along with actual values ​​612 for these properties. The computing device may compare the estimated properties 610 with the actual values ​​612 for these properties of the image capture device 602 and determine one or more corrections 620 for the virtual image display system based on gaze direction, depth plane switching, rendering center, convergence plane, etc.

[0047] 7 is a simplified flowchart 700 illustrating a method for determining and applying corrections to a virtual image display system according to some embodiments of the present disclosure. The virtual image display system may include at least a display device and a world camera. According to various embodiments, an image capture element simulating a human eye (or a pair of human eyes) may be coupled to the virtual image display system.

[0048] In block 702, a computing device including a processor receives a first image and a second image from a virtual image display system. The first image may be captured using a world camera of the virtual image display system, and the second image may be captured using an image capture device (e.g., an eye camera). The image capture device has one or more properties similar to a human eye, including, but not limited to, corneal position, corneal geometry, eye position, pupil size, pupil position, gaze distance, and / or gaze orientation. These properties of the image capture device may have predetermined values ​​or may be configurable.

[0049] At block 704, the computing device may determine visual properties of the first image and the second image. The computing device may initially determine a first set of visual properties associated with the first image and a second set of visual properties associated with the second image. The first set of visual properties and the second set of visual properties may include special positioning of the object (e.g., the location, coordinates, or distance between two objects or points in the image of the object), color of the object (e.g., the hue, saturation, or contrast of the object), geometric shape attributes of the object (collinearity, curvature, length, width, category, added marks, missing marks, multiplicity, shape, size, spatial grouping, and spatial orientation of the elements).

[0050] At block 706, the computing device may identify differences between the first set of visual properties of the first image and the second set of visual properties of the second image.

[0051] In block 708, the computing device may use one or more known properties of the image capture device to determine a correction to apply to the virtual image display system to at least partially correct for the discrepancy.

[0052] In block 710, the computing device may apply a correction to the virtual image display system based on one or more properties of the image capture device. The correction may calibrate a display device or a world camera of the virtual image display system.

[0053] It should be understood that the specific steps illustrated in FIG. 7 provide a particular method for determining and applying corrections to a virtual image display system according to some embodiments of the present disclosure. Other sequences of steps may also be performed in accordance with alternative embodiments. For example, alternative embodiments may perform the steps outlined in FIG. 7 in a different order. Furthermore, individual steps illustrated in FIG. 7 may include multiple sub-steps that may be performed in various sequences depending on the needs of the individual steps. Furthermore, additional steps may be added or removed depending on the particular application. Those skilled in the art will recognize numerous variations, modifications, and alternatives.

[0054] 8 is a simplified flowchart 800 illustrating a method for determining and applying corrections to a virtual image display system to adjust display color according to some embodiments of the present disclosure. The virtual image display system may include at least a display device and a world camera. According to various embodiments, an image capture element simulating a human eye (or a pair of human eyes) may be coupled to the virtual image display system.

[0055] In block 802, a computing device including a processor receives a first image and a second image from a virtual image display system. The first image may be captured using a world camera of the virtual image display system, and the second image may be captured using an image capture device (e.g., an eye camera). The image capture device has one or more properties similar to a human eye, including, but not limited to, corneal position, corneal geometry, eye position, pupil size, pupil position, gaze distance, and / or gaze orientation. These properties of the image capture device may have predetermined values ​​or may be configurable.

[0056] At block 804, the computing device may determine visual properties of the first image and the second image. The computing device may first determine the spatial location of a point in the first image and the spatial location of a corresponding point in the second image.

[0057] At block 806, the computing device may identify differences between the spatial positioning of the point in the first image and the spatial positioning of the corresponding point in the second image.

[0058] In block 808, the computing device may determine a correction to apply to the virtual image display system to align points in the first image with corresponding points in the second image.

[0059] In block 810, the computing device may apply a correction to the virtual image display system based on one or more properties of the image capture device. The correction may calibrate a display device or world camera of the virtual image display system and result in points of the first image more accurately aligning with corresponding points in the second image.

[0060] It should be understood that the specific steps illustrated in FIG. 8 provide a particular method for determining and applying corrections to a virtual image display system and adjusting display color according to some embodiments of the present disclosure. Other sequences of steps may also be performed in accordance with alternative embodiments. For example, alternative embodiments may perform the steps outlined in FIG. 8 in a different order. Furthermore, individual steps illustrated in FIG. 8 may include multiple sub-steps that may be performed in various sequences depending on the needs of the individual steps. Furthermore, additional steps may be added or removed depending on the particular application. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0061] 9 is a simplified flowchart 900 illustrating a method for determining and applying corrections to a virtual image display system and adjusting geometric attributes according to some embodiments of the present disclosure. The virtual image display system may include at least a display device and a world camera. According to various embodiments, an image capture element simulating a human eye (or a pair of human eyes) may be coupled to the virtual image display system.

[0062] In block 902, a computing device including a processor receives a first image and a second image from a virtual image display system. The first image may be captured using a world camera of the virtual image display system, and the second image may be captured using an image capture device (e.g., an eye camera). The image capture device has one or more properties similar to a human eye, including, but not limited to, corneal position, corneal geometry, eye position, pupil size, pupil position, gaze distance, and / or gaze orientation. These properties of the image capture device may have predetermined values ​​or may be configurable.

[0063] At block 904, the computing device may determine visual properties of the first image and the second image. The computing device may determine the color (e.g., hue, saturation, or contrast) of an area (e.g., a point, object, or pixel) in the first image and the color of a corresponding area in the second image.

[0064] At block 906, the computing device may identify a difference between a color of an area in the first image and a color of a corresponding area in the second image.

[0065] In block 908, the computing device may determine a correction to apply to the virtual image display system to match the color of an area in the first image with the color of a corresponding area in the second image.

[0066] In block 910, the computing device may apply a correction to the virtual image display system based on one or more properties of the image capture device. The correction may calibrate a display device or world camera of the virtual image display system and cause the color of an area in the first image to more accurately match the color of a corresponding area in the second image. The correction may adjust one or more of the hue, saturation, and contrast of the image displayed on the display device.

[0067] It should be understood that the specific steps illustrated in FIG. 9 provide a particular method for determining and applying corrections to a virtual image display system and adjusting geometric attributes according to some embodiments of the present disclosure. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments may perform the steps outlined in FIG. 9 in a different order. Furthermore, individual steps illustrated in FIG. 9 may include multiple sub-steps that may be performed in various sequences depending on the needs of the individual steps. Furthermore, additional steps may be added or removed depending on the particular application. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0068] 10 is a simplified flowchart 1000 illustrating a method for determining and applying corrections to a virtual image display system and adjusting spatial positioning according to some embodiments of the present disclosure. The virtual image display system may include at least a display device and a world camera. According to various embodiments, an image capture element simulating a human eye (or a pair of human eyes) may be coupled to the virtual image display system.

[0069] In block 1002, a computing device including a processor receives a first image and a second image from a virtual image display system. The first image may be captured using a world camera of the virtual image display system, and the second image may be captured using an image capture device (e.g., an eye camera). The image capture device has one or more properties similar to a human eye, including, but not limited to, corneal position, corneal geometry, eye position, pupil size, pupil position, gaze distance, and / or gaze orientation. These properties of the image capture device may have predetermined values ​​or may be configurable.

[0070] At block 1004, the computing device may determine visual properties of the first image and the second image. The computing device may determine geometric attributes (e.g., form, collinearity, curvature, length, width, category, added marks, missing marks, multiplicity, shape, size, spatial grouping, and / or spatial orientation) of objects in the first image and geometric attributes of corresponding objects in the second image.

[0071] At block 1006, the computing device may identify differences between geometric attributes of an object in the first image and geometric attributes of a corresponding object in the second image.

[0072] At block 1008, the computing device may determine a correction to apply to the virtual image display system to match geometric attributes of an object in the first image with geometric attributes of a corresponding object in the second image.

[0073] In block 1010, the computing device may apply a correction to the virtual image display system based on one or more properties of the image capture device. The correction may calibrate a display device or world camera of the virtual image display system and result in geometric attributes of objects in a first image more accurately matching geometric attributes of corresponding areas in a second image. The correction may adjust one or more of the collinearity, curvature, length, width, category, added marks, missing marks, multiplicity, shape, size, spatial grouping, and / or spatial orientation of objects displayed on the display device.

[0074] It should be understood that the specific steps illustrated in FIG. 10 provide a particular method for determining and applying corrections to a virtual image display system and adjusting spatial positioning according to some embodiments of the present disclosure. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments may perform the steps outlined in FIG. 10 in a different order. Furthermore, the individual steps illustrated in FIG. 10 may include multiple sub-steps that may be performed in various sequences depending on the needs of the individual steps. Furthermore, additional steps may be added or removed depending on the particular application. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0075] The embodiments provide several advantages over prior art systems. The embodiments allow for calibrating a real-world image capture device and / or an AR / VR display system using one or more cameras (e.g., eye cameras) that mimic the human eye. Properties of the eye cameras, such as one or more of corneal position, corneal geometry, eye position, pupil size, pupil position, line of sight distance, or line of sight orientation, can be known and controlled. Thus, calibration can be performed using a set of known and controllable parameters. For example, it is possible to fix the line of sight on infinity and use the data to calibrate the device. Thus, the embodiments provide a more accurate calibration of the AR / VR display system, thus resulting in a more accurate and seamless AR / VR experience.

[0076] Each of the processes, methods, and algorithms described herein and / or depicted in the accompanying figures may be embodied in code modules executed by one or more physical computing systems, hardware computer processors, application-specific circuits, and / or electronic hardware configured to execute specific and particular computer instructions, and thus may be fully or partially automated. For example, the computing systems may include general-purpose computers (e.g., servers) or special-purpose computers programmed with specific computer instructions, special-purpose circuits, etc. Code modules may be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language. In some implementations, certain operations and methods may be performed by circuitry specific to a given function.

[0077] Furthermore, certain implementations of the functionality of the present disclosure may be sufficiently mathematically, computationally, or technically complex that special purpose hardware (utilizing appropriate specialized executable instructions) or one or more physical computing devices may be required to implement the functionality, e.g., due to the amount or complexity of the calculations involved, or to provide results in substantially real-time. For example, a video may contain many frames, each frame may have millions of pixels, and specifically programmed computer hardware may be required to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time.

[0078] The code modules or any type of data may be stored on any type of non-transitory computer readable medium, such as physical computer storage devices, including hard drives, solid state memory, random access memory (RAM), read only memory (ROM), optical disks, volatile or non-volatile storage devices, combinations of the same, and / or the like. The methods and modules (or data) may also be transmitted (e.g., as part of a carrier wave or other analog or digital propagating signal) generated on a variety of computer readable transmission media, including wireless-based and wired / cable-based media, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored persistently or otherwise in any type of non-transitory tangible computer storage device or communicated via a computer readable transmission medium.

[0079] Any process, block, state, step, or functionality in the flow diagrams described herein and / or depicted in the accompanying figures should be understood as potentially representing a code module, segment, or portion of code that includes one or more executable instructions for implementing a specific function (e.g., logical or arithmetic) or step in the process. Various processes, blocks, states, steps, or functionality can be combined, rearranged, added to, removed from, modified, or otherwise altered from the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules may perform some or all of the functionality described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps, or states associated therewith can be performed in other sequences as appropriate, e.g., serially, in parallel, or in some other manner. Tasks or events can be added to or removed from the disclosed exemplary embodiments. Furthermore, the separation of various system components in the implementations described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods, and systems may generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variations are possible.

[0080] The process, method, and system may be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowdsourced computing networks, the Internet, and the World Wide Web. The network may be a wired or wireless network or any other type of communication network.

[0081] The systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or required for the desirable attributes disclosed herein. The various features and processes described above may be used independently of one another or may be combined in various ways. All possible combinations and subcombinations are intended to fall within the scope of the present disclosure. Various modifications of the implementations described in the present disclosure may be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other implementations without departing from the spirit or scope of the present disclosure. Thus, the claims are not intended to be limited to the implementations shown herein, but should be accorded the widest scope consistent with the present disclosure, the principles, and novel features disclosed herein.

[0082] Certain features described herein in the context of separate implementations can also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation can also be implemented separately in multiple implementations or in any suitable subcombination. Furthermore, although features may be described above as acting in a combination and may be initially claimed as such, one or more features from the claimed combination may in some cases be deleted from the combination, and the claimed combination may be subject to subcombinations or variations of the subcombination. No single feature or group of features is required or essential to every embodiment.

[0083] In particular, conditional statements used herein, such as "can," "could," "might," "may," "eg," and the like, are generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not, unless specifically stated otherwise or understood otherwise within the context as used. Thus, such conditional statements are generally not intended to agree that features, elements, and / or steps are in any way required for one or more embodiments, or that one or more embodiments necessarily include logic for determining whether those features, elements, and / or steps should be included or performed in any particular embodiment, with or without authorial input or prompting. The terms "comprising," "including," "having," and the like, are synonymous and used inclusively in a non-limiting manner and do not exclude additional elements, features, acts, operations, etc. Also, the term "or" is used in its inclusive sense (and not its exclusive sense), so that, for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, the articles "a," "an," and "the," as used in this application and the appended claims, should be interpreted to mean "one or more" or "at least one," unless otherwise specified.

[0084] As used herein, a phrase referring to a list of items "at least one of" refers to any combination of those items, including single elements. As an example, "at least one of A, B, or C" is intended to cover A, B, C, A and B, A and C, B and C, and A, B, and C. Transitive phrases such as "at least one of X, Y, and Z" are generally understood differently in the context in which they are used to convey that an item, term, etc. may be at least one of X, Y, or Z, unless specifically stated otherwise. Thus, such transitive phrases are generally not intended to suggest that an embodiment requires that at least one of X, at least one of Y, and at least one of Z, respectively, be present.

[0085] Similarly, although operations may be depicted in the figures in a particular order, it should be appreciated that such operations need not be performed in the particular order depicted, or in sequential order, or that all of the depicted operations need not be performed to achieve desirable results. Additionally, the figures may diagrammatically depict one or more exemplary processes in the form of a flow chart. However, other operations not depicted may be incorporated within the diagrammatically depicted exemplary methods and processes. For example, one or more additional operations may be performed before, after, simultaneously with, or during any of the depicted operations. Additionally, operations may be rearranged or reordered in other implementations. In some circumstances, multitasking and parallel processing may be advantageous. Additionally, the separation of various system components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products. Additionally, other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results.

Claims

1. 1. A system for applying corrections to a virtual image display system, the system comprising: a display device configured to display a virtual image; an image capture device positioned in front of the display device, the image capture device having one or more artificial eyes configured with one or more artificial corneas, artificial pupils, or artificial irises having one or more properties; and a processor coupled to the display device and the image capture device to receive image data from the image capture device, the processor comprising: Receiving a first image of a scene from a real world image capture device; determining a first set of visual properties associated with the first image; receiving a second image of the scene captured with the image capture device; and determining a second set of visual properties associated with the second image, wherein the first set of visual properties and the second set of visual properties include at least one of a spatial orientation, a color, or a geometric shape attribute; identifying differences between the first set of visual properties and the second set of visual properties; determining a correction to apply to the virtual image display system using one or more properties of the image capture device to at least partially correct for the difference; and applying the correction to the virtual image display system based on one or more characteristics of the image capture device; and a processor programmed to A system comprising:

2. 2. The system of claim 1, wherein the one or more properties of the image capture device include one or more of a corneal position of the artificial cornea, a corneal geometry of the artificial cornea, an ocular position of the artificial eye, a pupil size of the artificial pupil, a pupil position of the artificial pupil, a gaze distance of the artificial eye, or a gaze orientation of the artificial eye.

3. The system of claim 1 , wherein the image capture device is used in conjunction with an eye tracking camera to estimate the one or more characteristics of the image capture device.

4. 2. The system of claim 1, wherein the display device comprises a first display associated with a first eye of a user and a second display associated with a second eye of the user, and the image capture device includes a first image capture element positioned in front of the first display and a second image capture element positioned in front of the second display.

5. The system of claim 4 , wherein the first image capture element and the second image capture element are configured to move independently of one another.

6. 2. The system of claim 1, further comprising a real world image capture device coupled to the display device and configured to capture the first image, and wherein the correction is applied to the virtual image display system to calibrate the real world image capture device.

7. The first set of visual properties and the second set of visual properties include spatial positioning, and the processor further comprises: determining a spatial location of a point in the first image; identifying differences between spatial orientations of points in the first image and spatial orientations of corresponding points in the second image; determining a correction to apply to the virtual image display system to align points in the first image with corresponding points in the second image; applying said correction to said virtual image display system; and The system of claim 1 , programmed to:

8. The first set of visual properties and the second set of visual properties include color, and the processor further comprises: determining a color of an area in the first image; identifying a difference between a color of an area in the first image and a color of a corresponding area in the second image; determining a correction to apply to the virtual image display system to match a color of an area in the first image with a color of a corresponding area in the second image; applying said correction to said virtual image display system; and The system of claim 1 , programmed to:

9. The system of claim 8 , wherein the correction comprises one or more of a correction in hue, saturation, or contrast.

10. The first set of visual properties and the second set of visual properties include geometric shape attributes, and the processor further comprises: determining geometric attributes of an object in the first image; identifying differences between geometric attributes of objects in the first image and geometric attributes of corresponding objects in the second image; determining a correction to apply to the virtual image display system to match geometric attributes of objects in the first image with geometric attributes of corresponding objects in the second image; applying said correction to said virtual image display system; and The system of claim 1 , programmed to:

11. The system of claim 10 , wherein the correction comprises one or more of a correction in curvature, length, width, shape, size, or spatial orientation of an object in the second image.

12. 1. A method for applying a correction to a virtual image display system, the method comprising: receiving, by a processor, a first image of the scene from a real world image capture device; determining, by the processor, a first set of visual properties associated with the first image; receiving, by the processor, from an image capture device a second image of the scene captured with the image capture device having one or more artificial eyes configured with one or more artificial corneas, artificial pupils, or artificial irises having one or more properties, the image capture device being positioned in front of a display device; determining, by the processor, a second set of visual properties associated with the second image, the first set of visual properties and the second set of visual properties including at least one of spatial orientation, color, or geometric shape attributes; identifying, by the processor, differences between the first set of visual properties and the second set of visual properties; determining, by the processor, a correction to apply to the virtual image display system using one or more properties of the image capture device to at least partially correct for the difference; applying, by the processor, the correction to the virtual image display system based on one or more characteristics of the image capture device; A method comprising:

13. 13. The method of claim 12, wherein the one or more properties of the image capture device include one or more of a corneal position of the artificial cornea, a corneal geometry of the artificial cornea, an ocular position of the artificial eye, a pupil size of the artificial pupil, a pupil position of the artificial pupil, a gaze distance of the artificial eye, or a gaze orientation of the artificial eye.

14. The method of claim 12 , wherein the image capture device is used in conjunction with an eye-tracking camera such that the processor estimates the one or more characteristics of the image capture device.

15. 13. The method of claim 12, wherein the display device comprises a first display associated with a first eye of a user and a second display associated with a second eye of the user, and the image capture device includes a first image capture element positioned in front of the first display and a second image capture element positioned in front of the second display.

16. The method of claim 15 , wherein the first image capture element and the second image capture element are configured to move independently of one another.

17. The first image is received from a real world image capture device coupled to the display device, and applying the correction further comprises: The method of claim 12 , comprising calibrating the real world image capture device.

18. The first set of visual properties and the second set of visual properties include spatial positioning, and the method further comprises: determining, by the processor, a spatial location of a point in the first image; identifying, by the processor, a difference between a spatial positioning of a point in the first image and a spatial positioning of a corresponding point in the second image; determining, by the processor, a correction to apply to the virtual image display system to align points in the first image with corresponding points in the second image; applying, by the processor, the correction to the virtual image display system; The method of claim 12, comprising:

19. The first set of visual properties and the second set of visual properties include color, and the method further comprises: determining, by the processor, a color of an area in the first image; identifying, by the processor, a difference between a color of an area in the first image and a color of a corresponding area in the second image; determining, by the processor, a correction to apply to the virtual image display system to match a color of an area in the first image with a color of a corresponding area in the second image; applying, by the processor, the correction to the virtual image display system; The method of claim 12, comprising:

20. The method of claim 12 , wherein the correction comprises one or more of a correction in hue, saturation, or contrast.

21. The first set of visual properties and the second set of visual properties include geometric shape attributes, and the method further comprises: determining, by the processor, geometric attributes of an object in the first image; identifying, by the processor, differences between geometric attributes of objects in the first image and geometric attributes of corresponding objects in the second image; determining, by the processor, a correction to apply to the virtual image display system to match geometric attributes of objects in the first image with geometric attributes of corresponding objects in the second image; applying, by the processor, the correction to the virtual image display system; The method of claim 12, comprising:

22. The method of claim 21 , wherein the correction comprises one or more of a correction in curvature, length, width, shape, size, or spatial orientation of an object in the second image.

Citation Information

Patent Citations

  • Method and apparatus for aligning stereoscopic images

    JP2001517414A

  • Imaging device and imaging method, and display and display method

    JP2004015256A

  • Eyeglass-shaped binocular display driver

    JP2009536477A

  • Determination of model parameters based on model transformation of the object.

    JP2014511608A

  • Dynamic display calibration based on eye tracking

    JP2019501564A