Eye camera system and method for display system calibration

Abstract

Embodiments provide camera designs (e.g., eyeball cameras) that mimic the human eye in terms of geometry, optical properties, and / or motion. The eyeball camera uses the same cornea and pupil geometry as the human eye, and has an iris and pupil configured with a variety of textures, colors, or diameter options. The resolution of the eyeball camera is designed to match the acuity of typical 20 / 20 human vision, and the focal length is adjusted from 0 to 4 diopters. A pair of eyeball cameras are independently mounted on two hexapods to simulate the gaze and convergence of the human eye. The virtual and real worlds perceived are calibrated and evaluated based on eye conditions such as pupil position and gaze using eyeball cameras. The eyeball camera acts as a bridge to combine data from spatial computing (such as eye tracking, 3D geometry of the digital world, display color accuracy / uniformity, and display optical quality (clarity, contrast, etc.)).

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a non-provisional application and claims the benefit of U.S. Patent Application No. 62 / 873,641, filed on July 12, 2019, entitled “EYEBALL CAMERA SYSTEM AND METHODS FOR DISPLAY SYSTEM CALIBRATION,” the entire contents of which are incorporated herein by reference for all purposes. Background Art

[0003] Modern computing and display technologies have facilitated the development of systems for so-called "virtual reality" or "augmented reality" experiences, in which digitally generated images, or portions thereof, are presented to a user in a wearable device in a manner that appears or could be perceived as real. Virtual reality or "VR" scenarios generally involve the presentation of digital or virtual image information that is opaque to other actual real-world visual input; augmented reality or "AR" scenarios generally involve the presentation of digital or virtual image information as an enhancement to the visualization of the actual world around the user.

[0004] The wearable device may include augmented and / or virtual reality glasses. A camera may be coupled to the glasses. In order to view the actual position of a real-world object on the augmented and / or virtual reality glasses, the camera needs to be calibrated. Calibrating the camera may include determining intrinsic and / or extrinsic parameters of the camera. The intrinsic parameters represent a projective transformation from 3D camera coordinates to 2D image coordinates. The intrinsic parameters may include focal length (f x , f y ), principal point, and distortion coefficients. On the other hand, extrinsic parameters represent the transformation from the world coordinate system to the camera's coordinate system. The extrinsic parameters include the rotation matrix and the translation vector. The extrinsic parameters can help determine the location of the camera center and the camera's heading in world coordinates.

[0005] Therefore, camera calibration can estimate the parameters of the camera's lens and image sensor. The determined parameters can be used to correct lens distortion, measure the size of objects in world units, or determine the position of the camera in the scene in 3D scene reconstruction.

[0006] It turns out that the human visual perception system is very complex, and it is challenging for VR, AR or MR technologies to facilitate comfortable, natural feeling, rich presentation of virtual image elements within other virtual or real world images. The systems and methods disclosed herein address various challenges associated with VR, AR and MR technologies. Summary of the invention

[0007] Embodiments generally relate to image display systems and methods for display system calibration. Embodiments provide camera designs (e.g., eyeball cameras) that mimic human eyes in terms of geometry, optical performance, and / or motion. Eyeball cameras use the same cornea and pupil geometry as human eyes, and have irises and pupils configured with a variety of textures, colors, or diameter options. The resolution of the eyeball camera is designed to match the acuity of typical 20 / 20 human vision, and in some implementations, the focus is adjusted from 0 to 4 diopters. A pair of eyeball cameras can be independently mounted on two hexapods to simulate the gaze and convergence of the human eye. Therefore, eyeball cameras can be used to calibrate and evaluate the perceived virtual and real world based on eye conditions such as pupil position and gaze. Eyeball cameras act as a bridge to combine data from spatial computing (such as eye tracking, 3D geometry of the digital world, display color accuracy / uniformity, and display optical quality (clarity, contrast, etc.).

[0008] Various embodiments provide a system for applying correction to a virtual image display system. The system includes: a display device configured to display a virtual image; an image capture device positioned in front of the display device; and a processor coupled to the display device and the image capture device to receive image data from the image capture device. The image capture device has one or more characteristics similar to a human eye. The processor is programmed to (a) receive a first image of a scene; (b) determine a first set of visual characteristics associated with the first image; (c) receive a second image of the scene captured with the image capture device; (d) determine a second set of visual characteristics associated with the second image; (e) identify a difference between the first set of visual characteristics and the second set of visual characteristics; (f) determine a correction to be applied to the virtual image display system using one or more characteristics of the image capture device to at least partially correct the difference; and (g) apply the correction to the virtual image display system based on the one or more characteristics of the image capture device.

[0009] Various embodiments provide a method for applying correction to a virtual image display system. The method includes receiving, by a processor, a first image of a scene from a real-world image capture device. The processor determines a first set of visual characteristics associated with the first image. The method further includes receiving, by the processor, from the image capture device, a second image of the scene captured by an image capture device, the image capture device having one or more characteristics similar to a human eye. The image capture device is positioned in front of a display device. The method also includes determining, by the processor, a second set of visual characteristics associated with the second image; and identifying, by the processor, a difference between the first set of visual characteristics and the second set of visual characteristics. The processor determines, using one or more characteristics of the image capture device, a correction to be applied to the virtual image display system to at least partially correct the difference; and applies the correction to the virtual image display system based on the one or more characteristics of the image capture device.

[0010] Compared with conventional techniques, many benefits are achieved through the present disclosure. For example, embodiments provide methods and systems for calibrating real-world image capture devices and / or AR / VR display systems using one or more cameras (e.g., eyeball cameras) that simulate human eyes. The characteristics of the eyeball camera (such as one or more of corneal position, corneal geometry, eyeball position, pupil size, pupil position, gaze distance, or gaze orientation) are known and can be controlled. Therefore, calibration can be performed using a set of known and controllable parameters. For example, the gaze can be fixed at infinity and the device can be calibrated using data. Therefore, embodiments provide more accurate calibration of AR / VR display systems, resulting in a more accurate and seamless AR / VR experience.

[0011] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Neither this summary nor the following detailed description is intended to define or limit the scope of the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An exemplary system for calibrating a display device according to some embodiments of the present disclosure is shown;

[0013] Figure 2A-2B illustrates an exemplary image capture element according to some embodiments of the present disclosure;

[0014] Figure 3 is a simplified schematic diagram illustrating a camera rig according to some embodiments of the present disclosure;

[0015] Figure 4 is an exemplary block diagram illustrating a method for calibrating a virtual image display system according to some embodiments of the present disclosure;

[0016] Figure 5 is an exemplary block diagram illustrating various calibrations that may be performed in conjunction with a virtual image display system according to some embodiments of the present disclosure;

[0017] Figure 6 is an exemplary block diagram illustrating eye tracking verification performed in conjunction with a virtual image display system according to some embodiments of the present disclosure;

[0018] Figure 7 is a simplified flow chart illustrating a method for determining and applying corrections to a virtual image display system according to some embodiments of the present disclosure;

[0019] Figure 8 is a simplified flow chart illustrating a method for determining and applying corrections to a virtual image display system to adjust display color according to some embodiments of the present disclosure;

[0020] Fig. 9 is a simplified flow chart illustrating a method for determining and applying corrections to a virtual image display system to adjust geometric properties according to some embodiments of the present disclosure; and

[0021] Fig.10 is a simplified flow chart illustrating a method for determining and applying corrections to a virtual image display system to adjust spatial positioning according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] Embodiments relate to image display systems and methods for display system calibration. Spatial computing overlays digital world content on real world content in a spatially interactive manner by combining digital light fields, sensing, and computing. The digital content presented by spatial computing technology preferably works in coordination with the real world environment and more importantly with the human eye-brain system, which is the ultimate criterion for the success of the system. Therefore, to develop such spatial computing systems, it is necessary to have a proxy for the human eye-brain system to calibrate and verify the performance of the spatial computing system.

[0023] Embodiments provide a camera design that mimics human physiology (e.g., a human eye) in at least one or more of the following aspects: geometry, optical properties, and / or motion. Specifically, embodiments provide a camera (e.g., an eyeball camera) that not only employs the same cornea and pupil geometry as a human eye, but also has an iris and pupil that can be configured with a variety of texture, color, or diameter options. In addition, the resolution of the eyeball camera is designed to match the acuity of typical 20 / 20 human vision, and the focal length can be adjusted, for example, from 0 to 4 diopters, by a piezoelectric motor or other suitable mechanical system.

[0024] According to various embodiments, a pair of eye cameras are independently mounted on two hexapods to simulate human eye gaze and convergence. With the help of the eye cameras, both perceived virtual and real world content can be calibrated and evaluated under deterministic and quantifiable eye conditions (e.g., pupil position and gaze). According to various embodiments, the eye cameras act as a bridge, combining data from spatial computing (such as eye tracking, 3D geometry of the digital world, display color accuracy / uniformity, and display optical quality (clarity, contrast, etc.) for an overall view, which helps to effectively blend the virtual world and the real world seamlessly.

[0025] Figure 1 An exemplary system 100 for calibrating a virtual image display system 110 is shown. The system 100 may include a virtual image display system 110, an image capture device 120, and a data processing module 140. The virtual image display system 110 may include: a display device 112 configured to display a virtual image; a real-world image capture device 150 configured to capture and display an image on the display device 112; and various mechanical and electronic modules and systems that support the functionality of the virtual image display system 110. The display device 112 may be coupled to a frame 130 that may be worn by a display system user, wearer, or viewer and is configured to position the display device 112 in front of the wearer's eyes during use. The display device 112 may be a light field display. In some embodiments, a speaker may be coupled to the frame 130 and positioned near an ear canal of the user (in some embodiments, another speaker not shown is positioned near another ear canal of the user to provide stereo / shapeable sound control). The real-world image capture device 150 may be coupled to the display device 112 via a wired lead or a wireless connection. In some embodiments, the real-world image capture device 150 may be disposed on the frame 130. In other embodiments, the real-world image capture device 150 may be disposed separately from the frame 130 and configured to be worn by the user of the virtual image display system 110 (e.g., worn on the head as a head camera, worn on the body as a belt, wristband, or watch).

[0026] According to various embodiments, the image capture device 120 may be positioned in front of the display device 112. The system may also include a data processing module 140 operatively coupled to the virtual image display system 110 and the image capture device 120, such as by a wired lead or a wireless connection. The data processing module 140 may include various input / output devices and may receive data from external modules.

[0027] The data processing module 140 may include one or more processors configured to analyze and process data and / or image information, such as image or video information captured by the real-world image capture device 150. The image or video data may be stored locally in the data processing module 140 and / or remotely in a remote data repository. In some embodiments, the remote data repository may include a digital data storage facility that may be obtained via the Internet or other network configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the data processing module 140, allowing for fully autonomous use from the remote module, while in other embodiments, data storage and / or calculations are distributed. Those of ordinary skill in the art will recognize many variations, modifications, and alternatives.

[0028] The display device 112 may include a first display 106 (e.g., a left display element) and a second display 108 (e.g., a right display element). Similarly, the image capture device 120 may include a first image capture element 102 (e.g., a left image capture element) and a second image capture element 104 (e.g., a right image capture element). The image capture device 120 may be positioned in front of the display device 112 such that the first image capture element 102 is positioned in front of the first display 106 and the second image capture element 104 is positioned in front of the second display 108.

[0029] According to various embodiments, the image capture device 120 can simulate a human eye. That is, the image capture device 120 can include characteristics similar to a human eye. For example, each of the first image capture element 102 and the second image capture element 104 can include corneal position, corneal geometry, eyeball position, pupil size, pupil position, gaze distance, gaze orientation, and iris color. For each of the first image capture element 102 and the second image capture element 104, these characteristics can be configurable and controllable independently of each other. For example, the first image capture element 102 and the second image capture element 104 can include an eyeball camera.

[0030] Figure 2A and Figure 2BA first image capture element 102 is shown according to some embodiments of the present disclosure. The second image capture element 104 is similar (or identical) to the first image capture element 102, and the description of the first image capture element 102 applies to the second image capture element 104. The first image capture element 102 and / or the second image capture element 104 may simulate a human eye and include an artificial cornea 200, an artificial pupil 202, and an artificial iris 204 mounted at the end of a lens holder 206 coupled to a camera body 210. The artificial cornea 200, the artificial pupil 202, and the artificial iris 204 may be referred to as an artificial eyeball 220. The camera body 210 may include a lightproof box that houses camera elements such as a camera shutter and an image sensor.

[0031] Various characteristics of the artificial cornea 200, the artificial pupil 202, and the artificial iris 204 may be configured according to various embodiments. For example, the position and / or geometry of the artificial cornea 200, the position and size of the artificial pupil 202, and the position and / or color of the artificial iris 204 may be configured, customized, or otherwise adjusted according to various embodiments. Similarly, the position of the artificial eyeball 220 at the end of the lens holder 206 may also be configured. Embodiments allow for the determination and adjustment of the convergence (including the convergence-rotation of the eyes toward each other, which occurs when looking at closer objects, and the divergence-rotation of the eyes away from each other, which occurs when looking at distant objects), gaze distance, and / or gaze orientation of the first image capture element 102 and / or the second image capture element 104. For example, embodiments allow for the gaze distance to be fixed at infinity for a desired (predetermined) amount of time. Since the human eye cannot maintain a stable gaze at infinity, the first image capture element 102 and / or the second image capture element 104 may mimic and build on the capabilities of the human eye. Thus, embodiments allow for more accurate calibration of the virtual image display system 110 (ie, the real-world image capture device 150 of the virtual image display system 110 ).

[0032] According to various embodiments, the characteristics of the first image capture element 102 may be configured and adjusted independently of the second image capture element 104. Similarly, the first image capture element 102 may be controlled to move independently of the second image capture element 104. Figure 3 As shown in , this can be accomplished by placing each of the first image capture element 102 and the second image capture element 104 on separate, individual hexapods.

[0033] Figure 3300 is a simplified schematic diagram illustrating a camera rig 300 according to some embodiments of the present disclosure. The camera rig 300 may include a first controllable support 302 and a second controllable support 304. The controllable supports 302, 304 may be controlled to move independently of each other or simultaneously with each other. The controllable supports 302, 304 may be controlled to mimic human eye muscles as well as a human head.

[0034] like Figure 3 As shown in FIG, first image capture element 102 may be mounted on first controllable support 302, and second image capture element 104 may be mounted on second controllable support 304. Camera rig 300 and image capture elements 102, 104 may work together to mimic (e.g., simulate) a pair of human eyes moving in eye sockets.

[0035] According to various embodiments, the controllable supports 302, 304 may include one or more actuators 306 (e.g., linear actuators, prismatic joints, etc.) coupled to a bottom platform 308 and / or a top platform 310. In some embodiments, the controllable supports 302, 304 may be in the form of a parallel manipulator including six linear actuators.

[0036] The controllable supports 302, 304 may be controlled via a computing device 350, which is configured to communicate with the controllable supports 302, 304 via a wired or wireless connection. In some embodiments, the computing device 350 may include the data processing module 140, or may be coupled to the data processing module 140 to work in conjunction with the data processing module 140. According to various embodiments, the computing device 350 may send commands (e.g., in the form of signals) to the controllable supports 302, 304 and receive feedback (e.g., in the form of signals) from the controllable supports 302, 304.

[0037] According to various embodiments, the calibration of the virtual image display system may be fine-tuned using the camera rig 300 during production of the virtual image display system. However, in some embodiments, the size and dimensions of the camera rig 300 may be designed to be provided at a retail location where a user may bring their virtual image display system 110 for calibration or to fine-tune the virtual image display system according to the user's specific eye characteristics (e.g., geometry, color, line of sight).

[0038] Embodiments allow for calibration of a virtual image display system (e.g., a real-world image capture device of an AR / VR system) using an image capture device (including one or more image capture elements). In some embodiments, the image capture device may be controlled or configured to mimic the human eye.

[0039] Figure 4An exemplary block diagram illustrating a method for calibrating a virtual image display system according to some embodiments of the present disclosure is shown. An image 450 of a scene, environment, or object in the real world 400 can be captured using a real-world image capture device (e.g., a world camera) 402. The captured image 450 can be sent to a rendering engine 410, which then processes the received image and sends the processed image to a projector 412, which will then be displayed on a display device (e.g., an eyepiece of a virtual image display system) 414. The displayed image may correspond to a set of pixels 416 illuminated on the display device 414. However, the displayed image may not completely match (e.g., align) the real world 400 that the image should represent. For example, the outline of an object in the image may not align with the outline of the object in the real world. In order to correct this mismatch, it may be necessary to calibrate the real-world image capture device 402.

[0040] For example, images captured using real-world image capture device 402 may be processed (e.g., refined) using data from eye-tracking camera 404 and one or more eye-tracking light emitters (e.g., light emitting diodes (LEDs)) 406. Data from eye-tracking camera 404 and one or more eye-tracking light emitters 406 may include additional information about where the user is looking to render images at a particular depth or range. For example, eye-tracking camera 404 and one or more eye-tracking light emitters 406 may be used to determine the vergence of the eye, and a vergence / accommodation model may be used to determine calibration / adjustment of a virtual image display system.

[0041] According to various embodiments, the eye tracking camera 404 detects the user's corneal position and determines the eye aperture based on the corneal position, and calculates the user's gaze. In other words, the eye tracking camera 404 estimates the user's eye position. Therefore, it is necessary to know the details of the user's eye geometry, position, and gaze to more accurately calibrate the virtual image display system.

[0042] Embodiments may use an image capture device 408 that simulates a human eye (or a pair of human eyes) to calibrate a virtual image display system. Image capture device 408 may be used in conjunction with eye tracking camera 404. According to various embodiments, image capture device 408 may have the physical characteristics of a human eye. For example, in conjunction with Figure 2A-2B As described, image capture device 408 may have a geometry similar to that of a human eye, including an artificial cornea, an artificial pupil, and an artificial iris. Thus, eye tracking camera 404 and one or more eye tracking light emitters 406, when used in conjunction with image capture device 408, provide more accurate data for calibrating a virtual image display system.

[0043] Because the characteristics of image capture device 408 are known, eye tracking camera 404 and one or more eye tracking light emitters 406 use data for the characteristics (e.g., corneal position, corneal geometry, eyeball position, pupil size, pupil position, gaze distance, and gaze orientation) rather than estimating or determining these characteristics. Corrections to the virtual image display system may be determined based on one or more of these characteristics of image capture device 408 to more accurately align image 450 received from real-world image capture device 402 with image 460 of the same scene, environment, or object in real world 400 captured using image capture device 408.

[0044] The rendering engine 410 may receive the image 460 from the image capture device 408 and compare the image 460 to the image 450 received from the real-world image capture device 402. If the image 460 and the image 450 are determined not to overlap to a predetermined degree or target, the rendering engine 410 may determine one or more corrections to be applied to the real-world image capture device 402 to more accurately capture an image of a scene, environment, or object in the real world 400. The virtual image display system (and more specifically, the real-world image capture device 402 and / or the display device 414) may be calibrated until the image 450 captured by the real-world image capture device 402 is aligned with the image 460 captured by the image capture device 408 within a predetermined threshold.

[0045] Figure 5 An exemplary block diagram illustrating various calibrations that may be performed in conjunction with a virtual image display system is shown. Figure 1 As described, the virtual image display system 500 can have a sensing component 502 (e.g., real-world image capture device 150) and a display component 504 (e.g., display device 112). The sensing component 502 can be calibrated using eye tracking camera calibration 506 and eye tracking LED calibration 508 for verifying and improving eye tracking 514. The display component 504 can be calibrated using geometry calibration 510 for verifying and improving the geometry 516 of the displayed object and color calibration 512 for verifying and improving the color 518 of the displayed object. Overall, the display quality 520 of the virtual image display system can be verified and improved.

[0046] The image capture device simulating the human eye described in conjunction with the various embodiments may be used to combine and simultaneously perform all of the above calibrations to improve eye tracking, geometry, color, and display quality of a virtual image display system.

[0047] Figure 6An exemplary block diagram illustrating a detailed view of eye tracking verification is shown. The eye tracking camera 604 and the eye tracking LED 606 can estimate characteristics 610 of the image capture device 602, such as corneal position, corneal geometry, eyeball position, pupil size, pupil position, gaze distance, and gaze orientation. The estimated characteristics 610 can be provided to a computing device 614 (e.g., data processing module 140) along with actual values ​​612 of these characteristics. The computing device can compare the estimated characteristics 610 with the actual values ​​612 of these characteristics of the image capture device 602, and determine one or more corrections 620 of the virtual image display system based on gaze orientation, depth pane switching, rendering center, convergence plane, etc.

[0048] Figure 7 700 is a simplified flow chart illustrating a method for determining and applying corrections to a virtual image display system according to some embodiments of the present disclosure. The virtual image display system may include at least a display device and a world camera. According to various embodiments, an image capture element simulating a human eye (or a pair of human eyes) may be coupled to the virtual image display system.

[0049] At box 702, a computing device including a processor receives a first image and a second image from a virtual image display system. The first image can be captured using a world camera of the virtual image display system and the second image can be captured using an image capture device (e.g., an eyeball camera). The image capture device has one or more characteristics similar to a human eye, including but not limited to corneal position, corneal geometry, eyeball position, pupil size, pupil position, gaze distance, and / or gaze orientation. These characteristics of the image capture device can have predetermined values ​​and can be configurable.

[0050] At block 704, the computing device may determine visual characteristics of the first image and the second image. The computing device may determine a first set of visual characteristics associated with the first image and a second set of visual characteristics associated with the second image. The first set of visual characteristics and the second set of visual characteristics may include a particular positioning of the object (e.g., the location, coordinates of the object in the image, or the distance between two objects or points in the image), the color of the object (e.g., the hue, saturation, or contrast of the object), and geometric properties of the object (collinearity, curvature, length, width, breadth, added markings, missing markings, number, shape, size, spatial grouping, and spatial orientation of elements of the object).

[0051] At block 706 , the computing device may identify differences between a first set of visual characteristics of the first image and a second set of visual characteristics of the second image.

[0052] At block 708 , the computing device may determine a correction to apply to the virtual image display system using one or more known characteristics of the image capture device to at least partially correct for the disparity.

[0053] At block 710, the computing device may apply a correction to the virtual image display system based on one or more characteristics of the image capture device. The correction may calibrate a display device or a world camera of the virtual image display system.

[0054] It should be understood that Figure 7 The specific steps shown in provide a specific method for determining and applying corrections to a virtual image display system according to some embodiments of the present disclosure. According to alternative embodiments, other sequences of steps may also be performed. For example, alternative embodiments may be performed in a different order. Figure 7 In addition, Figure 7 The various steps shown in the embodiment may include multiple sub-steps, which may be performed in various orders suitable for the various steps. In addition, additional steps may be added or deleted depending on the specific application. Those of ordinary skill in the art will recognize many variations, modifications and substitutions.

[0055] Figure 8 800 is a simplified flow chart illustrating a method for determining and applying corrections to a virtual image display system to adjust display color according to some embodiments of the present disclosure. The virtual image display system may include at least a display device and a world camera. According to various embodiments, an image capture element simulating a human eye (or a pair of human eyes) may be coupled to the virtual image display system.

[0056] At box 802, a computing device including a processor receives a first image and a second image from a virtual image display system. The first image can be captured using a world camera of the virtual image display system, and the second image can be captured using an image capture device (e.g., an eyeball camera). The image capture device has one or more characteristics similar to a human eye, including but not limited to corneal position, corneal geometry, eyeball position, pupil size, pupil position, gaze distance, and / or gaze orientation. These characteristics of the image capture device can have predetermined values ​​and can be configurable.

[0057] At block 804, the computing device may determine visual characteristics of the first image and the second image.The computing device may determine a spatial location of a point in the first image and a spatial location of a corresponding point in the second image.

[0058] At block 806 , the computing device may identify differences between the spatial positioning of points in the first image and the spatial positioning of corresponding points in the second image.

[0059] At block 808 , the computing device may determine a correction to apply to the virtual image display system to align points in the first image with corresponding points in the second image.

[0060] At block 810, the computing device may apply a correction to the virtual image display system based on one or more characteristics of the image capture device. The correction may calibrate a display device or a world camera of the virtual image display system and cause points of the first image to more accurately align with corresponding points in the second image.

[0061] It should be understood that Figure 8 The specific steps shown in provide a specific method for determining and applying corrections to a virtual image display system to adjust display color according to some embodiments of the present disclosure. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments may be performed in a different order. Figure 8 In addition, Figure 8 The various steps shown in the embodiment may include multiple sub-steps, which may be performed in various orders suitable for the various steps. In addition, additional steps may be added or deleted depending on the specific application. Those of ordinary skill in the art will recognize many variations, modifications and substitutions.

[0062] Fig. 9 900 is a simplified flow chart illustrating a method for determining and applying corrections to a virtual image display system to adjust geometric properties according to some embodiments of the present disclosure. The virtual image display system may include at least a display device and a world camera. According to various embodiments, an image capture element simulating a human eye (or a pair of human eyes) may be coupled to the virtual image display system.

[0063] At box 902, a computing device including a processor receives a first image and a second image from a virtual image display system. The first image can be captured using a world camera of the virtual image display system, and the second image can be captured using an image capture device (e.g., an eyeball camera). The image capture device has one or more characteristics similar to a human eye, including but not limited to corneal position, corneal geometry, eyeball position, pupil size, pupil position, gaze distance, and / or gaze orientation. These characteristics of the image capture device can have predetermined values ​​and can be configurable.

[0064] At block 904, the computing device may determine visual characteristics of the first image and the second image. The computing device may determine a color (e.g., hue, saturation, or contrast) of an area (e.g., a point, object, or pixel) in the first image and a color of a corresponding area in the second image.

[0065] At block 906 , the computing device may identify a difference between a color of a region in the first image and a color of a corresponding region in the second image.

[0066] At block 908 , the computing device may determine a correction to apply to the virtual image display system to match the color of the region in the first image to the color of the corresponding region in the second image.

[0067] At block 910, the computing device may apply a correction to the virtual image display system based on one or more characteristics of the image capture device. The correction may calibrate a display device or a world camera of the virtual image display system and cause the color of an area in the first image to more accurately match the color of a corresponding area in the second image. The correction may adjust one or more of the hue, saturation, and contrast of an image displayed on the display device.

[0068] It should be understood that Fig. 9 The specific steps shown in provide a specific method for determining and applying corrections to a virtual image display system to adjust geometric properties according to some embodiments of the present disclosure. According to alternative embodiments, other sequences of steps may also be performed. For example, alternative embodiments may be performed in a different order. Fig. 9 In addition, Fig. 9 The various steps shown in the embodiment may include multiple sub-steps, which may be performed in various orders suitable for the various steps. In addition, additional steps may be added or deleted depending on the specific application. Those of ordinary skill in the art will recognize many variations, modifications and substitutions.

[0069] Fig.10 1000 is a simplified flow chart illustrating a method for determining and applying corrections to a virtual image display system to adjust spatial positioning according to some embodiments of the present disclosure. The virtual image display system may include at least a display device and a world camera. According to various embodiments, an image capture element simulating a human eye (or a pair of human eyes) may be coupled to the virtual image display system.

[0070] At box 1002, a computing device including a processor receives a first image and a second image from a virtual image display system. The first image can be captured using a world camera of the virtual image display system, and the second image can be captured using an image capture device (e.g., an eyeball camera). The image capture device has one or more characteristics similar to a human eye, including but not limited to corneal position, corneal geometry, eyeball position, pupil size, pupil position, gaze distance, and / or gaze orientation. These characteristics of the image capture device can have predetermined values ​​and can be configurable.

[0071] At block 1004, the computing device may determine visual characteristics of the first image and the second image. The computing device may determine geometric properties (e.g., form, colinearity, curvature, length, width, extent, added markings, missing markings, number, shape, size, spatial grouping, and / or spatial orientation) of objects in the first image and geometric properties of corresponding objects in the second image.

[0072] At block 1006 , the computing device may identify differences between geometric properties of an object in the first image and geometric properties of a corresponding object in the second image.

[0073] At block 1008 , the computing device may determine a correction to apply to the virtual image display system to match geometric properties of objects in the first image to geometric properties of corresponding objects in the second image.

[0074] At block 1010, the computing device may apply a correction to the virtual image display system based on one or more characteristics of the image capture device. The correction may calibrate a display device or a world camera of the virtual image display system and cause geometric properties of an object in the first image to more accurately match geometric properties of a corresponding region in the second image. The correction may adjust one or more of collinearity, curvature, length, width, breadth, added markings, missing markings, quantity, shape, size, spatial grouping, and / or spatial orientation of objects displayed on the display device.

[0075] It should be understood that Fig.10 The specific steps shown in provide a specific method for determining and applying corrections to a virtual image display system to adjust spatial positioning according to some embodiments of the present disclosure. According to alternative embodiments, other sequences of steps may also be performed. For example, alternative embodiments may be performed in a different order. Fig.10 In addition, Fig.10 The various steps shown in the embodiment may include multiple sub-steps, which may be performed in various orders suitable for the various steps. In addition, additional steps may be added or deleted depending on the specific application. Those of ordinary skill in the art will recognize many variations, modifications and substitutions.

[0076] Embodiments provide many advantages over existing systems. Embodiments allow the use of one or more cameras (e.g., eyeball cameras) that simulate the human eye to calibrate real-world image capture devices and / or AR / VR display systems. The characteristics of the eyeball camera (such as one or more of corneal position, corneal geometry, eyeball position, pupil size, pupil position, gaze distance, or gaze orientation) are known and can be controlled. Therefore, calibration can be performed using a set of known and controllable parameters. For example, the gaze can be fixed at infinity and the data can be used to calibrate the device. Therefore, embodiments provide more accurate calibration of the AR / VR display system, thereby producing a more accurate and seamless AR / VR experience.

[0077] Each process, method and algorithm described herein and / or depicted in the accompanying drawings may be embodied in a code module that is executed by one or more physical computing systems, hardware computer processors, special circuits and / or electronic hardware configured to execute specific and special computer instructions and may be fully or partially automated by them. For example, a computing system may include a general-purpose computer (e.g., a server) or a special-purpose computer, a special-purpose circuit, etc., programmed with specific computer instructions. The code module may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language. In some embodiments, specific operations and methods may be performed by circuits specific to a given function.

[0078] Furthermore, certain implementations of the functionality of the present disclosure may be sufficiently mathematically, computationally, or technically complex that specialized hardware or one or more physical computing devices (with appropriate specialized executable instructions) may be required to perform the functionality, e.g., due to the amount or complexity of the computations involved, or to provide results in substantially real time. For example, a video may include many frames, each with millions of pixels, and specially programmed computer hardware may be required to process the video data to provide the desired image processing task or application in a commercially reasonable amount of time.

[0079] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as physical computer storage devices including hard drives, solid-state memory, random access memory (RAM), read-only memory (ROM), optical disks, volatile or non-volatile storage devices, combinations thereof, and / or the like. Methods and modules (or data) may also be sent as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media (including wireless-based and wired / cable-based media) and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored permanently or otherwise in any type of non-transitory tangible computer storage device, or may be transmitted via a computer-readable transmission medium.

[0080] Any process, block, state, step or function in the flowchart described herein and / or depicted in the accompanying drawings should be understood as potentially representing a code module, code segment or code portion including one or more executable instructions for implementing a specific function (such as logic or arithmetic) or step in the process. Various processes, blocks, states, steps or functions can be combined, rearranged, added, deleted, modified or otherwise changed with the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules can perform some or all of the functions described herein. The methods and processes described herein are also not limited to any particular order, and the blocks, steps or states associated therewith can be executed in other appropriate orders, such as serial, parallel or in some other manner. Tasks or events can be added to or deleted from the disclosed example embodiments. In addition, the separation of various system components in the embodiments described herein is for illustrative purposes and should not be understood as requiring such separation in all embodiments. It should be understood that the described program components, methods and systems can generally be integrated together in a single computer product or packaged into multiple computer products. Many variations of the embodiments are possible.

[0081] The processes, methods and systems may be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowd-sourced computing networks, the Internet and the World Wide Web. The network may be a wired or wireless network or any other type of communication network.

[0082] The systems and methods of the present disclosure each have several innovative aspects, none of which bears sole responsibility or claim for the desired attributes disclosed herein. The various features and processes described above can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. Various modifications to the embodiments described in the present disclosure are obvious to those skilled in the art, and the general principles defined herein can be applied to other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the claims are not intended to be limited to the embodiments shown herein, but should be given the widest scope consistent with the present disclosure, principles and novel features disclosed herein.

[0083] Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. Moreover, although features may be described above as functioning in certain combinations and even initially claimed as such, in some cases one or more features in the claimed combination may be cut out of that combination and the claimed combination may be used in a sub-combination or variation of the sub-combination. No single feature or set of features is necessary or essential for every embodiment.

[0084] Conditional language used herein, especially such as "may", "can", "might", "can", "for example", etc., unless otherwise explicitly stated or otherwise understood in context, is generally intended to convey that certain embodiments include and other embodiments do not include certain features, elements and / or steps. Therefore, such conditional language is generally not intended to imply that features, elements and / or steps are necessary for one or more embodiments in any way, or that one or more embodiments necessarily include logic for determining (whether or not there is author input or prompt) whether these features, elements and / or steps will be included or performed in any particular embodiment. The terms "include", "comprise", "have", etc. are synonyms, used inclusively in an open-ended manner, and do not exclude additional elements, features, actions, operations, etc. Similarly, the term "or" is used in its inclusive meaning (rather than in its exclusive meaning), so that, for example, when used to connect a list of elements, the term "or" represents one, some or all of the elements in the list. In addition, "one", "an" and "the" used in the present application and the appended claims should be interpreted as meaning "one or more" or "at least one", unless otherwise stated.

[0085] As used herein, phrases referring to "at least one" of a list of items refer to any combination of those items, including single members. For example, "at least one of A, B, or C" is intended to cover: A, B, C, A and B, A and C, B and C, and A, B and C. Unless otherwise specifically stated, conjunctive language such as phrases "at least one of X, Y, and Z" should be understood with context, which context is generally used to convey that an item, term, etc. can be at least one of X, Y, or Z. Therefore, such conjunctive language is generally not intended to imply that certain embodiments require the presence of at least one of X, at least one of Y, and at least one of Z.

[0086] Similarly, although operations may be depicted in the accompanying drawings in a particular order, it is to be appreciated that such operations need not be performed in the particular order or sequence shown, or that all of the operations shown need not be performed to achieve the desired result. In addition, the accompanying drawings may schematically depict one or more example processes in the form of a flow chart. However, other operations not shown may be combined in the schematically illustrated example methods and processes. For example, one or more additional operations may be performed before, after, simultaneously, or between any of the illustrated operations. In addition, in other embodiments, these operations may be rearranged or reordered. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products. In addition, other embodiments are within the scope of the appended claims. In some cases, the actions recorded in the claims may be performed in different orders, and the desired results may still be achieved.

Claims

1. A system for applying correction to a virtual image display system, the system comprising: a display device configured to display a virtual image; an image capture device positioned in front of the display device, wherein the image capture device has one or more characteristics similar to a human eye; and a processor coupled to the display device and the image capture device to receive image data from the image capture device, wherein the processor is programmed to: receiving a first image of a scene from a real-world image capture device; determining a first set of visual characteristics associated with the first image; receiving a second image of the scene captured with the image capture device; determining a second set of visual characteristics associated with the second image, wherein the first set of visual characteristics and the second set of visual characteristics include at least one of: spatial location, color, or geometric properties; identifying a difference between at least one visual characteristic of the first set of visual characteristics of the first image and at least one visual characteristic of the second set of visual characteristics of the second image that corresponds to the at least one visual characteristic of the first set of visual characteristics of the first image; determining, using the one or more characteristics of the image capture device, a correction to be applied to the virtual image display system to at least partially correct for the difference, wherein the correction matches the spatial positioning, the color, or the geometric properties associated with the first image to the spatial positioning, the color, or the geometric properties associated with the second image; and Based on the one or more characteristics of the image capture device, the correction is applied to the virtual image display system.

2. The system according to claim 1, wherein: The one or more characteristics of the image capture device include one or more of: corneal position, corneal geometry, eyeball position, pupil size, pupil position, gaze distance, or gaze orientation.

3. The system according to claim 1, wherein: The one or more characteristics of the image capture device are configurable.

4. The system according to claim 1, wherein: The display device includes a first display associated with a first eye of a user and a second display associated with a second eye of the user, and wherein the image capture device includes a first image capture element positioned in front of the first display and a second image capture element positioned in front of the second display.

5. The system according to claim 4, wherein: The first image capturing element and the second image capturing element are configured to move independently of each other.

6. The system according to claim 4, wherein: One or more characteristics of the first image capture element are configurable independently of one or more characteristics of the second image capture element.

7. The system of claim 1, further comprising: A real-world image capture device is coupled to the display device and configured to capture the first image, wherein the correction is applied to the virtual image display system to calibrate the real-world image capture device.

8. The system according to claim 1, wherein: The first set of visual characteristics and the second set of visual characteristics include the spatial location, wherein the processor is further programmed to: determining a spatial location of a point in the first image; identifying a difference between a spatial location of the point in the first image and a spatial location of a corresponding point in the second image; determining a correction to be applied to the virtual image display system to align the point in the first image with the corresponding point in the second image; and The correction is applied to the virtual image display system.

9. The system according to claim 1, wherein: The first set of visual characteristics and the second set of visual characteristics include the color, wherein the processor is further programmed to: determining a color of a region in the first image; identifying a difference between a color of the region in the first image and a color of a corresponding region in the second image; determining a correction to be applied to the virtual image display system to match the color of the region in the first image to the color of the corresponding region in the second image; and The correction is applied to the virtual image display system.

10. The system according to claim 9, wherein: The correction includes correction of one or more of: hue, saturation, or contrast.

11. The system according to claim 1, wherein: The first set of visual characteristics and the second set of visual characteristics include the geometric attribute, wherein the processor is further programmed to: determining the geometric properties of objects in the first image; identifying differences between the geometric attributes of the object in the first image and the geometric attributes of the corresponding object in the second image; determining corrections to be applied to the virtual image display system to match the geometric properties of the objects in the first image with the geometric properties of the corresponding objects in the second image; and The correction is applied to the virtual image display system.

12. The system according to claim 11, wherein: The correction includes correction of one or more of: collinearity, curvature, length, width, extent, added markings, missing markings, number, shape, size, spatial grouping, or spatial orientation of the objects in the second image.

13. A method for applying correction to a virtual image display system, the method comprising: receiving, by a processor, a first image of a scene from a real-world image capture device; determining, by the processor, a first set of visual characteristics associated with the first image; receiving, by the processor, from an image capture device a second image of the scene captured by the image capture device, the image capture device having one or more characteristics similar to a human eye, wherein the image capture device is positioned in front of a display device; determining, by the processor, a second set of visual characteristics associated with the second image, wherein the first set of visual characteristics and the second set of visual characteristics include at least one of: spatial positioning, color, or geometric properties; identifying, by the processor, a difference between at least one visual characteristic of the first set of visual characteristics of the first image and at least one visual characteristic of the second set of visual characteristics of the second image that corresponds to the at least one visual characteristic of the first set of visual characteristics of the first image; determining, by the processor, using the one or more characteristics of the image capture device, a correction to be applied to the virtual image display system to at least partially correct for the difference, wherein the correction matches the spatial positioning, the color, or the geometric attributes associated with the first image to the spatial positioning, the color, or the geometric attributes associated with the second image; and The correction is applied, by the processor, to the virtual image display system based on the one or more characteristics of the image capture device.

14. The method according to claim 13, wherein: The one or more characteristics of the image capture device include one or more of: corneal position, corneal geometry, eyeball position, pupil size, pupil position, gaze distance, or gaze orientation.

15. The method according to claim 13, wherein: The one or more characteristics of the image capture device are configurable.

16. The method according to claim 13, wherein: The display device includes a first display associated with a first eye of a user and a second display associated with a second eye of the user, and wherein the image capture device includes a first image capture element positioned in front of the first display and a second image capture element positioned in front of the second display.

17. The method according to claim 16, wherein: The first image capturing element and the second image capturing element are configured to move independently of each other.

18. The method according to claim 16, wherein: One or more characteristics of the first image capture element are configurable independently of one or more characteristics of the second image capture element.

19. The method according to claim 13, wherein: The first image is received from a real-world image capture device coupled to the display device, wherein applying the correction further comprises: The real-world image capture device is calibrated.

20. The method according to claim 13, wherein: The first set of visual characteristics and the second set of visual characteristics include the spatial location, and the method further comprises: determining, by the processor, a spatial location of a point in the first image; identifying, by the processor, a difference between the spatial location of the point in the first image and the spatial location of a corresponding point in the second image; determining, by the processor, a correction to be applied to the virtual image display system to align the point in the first image with the corresponding point in the second image; and The correction is applied by the processor to the virtual image display system.

21. The method according to claim 13, wherein: The first set of visual properties and the second set of visual properties include the color, and the method further comprises: determining, by the processor, a color of an area in the first image; identifying, by the processor, a difference between a color of the region in the first image and a color of a corresponding region in the second image; determining, by the processor, a correction to be applied to the virtual image display system to match the color of the region in the first image to the color of the corresponding region in the second image; and The correction is applied by the processor to the virtual image display system.

22. The method according to claim 21, wherein: The correction includes correction of one or more of: hue, saturation, or contrast.

23. The method according to claim 13, wherein: The first set of visual properties and the second set of visual properties include the geometric attribute, and the method further comprises: determining, by the processor, the geometric properties of objects in the first image; identifying, by the processor, a difference between the geometric attributes of the object in the first image and the geometric attributes of the corresponding object in the second image; determining, by the processor, a correction to be applied to the virtual image display system to match the geometric properties of the object in the first image with the geometric properties of the corresponding object in the second image; and The correction is applied by the processor to the virtual image display system.

24. The method according to claim 23, wherein: The correction includes correction of one or more of: collinearity, curvature, length, width, extent, added markings, missing markings, number, shape, size, spatial grouping, or spatial orientation of the objects in the second image.

Images