Artificial reality system with varifocal display of artificial reality content

Abstract

The present disclosure describes artificial reality systems and techniques for providing artificial reality content to users. For example, the artificial reality system includes: a head-mounted display (HMD) configured to output artificial reality content, the HMD including: a set of second image capture devices configured to capture image data indicating the focus of a user's gaze and a zoom display having a focal length modifiable based on the user's focus. In addition, the system includes a depth engine configured to generate a three-dimensional (3D) scene of the user's physical environment based on real-world image data and depth data associated with the real-world image data and generate artificial reality content as an overlay on the 3D scene of the physical environment based on the user's focus to be displayed on the zoom display of the HMD.

Description

Technical Field

[0001] The present disclosure generally relates to artificial reality systems, such as augmented reality, mixed reality, and / or virtual reality systems. Background Art

[0002] Artificial reality systems are becoming increasingly common, with applications in many fields such as computer gaming, health and safety, industry, and education. Artificial reality systems are being incorporated into mobile devices, gaming consoles, personal computers, movie theaters, and theme parks, to name a few. Generally speaking, an artificial reality is a form of reality that has been modified in some way before being presented to a user, which may include, for example, virtual reality, augmented reality, mixed reality, hybrid reality, or some combination and / or derivative thereof.

[0003] A typical artificial reality system includes one or more devices for drawing and displaying content to a user. As an example, an artificial reality system may incorporate a head mounted display (HMD) worn by a user and configured to output artificial reality content to the user. The artificial reality content may include entirely content generated by the system, or may include content generated in combination with captured content (e.g., video and / or images of the real world). Summary of the invention

[0004] In general, the present disclosure describes an artificial reality system including a varifocal head-mounted display (HMD) configured to provide visually accurate artificial reality content to a user. For example, the artificial reality system can be used to provide a pass-through or mixed reality (MR) experience in which real-world image data is reconstructed with a different focal depth per frame based on where the user of the HMD is looking. Compared to current pass-through systems in which real-world image data is captured and reconstructed with a large depth of field (e.g., focused at infinity), the combination of a varifocal HMD and image processing techniques can provide a more realistic three-dimensional (3D) experience and reduce vergence-accommodation conflicts. In other examples, the varifocal head-mounted display and image processing techniques can be used to provide a virtual reality (VR) experience or an augmented reality experience.

[0005] In one or more examples where an artificial reality system provides a pass-through or MR experience, a zoom HMD may include: at least one pass-through camera configured to capture real-world image data in color (e.g., RGB) and at high resolution; an eye tracking camera configured to capture movement of a user's pupil; and a display panel configured to move mechanically relative to one or more lenses within an eyepiece of the HMD. In addition, the zoom HMD may include a depth sensor and / or a dual inside-out camera or a stereo inside-out camera configured to capture depth data corresponding to the real-world image data captured by the pass-through camera. In some examples, the zoom HMD may include an inside-out camera for each eye of the user.

[0006] According to the disclosed technology, the AR system can perform image processing techniques to generate a three-dimensional mesh of the real-world scene using the captured depth data, and overlay or overlay texture data generated using real-world images captured from a pass-through camera onto the 3D mesh to create a virtual scene for display on an HMD with real depth.

[0007] Once the virtual scene is generated, the techniques described herein modify the way the virtual scene is displayed to the user on the HMD so as to mimic the visual effects that the user corresponding to the virtual scene would experience when focusing on a particular object in the real-world environment. For example, one or more eye-tracking cameras of the HMD may capture focus data including the position of the user's pupil relative to a set of reference points. Using the focus data, AR may move the focus of the zoom display in real time or near real time to match the user's identified focus. In addition, the AR system may be configured to apply a depth blur or defocus filter to blur objects represented in the virtual scene that are outside the identified depth of field of the user of the HMD.

[0008] In some examples, an artificial reality system includes: a first image capture device configured to capture real-world image data representing a user's physical environment and a head-mounted display (HMD) configured to output artificial reality content, the HMD including: a set of second image capture devices configured to capture image data indicating a focus of the user's gaze, and a zoom display having a focal length modifiable based on the user's focus. In addition, the artificial reality system includes a depth engine configured to generate a three-dimensional (3D) scene of the user's physical environment based on the real-world image data and depth data associated with the real-world image data, and generate artificial reality content as an overlay on the 3D scene of the physical environment based on the user's focus for display on the zoom display of the HMD.

[0009] In some embodiments, the first image capture device may include at least one pass-through camera configured to capture real-world image data in color and at high resolution.

[0010] In some embodiments, the HMD may include one or more processors configured to execute a depth engine, wherein the one or more processors are located proximate to the at least one pass-through camera.

[0011] In some embodiments, the HMD may further include a depth sensor configured to generate depth data, and wherein to generate a 3D scene of the physical environment, the depth engine is configured to: use the depth data to generate a 3D mesh of the real-world scene; and overlay at least a portion of the real-world image data onto the 3D mesh of the real-world scene.

[0012] In some embodiments, the real-world image data may represent a real-world image data stream, wherein the depth data represents a depth data stream, and wherein, to generate artificial reality content, the depth engine is further configured to: generate an artificial reality content scene in near real time using the real-world image data stream and the depth data stream based on the position and orientation of the HMD relative to a 3D mesh of the real world.

[0013] In some embodiments, a set of second image capture devices may include: a first eye tracking camera configured to capture a first set of image data, the first set of image data including the position of a first pupil of the user relative to a first set of reference points; a second eye tracking camera configured to capture a second set of image data, the second set of image data including the position of a second pupil of the user relative to a second set of reference points, and wherein the depth engine is configured to determine the focus of the user based on the position of the first pupil relative to the first set of reference points and the position of the second pupil relative to the second set of reference points.

[0014] In some embodiments, the system may also include a gaze tracker configured to: track the position of a first pupil relative to a first set of reference points over a period of time; track the position of a second pupil relative to a second set of reference points over a period of time; determine an expected future movement of the position of the first pupil relative to the first set of reference points based on the movement of the position of the first pupil relative to the first set of reference points over a period of time; and determine an expected future movement of the position of the second pupil relative to the second set of reference points based on the movement of the position of the second pupil relative to the second set of reference points over the period of time, wherein the depth engine is further configured to determine an expected future focus of the user based on the expected future movement of the first pupil position and the expected future movement of the second pupil position.

[0015] In some embodiments, the depth engine may be further configured to: determine in near real time the focal length of the zoom display to match the user's focus; and direct the zoom display to move relative to the lens to achieve the determined focal length.

[0016] In some embodiments, the HMD may also include a motor configured to control a position of the zoom display relative to the lens, and wherein to instruct the zoom display to move, the depth engine is configured to: control the motor to place the zoom display at a certain distance from the lens to achieve the determined focal length.

[0017] In some embodiments, the image data may further indicate a depth of field of the user, and wherein to generate the artificial reality content, the depth engine is configured to: blur portions of the artificial reality content outside of the user's depth of field.

[0018] In some examples, a method includes: capturing, by a first image capture device, real-world image data representing a user's physical environment; capturing, by a set of second image capture devices of a head-mounted display (HMD) configured to output artificial reality content, image data indicating a focus of the user's gaze; modifying, by a depth engine of the head-mounted display, a focal length of a zoom display of the HMD based on the user's focus; generating, by the depth engine and based on the real-world image data and depth data associated with the real-world image data, a three-dimensional (3D) scene of the user's physical environment; and generating, by the depth engine, artificial reality content as an overlay on the 3D scene of the physical environment based on the user's focus for display on the zoom display of the HMD.

[0019] In some embodiments, the first image capture device may include at least one pass-through camera, and wherein the method further comprises capturing real-world image data in color and at high resolution using the pass-through camera.

[0020] In some embodiments, the method may further include executing the depth engine using one or more processors of the HMD, wherein the one or more processors are located proximate to the at least one pass-through camera.

[0021] In some embodiments, the method may further include: generating depth data using a depth sensor, and wherein generating a 3D scene of the physical environment includes: generating a 3D mesh of the real-world environment using the depth data; and overlaying at least a portion of the real-world image data onto the 3D mesh of the real-world scene.

[0022] In some embodiments, the real-world image data represents a real-world image data stream, wherein the depth data represents a depth data stream, and wherein generating the artificial reality content includes: generating the artificial reality content in near real time using the real-world image data stream and the depth data stream based on the position and orientation of the HMD relative to a 3D mesh of the real-world scene.

[0023] In some embodiments, the method may also include: capturing a first set of image data using a first eye tracking camera of a set of second image capture devices, the first set of image data including the position of a first pupil of the user relative to a first set of reference points; capturing a second set of image data using a second eye tracking camera of a set of second image capture devices, the second set of image data including the position of a second pupil of the user relative to a second set of reference points; and determining the focus of the user based on the position of the first pupil relative to the first set of reference points and the position of the second pupil relative to the second set of reference points using a depth engine.

[0024] In some embodiments, the method may also include: tracking the position of the first pupil relative to the first set of reference points over a period of time using a gaze tracker; tracking the position of the second pupil relative to the second set of reference points over a period of time using a gaze tracker; determining an expected future movement of the position of the first pupil relative to the first set of reference points based on the movement of the position of the first pupil relative to the first set of reference points over a period of time; determining an expected future movement of the position of the second pupil relative to the second set of reference points based on the movement of the position of the second pupil relative to the second set of reference points over a period of time; and determining an expected future focus of the user using a depth engine and based on the expected future movement of the first pupil position and the expected future movement of the second pupil position.

[0025] In some embodiments, the method may further include: determining, using the depth engine, in near real time, a focal length of the zoom display to match the user's focus; and directing, by the depth engine, movement of the zoom display relative to the lens to achieve the determined focal length.

[0026] In some embodiments, the image data may also indicate a depth of field of the user, and wherein generating the artificial reality content includes blurring portions of the artificial reality content outside of the depth of field of the user.

[0027] In some examples, a non-transitory computer-readable medium includes instructions that, when executed, cause one or more processors to: capture real-world image data representing a user's physical environment; capture image data indicating a focus of the user's gaze; modify a focal length of a varifocal display based on the user's focus; generate a three-dimensional (3D) scene of the user's physical environment based on the real-world image data and depth data associated with the real-world image data; and generate artificial reality content as an overlay on the 3D scene of the physical environment based on the user's focus for display on a varifocal display of a head-mounted display (HMD).

[0028] Further details of one or more examples of the disclosure are set forth in the accompanying drawings and the description that follows. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1A is a conceptual diagram illustrating an artificial reality system for presenting a virtual scene to a user according to one or more techniques of the present disclosure.

[0030] Figure 1B is a conceptual diagram illustrating an artificial reality system for presenting a virtual environment to more than one user, in accordance with one or more techniques of this disclosure.

[0031] Figure 2A is a conceptual diagram illustrating an example head mounted display (HMD) and example peripheral devices for presenting a virtual scene to a user, in accordance with one or more techniques of this disclosure.

[0032] Figure 2B is a conceptual diagram illustrating another example HMD in accordance with one or more techniques of this disclosure.

[0033] Figure 3 is a diagram illustrating one or more techniques according to the present disclosure Figure 1A , Figure 1B Block diagram of an example implementation of a console, HMD, and peripheral devices for one or more multi-device artificial reality systems.

[0034] Figure 4 is a diagram illustrating a method of performing a Figure 1A , Figure 1B Block diagram of an example of an artificial reality system HMD generating a virtual environment.

[0035] Figure 5 is a conceptual diagram illustrating example components of an HMD in accordance with one or more techniques of this disclosure.

[0036] Fig. 6Ais a conceptual diagram illustrating a focal length and a first depth of field within an environment in accordance with one or more techniques of this disclosure.

[0037] Figure 6B is a conceptual diagram illustrating focal length and a second depth of field within an environment in accordance with one or more techniques of this disclosure.

[0038] Figure 7 is a flow diagram illustrating example operations for providing artificial reality content in accordance with one or more techniques of this disclosure. DETAILED DESCRIPTION

[0039] Figure 1A is a conceptual diagram illustrating an artificial reality system 10 for presenting a virtual scene to a user according to one or more techniques of the present disclosure. Figure 1A In examples of , the artificial reality system 10 includes a head mounted display (HMD) 112 , peripheral devices 136 , and in some examples may include one or more external sensors 90 and / or a console 106 .

[0040] As shown, the HMD 112 is typically worn by the user 110 and includes a varifocal display and optical accessories for presenting artificial reality content 122 to the user 110. In addition, the HMD 112 includes one or more sensors (e.g., accelerometers) for tracking the movement of the HMD 112. The HMD 112 may include one or more "inside-out" image capture devices 138 for capturing image data representing the physical environment surrounding the user 110. Additionally or alternatively, the HMD 112 may include one or more "pass-through" image capture devices 139 configured to capture real-world image data so that the artificial reality system 10 can communicate at least some of the real-world image data to the user 110 via the HMD 112. In some examples, the resolution of the image data captured by the pass-through image capture device 139 is greater than the resolution of the image data captured by the inside-out image capture device 138. The image data captured by the pass-through image capture device 139 may include image data captured in color and at a higher resolution than the black-and-white image data captured by the inside-out image capture device 138.

[0041] In some examples, HMD 112 may include eye-tracking camera 140A and eye-tracking camera 140B (collectively, “eye-tracking cameras 140”). Each of eye-tracking cameras 140 may capture image data indicating a pupil of user 110 relative to a corresponding set of reference points. For example, eye-tracking camera 140A may capture image data indicating a position of a first pupil of user 110 relative to a first set of reference points, and eye-tracking camera 140B may capture image data indicating a position of a second pupil of user 110 relative to a second set of reference points. The image data captured by eye-tracking cameras 140 may be referred to herein as “focus data” because the position of the first pupil relative to the first set of reference points and the position of the second pupil relative to the second set of reference points may indicate a focus of user 110 and / or a depth of field of user 110. In some examples, an artificial reality application executing on HMD 112 and / or console 106 may use a gaze tracking algorithm and / or a neural network to generate focus data based on the image data captured by eye-tracking cameras 140. In some examples, focus data may be generated based on other information in addition to or instead of image data captured by eye tracking camera 140. For example, an artificial reality application may determine the gaze of user 110 based on one or more objects with which user 110 is interacting.

[0042] Additionally, HMD 112 may include one or more depth sensors that collect depth data indicating the depth of one or more points and / or objects in the real-world three-dimensional (3D) physical environment of user 110. In some examples, the depth sensor may include a set of at least two inside-outside image capture devices 138. In other examples, the depth sensor may include an independent depth sensor device. For example, the depth data may indicate that corner 152 is closer to user 110 than corner 154 (e.g., the depth of corner 152 is less than the depth of corner 154). Although illustrated as a head-mounted display, artificial reality system 10 may alternatively or additionally include glasses or other display devices for presenting artificial reality content 122 to user 110.

[0043] In this example, the console 106 is shown as a single computing device, such as a game console, a workstation, a desktop computer, or a laptop computer. In other examples, the console 106 can be distributed across multiple computing devices, such as a distributed computing network, a data center, or a cloud computing system. As shown in this example, the console 106, the HMD 112, and the sensor 90 can be communicatively coupled via a network 104, which can be a wired or wireless network, such as Wi-Fi, a mesh network, or a short-range wireless communication medium, or a combination thereof. Although the HMD 112 is shown in this example as communicating with the console 106, such as being connected to the console 106 or wirelessly communicating with the console 106, in some implementations, the HMD 112 operates as an independent mobile artificial reality system.

[0044] Generally speaking, the artificial reality system 10 uses information captured from the real world, 3D physical environment to render artificial reality content 122 for display to the user 110. Figure 1A In the example of FIG. 1 , user 110 views artificial reality content 122 constructed and drawn by an artificial reality application executed on HMD 112 and / or console 106. In some examples, artificial reality content 122 may include a mixture of real-world images (e.g., hand 132, peripheral device 136, wall 121) and virtual objects (e.g., virtual content items 124, 126 and virtual user interface 137) to produce mixed reality and / or augmented reality. In some examples, virtual content items 124, 126 may be mapped (e.g., fixed, locked, placed) to specific locations within artificial reality content 122. The location for the virtual content item may be fixed, for example, fixed relative to one of wall 121 or the earth. The location for the virtual content item may be variable, for example, variable relative to peripheral device 136 or the user. In some examples, the specific location of the virtual content item within artificial reality content 122 is associated with a location within the real-world 3D physical environment (e.g., on the surface of a physical object).

[0045] In this example, peripheral device 136 is a physical, real-world device having a surface on which artificial reality system 10 overlays virtual user interface 137. Peripheral device 136 may include one or more presence-sensitive surfaces for detecting user input by detecting the presence of one or more objects (e.g., fingers, stylus) touching or hovering above the location of the presence-sensitive surface. In some examples, peripheral device 136 may include an output display, which may be a presence-sensitive display. In some examples, peripheral device 136 may be a smart phone, tablet computer, personal data assistant (PDA), or other handheld device. In some examples, peripheral device 136 may be a smart watch, smart ring, or other wearable device. Peripheral device 136 may also be part of a kiosk or other fixed or mobile system. Peripheral device 136 may or may not include a display device for outputting content to a screen.

[0046] exist Figure 1A In the example artificial reality experience shown in , virtual content items 124 , 126 are mapped to locations on wall 121 . Figure 1A The example in also shows that the virtual content item 124 only partially appears on the wall 121 within the artificial reality content 122, which illustrates that the virtual content does not exist in the real world physical environment. The virtual user interface 137 is mapped to the surface of the peripheral device 136. As a result, the artificial reality system 10 draws the virtual user interface 137 at a user interface position that is locked relative to the position of the peripheral device 136 in the artificial reality environment for display at the HMD 112 as part of the artificial reality content 122. Figure 1A The virtual user interface 137 is shown to appear on the peripheral device 136 only within the artificial reality content 122, which illustrates that the virtual content does not exist in the real world, 3D physical environment.

[0047] The artificial reality system 10 may draw one or more virtual content items in response to determining that at least a portion of the location of the virtual content item is within the field of view 130 of the user 110. For example, the virtual user interface 137 of the artificial reality system 10 may draw the virtual user interface 137 on the peripheral device 136 only when the peripheral device 136 is within the field of view 130 of the user 110.

[0048] During operation, the artificial reality application constructs artificial reality content 122 for display to the user 110 by tracking and calculating pose information of a reference frame, which is typically the viewing perspective of the HMD 112. Using the HMD 112 as a reference frame, and based on the current field of view 130 determined by the current estimated pose of the HMD 112, the artificial reality application renders 3D artificial reality content, which in some examples may be at least partially overlaid on the real world 3D physical environment of the user 110. In some examples, the artificial reality application may use depth data captured by one or more depth sensors of the HMD 112 to generate a 3D mesh of the physical environment of the user 110. In some examples, the 3D mesh may include a set of data points, where the position of each data point of the set of data points is known relative to each other data point of the set of data points. For example, the 3D mesh may indicate that one or more data points representing corner 152 are closer to the user 110 than one or more data points representing corner 154. After generating the 3D mesh, the artificial reality application may overlay at least some of the real-world image data captured by the pass-through image capture device 139 on the 3D mesh to generate a 3D scene of the physical environment of the user 110. Additionally, the artificial reality application may generate artificial reality content 122 as an overlay on the 3D scene of the physical environment for display on the varifocal display of the HMD 112 based on the focus of the user 110.

[0049] In some examples, the artificial reality application uses sensory data received from HMD 112 (such as movement information and user commands), and in some examples, data from any external sensors 90 (such as external cameras), to capture 3D information within the real-world physical environment, such as the movement of user 110 and / or feature tracking information about user 110. Based on the sensory data, the artificial reality application can determine the current pose of the reference frame of HMD 112 and, based on the current pose, render artificial reality content 122.

[0050] The artificial reality application may trigger the generation and rendering of virtual content items based on the current field of view 130 of the user 110, which may be determined by near real-time gaze tracking of the user (e.g., tracking of image data collected by the eye tracking camera 140), or other conditions. More specifically, the pass-through image capture device 139 of the HMD 112 captures image data representing objects in the real-world 3D physical environment within the field of view 130 of the pass-through image capture device 139. The field of view 130 generally corresponds to the viewing perspective of the HMD 112. In some examples, the artificial reality application presents artificial reality content 122 including mixed reality and / or augmented reality. As shown in FIG. Figure 1A, the artificial reality application may draw images of real-world objects (such as portions of the user 110's peripherals 136, hands 132, and / or arms 134) within the field of view 130 along with virtual objects (such as objects 124, 126) within the artificial reality content 122. In other examples, the artificial reality application may draw virtual representations of portions of the user's 110 peripherals 136, hands 132, and / or arms 134 within the field of view 130 within the artificial reality content 122 (e.g., drawing the real-world objects as virtual objects). In either example, the user 110 is able to view portions of their hands 132, arms 134, peripherals 136, and / or any other real-world objects within the field of view 130 within the artificial reality content 122. In other examples, the artificial reality application may not draw representations of the user's hands 132 or arms 134.

[0051] During operation, the artificial reality application performs object identification within image data captured by the internal and external image capture devices 138 of the HMD 112 to identify the peripheral device 136, the hand 132, including optionally identifying a single finger or thumb of the user 110, and / or all or part of the arm 134. In addition, the artificial reality application tracks the position, orientation, and configuration of the peripheral device 136, the hand 132 (optionally including specific fingers of the hand), and / or parts of the arm 134 over a time sliding reference. In some examples, the peripheral device 136 includes one or more sensors (e.g., accelerometers) for tracking the motion or orientation of the peripheral device 136.

[0052] As described above, multiple devices of the artificial reality system 10 can work together in an AR environment, where each device can be a separate physical electronic device and / or a separate integrated circuit (e.g., SoC) within one or more physical devices. In this example, the peripheral device 136 is operatively paired with the HMD 112 to operate together within the artificial reality system 10 to provide an artificial reality experience. For example, the peripheral device 136 and the HMD 112 can communicate with each other as co-processing devices. As an example, when a user performs a user interface gesture in a virtual environment at a location corresponding to one of the virtual user interface elements of the virtual user interface 137 overlaid on the peripheral device 136, the artificial reality system 10 detects the user interface and performs an action drawn to the HMD 112.

[0053] In some example implementations, as described herein, the peripherals 136 and the HMD 112 may each include one or more SoC integrated circuits configured to support artificial reality applications, such as a SoC operating as a co-application processor, sensor aggregator, display controller, and the like.

[0054] In some examples, the artificial reality application of the artificial reality system 10 is configured to generate artificial reality content 122 for display by the HMD 112 based on real-world image data captured by the pass-through image capture device 139 and focus data captured by the eye tracking camera 140. For example, the artificial reality application can determine the focus of the user 110's gaze and / or the depth of field of the user 110 based on the focus data captured by the eye tracking camera 140. In some examples, the focus data can represent image data indicating the position of the first pupil of the user 110 relative to the first set of reference points and image data indicating the position of the second pupil of the user 110 relative to the second set of reference points. The artificial reality application can determine the focus of the user 110 based on the position of the first pupil relative to the first set of reference points and the position of the second pupil relative to the second set of reference points. Focus can refer to the point or plane on which the user 110 is focusing. Depth of field can refer to the distance between the nearest object and the farthest object that are in focus (e.g., appear sharp) in a scene or image perceived by the user. As described herein, depth of field may also include the distance between the closest in-focus object and the user, and the distance between the farthest in-focus object and the user.

[0055] Additionally or alternatively, the artificial reality application may determine the focus of the user 110 based on image data captured by the inside-out image capture device 138, the pass-through image capture device 139, the depth projector sensor, or any combination thereof. For example, the artificial reality application may determine the focus of the user 110 based on detecting the user's interaction with one or more objects in the artificial reality content 122. For example, the artificial reality application may determine that the hand 134 of the user 110 is holding the peripheral device 136 within the field of view 130. The artificial reality application may determine that the focus of the user 110 is near the peripheral device 136 based on identifying that the user 110 is interacting with the peripheral device 136 within the field of view 130. In some examples, the artificial reality application may determine the focus of the user 110 based on focus data captured by the eye tracking camera 140 and identifying the user's interaction with one or more objects within the field of view 130. In some examples, the artificial reality application may determine the focus of the user 110 based on the focus data captured by the eye tracking camera 140. In some examples, the artificial reality application may determine the focus of user 110 based on identifying the user's interactions with one or more objects within field of view 130 .

[0056] In some examples, the artificial reality application may determine the focus of user 110 based on information related to one or more pupils and / or corneas of user 110. For example, the image data captured by eye tracking camera 140 may include information related to one or more pupils and / or corneas of user 110. In some examples, the image data captured by eye tracking camera 140 may include the location of the center of the left pupil and / or the location of the center of the right pupil of user 110. The artificial reality application may determine the focus of user 110 based on the location of the center of the left pupil and / or the location of the center of the right pupil. In some cases, the artificial reality application may determine the focus and / or gaze direction of user 110 based on other information related to the pupils and / or corneas of user 110. For example, the artificial reality application may determine the focus and / or gaze direction of user 110 based on the shape of the left cornea, the shape of the right cornea, or information indicating infrared light reflected from one or both of the left cornea or the right cornea.

[0057] As described above, the artificial reality application can generate a 3D mesh based on depth data collected by one or more depth sensors (e.g., internal and external image capture devices 138) of the HMD 112. The 3D mesh can include a set of data points, where the position of each data point of the set of data points relative to each other data point of the set of data points is known. The 3D mesh can represent the topography of the physical environment of the user 110. For example, the 3D mesh includes one or more physical objects and / or points (e.g., wall 121, hand 132, arm 134, peripheral device 136, corner 152, and corner 154) within the real-world physical environment of the user 110, for example. The 3D mesh can indicate that the hand 132 is closer to the user 110 than the corner 152. In addition, in some cases, the 3D mesh can indicate the distance between any two or more objects or points within the real-world 3D physical environment of the user 110. The artificial reality application can generate the 3D mesh in near real time so that the 3D mesh reflects changes in the physical environment of the user 110. For example, if the user 110 moves the hand 132, the artificial reality application can update the 3D mesh in near real time to account for the movement of the hand 132. The artificial reality application can generate a 3D scene by overlaying at least a portion of the real-world image data collected by the pass-through image capture device 139 onto the 3D mesh. In addition, the artificial reality application can generate artificial reality content 122 as an overlay on the 3D scene based on the detected focus of the user 110.

[0058] The 3D mesh and eye-tracking camera 140 may allow the artificial reality application to provide a zoom experience to the user 110. For example, the artificial reality application may identify the depth of field of the user 110. As an example, the depth of field may include the corner 152 but not the corner 154, the peripheral device 136, the hand 132, and the arm 134. The artificial reality application may in turn generate artificial reality content 122 in near real time to match the depth of field of the user 110. In other words, the artificial reality application may generate artificial reality content 122 such that the user 110 perceives the corner 152 and other points within the depth of field as sharp or "in focus", while the corner 154, the peripheral device 136, the hand 132, the arm 134, and other objects or points outside the depth of field are perceived as blurry or "out of focus". The 3D mesh indicates the depth of each object included in the artificial reality content 122 relative to the position of the HMD 112. In this way, to generate artificial reality content 122 , the artificial reality application may blur or unblur objects in artificial reality content 122 based on whether the depth of the object indicated by the 3D mesh is within the detected depth of field of user 110 .

[0059] In some examples, the artificial reality application may calibrate the inside-out image capture device 138 and other depth sensors based on detecting the focus of the user 110 relative to one or more objects within the artificial reality content 122. In some examples, the artificial reality application may determine that the user 110 is focusing on an object in the artificial reality content 122, such as the corner 152. At this point, the artificial reality application may determine the focus of the user 110 while the user 110 is focusing on the corner 152. Based on the focus of the user 110 when the user 110 is focusing on the corner 152 and / or the focus of the user 110 when the user 110 is focusing on another object within the artificial reality content 122, the artificial reality application may calibrate the inside-out image capture device 138.

[0060] The HMD 112 may modify the focal length of the varifocal display of the HMD 112 based on the detected focus of the user 110. For example, the artificial reality application may detect the focus of the user 110 based on the image data collected by the eye tracking camera 140. In turn, the HMD 112 may move the varifocal display of the HMD 112 relative to the eyes of the user 110. For example, the HMD 112 may display artificial reality content 122 to the user 110 on a varifocal display that includes a motor (e.g., an electric motor). The artificial reality application may output instructions to move the display panel relative to the eyes of the user 110 so as to match the detected focus of the user 110. In this way, the HMD 112 may implement a mechanical zoom system that allows the artificial reality application to move the varifocal display. The artificial reality application may determine the focus of the user 110 in near real time. In this way, the artificial reality application may modify the focal length of the varifocal display in near real time based on the detected focus of the user 110. Additionally or alternatively, based on the determined focus of the user 110 , the HMD 112 may implement an optical zoom system that changes the properties of one or more imaging components of the HMD 112 .

[0061] In some examples, the artificial reality application may predict a future focus of the user 110 based on the focus data collected by the eye tracking camera 140. For example, the artificial reality application may track the position of a first pupil of the user 110 relative to a first set of reference points over a period of time, and track the position of a second pupil of the user 110 relative to a second set of reference points over a period of time. Subsequently, the artificial reality application may determine an expected future movement of the position of the first pupil relative to the first set of reference points based on the movement of the position of the first pupil relative to the first set of reference points over a period of time, and determine an expected future movement of the position of the second pupil relative to the second set of reference points based on the movement of the position of the second pupil relative to the second set of reference points over a period of time. The artificial reality application may determine an expected future focus of the user 110 based on the expected future movement of the position of the first pupil and the expected future movement of the position of the second pupil. In some examples, the artificial reality application may generate artificial reality content 122 and / or modify the focal length of a zoom display based on the expected future focus of the user 110. Additionally or alternatively, the artificial reality application may generate artificial reality content 122 and / or modify the focal length of the zoom display based on both the predicted future focus of the user 110 and the current focus of the user 110 .

[0062] One or more techniques described herein may provide one or more technical improvements that provide practical applications. For example, by using focus data captured by one or more eye tracking cameras 140 to determine one or both of a user's focus and depth of field, an artificial reality application may improve a virtual scene (e.g., artificial reality content 122) displayed by an HMD 112, compared to an artificial reality system that does not use focus data to generate a virtual scene. In other words, the artificial reality system 10 may customize the artificial reality content 122 to focus on objects within the depth of field of the user 110, thereby allowing the artificial reality system 10 to mimic a real-world environment. In addition, the three-dimensional mesh may include information indicating the depth of one or more objects that are part of the real-world environment near the user 110, which is used as the basis for the artificial reality content 122 displayed to the user 110 by the HMD 112. Therefore, when the user focuses on one or more objects in the artificial reality content 122, the HMD 112 may modify the focal length of the zoom display based on the detected focus of the user 112.

[0063] Figure 1B is a conceptual diagram illustrating an artificial reality system 20 for presenting a virtual environment to more than one user in accordance with one or more techniques of the present disclosure. Figure 1A The artificial reality system 10, in some examples, Figure 1B The artificial reality system 20 can generate and draw virtual content items within the multi-user artificial reality environment. In various examples, the artificial reality system 20 can also generate certain virtual content items and / or graphical user interface elements and draw them to the user in response to detecting one or more specific interactions of the user with the peripheral device 136.

[0064] exist Figure 1B In the example of FIG. 1 , the artificial reality system 20 includes external cameras 102A and 102B (collectively referred to as “external cameras 102”), HMDs 112A-112C (collectively referred to as “HMDs 112”), controllers 114A and 114B (collectively referred to as “controllers 114”), a console 106, and a sensor 90. Figure 1BAs shown in , artificial reality system 20 represents a multi-user environment in which an artificial reality application executed on console 106 and / or HMD 112 presents artificial reality content to each of users 110A-110C (collectively, “users 110”) based on a current viewing perspective of a corresponding reference frame for the respective user. That is, in this example, the artificial reality application constructs artificial reality content by tracking and calculating pose information for a reference frame for each HMD 112. Artificial reality system 20 uses data received from camera 102, HMD 112, and controller 114 to capture 3D information within the real-world environment, such as the motion of user 110 and / or tracking information about user 110 and object 108, for use in calculating updated pose information for a corresponding reference frame for HMD 112. As an example, the artificial reality application may render artificial reality content 122 having virtual objects 128A-128B (collectively, “virtual objects 128”) spatially overlaid on real-world objects 108A-108B (collectively, “real-world objects 108”) based on the current viewing perspective determined for HMD 112C. Additionally, from the perspective of HMD 112C, artificial reality system 20 renders avatars 120A, 120B based on estimated positions for users 110A, 110B, respectively. HMD 112C may be Figure 1A and Figure 1B An example of an HMD 112 .

[0065] Each of the HMDs 112 operates simultaneously within the artificial reality system 20. Figure 1B In the example of FIG. 1 , each user 110 may be a “player” or “participant” in the artificial reality application, and any user 110 may be a “spectator” or “observer” in the artificial reality application. By tracking the hand 132 and / or arm 134 of the user 110C and rendering the portion of the hand 132 within the field of view 130 as a virtual hand 132 within the artificial reality content 122, the HMD 112C may be substantially similar to the virtual hand 132 in FIG. Figure 1A HMD 112B may receive user input from controller 114 held by user 110B. In some examples, controllers 114A and / or 114B may correspond to Figure 1A The peripheral device 136 is substantially similar to Figure 1A The HMD 112A may also be substantially similar to Figure 1AThe HMD 112B is operated by the HMD 112A and receives user input in the form of gestures performed by the hands 132A, 132B of the user 110A on or with the peripheral device 136. The HMD 112B receives user input from a controller 114 held by the user 110B. The controller 114 may communicate with the HMD 112B using near field communication such as short-range wireless communication such as Bluetooth, using a wired communication link, or using other types of communication links.

[0066] In some examples, the console 106 and / or HMD 112C of the artificial reality system 20 may generate and render a virtual surface including a virtual content item 129 (e.g., a GIF, photo, application, live stream, video, text, web browser, drawing, animation, 3D model, representation of a data file (including two-dimensional and three-dimensional data sets), or any other visual media) that may be overlaid on the artificial reality content 122 displayed to the user 110C when a portion of the wall 121 associated with the virtual content item 129 enters the field of view 130 of the HMD 112C. Figure 1B As shown in , in addition to or in lieu of image data captured via the image capture device 138 of the HMD 112C, input data from the external camera 102 can be used to track and detect specific movements, configurations, positions and / or orientations of the hands and arms of the user 110 (such as the hand 132 of the user 110C) and / or the peripheral devices 136, including movement of individual fingers and / or finger combinations (fingers, thumbs) of the hands.

[0067] In some examples, an artificial reality application may be running on console 106 and may utilize external cameras 102A and 102B to analyze the configuration, position, and / or orientation of hand 132B to identify input gestures that may be performed by a user of HMD 112A. Similarly, HMD 112C may utilize image capture device 138 to analyze the configuration, position, and / or orientation of peripheral device 136 and hand 132C to input gestures that may be performed by a user of HMD 112C. In some examples, peripheral device 136 includes one or more sensors (e.g., accelerometers) for tracking the motion or orientation of peripheral device 136. In response to such gestures, motion, and orientation, the artificial reality application may perform gestures similar to those described above with respect to Figure 1A Describes the way to draw virtual content items and / or UI elements.

[0068] The external camera 102 and the internal and external image capture device 138 can capture images in the visible spectrum, infrared spectrum, or other spectrums. For example, the image processing described herein for identifying objects, object poses, and gestures can include processing infrared images, visible spectrum images, etc.

[0069] The devices of the artificial reality system 20 can work together in the AR environment. For example, the peripheral device 136 is paired with the HMD 112C to operate together within the AR system 20. Similarly, the controller 114 is paired with the HMD 112B to operate together within the AR system 20. The peripheral device 136, the HMD 112, and the controller 114 can each include one or more SoC integrated circuits configured to enable an operating environment for artificial reality applications.

[0070] In some examples, each of the HMDs 112 includes one or more image capture devices and one or more eye tracking cameras. For example, HMD 112C includes inside-out image capture device 138, pass-through image capture device 139, and eye tracking cameras 140A-140B (collectively, “eye tracking cameras 140”). The artificial reality application can generate artificial reality content for display by any one or more of the HMDs 112. Each HMD 112 can perform operations related to Figure 1A and Figure 1B Any one or more of the techniques described for HMD 112 may be used.

[0071] Figure 2A is a conceptual diagram illustrating an example HMD 112 and example peripheral devices 136 for presenting a virtual scene to a user, in accordance with one or more techniques of this disclosure. Figure 2A The HMD 112 may be Figure 1A and Figure 1B An example of any HMD 112. The HMD 112 may be part of an artificial reality system, such as Figure 1A , Figure 1B The artificial reality system 10, 20 may be operated as an independent, mobile artificial reality system configured to implement the techniques described herein.

[0072] In this example, the HMD 112 includes a front rigid body and a band for fixing the HMD 112 to the user. In addition, the HMD 112 includes an inward-facing zoom display 203, which is configured to present artificial reality content to the user. The zoom display 203 can be any suitable display technology, such as a liquid crystal display (LCD), a quantum dot display, a dot matrix display, a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a cathode ray tube (CRT) display, an electronic ink, or a monochrome, color, or any other type of display capable of generating visual output. In some examples, the zoom display is a stereoscopic display for providing a separate image to each eye of the user. In some examples, when tracking the position and orientation of the HMD 112 to draw artificial reality content according to the current viewing angle of the HMD 112 and the user, the known orientation and position of the display 203 relative to the front rigid body of the HMD 112 is used as a reference frame, also referred to as a local origin. In other examples, the HMD 112 can take the form of other wearable head-mounted displays, such as glasses or goggles. In some examples, HMD 112 may include a single varifocal display 203 to provide images to both eyes of the user sequentially or simultaneously. In some examples, HMD 112 may include two varifocal displays, or one varifocal display for each eye of user 110. In some examples, HMD 112 may include a pupil replica waveguide that is used to deliver images in an angular domain generated by a micro-projector directly to the user's eyes.

[0073] In some examples, the display 203 can be moved along the axis 204 within the HMD 112. For example, the movement of the display 203 can be controlled by a motor (e.g., a motor) configured to move the display 203 closer to or farther away from one (or both) eyes of the user of the HMD 112. By moving the display 203, the user of the HMD 112 can change the way the user focuses on the artificial reality content presented by the display 203. For example, moving the display 203 closer to the user's eyes or moving the display 203 away from the user's eyes can match the focal length of the display 203 to the user's focus. In this way, the HMD 112 can represent a zoom device because the HMD 112 is configured to change the focus of the artificial reality content presented by the display 203.

[0074] like Figure 2AAs further shown in , in this example, the HMD 112 also includes one or more motion sensors 206, such as one or more accelerometers (also referred to as inertial measurement units or "IMUs") that output data indicating the current acceleration of the HMD 112, a GPS sensor that outputs data indicating the position of the HMD 112, a radar or sonar that outputs data indicating the distance of the HMD 112 from various objects, or other sensors that provide an indication of the position or orientation of the HMD 112 or other objects within the physical environment. The HMD 112 can collect depth data that indicates the depth of one or more points and / or objects in the real-world 3D physical environment of the user of the HMD 112. In some examples, the depth sensor 208 can collect the depth data. In some examples, the internal and external image capture device 138 can collect the image data.

[0075] The HMD 112 may include integrated inside-out image capture devices 138A and 138B (collectively, “inside-out image capture devices 138 ”), such as cameras, laser scanners, Doppler radar scanners, depth scanners, etc., configured to capture image data representative of the physical environment. More specifically, the inside-out image capture devices 138 capture image data representative of objects (including the peripheral device 136 and / or the hand 132 ) in the physical environment that are within the field of view 130A, 130B of the inside-out image capture devices 138 (which generally corresponds to the viewing perspective of the HMD 112 ). In some examples, two or more of the inside-out image capture devices 138 may collect depth data that indicates the depth of one or more points and / or objects in the real-world 3D physical environment of the user of the HMD 112 . In this manner, the inside-out image capture devices 138 may represent the depth sensor 208 . In other cases, the inside-out image capture devices 138 and the depth sensor 208 may be separate devices.

[0076] The HMD 112 may include one or more pass-through image capture devices 139. The pass-through image capture devices 139 may be configured to capture real-world image data in color and at a higher resolution than that captured by the internal and external image capture devices 138, which allows the artificial reality system to display or "pass-through" at least some of the real-world image data to the user via the HMD 112. In other words, the HMD 112 may display a virtual scene to the user that includes one or more elements, objects, and aspects of the real-world scene that the user would see without the use of the HMD 112.

[0077] HMD 112 includes an internal control unit 210, which may include an internal power supply and one or more printed circuit boards having one or more processors, memory, and hardware to provide an operating environment for performing programmable operations to process sensory data and present artificial reality content on display 203. In some examples, one or more techniques described herein as being performed by control unit 210 may be performed by any one or a combination of console 106, peripherals 136, and HMD 112.

[0078] HMD 112 may include eye-tracking camera 140A and eye-tracking camera 140B (collectively, “eye-tracking cameras 140”). Each of eye-tracking cameras 140 may capture image data indicating a pupil of a user of HMD 112 relative to a corresponding set of reference points. For example, eye-tracking camera 140A may capture image data indicating a position of a first pupil (e.g., a left pupil) of the user relative to a first set of reference points, and eye-tracking camera 140B may capture image data indicating a position of a second pupil (e.g., a right pupil) of the user relative to a second set of reference points. In some examples, the image data captured by eye-tracking cameras 140 may represent focus data, as the position of the first pupil relative to the first set of reference points and the position of the second pupil relative to the second set of reference points may indicate a depth of field (e.g., focus) of the user.

[0079] The depth of field may represent the distance between the nearest object and the farthest object in the scene or image that is in focus (e.g., appears sharp) as perceived by the user. As described herein, the depth of field may also include the distance between the nearest focused object and the user, and the distance between the farthest focused object and the user. For example, if the closest focused object to the observer is 10 meters away from the observer, and the farthest focused object from the observer is 11.5 meters away from the observer, the observer's depth of focus is 1.5 meters. In such an example, objects closer than 10 meters and objects farther than 12 meters may appear blurry to the observer, while objects within the range of 10 meters to 12 meters may appear clear to the observer. In some examples, the depth of field of the image data captured by the pass-through image capture device 139 may be proportional to the "distance to the object", which is the distance between the pass-through image capture device 139 and the object on which the image capture device 139 focuses. In the case where the image capture device 139 is focused on a peripheral device within the artificial reality content 122, the depth of field of the image data captured by the image capture device 139 may be less than, for example, the case where the image capture device 139 is focused on a corner 154. In some cases, the optical power of a lens may also be a function of distance. For example, the optical power of a lens may be inversely proportional to the focal length of the lens. Diopter is a unit of measurement for the optical power of a lens.

[0080] In one example, the control unit 210 is configured to generate and render one or more virtual content items (e.g., a location contained within the field of view 130A, 130B of the inside-out image capture device 138 or within the field of view 130C of the image capture device 139) associated with the location based on the sensory data (e.g., image data captured by any one or more of the image capture device 102, the inside-out image capture device 138, and the pass-through image capture device 139 and location information from a GPS sensor). Figure 1A The virtual content items 124, 126) are displayed on the display 203. Figure 1A-1B As explained, a virtual content item may be associated with a location in a real-world environment, and the control unit 210 may be configured to draw the virtual content item (or portion thereof) for display on the display 203 in response to determining that the location associated with the virtual content (or portion thereof) is within the current field of view 130A, 130B.

[0081] In one example, the control unit 210 is configured to identify a particular gesture or combination of gestures performed by a user based on the sensed data, and in response, perform an action. For example, in response to an identified gesture, the control unit 210 may generate and draw a particular user interface to be displayed on the zoom display 203 at a user interface position locked relative to the position of the peripheral device 136. For example, the control unit 210 may generate and draw a user interface including one or more UI elements (e.g., virtual buttons) on the surface 220 of the peripheral device 136 or near the peripheral device 136 (e.g., above, below, or adjacent to the peripheral device 136). The control unit 210 may perform object identification within the image data captured by the internal and external image capture device 138 to identify the peripheral device 136 and / or the hand 132, finger, thumb, arm, or another part of the user, and track the movement, position, configuration, etc. of the peripheral device 136 and / or the identified part of the user to identify the predefined gesture performed by the user. In response to identifying a predefined gesture, the control unit 210 takes some action, such as selecting an option from a set of options associated with the user interface (e.g., selecting an option from a UI menu), converting the gesture into an input (e.g., a character), launching an application, manipulating virtual content (e.g., moving, rotating a virtual content item), generating and drawing a virtual marker, generating and drawing a laser pointer, or otherwise displaying content, etc. For example, in response to detecting a predefined gesture designated as a "trigger" for revealing a user interface (e.g., turning a peripheral device into a landscape or horizontal orientation (not shown)), the control unit 210 can dynamically generate and present a user interface such as a menu. In some examples, the control unit 210 detects user input (e.g., a tap gesture performed on a virtual UI element) with respect to the drawn user interface based on sensory data. In some examples, the control unit 210 performs such functions in response to instructions from an external device such as the console 106, which can perform object identification, motion tracking, and gesture detection, or any portion thereof.

[0082] As an example, the control unit 210 can utilize the internal and external image capture devices 138A and 138B to analyze the configuration, position, movement, and / or orientation of the peripheral device 136, the hand 132, and / or the arm 134 to identify user interface gestures, selection gestures, stamping gestures, translation gestures, rotation gestures, pull gestures, pointing gestures, etc. that can be performed by the user relative to the peripheral device 136. The control unit 210 can draw a UI menu (including UI elements) and enable the user to interact with the UI menu based on the detection of the user interface gestures, selection gestures, stamping gestures, translation gestures, rotation gestures, and pull gestures performed by the user relative to the peripheral device, as described in further detail below.

[0083] In one example, surface 220 of peripheral device 136 is a presence-sensitive surface, such as a surface that uses capacitive, conductive, resistive, acoustic, or other technology to detect touch and / or hover input. In some examples, surface 220 of peripheral device 136 is a touch screen (e.g., a capacitive touch screen, a resistive touch screen, a surface acoustic wave (SAW) touch screen, an infrared touch screen, an optical imaging touch screen, an acoustic pulse identification touch screen, or any other touch screen). In such an example, peripheral device 136 can draw a user interface or other virtual elements (e.g., a virtual marker) on surface 220 and detect user input (e.g., touch or hover input) on surface 220. In such an example, peripheral device 136 can use a wireless communication link (e.g., Wi-Fi, near field communication for short-range wireless communication such as Bluetooth), use a wired communication link (not shown), or use other types of communication links to transmit any detected user input to HMD 112 (and / or Figure 1A In some examples, the peripheral device may include one or more input devices (e.g., buttons, trackballs, scroll wheels) for interacting with virtual content (e.g., selecting virtual UI elements, scrolling virtual UI elements).

[0084] The control unit 210 may generate artificial reality content presented by the display 203. One way in which the control unit 210 may generate artificial reality content presented by the display 203 is to blur one or more portions of the artificial reality content outside the depth of field of the user of the HMD 112. The 3D grid may indicate the depth of each object included in the artificial reality content relative to the position of the HMD 112. In this way, in order to generate the artificial reality content, the control unit 210 may blur or unblur the objects in the artificial reality content based on whether the depth of the object indicated by the 3D grid is within the depth of field of the user. In addition, the control unit 210 may move the zoom display 203 along the axis 204 to modify the focal length of the zoom display 203. For example, the HMD 112 may present artificial reality content to the user on the zoom display 203 attached to a motor (e.g., a motor). The control unit 210 may move the zoom display 203 to reflect the detected focus of the user. In some examples, the focus of the user may be determined in near real time based on image data captured by the eye tracking camera 140. In this manner, control unit 210 may generate artificial reality content in near real time based on the user's depth of view and based on the position of one or more objects identified by the 3D mesh.

[0085] Figure 2B is a conceptual diagram illustrating another example HMD 112 according to one or more techniques of this disclosure. Figure 2B As shown in , HMD 112 may take the form of glasses. Figure 2AThe HMD 112 may be Figure 1A and Figure 1B The HMD 112 may be an artificial reality system such as Figure 1A , Figure 1B The artificial reality system 10, 20 may be part of an artificial reality system 10, 20), or may be operated as an independent, mobile artificial reality system configured to implement the techniques described herein.

[0086] In this example, HMD 112 is a pair of glasses that includes a front frame that includes a bridge that allows HMD 112 to rest on the user's nose and temples that extend over the user's ears to secure HMD 112 to the user. Figure 2B The HMD 112 includes inwardly facing varifocal displays 203A and 203B (collectively referred to as "varifocal displays 203") that are configured to present artificial reality content to the user. The varifocal displays 203 may be any suitable display technology, such as an LCD, a quantum dot display, a dot matrix display, an LED display, an OLED display, a CRT display, electronic ink, or a monochrome, color, or any other type of display capable of generating a visual output. Figure 2B In the example shown in , the varifocal display 203 forms a stereoscopic display for providing a separate image to each eye of the user. In some examples, when tracking the position and orientation of the HMD 112 to render artificial reality content according to the current viewing perspective of the HMD 112 and the user, the known orientation and position of the display 203 relative to the front frame of the HMD 112 is used as a reference frame, also referred to as a local origin. In some examples, the HMD 112 can include a pupil replica waveguide that is used to transmit the image in the angular domain generated by the micro-projector directly to the user's eyes.

[0087] like Figure 2BAs further shown in , in this example, the HMD 112 also includes one or more motion sensors 206, such as one or more accelerometers (also referred to as inertial measurement units or "IMUs") that output data indicating the current acceleration of the HMD 112, a GPS sensor that outputs data indicating the position of the HMD 112, a radar or sonar that outputs data indicating the distance of the HMD 112 from various objects, or other sensors that provide an indication of the position or orientation of the HMD 112 or other objects within the physical environment. The HMD 112 may include one or more depth sensors 208 that collect depth data indicating the depth of one or more points and / or objects in the real-world 3D physical environment of the user of the HMD 112. In addition, the HMD 112 may include integrated inside-outside image capture devices 138A and 138B (collectively referred to as "inside-outside image capture devices 138"), such as cameras, laser scanners, Doppler radar scanners, depth scanners, etc., which are configured to output image data representing the physical environment. The HMD 112 includes an internal control unit 210 , which may include an internal power supply and one or more printed circuit boards having one or more processors, memory, and hardware to provide an operating environment for performing programmable operations to process sensory data and present artificial reality content on the display 203 .

[0088] Similar to Figure 2A , the control unit 210 may generate artificial reality content presented by the varifocal display 203. The control unit 210 may generate artificial reality content presented by the display 203 by blurring one or more portions of the artificial reality content that are outside the depth of field of the user of the HMD 112. The 3D mesh may indicate the depth of each object included in the artificial reality content relative to the position of the HMD 112. In this manner, to generate the artificial reality content, the control unit 210 may blur or unblur the objects in the artificial reality content based on whether the depth of the object indicated by the 3D mesh is within the depth of field of the user. The control unit 210 may determine the user's depth of field based on image data collected by the eye tracking camera 140.

[0089] Figure 3 is a diagram illustrating one or more techniques according to the present disclosure Figure 1A , Figure 1B A block diagram of an example implementation of a console 106, an HMD 112, and peripheral devices 136 of a multi-device artificial reality system 10, 20. Figure 3In the example of , the console 106 sets the focal length of the varifocal display 203 of the HMD 112, thereby affecting the user's perception of the artificial reality content displayed by the HMD 112. Additionally or alternatively, the console 106 may blur portions of the artificial reality content displayed by the HMD 112 to mimic how a user of the HMD 112 would perceive the physical environment of the real world based on the user's focus (e.g., focal length and / or depth of field).

[0090] In this example, HMD 112 includes one or more processors 302 and memory 304, which in some examples provide a computer platform for executing an operating system 305, which may be, for example, an embedded real-time multitasking operating system, or other types of operating systems. Operating system 305 in turn provides a multitasking operating environment for executing one or more software components 307, including application engine 340. Figure 2A and Figure 2B As discussed in the example of , processor 302 is coupled to varifocal display 203, motion sensor 206, depth sensor 208, inside-out image capture device 138, pass-through image capture device 139, and eye tracking camera 140. In some examples, processor 302 and memory 304 may be separate, discrete components. In other examples, memory 304 may be on-chip memory collocated with processor 302 within a single integrated circuit.

[0091] Generally speaking, the console 106 is responsible for processing the data from the camera 102 ( Figure 1B ), the image and tracking information received from any one or more of the internal and external image capture devices 138, the pass-through image capture device 139, and the eye tracking camera 140, so as to set the focal length of the zoom display 203 of the HMD 112 and / or generate artificial reality content for display by the HMD 112. The zoom display 203 can be connected to the motor 306, wherein the motor 306 is configured to move the zoom display 203 to set the focal length of the zoom display 203. In some examples, the console 106 is a single computing device, such as a workstation, a desktop computer, a laptop computer, or a gaming system. In some examples, at least a portion of the console 106, such as the processor 312 and / or the memory 314, can be distributed on a cloud computing system, a data center, or a network, such as the Internet, another public or private communication network, such as broadband, cellular, Wi-Fi, and / or other types of communication networks for transmitting data between computing systems, servers, and computing devices.

[0092] exist Figure 3In the example of , console 106 includes one or more processors 312 and memory 314, which in some examples provide a computer platform for executing an operating system 316, which may be, for example, an embedded real-time multitasking operating system or other type of operating system. Operating system 316 in turn provides a multitasking operating environment for executing one or more software applications 317. Processor 312 is coupled to one or more I / O interfaces 315, which provide one or more I / O interfaces for communicating with external devices such as keyboards, game controllers, display devices, image capture devices, HMDs, peripheral devices, etc. In addition, one or more I / O interfaces 315 may include one or more wired or wireless network interface controllers (NICs) for communicating with a network such as network 104.

[0093] The software application 317 of the console 106 operates to provide an overall artificial reality application. In this example, the software application 317 includes an application engine 320, a rendering engine 322, a depth engine 324, and a gesture tracker 326.

[0094] Generally speaking, the application engine 320 includes functionality for providing and presenting artificial reality applications, such as teleconferencing applications, gaming applications, navigation applications, educational applications, training or simulation applications, etc. The application engine 320 may include, for example, one or more software packages, software libraries, hardware drivers, and / or application program interfaces (APIs) for implementing artificial reality applications on the console 106. In response to control of the application engine 320, the rendering engine 322 may generate 3D artificial reality content for display to the user by the application engine 340 of the HMD 112.

[0095] In some examples, the application engine 320 and the rendering engine 322 can construct artificial reality content for display to the user 110 based on the current pose information for a reference frame, which is typically the viewing angle of the HMD 112 determined by the pose tracker 326. Based on the current viewing angle, the rendering engine 322 constructs 3D artificial reality content, which in some cases can be at least partially overlaid on the real world 3D environment of the user 110. During this process, the pose tracker 326 operates on sensory data received from the HMD 112 (such as movement information and user commands), and in some examples, on any external sensors 90 ( Figure 1A , Figure 1B) (such as an external camera) to capture 3D information within the real-world environment, such as the movement of the user 110, and / or feature tracking information about the user 110. Based on the sensed data, the pose tracker 326 determines the current pose of the reference frame of the HMD 112, and, based on the current pose, constructs artificial reality content for transmission to the HMD 112 via one or more I / O interfaces 315 for display to the user 110.

[0096] The gesture tracker 326 can determine the current gesture of the peripheral device 136 and, based on the current gesture, trigger certain functions associated with any drawn virtual content (e.g., placing a virtual content item, manipulating a virtual content item, generating and drawing one or more virtual markers, generating and drawing a laser pointer). In some examples, the gesture tracker 326 detects whether the HMD 112 is near a physical location corresponding to a virtual surface (e.g., a virtual pinboard) to trigger the drawing of the virtual content.

[0097] The rendering engine 322 is configured to render virtual content in the artificial reality environment based on the current posture of the peripheral device 136, which virtual content can be locked relative to the position of another object in the artificial reality environment. The rendering engine 322 can scale, rotate, and otherwise transform the virtual content to match the position of the object so that the virtual content appears to be overlaid on or near the object in the artificial reality environment. In some examples, the rendering engine 322 can slide, translate, or otherwise move one or more virtual rendering cameras based on the posture data and / or based on the focus data generated by the image data captured by the image capture devices 138, 139.

[0098] In some examples, the depth engine 324 is configured to generate a 3D scene of the physical environment of the user 110 of the HMD 112 based on the real-world image data and the depth data associated with the real-world image data. The real-world image data can be captured by the pass-through image capture device 139 of the HMD 112, which represents a pass-through camera. In some examples, the depth data can be captured by the depth sensor 208 of the HMD 112. In some examples, the depth data can be captured by the inside-outside image capture device 138 of the HMD 112, which represents an inside-outside camera. The depth engine 324 can generate artificial reality content (e.g., artificial reality content 122) based on the focus of the user 110 as an overlay of the 3D scene of the physical environment for display on the zoom display of the HMD. In some examples, the depth engine 324 can determine the focus of the user 110 based on the image data captured by the eye tracking camera 140.

[0099] The console 106 may output the virtual user interface and other artificial reality content to the HMD 112 via the communication channel for display at the HMD 112. The rendering engine 322 receives gesture information for the peripherals 136 to continuously update the user interface position and gesture to match the position and gesture of the peripherals 136, such as the position and gesture of one of the presence-sensitive surfaces 220.

[0100] exist Figure 3 In the example shown in , the peripheral device 136 includes one or more processors 346 and memory 344, which in some examples provide a computer platform for executing an operating system 342, which may be, for example, an embedded real-time multitasking operating system or other type of operating system. The operating system 342, in turn, provides a multitasking operating environment for executing one or more software components. In some examples, the peripheral device 136 includes one or more presence-sensitive surfaces 220 (e.g., one or more surfaces that use capacitive, conductive, resistive, acoustic, and / or other technologies to detect touch and / or hover input). In one or more aspects, the peripheral device 136 can be configured to detect touch and / or hover input at the presence-sensitive surface 220, process the input (e.g., at the processor 346), and communicate the touch and / or hover input and information about the input (including location information about the input) to the console 106 and / or HMD 112. As described with respect to Figure 2A As discussed in the examples of , presence-sensitive surface(s) 220 may include a touch screen (e.g., a capacitive touch screen, a resistive touch screen, a surface acoustic wave (SAW) touch screen, an infrared touch screen, an optical imaging touch screen, an acoustic pulse identification touch screen, or any other touch screen).

[0101] like Figure 3As further shown in , in this example, the peripheral device 136 also includes one or more motion sensors 348, such as one or more accelerometers (also referred to as IMUs) that output data indicating the current acceleration of the peripheral device 136, a GPS sensor that outputs data indicating the location or position of the peripheral device, a radar or sonar that outputs data indicating the distance of the peripheral device from various objects (e.g., a wall or other surface), or other sensors that provide an indication of the location, position, and / or orientation of the peripheral device or other objects within the physical environment. In some examples, the processor 346 is coupled to the presence-sensitive surface(s) 220 and the motion sensor 246. In some examples, the processor 346 and the memory 344 can be separate, discrete components. In other examples, the memory 344 can be an on-chip memory co-located with the processor 346 within a single integrated circuit. In one or more aspects, the peripheral device 136 can coexist with the HMD, and in some examples, operate as an auxiliary input / output device of the HMD in a virtual environment. In some examples, the peripheral device 136 can operate as an artificial reality co-processing device to which some functions of the HMD are offloaded. In one or more aspects, peripheral device 136 may be a smartphone, tablet computer, or other handheld device.

[0102] In some examples, each of the processors 302, 312, 346 may include any one or more of the following: a multi-core processor, a controller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or equivalent discrete or integrated logic circuits. The memory 304, 314, 344 may include any form of memory for storing data and executable software instructions, such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electronically erasable programmable read-only memory (EEPROM), and flash memory.

[0103] The depth engine 324 may generate artificial reality content for display by the HMD 112 based on the real-world image data captured by the pass-through image capture device 139 and the focus data captured by the eye tracking camera 140. For example, the depth engine 324 may be configured to determine one or both of the focus of the user 110 and the depth of field of the user 110 based on the focus data captured by the eye tracking camera 140. In some examples, the focus data may represent image data indicating the position of a first pupil of the user relative to a first set of reference points, and image data indicating the position of a second pupil of the user relative to a second set of reference points. The depth engine 324 may determine the focus and / or depth of field of the user 110 based on the position of the first pupil relative to the first set of reference points and the position of the second pupil relative to the second set of reference points.

[0104] In some examples, the depth engine 324 may generate a 3D mesh based on depth data collected by the depth sensor 208 and / or the inside-out image capture device 138 of the HMD 112. The depth engine 324 may generate artificial reality content by overlaying at least a portion of the real-world image data collected by the pass-through image capture device 139 onto the 3D mesh. The 3D mesh and the eye-tracking camera 140 may allow the depth engine 324 to provide a zoom experience to the user of the HMD 112. For example, the depth engine 324 may set the focal length of the zoom display 203 of the HMD 112 in near real time to match the detected focus of the user. Additionally or alternatively, the depth engine 324 may blur portions of the artificial reality content displayed by the HMD 112 that are outside the detected depth of field of the user.

[0105] Figure 4 is a diagram illustrating a method of performing a Figure 1A , Figure 1B A block diagram of an example of an artificial reality system in which HMD 112 generates a virtual environment.

[0106] In this example, something like Figure 3 , HMD 112 includes one or more processors 302 and memory 304, which in some examples provide a computer platform for executing an operating system 305, which may be, for example, an embedded real-time multitasking operating system or other type of operating system. Operating system 305 in turn provides a multitasking operating environment for executing one or more software components 417. In addition, processor(s) 302 are coupled to zoom display 203, motion sensor 206, inside-out image capture device 138, pass-through image capture device 139, and eye tracking camera 140. Figure 4 In the example of FIG. 4 , software component 417 operates to provide an overall artificial reality application. In this example, software application 417 includes application engine 440, rendering engine 422, depth engine 424, and pose tracker 426. In various examples, software component 417 is similar to Figure 3 106 (e.g., application engine 320, rendering engine 322, depth engine 324, and gesture tracker 326, respectively) to set the focus of varifocal display 203 of HMD 112 or otherwise affect the user's perception of artificial reality content provided by HMD 112. In some examples, HMD 112 can communicate with one or more other devices such as peripheral devices 136.

[0107] Figure 5is a conceptual diagram illustrating example components of an HMD 560 in accordance with one or more techniques of this disclosure. The HMD 560 may be substantially similar to the Figure 1A-1B , Figure 2A-2B and Figure 3 To facilitate the description of the internal components of HMD 560, Figure 5 5. The HMD 560 in FIG. 5 is shown as having a headphone form factor. In other examples, the HMD 560 may include another form factor, including a glasses form factor.

[0108] HMD 560 includes eyepieces 562A, 562B, where right eyepiece 562A is configured to present images to the user's right eye and left eyepiece 562B is configured to present images to the user's left eye. As used herein, the term "eyepiece" means a three-dimensional geometric area in which images of acceptable quality can be presented to the user's eyes. Figure 5 In an example of, each of the eyepieces 562A, 562B includes a zoom display 564A, 564B coupled to an imaging assembly 566A, 566B for communicating an image generated by the zoom display 564A, 564B to the eyepieces 562A, 562B when the user wears the HMD 560, and the user's eyes are located in the eyepieces 562A, 562B. Each of the imaging assemblies 566A, 566B can be a lens, a mirror, or any other element with optical (i.e., focusing) power. Each of the imaging assemblies 566A, 566B can include a zoom optical element with tunable or switchable optical power. In some examples, the zoom display 564A, 564B can be attached to one or more motors that are configured to move the zoom display 564A, 564B relative to the imaging assembly 566A, 566B in a direction 590. For example, one or more motors may move the zoom displays 564A, 564B to match the focal length of the HMD 560 with the detected focus of the user.In some examples, the eyepieces 562A, 562B may be referred to as "eye cups."

[0109] In some examples, HMD 560 may include a single zoom display to provide images to both eyes of the user sequentially or simultaneously. In other examples, HMD 560 may not include imaging components 566A, 566B, but may include a pupil replica waveguide that is used to transmit images in an angular domain generated by a micro-projector directly to the user's eyes.

[0110] Each of the eyepieces 562A, 562B may include an eye tracking system for tracking the position and orientation of the user's eyes in real time. The eye tracking system may include an array of illuminators 567A, 567B for illuminating the user's eyes, typically illuminated with invisible light such as an infrared light array, and a hot mirror 565A, 565B for reflecting infrared light scattered by the user's eyes and the eye area of ​​the user's face when transmitting visible light from the zoom display 564A, 564B. The eye tracking system also includes eye tracking cameras 584A, 584B for detecting images of the user's eyes, which have pupils and reflections (i.e., so-called "flashes") from the user's eyes to the illuminators 567A, 567B, for determining eye position and orientation. In this article, the term "eye area" indicates an area of ​​the user's face that includes the eyes. The eye area includes the eye itself, which has a cornea, an iris, and a pupil. The HMD 560 includes internal and external cameras 582A, 582B for capturing image data representing the physical environment around the user. In some examples, HMD 560 may not include thermal mirrors 565A, 565B and may include an eye tracking camera configured to capture the eye region of the user's face without thermal mirrors 565A, 565B.

[0111] HMD 560 includes a control unit 580 that is coupled to other components of HMD 560, including zoom displays 564A, 564B, imaging components 566A, 566B, illuminators 567A, 567B, eye tracking cameras 584A, 584B, and inside-out cameras 582A, 582B. Control unit 580 may be substantially similar to that from Figure 2A and Figure 2B The control unit 580 may be operated by the internal control unit 210 of the HMD 112. For example, during operation of the HMD 560, the control unit 580 may generate an image to be displayed by the zoom display 564A, 564B, energize the illuminators 567A, 567B, obtain an image of the eye region from the corresponding eye tracking cameras 584A, 584B, and may determine one or more of the user's gaze direction, the convergence angle of the user's eyes from the eye pupil position, and the blinking position in the obtained image. In some examples, the image of the eye region captured by the eye tracking cameras 584A, 584B may include information indicating the user's depth of field. In some examples, once the convergence angle is determined, the control unit 580 may adjust the focal length of the imaging components 566A, 566B to reduce the convergence adjustment conflict, that is, the difference between the eye convergence angle and the eye focusing distance. In some examples, the focus of the artificial reality content may be set based on the user's depth of field.

[0112] In some examples, the HMD 580 includes a pass-through camera 588 configured to capture image data representing objects in a real-world 3D physical environment within a field of view of the pass-through camera 588. In some examples, the depth engine can overlay at least some of the image data captured by the pass-through camera 588 onto a 3D mesh generated using the depth data to create a 3D scene of the physical environment. The HMD 360 can display the 3D scene of the physical environment to the user on the zoom displays 564A, 564B.

[0113] Fig. 6A 6 is a conceptual diagram illustrating a focal length 610 and a first depth of field 620 within an environment 602 in accordance with one or more techniques of the present disclosure. An observer 604 can view the environment 602. The focal length 610 can represent a distance between an eye position 612 of the observer 604 and a focal point 614 of the observer 604. Additionally, the first depth of field 620 of the observer 604 can represent a distance between a first near depth of field point 622 and a first far depth of field point 624. The focal point 514 is within the first depth of field 620. As shown in Fig. 6A As seen in the first depth of field 620, objects within the first depth of field 620 appear in focus (e.g., sharp), while objects outside the first depth of field 620 appear out of focus (e.g., blurry). In some examples, the observer 604 may use the environment 602 to Figure 1A-Figure 4 An example HMD (e.g., HMD 112) is provided, and the example HMD can present at least some of the environment 602 as virtual reality content to an observer 604. The observer 604 can perceive the virtual reality content representing the environment 602, which has depth characteristics similar to that of the observer 604 perceiving the real environment 602.

[0114] Figure 6B is a diagram illustrating one or more techniques according to the present disclosure Fig. 6A Conceptual diagram of the focal length 610 and the second depth of field 640 within the environment 602. The second depth of field 640 of the observer 604 can represent the distance between the second near depth of field point 642 and the second far depth of field point 644. The focal point 614 is within the second depth of field 640. Figure 6B As seen in the second depth of field 640, objects within the second depth of field 640 appear in focus (e.g., sharp), while objects outside the second depth of field 640 appear out of focus (e.g., blurry). The second depth of field 640 is greater than (e.g., longer than) Fig. 6A Thus, when observer 604 views environment 602 using second depth of field 640, more objects of environment 602 are in focus than when observer 604 views environment 602 using first depth of field 620. Figure 1A-Figure 4An example HMD (e.g., HMD 112) is provided, and the example HMD can present at least some of the environment 602 as virtual reality content to an observer 604. The observer 604 can perceive the virtual reality content representing the environment 602, which has depth characteristics similar to that of the observer 604 perceiving the real environment 602.

[0115] A depth engine (e.g., Figure 3 The depth engine 324 and / or Figure 4 The depth engine 424 of the HMD 112 may set the focus of the varifocal display of the HMD 112 based on the detected focus of the observer 604. For example, the depth engine may set the focal length of the varifocal display of the HMD 112 to match the detected focus of the observer 604. Additionally or alternatively, based on the detected depth of field of the observer 604, the depth engine may blur or unblur objects within the environment 602 to mimic how the user perceives the environment 602 without the HMD 112.

[0116] Figure 7 1 is a flowchart illustrating example operations for providing artificial reality content according to one or more techniques of the present disclosure. Figure 7 .However, Figure 7 The techniques may be performed by different components of console 106, HMD 112, and peripherals 136, or by additional or alternative devices.

[0117] The first image capture device captures real-world image data representing a physical environment of the user 110 (702). For example, the first image capture device may include a pass-through image capture device 139 of the HMD 110, which represents a pass-through camera that captures the real-world image data. Additionally or alternatively, the first image capture device may include other image capture devices of the HMD 112 and / or other image capture devices not included in the HMD 112. In some cases, the real-world image data representing the physical environment is within the field of view 130 of the pass-through image capture device 139. Because a user (e.g., user 110) can wear the HMD 112, the field of view of the pass-through image capture device 139 can change based on where the user 110 is looking.

[0118] A set of second image capture devices of HMD 112 captures image data indicating a gaze focus of user 110 (704). In some examples, the set of second image capture devices includes eye tracking camera 140. For example, eye tracking camera 140A may capture image data indicating a position of a first pupil (e.g., a left pupil) of user 110 relative to a first set of reference points, and eye tracking camera 140B may capture image data indicating a position of a second pupil (e.g., a right pupil) of user 110 relative to a second set of reference points. In some examples, the image data captured by eye tracking camera 140 may represent focus data because the position of the first pupil relative to the first set of reference points and the position of the second pupil relative to the second set of reference points may indicate one or both of a focus of user 110 and a depth of field of user 110.

[0119] The HMD 112 may modify the focal length of the zoom display 203 of the HMD 112 based on the focus of the user 110 indicated by the image data captured by the set of second image capture devices (706). For example, a depth engine (e.g., Figure 3 The depth engine 324 and / or Figure 4 The depth engine 424 of the present invention can determine the focus of the user 110 based on the image data captured by the set of second image capture devices. Then, the depth engine can cause a motor (e.g., a motor) to move the zoom display 203 so as to set the focus of the zoom display 203 to match the focus of the user 110.

[0120] The depth engine generates a 3D scene of the physical environment of the user 110 based on the real-world image data captured by the first image capture device and the depth data associated with the real-world image data (708). In some examples, the depth data may be captured using the inside-out image capture device 138 and / or the depth sensor 308 of the HMD 112. The depth engine may use the depth data to generate a 3D mesh of the physical environment of the user 110. The 3D mesh may include a set of data points, where the position of each data point of the set of data points relative to each other data point of the set of data points is known. For example, the set of data points may represent a topography of at least a portion of the physical environment of the user 110. For example, the 3D mesh includes a digital representation of the position of one or more physical objects and / or points within the physical environment of the user 110 (e.g., at least some of the one or more points and / or objects represented in the depth data). The depth engine may overlay at least a portion of the real-world image data onto the 3D mesh to generate the 3D scene.

[0121] The depth engine generates artificial reality content 122 as an overlay on the 3D scene for display on the zoom display 203 of the HMD 112 based on the detected focus of the user 110 (710). Since the 3D mesh can include data representing the depth of one or more objects and / or points and the real-world image data includes image data representing at least some of the one or more objects and / or points, the artificial reality content can include data reflecting the appearance and depth of such objects and / or points. The depth engine can generate the artificial reality content 122 and modify the focus of the zoom display 203 in near real time to provide a realistic, zoomed, and direct experience to the user 110.

[0122] As described herein by various examples, the technology of the present disclosure may include an artificial reality system or be implemented in conjunction with an artificial reality system. As described, artificial reality is a form of reality that is adjusted in some way before being presented to a user, which may include, for example, virtual reality (VR), augmented reality, mixed reality (MR), mixed reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured content (e.g., a photo of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of them may be presented in a single channel or multiple channels (such as a stereoscopic video that produces a three-dimensional effect to the viewer). In addition, in some embodiments, artificial reality may be associated with an application, product, accessory, service, or some combination thereof, such as being used to create content in artificial reality and / or being used in artificial reality (e.g., performing activities therein). An artificial reality system that provides artificial reality content may be implemented on various platforms, including an HMD connected to a host computer system, an independent HMD, a mobile device or a computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0123] The techniques described in this disclosure may be implemented, at least in part, in hardware, software, firmware, or any combination thereof. For example, various aspects of the described techniques may be implemented within one or more processors, including one or more microprocessors, DSPs, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or any other equivalent integrated or discrete logic circuits, and any combination of these components. The term "processor" or "processing circuit" may generally refer to any of the aforementioned logic circuits, either alone or in combination with other logic circuits or any other equivalent circuits. A control unit including hardware may also perform one or more of the techniques disclosed herein.

[0124] Such hardware, software and firmware can be implemented in the same device or in separate devices to support the various operations and functions described in this disclosure. In addition, any described unit, module or component can be implemented together or separately as a discrete but interoperable logical device. Describing different features as modules or units is intended to highlight different functional aspects and does not necessarily mean that these modules or units must be implemented by separate hardware or software components. On the contrary, the functionality associated with one or more modules or units can be performed by separate hardware or software components or integrated in common or separate hardware or software components.

[0125] The techniques described in the present disclosure may also be embodied or encoded in a computer-readable medium such as a computer-readable storage medium containing instructions. The instructions embedded or encoded in the computer-readable storage medium may cause a programmable processor or other processor to perform the method, for example, when the instructions are executed. The computer-readable storage medium may include a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a hard disk, a CD-ROM, a floppy disk, a tape, a magnetic medium, an optical medium, or other computer-readable medium.

Claims

1. An artificial reality system, comprising: a first image capture device configured to capture real-world image data representative of a user's physical environment; A head mounted display (HMD), wherein the HMD is configured to output artificial reality content, and the HMD comprises: a set of second image capture devices configured to capture image data indicative of a focus of the user's gaze; a variable zoom display having a focal length modifiable based on the focus of the user; and a motor configured to control a position of the zoom display to modify the focal length; and A depth engine, the depth engine being configured to: determining the focal length of the zoom display in near real time to match the focus of the user; controlling the motor to move the zoom display relative to the lens along the longitudinal axis of the HMD to achieve the determined focal length; generating a three-dimensional (3D) scene of the physical environment of the user based on the real-world image data and depth data associated with the real-world image data; and Artificial reality content is generated based on the focus of the user as an overlay on the 3D scene of the physical environment for display on the varifocal display of the HMD. 2 . The system of claim 1 , wherein the first image capture device comprises a pass-through camera configured to capture the real-world image data in color and at a high resolution.

3. The system of claim 2, wherein the HMD comprises one or more processors configured to execute the depth engine, wherein the one or more processors are located proximate to the pass-through camera.

4. The system of claim 1 , wherein the HMD further comprises a depth sensor configured to generate the depth data, and wherein to generate the 3D scene of the physical environment, the depth engine is configured to: using the depth data to generate a 3D mesh of a real-world scene; and At least a portion of the real-world image data is overlaid onto the 3D mesh of the real-world scene.

5. The system of claim 4, wherein the real-world image data represents a real-world image data stream, wherein the depth data represents a depth data stream, and wherein to generate the artificial reality content, the depth engine is further configured to: The artificial reality content is generated in near real time using the real-world image data stream and the depth data stream based on the position and orientation of the HMD relative to the 3D mesh of the real-world scene.

6. The system of claim 1, wherein the set of second image capture devices comprises: a first eye-tracking camera configured to capture a first set of image data including a position of a first pupil of the user relative to a first set of reference points; and a second eye-tracking camera configured to capture a second set of image data, the second set of image data comprising a position of a second pupil of the user relative to a second set of reference points, and wherein the depth engine is configured to: The focus of the user is determined based on the position of the first pupil relative to the first set of reference points and the position of the second pupil relative to the second set of reference points.

7. The system of claim 6, further comprising a gaze tracker, the gaze tracker being configured to: tracking a position of the first pupil relative to the first set of reference points over a period of time; tracking a position of the second pupil relative to the second set of reference points over the period of time; determining an expected future movement of the position of the first pupil relative to the first set of reference points based on the movement of the position of the first pupil relative to the first set of reference points over the period of time; as well as determining an expected future movement of the position of the second pupil relative to the second set of reference points based on the movement of the position of the second pupil relative to the second set of reference points over the period of time, Wherein the depth engine is further configured to determine an expected future focus of the user based on the expected future movement of the position of the first pupil and the expected future movement of the position of the second pupil.

8. The system of claim 1 , wherein the image data further indicates a depth of field of the user, and wherein to generate the artificial reality content, the depth engine is configured to: Blurring portions of the artificial reality content that are outside the depth of field of the user.

9. A method for providing a zoom display of artificial reality, comprising: capturing, by a first image capture device, real-world image data representative of a user's physical environment; capturing, by a set of second image capture devices of a head mounted display (HMD) configured to output artificial reality content, image data indicating a focus of a gaze of the user; modifying, by a depth engine of the HMD, a focal length of a zoom display of the HMD based on the focus of the user, wherein modifying the focal length of the zoom display comprises: determining, by the depth engine in near real time, the focal length of the zoom display to match the focus of the user; and controlling a motor by the depth engine to move the zoom display relative to a lens along a longitudinal axis of the HMD to achieve the determined focal length; generating, by the depth engine and based on the real-world image data and depth data associated with the real-world image data, a three-dimensional 3D scene of the physical environment of the user; and Artificial reality content is generated by the depth engine based on the focus of the user as an overlay to the 3D scene of the physical environment for display on the varifocal display of the HMD.

10. The method of claim 9, wherein the first image capture device comprises a pass-through camera, and wherein the method further comprises: The real-world image data is captured in color and at high resolution using the pass-through camera.

11. The method according to claim 10, further comprising: The depth engine is executed using one or more processors of the HMD, wherein the one or more processors are located proximate to the pass-through camera.

12. The method according to claim 9, further comprising: The depth data is generated using a depth sensor, and wherein generating the 3D scene of the physical environment comprises: using the depth data to generate a 3D mesh of a real-world scene; and At least a portion of the real-world image data is overlaid onto the 3D mesh of the real-world scene.

13. The method of claim 12, wherein the real-world image data represents a real-world image data stream, wherein the depth data represents a depth data stream, and wherein generating the artificial reality content comprises: The artificial reality content is generated in near real time using the real-world image data stream and the depth data stream based on the position and orientation of the HMD relative to the 3D mesh of the real-world scene.

14. The method according to claim 9, further comprising: capturing a first set of image data using a first eye tracking camera of the set of second image capture devices, the first set of image data comprising a position of a first pupil of the user relative to a first set of reference points; capturing, using a second eye-tracking camera of the set of second image capture devices, a second set of image data including a position of a second pupil of the user relative to a second set of reference points; and The focus of the user is determined using the depth engine based on a position of the first pupil relative to the first set of reference points and a position of the second pupil relative to the second set of reference points.

15. The method according to claim 14, further comprising: tracking a position of the first pupil relative to the first set of reference points over a period of time using a gaze tracker; tracking, using the gaze tracker, a position of the second pupil relative to the second set of reference points over the period of time; determining an expected future movement of the position of the first pupil relative to the first set of reference points based on the movement of the position of the first pupil relative to the first set of reference points over the period of time; determining an expected future movement of the position of the second pupil relative to the second set of reference points based on the movement of the position of the second pupil relative to the second set of reference points over the period of time; as well as An expected future focus of the user is determined using the depth engine and based on the expected future movement of the position of the first pupil and the expected future movement of the position of the second pupil.

16. The method of claim 9, wherein the image data further indicates a depth of field of the user, and wherein generating the artificial reality content comprises: Blurring portions of the artificial reality content that are outside of the depth of field of the user.

17. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to: capturing real-world image data representing the user's physical environment; capturing image data indicative of a focus of a gaze of the user; modifying a focal length of a zoom display based on the focus of the user, wherein to modify the focal length of the zoom display, the instructions further cause one or more processors to: determining the focal length of the zoom display in near real time to match the focus of the user; as well as controlling a motor to move the zoom display relative to a lens along a longitudinal axis of the head mounted display HMD to achieve the determined focal length; generating a three-dimensional (3D) scene of the physical environment of the user based on the real-world image data and depth data associated with the real-world image data; as well as Artificial reality content is generated based on the focus of the user as an overlay on the 3D scene of the physical environment for display on a varifocal display of the HMD.

Images