Stereoscopic display including liquid crystal based lenses

Abstract

The stereoscopic display may include a two-dimensional display; a variable-focus optical system configured to receive image light from the two-dimensional display and focus the image light; and at least one processor configured to control the two-dimensional display so that the display displays multiple sub-frames associated with an image frame, wherein each of the multiple sub-frames includes a corresponding portion of image data associated with the image frame; and control the variable-focus optical system to a corresponding focus state for each sub-frame.

Description

Technical Field

[0001] The present disclosure relates generally to artificial reality systems, such as mixed reality and / or virtual reality systems. Background Art

[0002] Artificial reality (AR) systems have widespread applications in many fields, such as computer gaming, health and safety, industry, and education. As a few examples, AR systems are currently being integrated into mobile devices, game consoles, personal computers, movie theaters, and theme parks. Generally speaking, an AR is a form of reality that has been modified in some way before being presented to the user. It can include, for example, virtual reality, augmented reality, mixed reality, or some combination and / or derivative thereof.

[0003] A typical artificial reality system includes one or more devices for rendering and displaying content to a user. As an example, an artificial reality system may be combined with a head-mounted display (HMD) worn by a user and configured to output artificial reality content to the user. The artificial reality content may consist entirely of system-generated content, or may include generated content combined with real-world content (e.g., through a view or captured real-world video and / or image of the user's physical environment). During operation, the user typically interacts with the artificial reality system to select content, launch applications, configure the system, and generally experience the artificial reality environment. Summary of the Invention

[0004] In general, the present disclosure describes artificial reality systems, and more specifically, describes artificial reality systems that include stereoscopic displays. As used herein, a stereoscopic display is a display that forms a visual representation of an object or scene in apparent three dimensions rather than a two-dimensional flat image. The stereoscopic displays described herein may include a two-dimensional flat display and a variable focus optical system. One or more processors may be configured to process an image frame to generate a plurality of subframes. Each subframe includes only a portion of image data from the image frame. The portion of the image data of each subframe corresponds to a depth or depth range within the image frame. Taken together, the multiple subframes collectively represent all of the image data in the frame, but each subframe includes only a portion of the image data.

[0005] To generate a volumetric image for display, one or more processors are configured to coordinate the sub-frame output of the display with the focal length of the variable-focus optical system so that the focal length of the variable-focus optical system is correlated with the depth associated with the displayed sub-frame. To display the entire image frame, the one or more processors control the display to output the sub-frames sequentially and control the focal length of the variable-focus optical system as each sub-frame is displayed. This technique exploits persistence of vision, which allows the user's visual system to efficiently combine the partial image data included in the sub-frames to reconstruct the image frame.

[0006] In one or more example aspects, the present disclosure describes a stereoscopic display comprising: a two-dimensional display; a variable-focus optical system configured to receive image light from the two-dimensional display and focus the image light; and at least one processor configured to: control the two-dimensional display so that the display displays a plurality of sub-frames associated with an image frame, wherein each of the plurality of sub-frames includes a corresponding portion of image data associated with the image frame; and control the variable-focus optical system to a corresponding focus state for each corresponding sub-frame.

[0007] In some embodiments, the variable focus optical system includes a plurality of optical stages, and wherein each optical stage of the plurality of optical stages includes a focusing optical element.

[0008] In some embodiments, the focusing optical element comprises a polarization-sensitive focusing optical element, and wherein at least one of the plurality of optical stages further comprises a switchable wave retarder.

[0009] In some embodiments, the switchable wave retarder comprises a switchable half-wave plate.

[0010] In some embodiments, the switchable wave retarder includes at least one ferroelectric liquid crystal cell.

[0011] In some embodiments, the polarization-sensitive focusing optical element includes at least one of the following: a Pancharatnam-Berry Phase (PBP) lens, a polarization-sensitive holographic (PSH) lens, a metamaterial, or a liquid crystal optical phase array.

[0012] In some embodiments, the at least one processor is further configured to generate the plurality of sub-frames by binning pixels of the image frame into a plurality of bins based on depth values associated with the pixels.

[0013] In some embodiments, multiple sub-frames collectively reconstruct an image frame.

[0014] In some examples, the present disclosure describes a system comprising a head-mounted display and at least one processor. The head-mounted display comprises: a housing; a two-dimensional display mechanically coupled to the housing; and a variable-focus optical system mechanically coupled to the housing and configured to receive image light from the two-dimensional display and focus the image light. The at least one processor is configured to: cause the display to display a plurality of sub-frames associated with an image frame, wherein each of the plurality of sub-frames comprises a corresponding portion of image data associated with the image frame, and wherein the plurality of sub-frames collectively comprise all image data associated with the image frame; and control the variable-focus optical system to a corresponding focus state for each respective sub-frame.

[0015] In some embodiments, the variable focus optical system includes a plurality of optical stages, and wherein each optical stage of the plurality of optical stages includes a focusing optical element.

[0016] In some embodiments, the focusing optical element comprises a polarization-sensitive focusing optical element, and wherein at least one of the plurality of optical stages further comprises a switchable wave retarder.

[0017] In some embodiments, the switchable wave retarder comprises a switchable half-wave plate.

[0018] In some embodiments, the switchable wave retarder includes at least one ferroelectric liquid crystal cell.

[0019] In some embodiments, the polarization-sensitive focusing optical element comprises at least one of a Pancharatnam-Berry phase (PBP) lens, a polarization-sensitive holographic (PSH) lens, a metamaterial, or a liquid crystal optical phase array.

[0020] In some embodiments, the at least one processor is further configured to generate the plurality of sub-frames by binning pixels of the image frame into a plurality of bins based on depth values associated with the pixels.

[0021] In some embodiments, the system further includes a console communicatively coupled to the head mounted display, wherein the console is configured to generate the plurality of sub-frames by binning pixels of the image frame into a plurality of bins based on depth values associated with the pixels.

[0022] In some embodiments, multiple sub-frames collectively reconstruct an image frame.

[0023] In some examples, the present disclosure describes a method comprising: one or more processors causing a plurality of subframes associated with an image frame to be displayed by a two-dimensional display, wherein each of the plurality of subframes includes a corresponding portion of image data associated with the image frame, and wherein the plurality of subframes collectively include all of the image data associated with the image frame; and the one or more processors controlling a variable-focus optical system to a corresponding focus state for each corresponding subframe, wherein the variable-focus optical system is configured to receive image light from the two-dimensional display and to focus the image light.

[0024] In some embodiments, the variable focus optical system includes a plurality of optical stages, and wherein each optical stage of the plurality of optical stages includes a focusing optical element.

[0025] In some embodiments, the focusing optical element comprises a polarization-sensitive focusing optical element, wherein at least one of the plurality of optical stages further comprises a switchable wave retarder, and wherein controlling the variable focus optical system comprises controlling a state of the switchable wave retarder.

[0026] In some embodiments, the polarization-sensitive focusing optical element comprises at least one of: a Pancharatnam-Berry phase (PBP) lens, a polarization-sensitive holographic (PSH) lens, a metamaterial, or a liquid crystal optical phase array.

[0027] In some embodiments, the polarization-sensitive focusing optical element comprises an active polarization-sensitive focusing optical element, including controlling a state of the active polarization-sensitive focusing optical element.

[0028] In some embodiments, the method further includes the one or more processors generating the plurality of sub-frames by binning pixels of the image frame into a plurality of bins based on depth values associated with the pixels.

[0029] In some embodiments, the method further includes binning, by the console, pixels of the image frame into a plurality of bins based on depth values associated with the pixels to generate the plurality of sub-frames.

[0030] It will be appreciated that any feature described herein as suitable for incorporation into one or more aspects or embodiments of the present invention is intended to be common to any and all aspects and embodiments of the disclosure.

[0031] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a diagram depicting an exemplary artificial reality system including a stereoscopic display according to techniques described in this disclosure.

[0033] Figure 2A is a diagram depicting an example HMD including a stereoscopic display according to techniques described in this disclosure.

[0034] Figure 2B is a diagram depicting another example HMD that includes a stereoscopic display in accordance with techniques described in this disclosure.

[0035] Figure 3 is a diagram showing the technology described in this disclosure Figure 1 Block diagram of an example implementation of a console and HMD for a multi-device artificial reality system.

[0036] Figure 4 is a diagram depicting a method according to the technology described in this disclosure. Figure 1 Block diagram of an example of an artificial reality system with an HMD performing gesture detection, user interface generation, and virtual surface functions.

[0037] Figure 5 is an isometric view of an example stereoscopic display according to some examples of the present disclosure.

[0038] Figure 6 is a conceptual diagram illustrating an example variable-focus optical system according to some examples of the present disclosure.

[0039] Figure 7 is a conceptual diagram of an example optical stage of a variable-focus optical system according to some examples of the present disclosure, the optical stage including a first optical element and a second optical element in optical series with the first optical element.

[0040] Figures 8A-8D is a schematic diagram illustrating an exemplary Pancharatnam-Berry phase (PBP) lens according to some examples of the present disclosure.

[0041] Figures 9A-9D is a schematic diagram illustrating an example polarization-sensitive holographic (PSH) lens according to some examples of the present disclosure.

[0042] Figures 10A-10D are images of example image frames and multiple sub-frames generated from the image frames according to some examples of the present disclosure.

[0043] Figure 11 is a flow chart illustrating an example technique for generating a volumetric display using a display and a variable focus optical system according to some examples of the present disclosure. DETAILED DESCRIPTION

[0044] Figure 1 is a diagram depicting an exemplary artificial reality system including a stereoscopic display according to the techniques described in this disclosure. Figure 1 In examples of , artificial reality system 100 includes HMD 112, one or more controllers 114A and 114B (collectively referred to as "controllers 114"), and in some examples may include one or more external sensors 90 and / or console 106.

[0045] The HMD 112 is typically worn by the user 110 and includes an electronic display and an optical transmission system for presenting artificial reality content 122 to the user 110. In addition, the HMD 112 includes one or more sensors (e.g., accelerometers) for tracking the movement of the HMD 112 and may include one or more image capture devices 138 (e.g., cameras, line scanners) for capturing image data of the surrounding physical environment. Although illustrated as a head-mounted display, the AR system 100 may alternatively or additionally include glasses or other display devices to present artificial reality content 122 to the user 110.

[0046] Each controller 114 is an input device that the user 110 can use to provide input to the console 106, the HMD 112, or another component of the artificial reality system 100. The controller 114 may include one or more presence-sensitive surfaces for detecting user input by detecting the presence of one or more objects (e.g., a finger, a stylus) touching or hovering over the presence-sensitive surface. In some examples, the controller 114 may include an output display, which may be a presence-sensitive display. In some examples, the controller 114 may be a smartphone, tablet computer, personal data assistant (PDA), or other handheld device. In some examples, the controller 114 may be a smartwatch, smart ring, or other wearable device. The controller 114 may also be part of a kiosk or other fixed or mobile system. Alternatively or additionally, the controller 114 may include other user input mechanisms, such as one or more buttons, triggers, joysticks, a numeric keypad, etc., to enable the user to interact with and / or control each corresponding aspect of the artificial reality content 122 presented to the user 110 by the artificial reality system 100.

[0047] In this example, console 106 is shown as a single computing device, such as a game console, workstation, desktop computer, or laptop computer. In other examples, console 106 can be distributed across multiple computing devices, such as a distributed computing network, a data center, or a cloud computing system. As shown in this example, console 106, HMD 112, and sensor 90 can be communicatively coupled via network 104, which can be a wired or wireless network, such as Wi-Fi, a mesh network, or a short-range wireless communication medium, or a combination thereof. Although HMD 112 is shown in this example as communicating with console 106, such as being connected to console 106 or wirelessly communicating with console 106, in some embodiments, HMD 112 operates as a standalone mobile artificial reality system, and artificial reality system 100 can omit console 106.

[0048] Typically, the artificial reality system 100 renders artificial reality content 122 for display to the user 110 on the HMD 112. Figure 1In the example of FIG1 , user 110 views artificial reality content 122 constructed and rendered by an artificial reality application executing on HMD 112 and / or console 106. In some examples, artificial reality content 122 may be completely artificial, i.e., images that are unrelated to the environment in which user 110 is located. In some examples, artificial reality content 122 may include a mix of real-world images (e.g., user 110's hands, controller 114, other environmental objects near user 110) and virtual objects to produce a mixed reality and / or augmented reality. In some examples, virtual content items may be mapped (e.g., pinned, locked, placed), for example, to specific locations within artificial reality content 122 relative to real-world images. The location of the virtual content item may be fixed, for example, relative to a wall or the ground. For example, the location of the virtual content item may be variable relative to controller 114 or the user. In some examples, the specific location of the virtual content item in artificial reality content 122 is associated with a location in the real-world physical environment (e.g., on a surface of a physical object).

[0049] During operation, the artificial reality application constructs artificial reality content 122 for display to the user 110 by tracking and calculating pose information of a reference frame (typically the viewing angle of the HMD 112). Using the HMD 112 as the reference frame and based on the current field of view determined by the current estimated pose of the HMD 112, the artificial reality application renders 3D artificial reality content, which in some examples may be at least partially overlaid on the real-world 3D physical environment of the user 110. During this process, the artificial reality application uses sensory data (e.g., motion information and user commands) received from the HMD 112, and in some examples, data from any external sensors 90 (e.g., external cameras) to capture 3D information within the real-world physical environment, such as the motion of the user 110 and / or feature tracking information about the user 110. Based on the sensed data, the artificial reality application determines the current pose of the reference frame of the HMD 112 and renders the artificial reality content 122 according to the current pose.

[0050] The artificial reality system 100 can trigger the generation and rendering of virtual content items based on the user's 110 current field of view 130, which can be determined through real-time gaze tracking of the user or other conditions. More specifically, the image capture device 138 of the HMD 112 captures image data representing objects in the real-world physical environment that are within the field of view 130 of the image capture device 138. The field of view 130 generally corresponds to the viewing perspective of the HMD 112. In some examples, the artificial reality application presents artificial reality content 122, including mixed reality and / or augmented reality. The artificial reality application can render images of real-world objects within the field of view 130 (such as portions of the user's 110 peripheral device 136, hand 132, and / or arm 134) alongside virtual objects within the artificial reality content 122. In other examples, the artificial reality application can render virtual representations of portions of the user's 110 peripheral device 136, hand 132, and / or arm 134 within the field of view 130 within the artificial reality content 122 (e.g., presenting real-world objects as virtual objects). In either example, user 110 is able to view portions of their hand 132, arm 134, peripheral device 136, and / or any other real-world objects within field of view 130 within artificial reality content 122. In other examples, the artificial reality application may not render representations of the user's hand 132 or arm 134.

[0051] According to the technology of the present disclosure, HMD 112 includes a stereoscopic display, which includes a display and a variable focus optical system. The display can be a two-dimensional flat display, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, an active matrix OLED (AMOLED) display, a micro light emitting diode (microLED) display, etc. The variable focus optical system can include multiple optical elements, at least some of which include controllable focal lengths. For example, the variable focus optical system can include multiple polarization-sensitive optical elements, such as lenses formed from liquid crystals, such as Pancharatnam-Berry phase (PBP; also known as geometric phase) lenses, polarization-sensitive holographic (PSH) lenses, metamaterials, and / or liquid crystal optical phase arrays. By controlling the polarization of light incident on each corresponding polarization-sensitive optical element and / or the state of the polarization-sensitive optical element, the optical system can be controlled to have a selected total optical power. In this way, the optical system can be a variable focus optical system.

[0052] One or more processors (e.g., HMD 112, console 116, etc.) can be configured to process an image frame of artificial reality content 122 to generate a plurality of sub-frames. Each sub-frame includes a portion of image data from the image frame. The portion of the image data of each sub-frame corresponds to a depth or depth range within the image frame. For example, each rendered pixel can have multiple values associated with it (e.g., color, brightness, position coordinates, etc.), one of which can represent a depth within the image. The one or more processors can group pixels having depth values within a selected range into a frame. The depth range can be selected so that the multiple sub-frames collectively represent substantially all of the image data in the frame, but each sub-frame includes only some of the image data.

[0053] To generate a volumetric image for display, the one or more processors are configured to coordinate the sub-frame output of the display and the focal length of the variable focus optical system (e.g., by controlling the state of the polarization-sensitive optical element and / or other optical elements to control the polarization of light incident on each polarization-sensitive optical element) so that the focal length of the variable focus optical system is related to the depth associated with the displayed sub-frame (e.g., the one or more processors control the focal length of the variable focus optical system to be related to the depth of the sub-frame being displayed, so that when the variable focus optical system is in a state that makes the sub-frame appear farther away from the viewer, a sub-frame with a higher depth value is displayed. To display the entire image frame, the one or more processors control the display to output the sub-frames in sequence and control the focus state of the variable focus optical system when displaying each sub-frame. This technology takes advantage of the persistence of vision, which allows the user's visual system to effectively combine portions of the image data included in the sub-frames to reconstruct the image frame.

[0054] Figure 2A is a diagram depicting an example HMD 112 including a stereoscopic display including a display and a variable focus optical system according to techniques described in this disclosure. Figure 2A The HMD 112 may be Figure 1 HMD 112 can be an example of an artificial reality system (such as Figure 1 ) or can operate as a standalone mobile artificial reality system configured to implement the techniques described herein.

[0055] In this example, HMD 112 includes a front rigid body and a strap that secures HMD 112 to the user. Furthermore, HMD 112 includes an inward-facing electronic display 203 configured to present artificial reality content to the user via a variable-focus optical system 205. As described above, electronic display 203 can be any suitable display technology. In some examples, the electronic display is a stereoscopic display, providing a separate image to each eye of the user. In some examples, the known orientation and position of display 203 relative to the front rigid body of HMD 112 is used as a reference system, also referred to as a local origin, when tracking the position and orientation of HMD 112 to render artificial reality content based on the HMD 112 and the user's current viewing perspective. In other examples, HMD 112 can take the form of other wearable head-mounted displays, such as glasses or goggles.

[0056] The variable focus optical system 205 includes a lens configured to manage the light output by the electronic display 203 for viewing by a user of the HMD 112 (e.g., Figure 1 The optical elements may include, for example, one or more lenses, one or more diffractive optical elements, one or more reflective optical elements, one or more waveguides, etc., that manipulate (e.g., focus, defocus, reflect, refract, diffract, etc.) light output by the electronic display 203. For example, the variable focus optical system 205 may be a lens as described herein. Figure 1 、 6 , 7 and 8A-8D describe any variable focus optical system.

[0057] like Figure 2AAs further shown, in this example, the HMD 112 also includes one or more motion sensors 206, such as one or more accelerometers (also known as inertial measurement units or "IMUs") that output data indicating the current acceleration of the HMD 112, a GPS sensor that outputs data indicating the position of the HMD 112, a radar or sonar that outputs data indicating the distance between the HMD 112 and various objects, or other sensors that provide an indication of the position or orientation of the HMD 112 or other objects in the physical environment. Furthermore, the HMD 112 may include integrated image capture devices 138A and 138B (collectively, "image capture devices 138"), such as cameras, laser scanners, Doppler radar scanners, depth scanners, etc., that are configured to output image data representing the physical environment. More specifically, the image capture devices 138 capture image data representing objects in the physical environment (including peripheral devices 136 and / or hand 132) that are within a field of view 130A, 130B of the image capture devices 138, which generally corresponds to the viewing angle of the HMD 112. The HMD 112 includes an internal control unit 210, which may include an internal power supply; and one or more printed circuit boards having one or more processors, memory, and hardware to provide an operating environment for performing programmable operations to process sensed data and present artificial reality content on the display 203.

[0058] Figure 2B is a diagram depicting another example HMD 112 according to the techniques described in this disclosure. Figure 2B As shown, HMD 112 may take the form of glasses. Figure 2A The HMD 112 may be Figure 1 An example of an HMD 112. The HMD 112 may be an artificial reality system such as Figure 1 ) or can operate as a standalone mobile artificial reality system configured to implement the techniques described herein.

[0059] In this example, the HMD 112 is a pair of glasses that includes a front frame including a bridge that allows the HMD 112 to rest on the user's nose and temples (or "arms") that extend over the user's ears to secure the HMD 112 to the user. Figure 2BThe HMD 112 includes one or more inward-facing electronic displays 203A and 203B (collectively, “electronic displays 203”) configured to present artificial reality content to the user, and one or more variable-focus optical systems 205A and 205B (collectively, “variable-focus optical systems 205”) configured to manage light output by the inward-facing electronic displays 203. In some examples, the known orientation and position of the displays 203 relative to the front frame of the HMD 112 is used as a reference system, also referred to as a local origin, when tracking the position and orientation of the HMD 112 to render artificial reality content based on the current viewing perspective of the HMD 112 and the user.

[0060] like Figure 2B As further shown in FIG, in this example, the HMD 112 also includes one or more motion sensors 206, one or more integrated image capture devices 138A and 138B (collectively referred to as “image capture devices 138”), an internal control unit 210 that may include an internal power supply, and one or more printed circuit boards having one or more processors, memory, and hardware to provide an operating environment for performing programmable operations to process sensed data and present artificial reality content on the display 203.

[0061] Figure 3 is a block diagram illustrating an example implementation of an artificial reality system including a console 106 and an HMD 112 according to the techniques described in this disclosure. Figure 3 In the example of , console 106 performs gesture tracking, gesture detection, and user interface generation and rendering of HMD 112 based on sensed data (eg, motion data and image data received from HMD 112 and / or external sensors).

[0062] In this example, the HMD 112 includes one or more processors 302 and memory 304, which in some examples provide a computer platform for executing an operating system 305, which may be, for example, an embedded real-time multitasking operating system or other type of operating system. In turn, the operating system 305 provides a multitasking operating environment for executing one or more software components 307, including an application engine 340. Figure 2A and 2B As discussed in the example of FIG, processor 302 is coupled to electronic display 203, motion sensor 206, image capture device 138, and, in some examples, to optical system 205. In some examples, processor 302 and memory 304 can be separate discrete components. In other examples, memory 304 can be on-chip memory collocated with processor 302 within a single integrated circuit.

[0063] Typically, console 106 is a computing device that processes images and tracking information received from image capture device 138 to perform gesture detection and user interface and / or virtual content generation for HMD 112. In some examples, console 106 is a single computing device, such as a workstation, desktop computer, laptop computer, or gaming system. In some examples, at least a portion of console 106 (e.g., processor 312 and / or memory 314) can be distributed across a cloud computing system, a data center, or a network, such as the Internet, another public or private communication network, such as broadband, cellular, Wi-Fi, and / or other types of communication networks for transmitting data between computing systems, servers, and computing devices.

[0064] exist Figure 3 In the example of , the console 106 includes one or more processors 312 and memory 314, which in some examples provide a computer platform for executing an operating system 316, which may be, for example, an embedded real-time multitasking operating system or other type of operating system. In turn, the operating system 316 provides a multitasking operating environment for executing one or more software components 317. The processor 312 is coupled to one or more I / O interfaces 315, which provide one or more I / O interfaces for communicating with external devices (e.g., a keyboard, a game controller, a display device, an image capture device, an HMD, peripheral devices, etc.). In addition, the one or more I / O interfaces 315 may include one or more wired or wireless network interface controllers (NICs) for communicating with a network, such as the network 104.

[0065] The software application 317 of the console 106 operates to provide the overall artificial reality application. In this example, the software application 317 includes an application engine 320, a rendering engine 322, a gesture detector 324, a pose tracker 326, and a user interface engine 328.

[0066] Generally, the application engine 320 includes functionality for providing and presenting artificial reality applications, such as teleconferencing applications, gaming applications, navigation applications, educational applications, training or simulation applications, etc. The application engine 320 may include, for example, one or more software packages, software libraries, hardware drivers, and / or application programming interfaces (APIs) for implementing artificial reality applications on the console 106. In response to control by the application engine 320, the rendering engine 322 generates 3D artificial reality content for display to the user by the application engine 340 of the HMD 112.

[0067] The application engine 320 and the rendering engine 322 construct artificial content for display to the user 110 based on the current pose information of the reference frame (typically the viewing angle of the HMD 112) determined by the pose tracker 326. Based on the current viewing angle, the rendering engine 322 constructs 3D, artificial reality content, which in some cases may be at least partially overlaid on the real world 3D environment of the user 110. In this process, the pose tracker 326 combines sensory data (such as motion information and user commands) received from the HMD 112 and, in some examples, from any external sensors 90 ( Figure 1 ) (e.g., an external camera) to capture 3D information in the real-world environment, such as the motion of the user 110 and / or feature tracking information about the user 110. Based on the sensed data, the pose tracker 326 determines the current pose of the reference frame of the HMD 112 and, based on the current pose, constructs artificial reality content for transmission to the HMD 112 via the one or more I / O interfaces 315 for display to the user 110.

[0068] The pose tracker 326 can determine the current pose of the HMD 112 and, based on the current pose, trigger certain functions associated with any rendered virtual content (e.g., placing a virtual content item on a virtual surface, manipulating a virtual content item, generating and rendering one or more virtual markers, generating and rendering a laser pointer). In some examples, the pose tracker 326 detects whether the HMD 112 is in proximity to a physical location corresponding to a virtual surface (e.g., a virtual pinboard) to trigger the rendering of the virtual content.

[0069] The user interface engine 328 is configured to generate a virtual user interface for rendering in an artificial reality environment. The user interface engine 328 generates the virtual user interface to include one or more virtual user interface elements 329, such as a virtual drawing interface, a selectable menu (e.g., a drop-down menu), a virtual button, a directional pad, a keyboard or other user-selectable user interface elements, glyphs, display elements, content, user interface controls, etc.

[0070] The console 106 may output the virtual user interface and other artificial reality content to the HMD 112 via a communication channel for display on the HMD 112 .

[0071] Based on the sensory data from any image capture device 138 or other sensor device, the gesture detector 324 analyzes the tracked motion, configuration, position, and / or orientation of an object (e.g., hand, arm, wrist, fingers, palm, thumb) of the controller 114 and / or user 110 to identify one or more gestures performed by the user 110. More specifically, the gesture detector 324 analyzes objects identified in image data captured by the image capture device 138 and / or sensor 90 of the HMD 112 and the external camera 102 to identify the controller 114 and / or the hand and / or arm of the user 110, and tracks the movement of the controller 114, hand, and / or arm relative to the HMD 112 to identify gestures performed by the user 110. In some examples, the gesture detector 324 may track the movement of the controller 114, hand, fingers, and / or arm, including changes in position and orientation, based on the captured image data, and compare the motion vector of the object to one or more entries in the gesture library 330 to detect a gesture or combination of gestures performed by the user 110. In some examples, gesture detector 324 may receive user input detected by a presence-sensitive surface of controller 114 and process the user input to detect one or more gestures performed by user 110 relative to controller 114 .

[0072] According to the techniques described herein, electronic display 203 and variable focus optical system 205 provide a volumetric display for HMD 112. Electronic display 203 may be any display described herein, and variable focus optical system 205 may be any variable focus optical system described herein.

[0073] Figure 4 is a block diagram depicting an example of the technology described in this disclosure, where HMD 112 is a standalone artificial reality system. In this example, similar to Figure 3 , the HMD 112 includes one or more processors 302 and memory 304. In some examples, the processor 302 and memory 304 provide a computer platform for executing an operating system 305, which may be, for example, an embedded real-time multitasking operating system or other type of operating system. In turn, the operating system 305 provides a multitasking operating environment for executing one or more software components 417. In addition, the processor 302 is coupled to the electronic display 203, the variable focus optical system 205, the motion sensor 206, and the image capture device 138.

[0074] exist Figure 4 In the example of FIG, software components 417 operate to provide an overall artificial reality application. In this example, software components 417 include application engine 440, rendering engine 422, gesture detector 424, posture tracker 426, and user interface engine 428. In various examples, the operation of software components 417 is similar to Figure 3 The corresponding components of the console 106 (e.g., the application engine 320, the rendering engine 322, the posture detector 324, the posture tracker 326, and the user interface engine 328) are used to build a virtual user interface that is overlaid on or as part of the artificial content for display to the user 110.

[0075] Similar to About Figure 3 In the described example, based on sensory data from any of the image capture devices 138 or 102, the controller 114, or other sensor devices, the gesture detector 424 analyzes the tracked motion, configuration, position, and / or orientation of the controller 114 and / or the user's object (e.g., hand, arm, wrist, fingers, palm, thumb) to identify one or more gestures performed by the user 110.

[0076] Figure 5 is an isometric view of an example stereoscopic display 500 according to some examples of the present disclosure. In some examples, stereoscopic display 500 includes a light emitting device array 510 and a variable focus optical system 530. Light emitting device array 510 emits image light toward a viewing user. Light emitting device array 510 can be, for example, an LED array, a micro-LED array, an OLED array, or some combination thereof. Light emitting device array 510 includes light emitting devices 520 that emit light in the visible range.

[0077] In some examples, the stereoscopic display 500 includes an emission intensity array that is configured to selectively attenuate light emitted from the light emitting array 510. In some examples, the emission intensity array is composed of a plurality of liquid crystal cells or pixels, groups of light emitting devices, or some combination thereof. Each liquid crystal cell, or in some examples, groups of liquid crystal cells, is addressable to have a specific attenuation level. For example, at a given time, some liquid crystal cells can be set to no attenuation, while other liquid crystal cells can be set to maximum attenuation. In this way, the emission intensity array can control which portion of the image light emitted from the light emitting device array 510 is passed to the variable focus optical system 530. In some examples, the stereoscopic display 500 uses the emission intensity array to facilitate providing image light to the location of the pupil 550 of the user's eye 540 and minimize the amount of image light provided to other areas of the eye.

[0078] Variable focus optical system 530 receives image light (eg, attenuated light) from the emission intensity array (or directly from emission device array 510) and directs the image light to the location of pupil 550 in such a manner that the perceived image is at the appropriate distance.

[0079] In some examples, stereoscopic display 500 includes one or more broadband sources (eg, one or more white LEDs) coupled to a plurality of color filters in addition to or in place of light emitting device array 510 .

[0080] The variable-focus optical system 530 includes a plurality of polarization-sensitive lenses, such as PBP lenses (also known as geometric phase lenses), PSH lenses, metamaterials, and / or liquid crystal optical phase arrays. By controlling the polarization of light incident on each corresponding lens and / or the state of the lens, the optical system can be controlled to have a selected total optical power. In this way, the optical system can be a variable-focus optical system.

[0081] The stereoscopic display 500 is coupled to one or more processors 560. The processor 560 is configured to control the light emitting device array 510 to display an image and is configured to control the variable focus optical system 530 to set the focal length of the variable focus optical system 530. The processor 560 can represent any processor described herein, including Figure 3 and 4 Processors 302 and 312 are shown. Processor 560 may include dedicated or other control circuitry, a processing core for executing instructions, discrete logic, or other hardware elements.

[0082] The processor 560 may be configured to process an image frame of artificial reality content to generate a plurality of sub-frames for display at the light emitting device array 510. The processor 560 may process the image frame such that each sub-frame includes only a portion of the image data from the image frame. For example, the processor 560 may bin the pixels from the image frame based on depth information associated with the image frame. Each pixel output for display may include an associated depth value that may correspond to the distance of the respective pixel from the virtual camera position. The virtual camera position may be aligned with the HMD 112 ( Figure 1 )'s viewing angle.

[0083] The processor 560 may group pixels having depth values within a selected range to generate subframes. For example, the processor 560 may be configured to generate a predetermined number of subframes and may bin the pixels into a corresponding number of depth ranges. As a specific example, the processor 560 may be configured to generate eight subframes for each image frame. The processor 560 may bin the pixels into eight ranges based on the depth information associated with the pixels. The first bin may include pixels associated with low depth values (e.g., pixels closest to the virtual camera coordinates, associated with depth values between, for example, 0 and D1), the second bin may include pixels associated with depth values within a second range (e.g., between depth value D1 and depth value D2), the third bin may include pixels associated with depth values within a third range (e.g., between depth value D2 and depth value D3), and so on. These ranges may be predetermined, determined based on the total depth range within the image frame, and so on. The depth ranges may be selected such that multiple subframes (e.g., eight subframes) collectively represent all image data in the image frame, but each subframe includes only some image data (e.g., pixels having depth values within the range associated with the subframe).

[0084] To generate a volumetric image for display, processor 560 is configured to coordinate the output of sub-frames for display at light emitting device array 510 and the focal length of variable-focus optical system 530. For example, processor 560 can be configured to control the state of lenses and / or other optical elements within variable-focus optical system 530 so that the focal length of the variable-focus optical system is correlated with the depth associated with the displayed sub-frame. To display the entire image frame, processor 560 is configured to control light emitting device array 510 to output the sub-frames in sequence and control the focus state of variable-focus optical system 530 in a corresponding sequence so that the focal length of the variable-focus optical system is correlated with the depth associated with the displayed sub-frame. This technique exploits persistence of vision, which allows a user's visual system to efficiently combine portions of image data included in the sub-frames to reconstruct the image frame.

[0085] Figure 6 FIG2 is a conceptual diagram illustrating a variable focus optical system 600 according to some examples of the present disclosure. Variable focus optical system 600 is an example of variable focus optical system 205 or 530. Optical systems can generally be used to provide focusing capabilities for display devices (e.g., display device 203). The disclosed examples utilize variable focus optical assembly 600 to enable display devices with adjustable optical power to support volumetric displays.

[0086] like Figure 6As shown, the variable-focus optical system 700 includes a plurality of sequential optical stages 602A, 602B, ... 602N (also referred to herein as "optical stages 602") configured to transmit light (e.g., light 604A-604Q) at various optical powers. With the exception of the first optical stage 602A, each of the sequential optical stages receives incident light output from the previous optical stage. For example, as shown, the second optical stage 602B receives light 604C output from the first optical stage 602A. In some examples, each stage of the sequential optical stages 602 can be configured to be in any one of a plurality of states, including at least a first state and a second state. In the first state, each respective optical stage has a first corresponding optical power for light of a first polarization and a second corresponding optical power, different from the first corresponding optical power, for light of a second polarization orthogonal to the first polarization. In the second state, each respective optical stage has a third optical power for light of the first polarization and a fourth optical power for light of the second polarization. As a result, by configuring one or more consecutive optical stages 602, the total optical power of the variable focus optical system 600 is variable.

[0087] The zoom optical assembly 600 is configured to have a total optical power that can be any of at least two different optical power levels for one or more of the two optical stages. The total optical power can have more different optical power levels by adding more stages, or by including one or more stages (where each stage has an electronic lens with a fast response time, such as an active liquid crystal optical phased array tunable lens with continuously adjustable optical power over a certain range). In some examples, the zoom optical system 600 can also include one or more optical elements 606 before the first optical stage and / or one or more optical elements 608 after the last optical stage 602N.

[0088] Each optical stage 602 can include at least one optical element. For example, an optical stage can include a pair of optical elements. Figure 7 is a conceptual diagram of an example optical stage 602 including a first optical element 712 and a second optical element 714 in optical series with the first optical element 712 .

[0089] The first optical element 712 can be configured to be in a first optical element state or a second optical element state by a controller 716. The controller 716 is Figure 5, an example of a processor 560 is shown. The first optical element 712 can be a switchable optical retarder, such as a switchable half-wave plate. In the first optical element state (e.g., the "off" state), the first optical element 712 can be configured to convert light of the first polarization or the second polarization into light of the second polarization or the first polarization, respectively. The first polarization can be substantially orthogonal to the second polarization (e.g., orthogonal or nearly orthogonal). In the second optical element state, the first optical element 712 transmits incident light without changing the polarization of the light. For example, when the controller 716 sets the first optical element 712 to the first state (e.g., by not applying a voltage across the first optical element 712), left circularly polarized (LCP) light incident on the first optical element 712 will be output as right circularly polarized (RCP) light, and vice versa. Conversely, when the controller 716 sets the first optical element 712 to the second state (e.g., by applying a voltage across the first optical element 712), light incident on the first optical element 712 will be transmitted without changing its polarization (e.g., LCP light remains LCP, and RCP light remains RCP).

[0090] The first optical element 712 may include a liquid crystal (LC) cell, such as a nematic LC cell, a nematic LC cell with a chiral dopant, a chiral LC cell, a uniformly helical (ULH) LC cell, a π-type LC cell, a ferroelectric LC cell, and the like. In other examples, the LC cell includes an electrically drivable birefringent material. In some examples, the LC cell can be switched at a relatively high speed (e.g., less than 5 milliseconds, less than 2 milliseconds, less than 1 millisecond, etc.). In some examples, the first optical element 712 includes at least one ferroelectric LC cell. The ferroelectric LC cell is based on a smectic C* LC (chiral smectic LC) that exhibits a bi-stable configuration. The two stable configurations can be switched using an applied voltage. The ferroelectric LC cell can exhibit fast switching times, such as less than 1 microsecond or less than 100 microseconds. The fast switching time can support relatively high frame rates for volumetric displays, particularly in examples where the image frame is divided into many subframes. For example, to support a volumetric display with a 100 Hz frame rate and 10 subframes per frame, a switching rate of less than 1 microsecond may be required to reduce or substantially eliminate optical artifacts caused by the switching incoordination of the variable focus optical system 600 and the subframe display of the light emitting device array 510.

[0091] The second optical element 714 is configured to receive light transmitted through the first optical element 712. The second optical element 714 can be a focusing optical element (e.g., a lens). In some examples, the second optical element 714 is a polarization-sensitive optical element. For example, the second optical element 714 can include one or more of a PBP lens (also known as a geometric phase lens), a PSH lens, a metamaterial or metasurface, and a liquid crystal optical phase array. Figures 9A-9D and Figures 10A-10D Provides details about PBP lenses and PSH lenses.

[0092] The second optical element 714 can be passive (e.g., not connected to a controller 718 configured to selectively apply a voltage to the second optical element 714 to change a property of the second optical element 714), or active (e.g., connected to a controller 718 configured to selectively apply a voltage to the second optical element 714 to change a property of the second optical element 714). In examples where the second optical element 714 is passive, the second optical element 714 has a first optical focal length for light of the first polarization and a second optical focal length for light of the second polarization that is different from the first optical focal length. In some examples, the second corresponding optical focal length is less than the first corresponding optical focal length. For example, the second corresponding optical focal length can be zero. For example, the second optical element 714 can exhibit a non-zero first optical focal length for RCP light and be configured to convert the RCP light into LCP light while converging or diverging (depending on the first optical focal length) the RCP light. The second optical element 714 can be configured to transmit the LCP light without focusing or changing the polarization of the LCP light.

[0093] In other examples, the second corresponding optical power is approximately equal in magnitude to the first corresponding optical power, but opposite in sign (effect) to the first corresponding optical power. For example, second optical element 714 can function as a positive lens having a power of +0.5 diopters for incident light RCP, and can function as a negative lens having a power of -0.5 diopters for incident light LCP. Thus, the optical power of second optical element 714 (and therefore the optical power of optical stage 602) can be based on the state of first optical element 712 and the polarization of light incident to optical stage 602.

[0094] In some examples, the second optical element 714 is an active optical element that can be configured to be in a third optical element state (e.g., an "off" state) or a fourth optical element state (e.g., an "on" state) via a controller 718. The controller 718 can be Figure 5 . An example of processor 560 is shown. In the third optical element state, active second optical element 714 is configured to have a first corresponding optical power for incident light having a first polarization and a second corresponding optical power for incident light having a second polarization, as described above with respect to the example in which second optical element 714 is passive. In the fourth optical element state, active second optical element 714 is configured to have zero optical power and is configured to transmit incident light without applying optical power regardless of the polarization of the incident light. As a result, depending on the states of first optical element 712 and active second optical element 714, optical stage 602, including first optical element 712 and active second optical element 714, can assume two or more different states.

[0095] In some examples, the second optical element 714 is a thin film on the surface of the first optical element 712 .

[0096] Second optical element 714 has an associated optical power (or multiple associated optical powers), which can be the same as or different from the optical powers of second optical elements in other optical stages 602. In some examples, the optical power of second optical element 714 has a magnitude no greater than 2.0 diopters (e.g., the optical power is no greater than -2 diopters or +2 diopters).

[0097] In some examples, continuous optical stage 602 ( Figure 6 ) includes only one of the first optical element 712 and the active second optical element 714. For example, an optical stage of the continuous optical stage 702 may include the active second optical element 714 but not the first optical element 712.

[0098] Therefore, controllers 716 and 718 (which are Figure 5 An example of a processor 560) can control the zoom optical assembly 500 ( Figure 5 ) of the total optical power, the total optical power can be adjusted by controlling each corresponding state of the optical stage 602, as shown in FIG. Figure 7 As stated.

[0099] Figures 8A-8D is a schematic diagram illustrating a Pancharatnam-Berry phase (PBP) lens 800 according to some examples. In some embodiments, the above reference Figure 6 and 7 Second optical element 714 of optical stage 602 in zoom optical assembly 600 is depicted as including a PBP lens 800. In some examples, PBP lens 800 is a liquid crystal optical element including a liquid crystal layer. In some examples, PBP lens 800 includes other types of substructure layers, such as nanorods composed of a high refractive index material.

[0100] The PBP lens 800 increases or decreases the optical power based in part on the polarization of the incident light. For example, if RCP light is incident on the PBP lens 800, the PBP lens 800 acts as a positive lens (i.e., it converges the light). If LCP light is incident on the PBP lens 800, the PBP lens 800 acts as a negative lens (i.e., it diverges the light). The PBP lens 800 also changes the handedness of the light to an orthogonal handedness (e.g., changes LCP to RCP, and vice versa). The PBP lens also has wavelength selectivity. If the incident light is of the designed wavelength, the LCP light is converted to RCP light, and vice versa. Conversely, if the wavelength of the incident light is outside the designed wavelength range, at least a portion of the light is transmitted without changing its polarization and without focusing or converging. The PBP lens can have a large aperture size and can be made of a very thin liquid crystal layer. The optical properties of the PBP lens (e.g., focusing power or diffraction power) are based on changes in the azimuth angle (θ) of the liquid crystal molecules. For example, for a PBP lens, the azimuth angle θ of the liquid crystal molecules is determined based on equation (1):

[0101]

[0102] Where r represents the radial distance between the liquid crystal molecules and the optical center of the PBP lens, f represents the focal length, and λ represents the wavelength of light for which the PBP lens is designed. In some examples, the azimuth angle of the liquid crystal molecules in the xy plane increases from the optical center to the edge of the PBP lens. In some examples, as shown in equation (1), the rate of increase of the azimuth angle between adjacent liquid crystal molecules also increases with the distance from the optical center of the PBP lens 800. The PBP lens 800 is based on the liquid crystal molecules in the Figure 8A In contrast, (non-PBP) liquid crystal lenses form their lens profiles through the birefringence properties (the liquid crystal molecules are oriented out of the xy plane, e.g., at a non-zero tilt angle to the xy plane) and the thickness of the liquid crystal layer.

[0103] Figure 8A A three-dimensional view of a PBP lens 800 is shown, with incident light 804 entering the lens along the z-axis.

[0104] Figure 8B The xy plane view of a PBP lens 800 having multiple differently oriented liquid crystals (eg, liquid crystals 802A and 802B) is shown. The orientation of the liquid crystals (ie, azimuth angle θ) varies from the center of the PBP lens 800 to the periphery of the PBP lens 800 along a reference line between A and A'.

[0105] Figure 8C FIG shows an xz cross-sectional view of the PBP lens 800. Figure 8C As shown, the orientation of the liquid crystals (eg, liquid crystals 802A and 802B) remains unchanged along the z-direction. Figure 8C An example of a PBP structure is shown, where the orientation along the z-axis is constant and the birefringence thickness (Δn×t) is ideally half the design wavelength, where Δn is the birefringence of the liquid crystal material and t is the physical thickness of the plate.

[0106] In some examples, a PBP optical element (e.g., a lens) may have a different Figure 8C For example, a PBP optical element may include a double-twisted liquid crystal structure along the z-direction. In another example, a PBP optical element may include a three-layer alternating structure along the z-direction to provide an achromatic response over a wide spectral range.

[0107] Figure 8D The diagram shows the Figure 8B Detailed plan view of the liquid crystal with reference lines between A and A' shown. Pitch 806 is defined as the distance the azimuth angle θ of the liquid crystal is rotated 180 degrees along the x-axis. In some examples, pitch 806 varies as a function of distance from the center of the PBP lens 800. In the case of the lens, the azimuth angle θ of the liquid crystal varies according to equation (1) shown above. In this case, the pitch is longest at the center of the lens and shortest at the edge of the lens.

[0108] Figures 9A-9D is a schematic diagram illustrating a polarization sensitive holographic (PSH) lens according to some examples. In some examples, the above reference Figure 6 and 7 The second optical element 714 of the optical stage 602 in the depicted variable focus optical system 600 includes a PSH lens 900. The PSH lens 900 is a liquid crystal PSH lens that includes a liquid crystal layer (e.g., formed of cholesteric liquid crystals) arranged in a helical structure. Figures 8A-8D Similar to the optical power lens described above, the PSH lens 900 increases or decreases the optical power based in part on the polarization of the incident light. However, the PSH lens 900 is selective for the circular polarization of light. When the state (handedness) of the circularly polarized light is along the helical axis of the liquid crystal, the PSH lens 900 interacts with the circularly polarized light, thereby changing the direction of the light (e.g., refracting or diffracting the light). At the same time, while transmitting the light, the PSH lens 900 also changes the polarization of the light. Conversely, the PSH lens 900 transmits light with opposite circular polarization without changing its direction or polarization. For example, the PSH lens 900 can change the polarization of RCP light to LCP light, and focus or defocus the light while transmitting the LCP light without changing its polarization or direction. The optical properties of the PSH lens 900 (e.g., focusing power or diffraction power) are based on changes in the azimuth angle of the liquid crystal molecules. In addition, the optical properties of the PSH are based on the helical axis and / or helical pitch of the liquid crystal.

[0109] Figure 9AA three-dimensional view of the PSH lens 800 is shown, with incident light 904 entering the lens along the z-axis. Figure 9B The xy plane of a PSH lens 900 having multiple differently oriented liquid crystals (eg, liquid crystals 902A and 902B) is shown. The orientation of the liquid crystals (ie, azimuthal angle θ) varies from the center of the PSH lens 900 to the periphery of the PSH lens 900 along a reference line between B and B'.

[0110] Figure 9C The xz cross-section of the PSH lens 900 is shown. Figure 9C As shown, with regard to Figure 8C In contrast to the PBP lens 800 described above, the liquid crystal of the PSH lens 900 (e.g., Figure 9B The liquid crystals 902A and 902B in the PSH lens 900 are arranged into a spiral structure 918. The spiral structure 918 has a spiral axis aligned parallel to the z-axis. When the azimuth angles of the individual liquid crystals in the xy plane vary, the spiral structure forms a volume grating having multiple diffraction planes (e.g., planes 920A and 920B) that form a cycloidal pattern. The diffraction planes (e.g., Bragg diffraction planes) defined in the volume of the PSH lens 900 are the result of periodic changes in the refractive index. The spiral structure 918 defines the polarization selectivity of the PSH lens 900 because light with a circular polarization handedness corresponding to the spiral axis is diffracted, while light with an opposite circular polarization handedness is not diffracted. The spiral structure 918 also defines the wavelength selectivity of the PSH lens 900 because the spiral pitch 922 determines which wavelengths are diffracted by the PSH lens 900 (light with other wavelengths is not diffracted). For example, for a PSH lens, the design wavelength of light that the PSH lens will diffract is determined based on equation (2):

[0111] λ=2n eff P z (2)

[0112] Where λ represents the wavelength of light for which the PSH lens 900 is designed, P z is the distance of the spiral pitch 922, n eff is the effective refractive index of the liquid crystal medium as a birefringent medium. The helical pitch is the distance between the helix and the helix axis (e.g. Figure 9C The effective refractive index of the birefringent liquid crystal medium is determined based on equation (3):

[0113]

[0114] where n0 is the ordinary refractive index of the birefringent medium, n e is the extraordinary refractive index of the birefringent medium.

[0115] Figure 9D The diagram shows the Figure 9BDetailed plan view of the liquid crystal with reference lines between B and B' in FIG. Spacing 906 is defined as the distance along the x-axis where the azimuth angle of the liquid crystal is rotated 180 degrees from its initial orientation. In some embodiments, spacing 906 varies as a function of distance from the center of PSH lens 900. In the case of the lens, the azimuth angle of the liquid crystal varies according to equation (1) shown above. In this case, the spacing is longest at the center of the lens and shortest at the edge of the lens.

[0116] Figures 10A-10D 10 is an image of an example image frame 1002 and a plurality of sub-frames generated from the image frame. Image frame 1002 includes a plurality of objects, including a first character 1004, a box 1006, a second character 1008, a third character 1010, a well 1012, and a sun 1014. Figures 10A-10D The objects shown in the are examples only and refer to Figures 10A-10D The described concepts can be applied to any object in any image frame. Figures 10A-10D Each object shown in is a combination of multiple pixels.

[0117] As described above, the processor 560 may be configured to bin pixels based on depth information associated with the pixels. Figures 10A-10D In the example of FIG5 , the processor 560 is configured to bin the pixels into three bins. In other examples, the processor 560 can be configured to bin the pixels into any number of bins (e.g., at least two bins). The bins can be predefined and static (e.g., each bin is associated with a predefined static range of depth values), or can be dynamic (e.g., the range of each bin is determined on a frame-by-frame or other basis, e.g., depending on scene complexity as a function of depth).

[0118] like Figure 10B As shown, the processor 560 may be configured to group pixels whose associated depth values indicate a relative proximity to the virtual camera position into a first bin. As described above, the virtual camera position may be associated with the HMD 112 ( Figure 1 ). These pixels may represent objects 1004 and 1006 that are relatively close to the virtual camera position and the viewpoint of the user of the HMD 112. The first subframe 1022A may include only pixels associated with a first bin and may not display pixels associated with depth values that fall outside the first bin. For example, the first bin may include pixels associated with depth values between 0 and the first depth value D1.

[0119] like Figure 10CAs shown, processor 560 may be configured to group pixels whose associated depth values indicate a relatively moderate distance from the virtual camera position into a second bin. These pixels may represent objects 1008 and 1010. Second subframe 1022B may include only pixels associated with the second bin and may not display pixels associated with depth values that fall outside the second bin. For example, the second bin may include pixels associated with depth values between the first depth value D1 and the second depth value D2.

[0120] like Figure 10D As shown, processor 560 may be configured to group pixels associated with depth values indicating relatively moderate distances from the virtual camera position into a second bin. These pixels may represent objects 1012 and 1014. Third subframe 1022C may include only pixels associated with the third bin and may not display pixels associated with depth values that fall outside the third bin. For example, the third bin may include pixels associated with depth values greater than second depth value D2.

[0121] exist Figures 10A-10D In the example shown, the pixels representing each object are grouped within a single bin. For example, all pixels representing the first character 1004 are grouped within a first bin and used to generate the first sub-frame 1022A. However, this may not be the case in all cases, depending, for example, on the number of bins (and sub-frames) and / or the size and position of the objects within the image frame. For example, if the objects include a large building extending from near the virtual camera position into the background, the pixels representing the large building may be grouped into multiple bins.

[0122] In this way, processor 560 can generate multiple sub-frames from an image frame. Processor 560 can perform a similar binning technique to generate a sub-frame for each image frame in the image sequence that constitutes the virtual reality viewing experience. In other examples, the sub-frames can be pre-generated (e.g., generated by another device), and processor 560 can simply output the pre-generated sub-frames for display at light emitting device array 510.

[0123] Figure 11 is a flow chart illustrating an example technique for generating a volumetric display using a display and a variable focus optical system. Figure 5-7 The stereoscopic display 500, the variable focus optical system 600 and the optical stage 602 are described Figure 11 technology, however, it is clear that other systems can be used to perform Figure 11 technology.

[0124] In some examples, the processor 560 may generate a plurality of sub-frames from an image frame (1102), as described above with reference to Figures 10A-10DIn other examples, the subframes may be pre-generated (eg, by another device, such as console 106).

[0125] Processor 560 causes the sub-frame to be displayed at light emitting device array 510 (1104). Processor 560 also controls variable-focus optical system 600 to a corresponding focus state (1106). For example, processor 560 can output a control signal to each optical stage 602 (e.g., each first optical element 712 and each second optical element 714) to set each optical stage 602 to a state that causes variable-focus optical system 600 to produce a selected optical power corresponding to the sub-frame displayed by light emitting device array 510.

[0126] In some examples, the variable-focus optical system 600 can have a defined number of focal powers, and each focal power can be associated with a corresponding subframe within an image frame. For example, each first subframe can include pixels representing objects closest to the virtual camera position within the image frame, each second subframe can include pixels representing objects intermediate in distance from the virtual camera position within the image frame, and each third subframe can include pixels representing objects farther from the virtual camera position within the image frame. For each image frame, the subframes can be output in this defined order. In this way, the processor 560 can output a control signal that causes the variable-focus optical system 600 to cycle through the three focal powers in a manner coordinated with the display of the three subframes.

[0127] In some examples, the processor 560 coordinates the focus state of the variable-focus optical system 600 and the display of the substrate by controlling the variable-focus optical system 600 to achieve a selected optical state (power) associated with a sub-frame before the image light associated with the sub-frame propagates through the variable-focus optical system 600. This can reduce or substantially eliminate optical artifacts that may be caused by changes in focus as the image light propagates through the variable-focus optical system 600. For example, during the period from when the light-emitting device array 510 ends outputting image light associated with a first sub-frame to when the light-emitting device array 510 begins outputting image light associated with a second sub-frame, the processor 560 can control the variable-focus optical system 600 to change from the focus state associated with the first sub-frame to the focus state associated with the second sub-frame.

[0128] For each subframe of the image frame, processor 560 may cause the subframe to be displayed at light emitting device array 510 (1104) and control zoom optical system 600 to a corresponding focus state (1106). Once all subframes have been displayed ("no" branch of decision block 1108), processor 560 may continue processing subsequent image frames (1110). In this manner, by dividing the image frame into subframes (each subframe comprising only some of the pixels of the image frame) and coordinating the focus state of the zoom optical assembly, a volumetric display may be generated. This display may create the appearance of three-dimensional depth to a user using a two-dimensional display.

[0129] As described herein through various examples, the technology of the present disclosure may include or be implemented in conjunction with an artificial reality system. As described above, artificial reality is a form of reality that has been adjusted in some way before being presented to the user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured content (e.g., real-world photos or videos). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of these contents may be presented in a single channel or multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the audience). In addition, in some embodiments, artificial reality may be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in artificial reality and / or to use in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on various platforms, including a head-mounted device (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0130] The techniques described in this disclosure may be implemented, at least in part, in hardware, software, firmware, or any combination thereof. For example, various aspects of the described techniques may be implemented in one or more processors, including one or more microprocessors, DSPs, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or any other equivalent integrated or discrete logic circuits, as well as any combination of these components. The term "processor" or "processing circuitry" may generally refer to any of the foregoing logic circuits, alone or in combination with other logic circuits, or any other equivalent circuits. A control unit comprising hardware may also perform one or more of the techniques of this disclosure.

[0131] Such hardware, software, and firmware may be implemented within the same device or within separate devices to support the various operations and functions described in this disclosure. In addition, any of the described units, modules, or components may be implemented together or separately as discrete but interoperable logic devices. Describing different features as modules or units is intended to highlight different functional aspects and does not necessarily mean that these modules or units must be implemented by separate hardware or software components. Instead, the functions associated with one or more modules or units may be performed by separate hardware or software components, or integrated in common or separate hardware or software components.

[0132] The techniques described in this disclosure may also be implemented or encoded in a computer-readable medium (e.g., a computer-readable storage medium) containing instructions. The instructions embedded or encoded in the computer-readable storage medium, when executed by a programmable processor or other processor, cause the programmable processor or other processor to perform the method. The computer-readable storage medium may include a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electronically erasable programmable read-only memory (EEPROM), a flash memory, a hard disk, a CD-ROM, a floppy disk, a box, a magnetic medium, an optical medium, or other computer-readable medium.

[0133] As described herein through various examples, the technology of the present disclosure may include or be implemented in conjunction with an artificial reality system. As described above, artificial reality is a form of reality that has been adjusted in some way before being presented to the user, which may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), or some combination and / or derivative thereof. Artificial reality content may include fully generated content or generated content combined with captured content (e.g., photos of the real world). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, and any of these contents may be presented in a single channel or multiple channels (e.g., stereoscopic video that produces a three-dimensional effect to the audience). In addition, in some embodiments, artificial reality may be associated with applications, products, accessories, services, or some combination thereof, which are used, for example, to create content in artificial reality and / or to use in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on various platforms, including a head-mounted device (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

Claims

1. A stereoscopic display comprising: 2D displays; a variable-focus optical system configured to receive image light from the two-dimensional display and focus the image light; as well as At least one processor configured to: controlling the two-dimensional display so that a plurality of sub-frames associated with an image frame are sequentially displayed by the display, wherein each sub-frame of the plurality of sub-frames includes a corresponding portion of image data associated with the image frame; controlling the variable focus optical system to a corresponding focus state for each corresponding subframe; as well as coordinating a focus state of the variable focus optical system and the two-dimensional display by controlling the variable focus optical system to achieve a selected optical state associated with a sub-frame before image light associated with the sub-frame propagates through the variable focus optical system, The at least one processor is further configured to generate the plurality of sub-frames by binning pixels of the image frame into a plurality of bins based on depth values associated with the pixels. 2 . The stereoscopic display of claim 1 , wherein the variable focus optical system comprises a plurality of optical stages, and wherein each of the plurality of optical stages comprises a focusing optical element.

3. The stereoscopic display of claim 2, wherein the focusing optical element comprises a polarization-sensitive focusing optical element, and wherein at least one of the plurality of optical stages further comprises a switchable wave retarder.

4. A stereoscopic display according to claim 3, wherein the switchable wave retarder comprises a switchable half-wave plate.

5. The stereoscopic display of claim 3, wherein the switchable wave retarder comprises at least one ferroelectric liquid crystal cell.

6. The stereoscopic display of claim 3, wherein the polarization-sensitive focusing optical element comprises at least one of the following: a Pancharatnamberry phase lens, a polarization-sensitive holographic lens, a metamaterial, or a liquid crystal optical phase array. 7 . The stereoscopic display according to claim 1 , wherein the plurality of sub-frames jointly reconstruct the image frame.

8. A display system comprising: Head-mounted display, including: shell; a two-dimensional display mechanically coupled to the housing; a variable focus optical system mechanically coupled to the housing and configured to receive image light from the two-dimensional display and focus the image light; and At least one processor configured to: causing a plurality of sub-frames associated with an image frame to be sequentially displayed by the display, wherein each of the plurality of sub-frames includes a respective portion of image data associated with the image frame, and wherein the plurality of sub-frames in the aggregate include all of the image data associated with the image frame; controlling the variable focus optical system to a corresponding focus state for each corresponding subframe; and coordinating a focus state of the variable focus optical system and the two-dimensional display by controlling the variable focus optical system to achieve a selected optical state associated with a sub-frame before image light associated with the sub-frame propagates through the variable focus optical system, The at least one processor or a console communicatively coupled to the head mounted display is further configured to generate the plurality of sub-frames by binning the pixels of the image frame into a plurality of bins based on depth values associated with the pixels.

9. The display system of claim 8, wherein the variable focus optical system comprises a plurality of optical stages, and wherein each of the plurality of optical stages comprises a focusing optical element.

10. The display system of claim 9, wherein the focusing optical element comprises a polarization-sensitive focusing optical element, and wherein at least one of the plurality of optical stages further comprises a switchable wave retarder.

11. The display system of claim 10, wherein the switchable wave retarder comprises a switchable half-wave plate.

12. The display system of claim 10, wherein the switchable wave retarder comprises at least one ferroelectric liquid crystal cell.

13. The display system of claim 10, wherein the polarization-sensitive focusing optical element comprises at least one of a Pancharatnamberry phase lens, a polarization-sensitive holographic lens, a metamaterial, or a liquid crystal optical phase array.

14. The display system according to any one of claims 8 to 13, wherein the plurality of sub-frames together reconstruct an image frame.

15. A display method, comprising: causing, by one or more processors, a plurality of sub-frames associated with an image frame to be sequentially displayed by a two-dimensional display, wherein each of the plurality of sub-frames includes a respective portion of image data associated with the image frame, and wherein the plurality of sub-frames in the aggregate include all of the image data associated with the image frame; as well as controlling, by the one or more processors, a variable-focus optical system to a corresponding focus state for each respective subframe, wherein the variable-focus optical system is configured to receive image light from the two-dimensional display and focus the image light; as well as coordinating, by the one or more processors, a focus state of the variable focus optical system and the two-dimensional display by controlling the variable focus optical system to achieve a selected optical state associated with a sub-frame before image light associated with the sub-frame propagates through the variable focus optical system, The method further includes generating, by the one or more processors or consoles, the plurality of sub-frames by binning pixels of the image frame into a plurality of bins based on depth values associated with the pixels.

16. The display method of claim 15, wherein the variable focus optical system comprises a plurality of optical stages, and wherein each of the plurality of optical stages comprises a focusing optical element.

17. The display method of claim 16, wherein the focusing optical element comprises a polarization-sensitive focusing optical element, wherein at least one of the plurality of optical stages further comprises a switchable wave retarder, and wherein controlling the variable focus optical system comprises controlling a state of the switchable wave retarder.

18. The display method according to claim 17, wherein the polarization-sensitive focusing optical element comprises at least one of the following: a Pancharatnamberry phase lens, a polarization-sensitive holographic lens, a metamaterial, or a liquid crystal optical phase array.

19. The display method according to claim 18, wherein the polarization-sensitive focusing optical element comprises an active polarization-sensitive focusing optical element, comprising controlling a state of the active polarization-sensitive focusing optical element.

Images