Selective up-sampling of vision sensor data for efficient display

Abstract

The present disclosure provides for selective upsampling of visual sensor data for efficient display. The efficiency manager may selectively upsample portions of the captured data, e.g., to obtain higher quality visual content for certain portions of the user's field of view (e.g., unvarnished display). For example, an efficiency manager may select portions of captured visual data for up-sampling based on user eye movement tracking, and omit up-sampling of other portions. An efficiency manager may implement selective upsampling using a hierarchical texture map. The texture map may store visual data at different quality levels. The efficiency manager may sparsely fill high quality levels of the texture map with the selectively upsampled visual data. Rendering techniques (e.g., graphics processor unit hardware and / or rendering pipeline) may utilize the texture map to efficiently render visual data for display.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 578,054, filed on August 22, 2023, and priority to U.S. Non-Provisional Patent Application No. 18 / 802,361, filed on August 13, 2024, the contents of both applications are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure relates to selective upsampling of vision sensor data for efficient display. Background Art

[0004] Hyper-reality systems are becoming increasingly common. The visual displays generated by these systems involve a high degree of visual data processing, such as when the system displays an immersive environment. Due to these visual data processing requirements, system resource efficiency (e.g., processor utilization, power usage, storage utilization, etc.) plays an important role in the design, production, and functionality of these systems. Artificial reality systems that can generate high-quality visual displays while maintaining effective performance may be particularly beneficial to the user experience. Summary of the invention

[0005] Various aspects of the present disclosure relate to selectively upsampling visual sensor data to achieve efficient display. A hyper-reality system may generate a display using visual data captured by the system and / or captured by a connected image capture device. An example of such a technique is a transparent visualization, in which a hyper-reality system generates a display of a user's real-world environment (or an enhanced version of the real-world environment). A hyper-reality system may benefit from selectively upsampling the captured visual data when generating a transparent visualization. For example, selective upsampling can reduce the hardware requirements of one or more image acquisition devices of the system and can reduce the processing load. Thus, a higher resolution transparent display can be generated using visual data captured at a lower resolution. Various embodiments selectively upsample portions of the captured visual data to improve the quality of display to the user while maintaining efficient processing and / or resource utilization levels. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a block diagram showing an overview of a device on which some embodiments of the present technology may operate.

[0007] Figure 2A is a line diagram illustrating a virtual reality head mounted viewer that may be used in some embodiments of the present technology.

[0008] Figure 2Bis a line diagram illustrating a mixed reality head mounted viewer that may be used in some embodiments of the present technology.

[0009] Figure 2C is a line diagram illustrating a plurality of controllers that, in some implementations, a user may hold with one or both hands to interact with a hyper-reality environment.

[0010] Figure 3 is a block diagram illustrating an overview of an environment in which some embodiments of the present technology may operate.

[0011] Figure 4 is a block diagram illustrating various components that may be used in a system employing the disclosed technology in some implementations.

[0012] Figure 5 is a conceptual diagram showing a pass-through visual display of a hyper-reality system.

[0013] Fig. 6A is a conceptual diagram illustrating example quality level patterns for a foveal display of an artificial reality system.

[0014] Figure 6B is a conceptual diagram illustrating a user's field of view with different levels of display quality.

[0015] Figure 7 is a flow chart illustrating a process used in some embodiments of the present technology for selectively upsampling vision sensor data for efficient display.

[0016] The technology described herein may be better understood by referring to the following detailed description in conjunction with the accompanying drawings, in which like reference numbers refer to the same elements or elements with similar functions. DETAILED DESCRIPTION

[0017] Various aspects of the present disclosure relate to selectively upsampling visual sensor data to achieve efficient display. A hyper-reality system may generate a display using visual data captured by the system and / or captured by a connected image capture device. An example of such a technique is a transparent visualization, in which a hyper-reality system generates a display of a user's real-world environment (or an enhanced version of the real-world environment). A hyper-reality system may benefit from selectively upsampling the captured visual data when generating a transparent visualization. For example, selective upsampling can reduce the hardware requirements of one or more image acquisition devices of the system and can reduce the processing load. Thus, a higher resolution transparent display can be generated using visual data captured at a lower resolution. Various embodiments selectively upsample portions of the captured visual data to improve the quality of display to the user while maintaining efficient processing and / or resource utilization levels.

[0018] The efficiency manager may selectively upsample portions of the captured visual data, for example to obtain higher quality visual content for certain portions of a user's field of view (e.g., a transparent display). Example techniques involve selecting visual data via a foveated display. In a foveated display, different portions of a user's field of view are selected for different quality levels based on a user's focus. For example, in response to user eye tracking, a central portion of a user's field of view may be targeted for high quality visual content, while other portions may be targeted for medium and / or low quality levels. The efficiency manager may select portions of the captured visual data for upsampling based on user eye tracking, and may omit upsampling of other portions of the captured visual data.

[0019] The efficiency manager may use a hierarchical texture map (mipmap) to implement selective upsampling. A texture map may store visual data of different quality levels. A rendering technology (e.g., graphics processor unit hardware and / or a rendering pipeline) may utilize the texture map to efficiently render visual data for display. The efficiency manager may sparsely fill the high quality level of the texture map with the selectively upsampled visual data. In other words, portions of the captured visual data / user field of view are selected for upsampling, and the high quality level of the texture map may be filled for these selected portions. Other levels of the hierarchical texture map (e.g., mid-level quality, low-level quality, etc.) may be filled with: A) captured visual data; and / or B) a downsampled version of the captured visual data. In some embodiments, the efficiency manager may not fill the high quality level of the texture map for portions of the captured visual data / user field of view that are not selected for upsampling, thereby resulting in a sparsely filled texture map.

[0020] The efficiency manager can then render a see-through visual display to a user of the artificial reality system. Portions of the see-through visual display can be selectively upsampled while other portions are not upsampled. For example, the rendered see-through display can be a foveated display. The efficiency manager can access a sparsely populated texture map to render the see-through display. The high quality level of selectively upsampling captured visual data assigned to a high quality level and sparsely populating a texture map with that data can support, for example, efficient rendering of a foveated display using artificial reality system hardware.

[0021] The technology described herein is generally described as being used with an artificial reality system. However, it is contemplated that the technology may also be used with other sensors and / or display systems, such as mobile devices, monitors, projectors, automotive displays, televisions, cameras, depth sensors, infrared (IR) sensors, sonar systems, laser radar (LIDAR) systems, audio sensors, etc.

[0022] Embodiments of the disclosed technology may include an artificial reality system, or may be implemented in conjunction with an artificial reality system. Artificial reality or hyperreality (XR) is a form of reality that has been adjusted in some way before being presented to a user, and may include, for example, virtual reality (VR), augmented reality (AR), mixed reality (MR), hybrid reality, or some combination and / or derivative thereof. Artificial reality content may include fully generated content, or generated content combined with captured content (e.g., real-world photos). Artificial reality content may include video, audio, tactile feedback, or some combination thereof, any of which may be presented in a single channel or multiple channels (e.g., stereoscopic video that gives the viewer a three-dimensional effect). In addition, in some embodiments, artificial reality may be associated with an application, product, accessory, service, or some combination thereof, which is used, for example, to create content in artificial reality and / or to be used in artificial reality (e.g., to perform activities in artificial reality). Artificial reality systems that provide artificial reality content can be implemented on a variety of platforms, including a head-mounted display (HMD) connected to a host computer system, a stand-alone HMD, a mobile device or computing system, a "cave" environment or other projection system, or any other hardware platform capable of providing artificial reality content to one or more viewers.

[0023] As used herein, "virtual reality" or "VR" refers to an immersive experience in which the user's visual input is controlled by a computing system. "Augmented reality" or "AR" refers to a system in which a user views images of the real world after they have passed through a computing system. For example, a tablet with a camera on the back can capture multiple images of the real world and then display them on a screen on the side of the tablet opposite the camera. The tablet can process and adjust or "enhance" the images as they pass through the system, for example by adding virtual objects. "Mixed reality" or "MR" refers to a system in which the light entering the user's eyes is generated in part by a computing system and in part from light reflected from objects in the real world. For example, an MR head-mounted viewer can be shaped as a pair of glasses with a transparent display that allows light from the real world to pass through a waveguide that simultaneously emits light from a projector in the MR head-mounted viewer, thereby allowing the MR head-mounted viewer to present virtual objects mixed with real objects that the user can see. As used herein, “artificial reality,” “hyperreality,” or “XR” refers to any of the following: VR, AR, MR, or any combination or hybrid thereof.

[0024] Embodiments improve processing efficiency by selectively upsampling the captured resolution and sparsely populating one or more visual data structures (e.g., texture map levels) when generating transparent visual content for high-resolution display. The evolution of XR head-mounted view devices toward high-resolution displays has increased the practicality of upsampling captured visual content, as sensor resolution may be limited due to cost, size, power usage, data bandwidth, and / or inefficient hardware usage. Embodiments use the fovea to target certain portions of the user's field of view with a high level of resolution, while downgrading other portions to a medium or low level of resolution. Embodiments then perform selective upsampling based on the resolution for that portion, such as using bicubic sampling and linear sampling (e.g., bilinear sampling, trilinear sampling, etc.). One or more visual data structures (e.g., texture map levels) used to generate an XR display may be sparsely populated so that hardware resources are not wasted on unnecessary upsampling for display areas at medium or low resolution levels.

[0025] In some embodiments, the sparsely populated texture map levels are combined with the rendering pipeline of the XR display. Specifically, hardware resources can execute a rendering pipeline that accesses the sparsely populated texture map levels based on the quality level assigned to the portion of visual data. This solution provides flexibility for artificial reality system design (e.g., systems with high-resolution displays). For example, the sensor resolution can be lower than the display resolution, and various embodiments can effectively upsample the visual data. Assigning different portions of the transparent visual content to different quality levels can save processing efficiency and improve system resource utilization.

[0026] Several embodiments are discussed in more detail below with reference to the accompanying drawings. Figure 1 is a block diagram showing an overview of multiple devices on which some embodiments of the disclosed technology may run. These devices may include hardware components of the computing system 100 that selectively upsample visual sensor data for efficient display. In various embodiments, the computing system 100 may include a single computing device 103 or multiple computing devices (e.g., computing device 101, computing device 102, and computing device 103) that communicate via a wired channel or a wireless channel to distribute processing and share input data. In some embodiments, the computing system 100 may include a standalone head-mounted viewer that is capable of providing a computer-created or enhanced experience to a user without the need for external processing or external sensors. In other embodiments, the computing system 100 may include multiple computing devices, such as a head-mounted viewer and a core processing component (e.g., a console, a mobile device, or a server system), wherein some processing operations are performed on the head-mounted viewer and other processing operations are offloaded to the core processing component. The following is combined with Figure 2A and Figure 2B An example head mounted view device is described. In some implementations, location data and environmental data may be collected solely by sensors incorporated into the head mounted view device, while in other implementations, one or more of the plurality of non-head mounted view computing devices may include sensor components that may track environmental data or location data.

[0027] The computing system 100 may include one or more processors 110 (e.g., a central processing unit (CPU), a graphical processing unit (GPU), a holographic processing unit (HPU), etc.). The processor 110 may be a single processing unit or multiple processing units, which are located in one device or distributed on multiple devices (e.g., distributed on two or more computing devices among the computing devices 101 to 103).

[0028] The computing system 100 may include one or more input devices 120 that provide input to the processors 110, thereby informing the processors of actions. These actions may be communicated by a hardware controller that interprets signals received from the input devices and transmits the information to the processors 110 using a communication protocol. Each input device 120 may include, for example, a mouse, a keyboard, a touch screen, a touchpad, a wearable input device (e.g., a tactile glove, a bracelet, a ring, an earring, a necklace, a watch, etc.), a camera (or other light-based input device, such as an infrared sensor), a microphone, or other user input device.

[0029] The processor 110 can be coupled to other hardware devices, for example, by using an internal bus or an external bus, or a wireless connection, such as a peripheral component interconnect standard (PCI) bus, a small computer system interface (SCSI) bus. The processor 110 can communicate with the hardware controller of each device (e.g., a display 130). The display 130 can be used to display text and graphics. In some embodiments, the display 130 includes an input device, such as when the input device is a touch screen or is equipped with an eye movement direction monitoring system, the input device is part of the display. In some embodiments, the display is separated from the input device. Examples of display devices include: a liquid crystal display (LCD) display screen; a light emitting diode (LED) display screen; and a projection, holographic, or augmented reality display (e.g., a head-up display device or a head-mounted device), etc. Other input / output (I / O) devices 140 may also be coupled to the processor, such as a network chip or network card, a video chip or video card, an audio chip or audio card, a universal serial bus (USB), FireWire or other external devices, a camera, a printer, a speaker, a compact disc read-only memory (CD-ROM) drive, a digital video disc (DVD) drive, a disk drive, etc.

[0030] In some embodiments, input from I / O devices 140 (e.g., cameras, depth sensors, inertial motion unit (IMU) sensors, global positioning system (GPS) units, laser radar (LiDAR) or other time-of-flight sensors, etc.) can be used by computing system 100 to identify and map the user's physical environment while tracking the user's position within the environment. A simultaneous localization and mapping (SLAM) system can generate a map (e.g., topology, grid, etc.) of an area (which can be a room, building, outdoor space, etc.) and / or obtain a map previously generated by computing system 100 or another computing system that maps the area. The SLAM system can track users within the area based on factors such as GPS data, matching identified objects and structures with mapped objects and structures, monitoring acceleration and other changes in position, etc.

[0031] The computing system 100 may include such a communication device: the communication device can communicate with other local computing devices or network nodes wirelessly or based on wired communication. The communication device can communicate with another device or server through a network, for example, using the Transmission Control Protocol / Internet Protocol (TCP / IP) protocol. The computing system 100 can use the communication device to distribute operations on multiple network devices.

[0032] Processor 110 can access memory 150, which can be included in one of the multiple computing devices of computing system 100, or can be distributed across multiple computing devices of computing system 100, or other external devices. Memory includes one or more hardware devices for volatile storage or non-volatile storage, and can include both read-only memory and writable memory. For example, memory can include one or more of the following: random access memory (RAM), various cache memories, CPU registers, read-only memory (ROM), and writable non-volatile memory, such as flash memory, hard drive, floppy disk, compact disk (CD), DVD, magnetic storage device, and tape drive. Memory is not a propagation signal that is separated from the underlying hardware; therefore, memory is non-transitory. Memory 150 can include program memory 160 that stores programs and software, such as operating system 162, efficiency manager 164, and other applications 166. Memory 150 may also include data storage 170, which may include, for example, captured visual data, upsampled visual data and / or downsampled visual data, one or more quality maps, configuration data, settings, user options or preferences, etc., which may be provided to program memory 160 or any element of computing system 100.

[0033] Some embodiments may operate with many other computing system environments or configurations. Examples of computing systems, environments, and / or configurations suitable for use with the technology include, but are not limited to, XR headsets, personal computers, server computers, handheld or laptop devices, cellular phones, wearable electronic devices, game consoles, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, networked personal computers (PCs), minicomputers, mainframe computers, or distributed computing environments that include any of the above systems or devices, etc.

[0034] Figure 2A2 is a line diagram of a virtual reality head mounted display (HMD) 200 according to some embodiments. The virtual reality HMD 200 includes a front rigid body 205 and a strap 210. The front rigid body 205 includes one or more electronic display elements of an electronic display 245, an inertial motion unit (IMU) 215, one or more position sensors 220, a locator 225, and one or more computing units 230. The position sensor 220, the IMU 215, and the computing unit 230 may be located inside the virtual reality HMD 200 and may not be visible to the user. In various embodiments, the IMU 215, the position sensor 220, and the locator 225 may track the movement and position of the virtual reality HMD 200 in the real world and in an artificial reality environment with three degrees of freedom (3DoF) or six degrees of freedom (6DoF). For example, the locator 225 may emit infrared beams that produce light spots on real objects around the virtual reality HMD 200. As another example, the IMU 215 may include, for example: one or more accelerometers; one or more gyroscopes; one or more magnetometers; one or more other non-camera-based position sensors, force sensors, or orientation sensors; or a combination thereof. One or more cameras (not shown) integrated with the virtual reality HMD 200 may detect light points. The computing unit 230 in the virtual reality HMD 200 may use the detected light points to infer the position and movement of the virtual reality HMD 200, and to recognize the shapes and positions of real objects around the virtual reality HMD 200.

[0035] The electronic display 245 can be integrated with the front rigid body 205 and can provide image light to the user as directed by the computing unit 230. In various embodiments, the electronic display 245 can be a single electronic display or multiple electronic displays (e.g., one display for each eye of the user). Examples of the electronic display 245 include: a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode display (AMOLED), a display including one or more quantum dot light-emitting diode (QOLED) sub-pixels, a projector unit (e.g., micro-LED, laser (LASER)), some other display, or some combination thereof.

[0036] In some embodiments, the virtual reality HMD 200 can be coupled to a core processing component such as a personal computer (PC) (not shown) and / or one or more external sensors (not shown). The external sensors can monitor the virtual reality HMD 200 (e.g., via light emitted from the virtual reality HMD 200), and the PC can use the HMD in combination with the output from the IMU 215 and the position sensor 220 to determine the position and movement of the virtual reality HMD 200.

[0037] Figure 2B 2 is a line diagram of a mixed reality HMD system 250, which includes a mixed reality HMD 252 and a core processing component 254. The mixed reality HMD 252 and the core processing component 254 can communicate via a wireless connection (e.g., a 60 Gigahertz (GHz) link) as indicated by link 256. In other embodiments, the mixed reality HMD system 250 includes only a head-mounted viewer without an external computing device, or includes other wired or wireless connections between the mixed reality HMD 252 and the core processing component 254. The mixed reality HMD 252 includes a transparent display 258 and a frame 260. The frame 260 can accommodate various electronic components (not shown), such as light projectors (e.g., lasers, LEDs, etc.), cameras, eye tracking sensors, micro-electromechanical systems (MEMS) components, network components, etc.

[0038] The projector can be coupled to the transparent display 258, for example, via an optical element to display media to the user. The optical element may include one or more waveguide components, one or more reflectors, one or more lenses, one or more mirrors, one or more collimators, one or more gratings, etc. for guiding light from the projector to the user's eyes. The image data can be transmitted from the core processing component 254 to the mixed reality HMD 252 via the link 256. The controller in the mixed reality HMD 252 can convert the image data into a plurality of light pulses from the projector, which can be transmitted to the user's eyes as output light via the optical element. The output light can be mixed with the light passing through the transparent display 258, thereby allowing the output light to present virtual objects as follows: these virtual objects appear to exist in the real world.

[0039] Similar to the virtual reality HMD 200, the mixed reality HMD system 250 may also include motion and position tracking units, cameras, light sources, etc., which allow the mixed reality HMD system 250 to track itself, for example, with 3DoF or 6DoF, track multiple parts of the user (e.g., hands, feet, head, or other body parts), draw virtual objects to appear stationary when the mixed reality HMD 252 moves, and make virtual objects react to gestures and other real-world objects.

[0040] Figure 2C A plurality of controllers 270 (including controllers 276A and 276B) are shown, which in some embodiments can be held by a user with one or both hands to interact with an artificial reality environment presented by the virtual reality HMD 200 and / or the mixed reality HMD system 250. The controllers 270 can communicate with the HMD directly or via an external device (e.g., the core processing component 254). The controllers can have their own IMU units, position sensors, and / or can emit additional light points. The virtual reality HMD 200 or the mixed reality HMD system 250, external sensors, or sensors in the controllers can track these controller light points to determine the position and / or orientation of the controllers (e.g., to track the controllers in 3DoF or 6DoF). The computing unit 230 or the core processing component 254 in the virtual reality HMD 200 can use this tracking to monitor the user's hand position and movement in combination with the IMU output and the position output. The controllers may also include various buttons (eg, buttons 272A-272F) and / or levers (eg, levers 274A and 274B) that a user may actuate to provide input and interact with objects.

[0041] In various embodiments, the virtual reality HMD 200 or mixed reality HMD system 250 may also include additional subsystems, such as an eye tracking unit, an audio system, various network components, etc., to monitor the user's interactions and indications of intent. For example, in some embodiments, instead of or in addition to the controller, one or more cameras included in the virtual reality HMD 200 or mixed reality HMD system 250 or from an external camera can monitor the position and posture of the user's hands to determine gestures and other hand and body movements. As another example, one or more light sources can illuminate either or both of the user's eyes, and the virtual reality HMD 200 or mixed reality HMD system 250 can use eye-facing cameras to capture reflections of that light to determine eye position (e.g., based on a set of reflections around the user's cornea), model the user's eyes, and determine gaze direction.

[0042] Figure 3 305B may be a virtual reality HMD 200 or a mixed reality HMD system 250. The client computing device 305 may be a network environment using a logical connection to one or more remote computers (e.g., a server computing device) through a network 330.

[0043] In some implementations, server computing device 310 may be an edge server that receives client requests and coordinates fulfillment of these requests through other servers (e.g., server computing devices 320A-320C). Server computing devices 310 and 320 may include computing systems, such as computing system 100. Although each server computing device 310 and 320 is logically shown as a single server, each of these server computing devices may be a distributed computing environment including multiple computing devices that are located at the same physical location or at geographically different physical locations.

[0044] Client computing device 305, and server computing devices 310 and 320 can all act as servers or clients of one or more other server / client devices. Server computing device 310 can be connected to database 315. Server computing devices 320A to 320C can each be connected to corresponding databases 325A to 325C. As described above, each server computing device 310 or 320 can correspond to a group of servers, and each of these servers can share a database or can have its own database. Although databases 315 and 325 are logically shown as a single unit, databases 315 and 325 can all be distributed computing environments that include multiple computing devices, which can be located in their corresponding servers, or can be located at the same physical location or at geographically different physical locations.

[0045] Network 330 may be a local area network (LAN), a wide area network (WAN), a mesh network, a hybrid network, or other wired or wireless network. Network 330 may be the Internet or some other public or private network. Client computing device 305 may be connected to network 330 via a network interface, such as by wired or wireless communications. Although the connections between server computing device 310 and multiple server computing devices 320 are shown as separate connections, these connections may be any type of local area network, wide area network, wired network, or wireless network including network 330 or a separate public or private network.

[0046] Figure 4 4 is a block diagram showing a component 400 that can be used in a system that employs the disclosed technology in some embodiments. The component 400 can be included in one device of the computing system 100, or can be distributed across multiple devices of the computing system 100. The component 400 includes hardware 410, middleware 420, and multiple dedicated components 430. As discussed above, a system that implements the disclosed technology can use various hardware, including a processing unit 412, a working memory 414, input and output devices 416 (e.g., a camera, a display, an IMU unit, a network connection, etc.) and a storage memory 418. In various embodiments, the storage memory 418 can be one or more of the following: a local device, an interface to a remote storage device, or a combination thereof. For example, the storage memory 418 can be one or more hard drives or flash drives accessible via a system bus, or can be a cloud storage provider (e.g., in a database 315 or 325) or other network storage, which can be accessed via one or more communication networks. In various implementations, component 400 may be implemented in a client computing device (eg, client computing device 305 ), or on a server computing device (eg, server computing devices 310 or 320 ).

[0047] The middleware 420 may include components that transfer resources between the hardware 410 and the plurality of specialized components 430. For example, the middleware 420 may include an operating system, a service system (service), a driver, a basic input output system (BIOS), a controller circuit, or other hardware or software systems.

[0048] Specialized components 430 may include software or hardware configured to perform operations for selectively upsampling visual sensor data for efficient display. Specialized components 430 may include: an eye tracker 434; a visual data manager 436; one or more texture maps 438; a pass-through renderer 440; and components and application programming interfaces (APIs) (e.g., interface 432) that can be used to provide a user interface, transfer data, and control the specialized components. In some embodiments, multiple components 400 may be located in a computing system distributed across multiple computing devices, or may be an interface to a server-based application that executes one or more of multiple specialized components 430. Although multiple specialized components 430 are depicted as separate components, these specialized components may be logical or other non-physical distinctions of functionality, and / or may be sub-modules or code blocks of one or more applications.

[0049] The eye tracker 434 can use captured images of the user's face to track one or both eyes of the user. Various embodiments of the eye tracker 434 receive images of the user's eyes from an HMD that captures images of the user's eyes. The eye tracker 434 may include one or more estimation models (e.g., a trained machine learning model or any other suitable model) that estimate the user's eye position (e.g., pupil position, user gaze, etc.) based on the user's eye images and / or features extracted from the user's eye images. In some embodiments, one or more lights of the HMD can illuminate either or both of the user's eyes, and one or more cameras can capture reflections of the light to determine the eye position (e.g., based on a set of reflections around the user's cornea, etc.). For example, one or more estimation models can estimate the eye position based on these captured images of one or both of the user's eyes. Any other suitable eye tracking technology can be used to track one or more eye positions of the user. The following is about Figure 7 and box 704 provide additional details regarding eye tracker 434 .

[0050] Visual data manager 436 may selectively upsample portions of the captured visual data and sparsely populate one or more data structures (e.g., one or more texture maps 438) with the upsampled visual data. For example, visual data manager 436 may assign one or more portions of the captured visual data to quality levels and selectively upsample one or more portions of the captured visual data corresponding to a high quality level (e.g., a quality level that satisfies a threshold). Visual data manager 436 may then sparsely populate one or more data structures with the selectively upsampled visual data. Figure 7Boxes 706 , 708 , and 710 provide additional details regarding the visual data manager 436 .

[0051] One or more texture maps 438 can store visual data at different quality levels. For example, one or more texture maps 438 can be hierarchical, such that different map levels store visual data at different (e.g., high, medium, and low) quality levels. Portions of a map level can be populated with raw sensor data, and portions of one or more map levels can be populated with selectively upsampled visual data and / or selectively downsampled visual data. One or more texture maps 438 can be implemented by any suitable data structure and / or computer graphics software technology. Figure 7 Boxes 708 , 710 , and 712 provide additional details regarding the one or more texture maps 438 .

[0052] The transparent renderer 440 can use the visual data captured from the image sensing device to render a visual transparent display. For example, the captured and processed (e.g., upsampled, downsampled, etc.) visual data can be rendered to display an XR environment to the user (e.g., the XR environment from the user's perspective). The transparent renderer 440 can utilize one or more GPUs and other suitable one or more processors to execute the rendering pipeline and render the transparent display. In some embodiments, the rendered transparent display may include multiple parts of different quality levels, such as a foveated transparent display. The following is about Figure 7 and block 712 provide additional details regarding the pass-through renderer 440 .

[0053] Figure 5 500 illustrates an XR system 502, an image capture device 504, and a transparent display 506. The image capture device 504 can capture the real-world environment of the XR system 502. The XR system 502 can process the captured visual data and generate the transparent display 506. For example, the transparent display 506 can include an XR environment from the user's perspective of the user's real-world environment.

[0054] In some embodiments, the transparent display 506 can include multiple portions with different resolutions. For example, the XR system 502 can selectively upsample portions of the visual data captured via the image capture device 504. The portions can be selected for upsampling in response to user eye tracking. For example, portions of the visual data corresponding to a focused (e.g., central) portion of the user's field of view can be selectively upsampled. In some embodiments, the transparent display 506 can include a foveated display.

[0055] In some embodiments, the configuration of one or more portions of the user's field of view that are upsampled (e.g., size, orientation, etc.) and / or the degree of upsampling can be based on an eye tracking accuracy and / or confidence metric. For example, eye tracking with a high confidence metric can be targeted to one or more smaller portions for upsampling, so that a higher degree of upsampling can be performed. When eye tracking is reported with a lower confidence metric, upsampling can be performed on one or more larger portions, so that the degree of upsampling can be reduced (e.g., to manage the processing load imposed on the hardware).

[0056] Fig. 6A 600A illustrates a conceptual diagram of an example quality level pattern for a foveal display for an artificial reality system. Diagram 600A illustrates multiple portions of a foveal display corresponding to different quality levels. In the example shown, level 0 corresponds to a high quality level, level 1 corresponds to a medium quality level, and level 2 corresponds to a low quality level. Any other suitable quality level assignments may be implemented. The foveal display is generated in response to eye tracking. In the example shown, the center of diagram 600A represents the center of the user's field of view, and thus the center is assigned to level 0. The periphery of diagram 600A represents the periphery of the user's field of view, and different portions of the periphery are assigned to level 1 and level 2. Any other suitable quality level pattern may be implemented that assigns different quality levels to one or more different portions of diagram 600A.

[0057] In some embodiments, in response to user eye tracking, the quality level is dynamically assigned to the captured visual data. For example, eye tracking can be used to identify the central part of the user's field of view and / or the peripheral part of the user's field of view. The captured visual data corresponding to the central area of ​​the user's field of view can be assigned to level 0, and the captured visual data corresponding to the peripheral area of ​​the user's field of view can be assigned to level 1 or level 2. Each embodiment may include two, three, four or more quality levels. The captured visual data assigned to level 0 can be selectively upsampled to fill the high quality level of the hierarchical texture map. In this example, the captured visual data assigned to level 1 or level 2 is not selectively upsampled, and therefore the high quality level of the hierarchical texture map of the captured visual data may be empty (for example, the high quality level of the hierarchical texture map can be sparsely filled). Other levels of the hierarchical texture map below the high quality level can be filled with captured visual data and / or captured downsampled visual data.

[0058] The quality level may correspond to a combination of multiple quality factors, such as resolution, variable rate shading (VRS), or any other suitable quality factor. Example quality factors for each quality level are shown in Table 1 below:

[0059] Table 1

[0060]

[0061] The upsampled data may include visual data captured at a given resolution (e.g., corresponding to the resolution of one or more image capture devices), for example, the visual data is selectively upsampled using bicubic sampling or any other suitable upsampling technique. The raw data may include visual data with a resolution captured by one or more image capture devices. For example, one or more image capture devices may include the following physical components and / or software components: these physical components and / or software components support image capture at a given resolution (e.g., 2 megapixels (MP), 4MP, 8MP, 16MP, etc.). In some embodiments, by performing selective upsampling, the image resolution supported by one or more image capture devices may be twice (e.g., 4MP to 8MP, 8MP to 16MP, etc.), three times, and four times (e.g., 4MP to 16MP, etc.), etc. In this example, an XR system with one or more high-resolution displays can utilize one or more image capture devices of lower resolution and effectively perform upsampling to provide a high-resolution visual display. In some embodiments, the foveal display may be a transparent display of the user's real-world environment, and the upsampled image resolution may provide a display of the user's real-world environment at an increased resolution.

[0062] Figure 6B 600B illustrates a conceptual diagram of a user's field of view with different display quality levels. Diagram 600B illustrates quality levels 602 (including a high quality level 610, a medium quality level 612, and a low quality level 614) and captured visual data 604. The captured visual data 604 may represent visual data captured by one or more image capture devices that are part of or in communication with the XR system. The quality level 602 may be applied to the visual data to generate a foveal display. For example, portions of the captured visual data (e.g., portions assigned the high quality level 610) may be upsampled. In some embodiments, the captured visual data 604 may be processed using the quality level 602 to generate multiple foveal displays (e.g., one display for each eye of the user). Such a display may enable an immersive foveal display.

[0063] In some embodiments, applying the quality level 602 to the captured visual data 604 includes: sparsely filling the hierarchical texture map with the captured visual data 604 and / or the captured upsampled visual data. For example, the captured visual data 604 may be captured at a given resolution (e.g., the resolution of one or more image capture devices). The medium quality level and / or the low quality level of the hierarchical texture map may be filled with the captured original visual data and / or the downsampled visual data. One or more high quality levels of the texture map may be selectively filled with upsampled visual data. For example, a portion of the captured visual data 604 assigned to the high quality level 610 may be selectively upsampled, and the upsampled data may sparsely fill the high quality level of the hierarchical texture map. In other words, for portions of the captured visual data 604 that are not assigned to the high quality level 610, the high quality level of the texture map may be empty. In some embodiments, the quality level of the hierarchical texture map that is lower than the resolution of the captured visual data 604 (e.g., the resolution of one or more image capture devices) may be automatically filled by an automatic MIP generation technique.

[0064] In some embodiments, the manner in which the captured visual data is assigned to different quality levels can be based on user eye tracking and / or the user's field of view. For example, the captured visual data 604 can include multiple parts, each of which can include a pixel area (e.g., a 16×16 pixel area, a 32×32 pixel area, etc.). With respect to Figure 600B, the quality level 602 can include an example pattern such that each of the quality levels 610, 612, and 614 corresponds to a pixel area associated with a given field of view of the user. In this example, given the user's field of view, the captured visual data is assigned to different quality levels based on the pixel area in which the captured visual data is located. Any other suitable pixel area size and / or quality level pattern can be used to assign the captured visual data to quality levels.

[0065] In some embodiments, the manner in which pixel areas are assigned to quality levels can be based on eye tracking confidence levels. For example, eye tracking with a high confidence metric (e.g., an eye tracking metric that meets a confidence standard or threshold) can be targeted at one or more smaller portions (e.g., smaller pixel area sizes) for upsampling. This can produce a higher degree of visual data captured that is assigned to a high quality level, and thus a higher degree of upsampling can be performed. Eye tracking with a low confidence metric (e.g., an eye tracking metric that does not meet a confidence standard or threshold) can be targeted at one or more larger portions (e.g., larger pixel area sizes) for upsampling. This can produce a lower degree of visual data captured that is assigned to a high quality level, and thus the degree of upsampling can be reduced (e.g., to manage the processing load imposed on the hardware). In other words, the captured visual data can be grouped according to pixel area and assigned to each quality level by pixel area. In this example, the pixel area size can change dynamically based on the confidence metric of the user's eye tracking.

[0066] Downsampling can be performed by discarding the captured visual data (e.g., selecting a subset of multiple pixels) (e.g., reducing 512×512 pixels of visual data to 256×256 pixels) to reduce the resolution. Upsampling can be performed by increasing the captured visual data (e.g., increasing 512×512 pixels of visual data to 1024×1024 pixels) to increase the resolution. In some examples, upsampling includes increasing pixel data based on aggregating pixel values ​​(e.g., RGB values) from nearby pixels. Examples of upsampling include bicubic sampling and linear sampling, among others. Bilinear sampling uses visual data coordinates to find multiple nearest samples (e.g., pixel values) (e.g., four nearest pixels) and blends these samples together for linear interpolation based on the distance to these samples. Various embodiments may also implement trilinear sampling. Bicubic interpolation extends it from linear interpolation on each axis (which is a straight line in 2D space) to cubic interpolation on each axis (which is a third-order polynomial). Where two points are used to form a straight line, four points are used to form a cubic polynomial. In some embodiments, cubic curves may be calculated as splines to provide smoother transitions between points. In some embodiments, the hardware may be configured with a bilinear texture sampler, and multiple (eg, four) reads via the bilinear texture sampler may be used to accomplish cubic sampling.

[0067] In some embodiments, a hardware resource (e.g., a processor such as a CPU and / or a GPU) can execute a rendering pipeline using a sparsely populated texture map. For example, the rendering pipeline can render portions of a display according to the assigned quality level by accessing a texture map level corresponding to the assigned quality level. In some embodiments, the assigned quality level can also configure a VRS during execution of the rendering pipeline.

[0068] In some embodiments, the pass-through visualization can be rendered according to a frequency (e.g., 60 hertz (hz)), and upsampling can be performed on a subset of the rendered multiple instances (e.g., 30hz, 15hz, etc.), or upsampling can be performed on each of the rendered multiple instances (e.g., 60hz). For example, each instance can be selectively upsampled, every other instance can be selectively upsampled (e.g., according to upsampling one or more portions for a high quality level), etc. In some embodiments, upsampling can be paused in response to detecting certain user conditions (e.g., large user head movements that cause content distortion).

[0069] Those skilled in the art will understand that in the above Figures 1 to 5 , Fig. 6A as well as Figure 6B The various components shown in the flowcharts and in each of the flowcharts discussed below may be changed in various ways. For example, the order of the logic may be rearranged, multiple sub-steps may be performed in parallel, the logic shown may be omitted, other logic may be included, etc. In some embodiments, one or more of the above components may perform one or more of the processes described below.

[0070] Figure 7 700 is a flowchart illustrating a process for selectively upsampling visual sensor data for efficient display used in some embodiments of the present technology. In some embodiments, process 700 is performed by an XR system. Process 700 may be triggered by initiating display of an XR environment to a user or continuing to display an XR environment.

[0071] At block 702, process 700 may capture visual data via one or more image capture devices. For example, the XR system may include or may communicate with one or more cameras that capture the real-world environment of the user / XR system. In some embodiments, the XR system may use the captured visual data to generate a transparent display of the user's real-world environment. The one or more image capture devices may include a capture resolution, and embodiments may selectively upscale portions of the captured visual data so that the generated transparent display of the user's real-world environment includes a portion having a display resolution greater than the capture resolution.

[0072] At box 704, process 700 may assign quality levels to the captured visual data. For example, the captured visual data may be processed to render visual data that immerses the user in a transparent visualization. Quality levels or categories may be assigned to portions of the captured visual data based on user eye tracking. The XR system may track the user's eyes and identify which portion of the transparent visual content the user is focusing on (e.g., which portions are located at the center of the user's field of view based on eye tracking). The captured visual data may be assigned different quality levels or categories (e.g., low quality, medium quality, and high quality) based on eye tracking. For example, the center where the user is focused may be assigned a high quality level, while the periphery may be assigned a medium quality level or a low quality level. In some embodiments, the captured visual data may be configured as a plurality of pixel regions, and different pixel regions may be assigned different quality levels. For example, a predetermined and / or dynamic mode may assign pixel regions to quality levels based on which regions are located at the center where the user is focused and which regions are located at the periphery where the user is focused.

[0073] In some embodiments, the portion of the captured visual data assigned to the high quality level corresponds to a central area of ​​the user's field of view associated with the eye tracking performed, and the portion of the captured visual data assigned to the medium quality level or the low quality level corresponds to a peripheral area of ​​the user's field of view associated with the eye tracking performed. In some embodiments, one or more quality levels can be assigned to the captured visual data (e.g., pixel areas) based on an eye tracking confidence metric.

[0074] At block 706, process 700 can selectively upsample the captured portion of the visual data assigned to the high quality level. For example, the captured portion of the visual data assigned to the high quality level can be selectively upsampled. Examples of upsampling include bicubic sampling and linear sampling, among others.

[0075] At block 708, process 700 can selectively populate high quality levels of a texture map (e.g., a data structure) with the upsampled visual data. For example, a hierarchical texture map can store versions of the visual data at different quality levels (e.g., resolutions). For portions of the captured visual data that are assigned to one or more high quality levels, one or more high quality levels of the hierarchical texture map can be populated with the selectively upsampled visual data. For other portions of the captured visual data that are assigned to other quality levels, one or more high quality levels of the hierarchical texture map can be left empty, thereby achieving a sparsely populated hierarchical texture map.

[0076] At block 710, process 700 can populate other levels of the hierarchical texture map. For example, a medium quality level and / or a low quality level of the texture map can be populated with captured visual data and / or a downsampled version of the captured visual data.

[0077] At block 712, process 700 can render a transparent display for the user using the layered texture map. For example, a hardware resource (e.g., one or more processors such as a CPU and / or a GPU) can execute a rendering pipeline to render a display having different quality levels (e.g., resolution levels, etc.). The display can include transparent visual content of the user's real-world environment, where the portion of the display on which the user is focused (e.g., the center of the user's field of view according to eye tracking) is displayed at a higher resolution than the periphery of the user's field of view. In some implementations, the generated display is a foveated display.

[0078] In some embodiments, a portion of the transparent visual display corresponding to the captured visual data assigned to the high quality level is rendered using a sparsely populated high quality level of the layered texture map. In some embodiments, a portion of the transparent visual display corresponding to a portion of the captured visual data not assigned to the high quality level (e.g., a medium quality level and / or a low quality level) is rendered using one or more other populated levels of the layered texture map. In some embodiments, a hardware resource renders the transparent visual display according to a frequency (e.g., 60hz) and performs selective upsampling for a subset of a plurality of instances rendered according to the frequency or for each instance of a plurality of instances rendered according to the frequency.

[0079] References in this specification to "multiple embodiments" (e.g., "some embodiments," "various embodiments," "one embodiment," "an embodiment," etc.) mean that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present disclosure. The appearances of these phrases in different places in the specification do not necessarily all refer to the same embodiment, nor do they necessarily refer to separate embodiments or alternative embodiments that are mutually exclusive of other embodiments. In addition, various features are described that may be present in some embodiments but not in other embodiments. Similarly, various requirements are described that may be requirements of some embodiments but not in other embodiments.

[0080] As used herein, being higher than a threshold value means that the value of the item being compared is higher than other values ​​specified, the item being compared is located in an item with a maximum value, a certain specified number, or the item being compared has a value within a specified maximum percentage value. As used herein, being lower than a threshold value means that the value of the item being compared is lower than other values ​​specified, the item being compared is located in an item with a minimum value, a certain specified number, or the item being compared has a value within a specified minimum percentage value. As used herein, being located within a threshold value means that the value of the comparison item is between two other values ​​specified, the comparison item is located in an item with a middle specified number, or the value of the comparison item is within a middle specified percentage range. When not otherwise defined, relative terms such as high or unimportant may be understood to assign a value and determine how the value is compared with the established threshold value. For example, the phrase "selecting fast connection" may be understood to mean selecting a connection with a value corresponding to the connection speed with which it is higher than a threshold value that is assigned.

[0081] As used herein, the word "or" refers to any possible arrangement of a group of items. For example, the phrase "A, B or C" refers to at least one of A, B, C, or any combination thereof, such as any of the following: A; B; C; A and B; A and C; B and C; A, B and C; or a plurality of any items, such as A and A; B, B and C; A, A, B, C and C; etc.

[0082] Although the subject matter has been described in language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Specific embodiments and implementations have been described herein for illustrative purposes, but various modifications may be made without departing from the scope of these embodiments and implementations. The specific features and actions described above are disclosed as example forms of implementing the appended claims. Therefore, these embodiments and implementations are not limited except as limited by the appended claims.

[0083] Any patent, patent application and other references mentioned above are incorporated herein by reference. If necessary, various aspects may be modified to provide further embodiments using the systems, functions and concepts of the various references mentioned above. If the statements or themes in the documents incorporated by reference conflict with the statements or themes of the present application, the present application shall prevail.

Claims

1. A method for selectively upsampling sensor data for display via an artificial reality (XR) system, the method comprising: capturing visual data of a real-world environment of the XR system using one or more image capture devices; assigning captured portions of the visual data to two or more categories; selectively upsampling a resolution of the captured visual data for a portion of the captured visual data that is assigned to a first category of the two or more categories; Populate the data structure in the following way: sparsely populate the high quality level of the data structure with the selectively upsampled data; as well as populating one or more other levels of the data structure using the captured visual data; as well as A transparent visual display of the real-world environment of the XR system captured by the XR system rendering, wherein a portion of the transparent visual display corresponding to the captured visual data assigned to the first category is rendered using the sparsely populated high quality level of the data structure.

2. The method according to claim 1, wherein: Portions of the pass-through visual display corresponding to portions of the captured visual data not assigned to the first category are rendered using the populated one or more other levels of the data structure.

3. The method according to claim 1, wherein: Bicubic sampling is performed to selectively upsample the captured visual data.

4. The method according to claim 1, further comprising: Eye tracking is performed on a user of the XR system, wherein the portions of the captured visual data are assigned to the two or more categories based on the eye tracking performed.

5. The method according to claim 4, wherein: the portion of the captured visual data assigned to the first category corresponds to a central area of ​​the user's field of view relevant to the eye tracking performed, The portion of the captured visual data assigned to another of the two or more categories corresponds to a peripheral area of ​​the user's field of view that is relevant to the eye tracking being performed, and The rendered pass-through visual display includes a foveated display including a plurality of portions having different resolution levels.

6. The method according to claim 1, further comprising: The rendered pass-through visual display is displayed via display hardware of the XR system, wherein a resolution of the one or more image capture devices is less than a resolution of the display hardware.

7. The method according to claim 1, wherein: The data structure comprises a hierarchical texture map, and the high quality level of the hierarchical texture map is sparsely populated by selectively upsampling portions of the captured visual data assigned to the first category.

8. The method according to claim 7, wherein: The one or more other levels of the hierarchical texture map are populated using the captured visual data and / or the captured visual data after downsampling.

9. The method according to claim 1, wherein: The pass-through visual display is rendered according to a frequency, and the selective upsampling is performed for a subset of a plurality of instances rendered according to the frequency or for each of the plurality of instances rendered according to the frequency.

10. An artificial reality (XR) system for selectively upsampling sensor data, the XR system comprising: one or more processors; as well as one or more memories storing instructions that, when executed by the one or more processors, cause the XR system to: capturing data of a real-world environment of the one or more sensor devices using one or more image capture devices; assigning portions of the captured data to two or more categories; selectively upsampling a resolution of the captured data for a portion of the captured data that is assigned to a first category of the two or more categories; Populate the data structure in the following way: sparsely populate the high quality level of the data structure with the selectively upsampled data; as well as populating one or more other levels of said data structure with said captured data; as well as An output of the real-world environment is provided using the populated data structure, wherein a portion of the output corresponding to a portion of the captured data assigned to the first category is rendered using the sparsely populated high quality level of the data structure.

11. The XR system of claim 10, wherein: The output comprises a rendered transparent visual display of the real-world environment, and Portions of the pass-through visual display corresponding to portions of the captured data not assigned to the first category are rendered using the populated one or more other levels of the data structure.

12. The XR system of claim 10, wherein: Bicubic sampling is performed to selectively upsample the captured data.

13. The XR system of claim 10, wherein: The instructions, when executed by the one or more processors, further cause the XR system to: Eye tracking is performed on a user of the XR system, wherein the portions of the captured data are assigned to the two or more categories based on the eye tracking performed.

14. The XR system of claim 13, wherein: the portion of the captured data assigned to the first category corresponds to a central area of ​​the user's field of view relevant to the eye tracking performed, The portion of the captured data assigned to another of the two or more categories corresponds to a peripheral area of ​​the user's field of view that is relevant to the eye tracking being performed, and The output includes a foveated display including a plurality of portions having different resolution levels.

15. The XR system of claim 10, wherein: The instructions, when executed by the one or more processors, further cause the XR system to: The output is displayed via display hardware of the XR system, wherein a resolution of the one or more image capture devices is less than a resolution of the display hardware.

16. The XR system of claim 10, wherein: The data structure comprises a hierarchical texture map, and the high quality level of the hierarchical texture map is sparsely populated by selectively upsampling portions of the captured data assigned to the first category.

17. The XR system of claim 16, wherein: The one or more other levels of the layered texture map are populated using the captured data and / or the captured downsampled data.

18. The XR system of claim 10, wherein: The output comprises a rendered transparent visual display of the real-world environment, and The transparent visual display is rendered according to the frequency, and The selective upsampling is performed on a subset of the plurality of instances rendered according to the frequency or on each of the plurality of instances rendered according to the frequency.

19. A computer-readable storage medium storing instructions for selectively upsampling sensor data, the instructions, when executed by a computing system, causing the computing system to: capturing data of a real-world environment of the one or more image capture devices using one or more image capture devices; assigning portions of the captured data to two or more categories; selectively upsampling a resolution of the captured data for a portion of the captured data that is assigned to a first category of the two or more categories; Populate the data structure in the following way: sparsely populate the high quality level of the data structure with the selectively upsampled data; as well as populating one or more other levels of said data structure with said captured data; as well as An output of the real-world environment is provided using the populated data structure, wherein a portion of the output corresponding to a portion of the captured data assigned to the first category is rendered using the sparsely populated high quality level of the data structure.

20. The computer-readable storage medium of claim 19, wherein: The data structure comprises a hierarchical texture map, and the high quality level of the hierarchical texture map is sparsely populated by selectively upsampling portions of the captured data assigned to the first category.

Images