Late-masking-based rendering for augmented reality (XR)

Abstract

Certain aspects of the present disclosure propose a method of graphics processing at a wearable display device. The method generally includes: rendering a first image based on a location of the wearable display device and contour and depth information for one or more real-world objects associated with the location, the rendered first image including at least one or more virtual objects; re-projecting the first image based on an updated position of the wearable display device different from the position; using the re-projected contours of the one or more real-world objects in the first image for the updated depth information of the one or more real-world objects in the first image, rendering a second image for the updated depth information of the one or more virtual objects in the first image and the warped one or more virtual objects; and displaying the second image on a display of the wearable display device.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application serial number 17 / 813,900, filed on July 20, 2022, which is hereby incorporated by reference into this application. Background Art Technical Field

[0003] Aspects of the present disclosure relate to techniques for graphics processing.

[0004] Related technologies

[0005] Extended reality (XR) refers to real and virtual environments generated by computer graphics. XR is an umbrella category covering various forms of computer-altered reality, including: augmented reality (AR), mixed reality (MR), and virtual reality (VR). There are VR technologies for simulating virtual environments in which users can be immersed. VR technologies typically use headsets, which block visual stimuli from the real world. AR technologies allow users to see the real world, but overlay virtual elements. In particular, an AR display is a display in which the image is a main real environment enhanced or augmented with computer-generated imagery. MR technology allows real objects and virtual objects to interact with each other, and allows users of MR technology to interact with virtual elements as they would in the real world.

[0006] With AR, images of virtual objects and real-world scenes are combined to generate an augmented reality image in which the virtual objects appear to have been included in the real-world scene to an observer. The augmented reality image may be generated by rendering the virtual object within a portion of the captured real-world scene. When rendering the virtual object in the scene, the relative depth of the virtual object relative to the depth of the scene is taken into account to ensure that portions of the virtual object and / or portions of the real-world scene are correctly obscured relative to each other. By obscuring the image in this manner, a realistic depiction of the virtual object within the scene may be achieved.

[0007] Determining correct occlusion in an AR image may be performed by comparing corresponding depth values ​​for (1) an image of a virtual object to be placed in a real-world scene with (2) the real-world scene itself. Based on such comparison, the virtual object and the real-world scene may be combined by rendering for each pixel of the display (either a pixel from the image of the virtual object or a pixel for the real-world scene) based on which of the virtual object or the real-world object corresponding to the pixel location has a smaller depth value relative to a specified viewpoint. Such comparison and rendering may be performed for each pixel to create an image augmented with the virtual object.

[0008] Thus, the quality of occlusion processing may depend on the quality of depth estimates for real and virtual objects (e.g., low-quality depth estimates for real and virtual objects may result in improper blending between object boundaries). The quality of occlusion processing may also be affected by the latency between the time a virtual object is rendered and the time the virtual object is displayed (e.g., Motion2Render2PhotonLatency). For example, a high latency between rendering and display may result in gaps and / or improper coverage between real and virtual objects. Summary of the invention

[0009] The following presents the summary of one or more aspects in order to provide a basic understanding of these aspects. This summary is not an extensive overview of all expected aspects, and is neither intended to identify the key elements of all aspects nor to describe the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to a more detailed description presented later.

[0010] In some aspects, a method for performing graphics processing at a wearable display device is disclosed. The method includes: rendering a first image based on a position of the wearable display device and contours and depth information for one or more real-world objects associated with the position, the rendered first image including at least one or more virtual objects; reprojecting the first image based on an updated position of the wearable display device that is different from the position, wherein the reprojection includes: reprojecting the contours of the one or more real-world objects based on the updated position; updating the depth information for the one or more real-world objects based on the updated position; updating the depth information for the one or more virtual objects in the first image based on the updated position; and distorting the one or more virtual objects in the first image based on the updated position; rendering a second image using the reprojected contours of the one or more real-world objects, the updated depth information for the one or more real-world objects, the updated depth information for the one or more virtual objects, and the distorted one or more virtual objects; and displaying the second image on a display of the wearable display device.

[0011] Other aspects provide: an apparatus operable to, configured to, or otherwise adapted to perform the foregoing method and / or those methods described elsewhere herein; a non-transitory computer-readable medium comprising instructions that, when executed by a processor of the apparatus, cause the apparatus to perform the foregoing method and those methods described elsewhere herein; a computer program product embodied on a computer-readable storage medium, the computer-readable storage medium comprising code for performing the foregoing method and those methods described elsewhere herein; and / or an apparatus comprising components for performing the foregoing method and those methods described elsewhere herein. By way of example, an apparatus may include a processing system, a device having a processing system, or a processing system cooperating through one or more networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] For purposes of illustration, the following description and drawings set forth certain features.

[0013] Figure 1 is a simplified illustration of an example of a wearable display device worn on a user's head in accordance with certain aspects of the present disclosure.

[0014] Figure 2A and Figure 2B is a block diagram illustrating an example rendering and display system configured to implement one or more aspects of the present disclosure.

[0015] Figure 3A Example augmented reality (AR) images generated without using occlusion techniques in accordance with certain aspects of the present disclosure are illustrated.

[0016] Figure 3B Illustrated are example AR images generated using occlusion techniques in accordance with certain aspects of the present disclosure.

[0017] Figure 4 is a timing diagram illustrating example latency caused by split rendering according to certain aspects of the present disclosure.

[0018] Figure 5 Example image degradation in a high-latency split extended reality (XR) system in accordance with certain aspects of the present disclosure is illustrated.

[0019] Figure 6 is a conceptual diagram illustrating an example process for occlusion-based rendering of real and virtual objects using depth maps and / or depth textures in accordance with certain aspects of the present disclosure.

[0020] Figure 7 is a conceptual diagram illustrating an example process for reprojection in occlusion-based rendering using a depth map and / or a depth texture, in accordance with certain aspects of the present disclosure.

[0021] Figure 8 is a conceptual diagram illustrating an example process for occlusion-based rendering of real and virtual objects using approximate depth values, in accordance with certain aspects of the present disclosure.

[0022] Fig. 9 is a conceptual diagram illustrating an example process for reprojection in occlusion-based rendering using approximate depth values, in accordance with certain aspects of the present disclosure.

[0023] Fig.10 is a conceptual diagram illustrating an example process for occlusion-based rendering of real and virtual objects using approximate depth values ​​and outlines of virtual objects in accordance with certain aspects of the present disclosure.

[0024] Fig.11 is a conceptual diagram illustrating an example process for reprojection in occlusion-based rendering using approximate depth values ​​and silhouettes of virtual objects in accordance with certain aspects of the present disclosure.

[0025] Fig.12 is a conceptual diagram illustrating an example process for reprojection in occlusion-based rendering using a depth map and / or depth texture for mixed reality (MR) applications, in accordance with certain aspects of the present disclosure.

[0026] Fig.13 is a conceptual diagram illustrating an example process for reprojection in occlusion-based rendering using approximate depth values ​​for MR applications, in accordance with certain aspects of the present disclosure.

[0027] Fig.14 is a flow diagram illustrating example operations for graphics processing at a wearable display device in accordance with certain aspects of the present disclosure. DETAILED DESCRIPTION

[0028] The present disclosure provides techniques for graphics processing, and more particularly, provides techniques for post-occlusion rendering in extended reality (XR) systems.

[0029] For example, optical see-through (OST) AR is an example of XR in which virtual content such as computer graphics, text, or video is presented to a user (e.g., a user of an OST display device) via a transparent optical system through which the real world remains visible. On the other hand, video see-through (VST) MR is an example of XR in which virtual content and real images are merged and displayed to a user (e.g., a user of a VST display device). In either system (e.g., AR or MR), when rendering virtual content in a real-world scene (e.g., performing graphics processing), the relative depth of the virtual content relative to the depth of real-world objects in the scene may be considered to ensure that the virtual content is properly occluded relative to the real-world objects in the scene. By occluding the virtual content in this manner, a realistic depiction of the virtual content within the scene may be achieved.

[0030] In some cases, such graphics processing for an XR system is performed at a computing system, such as integrated into and / or communicatively coupled to an OST (e.g., in AR) device, a VST (e.g., in MR) device (referred to herein as a wearable display device), etc. In some cases, such graphics processing may be distributed between a computing system and a rendering server communicatively coupled to the computing system, such as using wired or wireless communications (e.g., over a network). While utilizing split rendering may help offload portions of the graphics processing performed at the computing system, split rendering may result in increased latency (e.g., due to processing latency, network jitter, and / or network irregularities when communicating rendered images from the rendering server to the computing system of the wearable display device).

[0031] In the case of rendering latency, the real-world scene may change between the time when the real-world scene is first captured and used to render the virtual object and the time when the rendered virtual object is displayed as an augmented object in the real-world scene. For example, in some aspects, the real-world scene may be updated (e.g., between these two times) due to a change in the position of the wearable display device (e.g., due to a head movement of a user of the wearable display device), even if the object in the real-world scene remains stationary. In some aspects, the real-world scene may be updated due to one or more moving real-world objects in the real-world scene while the wearable display device remains in a constant position. In some aspects, the real-world scene may be updated due to both a change in the position of the wearable display device and one or more moving real-world objects in the real-world scene. Therefore, when the virtual object is displayed on the display of the wearable display device as an augmented object in the updated real-world scene, the rendered virtual object may not be properly aligned with the real-world object in the updated scene. Therefore, a user viewing the displayed image may see a gap between the virtual object and the real-world object and / or an improper overlap of the virtual object and the real-world object in the real-world scene displayed for the user.

[0032] Therefore, aspects of the present disclosure provide techniques for post-occlusion rendering in XR applications. The post-occlusion rendering techniques described herein can be used to reproject depth information for real-world objects in addition to virtual objects in a rendered image due to changes in real-world objects displayed at a wearable display device after the image has been rendered (e.g., due to changes in the position of the wearable display device and / or changes in the position of the real-world objects). When modifying a previously rendered image, the reprojected real depth and virtual depth can be used to perform occlusion rendering. In addition, in some aspects, the outline of the real-world object can also be reprojected based on the reprojected depth to improve occlusion at the updated position.

[0033] Reprojection of real-world objects (e.g., updating depth information for real-world objects and reprojecting outlines of real-world objects) can allow for more accurate depth and outline information of real-world objects when occlusion is performed prior to display of an image. Thus, the techniques presented herein can help ensure that virtual content and / or real-world scenes are correctly occluded relative to each other (even in the case of high latency due to the implementation of split rendering techniques). As a result, a more realistic depiction of virtual objects within a real-world scene can be achieved.

[0034] Various aspects of the system, device, computer program product, and method will be described more fully below for image processing. However, the present disclosure can be embodied in many different forms and should not be construed as being limited to any specific structure or function presented throughout the present disclosure. On the contrary, these aspects are provided so that the present disclosure will be thorough and complete, and the scope of the present disclosure will be fully conveyed to those skilled in the art. Based on the teachings of this article, those skilled in the art should understand that the scope of the present disclosure is intended to cover any aspect of the system, device, computer program product, and method disclosed herein, whether it is implemented independently of other aspects of the present disclosure or implemented in combination with other aspects of the present disclosure. For example, any number of aspects set forth herein may be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover the use of other structures, functionality, or structures and functionality other than the various aspects of the present disclosure set forth herein to implement such devices or practice methods. Any aspect disclosed herein may be embodied by one or more elements of the claims.

[0035] Although various aspects are described herein, many variations and permutations of these aspects fall within the scope of the present disclosure. Although some potential benefits and advantages of various aspects of the present disclosure are mentioned, the scope of the present disclosure is not intended to be limited to specific benefits, uses or objectives. Specifically, various aspects of the present disclosure are intended to be broadly applicable to different graphics technologies, system configurations, etc., some of which are illustrated by way of example in the drawings and the description below. The specific embodiments and drawings merely illustrate the present disclosure and do not limit the present disclosure. The scope of the present disclosure is defined by the attached claims and their equivalent technical solutions.

[0036] Several aspects are presented with reference to various devices and methods. These devices and methods are described in the following detailed description and illustrated in the accompanying drawings by various blocks, components, circuits, processes, algorithms, etc. (collectively referred to as "elements"). These elements can be implemented using electronic hardware, computer software, or any combination thereof. Whether these elements are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0037] For example, an element or any part of an element or any combination of elements may be implemented as a "processing system" including one or more processors (which may also be referred to as a processing unit). Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), general-purpose GPUs, central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems on chip (SOCs), baseband processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gating logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionalities described in this disclosure. One or more processors in a processing system may execute software. Software may be broadly interpreted as instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software components, applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, processes, functions, etc., whether referring to software, firmware, middleware, microcode, hardware description languages, or other names. The term "application" may refer to software. As described herein, one or more technologies may refer to applications configured to perform one or more functions, i.e., software. In such examples, the application may be stored in a memory (e.g., on-chip memory of a processor, system memory, or any other memory). The hardware described herein (such as a processor) may be configured to execute the application. For example, an application may be described as including code that, when executed by hardware, causes the hardware to perform one or more techniques described herein. As an example, the hardware may access code from the memory and execute the code accessed from the memory to perform one or more techniques described herein. In some examples, components are identified in the present disclosure. In such examples, the components may be hardware, software, or a combination thereof. Each component may be a separate component or a subcomponent of a single component.

[0038] Therefore, in one or more examples described herein, the functions described may be implemented in hardware, software, or any combination thereof. If implemented in software, the functions may be stored or encoded on a computer-readable medium as one or more instructions or codes. Computer-readable media include computer storage media. Storage media may be any available medium that can be accessed by a computer. By way of example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, a combination of computer-readable media of the above types, or any other medium that may be used to store computer executable code in the form of instructions or data structures that can be accessed by a computer.

[0039] Figure 11 is a simplified illustration 100 of an example of a wearable display device 120 worn on the head of a user 110 that can be used with or implement one or more of the aspects described herein. The wearable display device 120 includes a body or frame similar to a pair of glasses that can rest on the nose and ears of the user. Portions of the display of the wearable display device 120 can be positioned in front of the user's eyes like the lenses of a pair of glasses. In some aspects, the display is opaque (e.g., for MR). In some aspects, at least a portion of the display can be transparent (e.g., for AR) so that within the user's field of view, the user 110 can see both the user's surroundings and the image (e.g., data elements) displayed by the display. The level of transparency of the image shown on the display can vary depending on the desired functionality of the display, the settings of a graphical user interface (GUI) shown with the display, and / or the software application (e.g., video, map, Internet browser, etc.) executed by the wearable display device 120.

[0040] In addition, the wearable display device 120 may include an interface such as a speaker (and / or headphones), a touchpad, buttons, etc. to interact with the user 110. On the other hand, the wearable display device 120 may include a wireless interface for connecting to the Internet, a local wireless network, or another computing device. The wearable display device 120 may also be operable to receive audio and / or gesture input from the user. Such gestures or audio inputs may be spoken voice commands or identified user gestures, which, when identified by the computing device, may cause the device to execute a corresponding command. Gestures may be received from one or more motion sensors and / or cameras integrated into the wearable display device 120 and / or communicatively coupled to the wearable display device to allow the wearable display device to track one or more objects, such as a user's hand, and determine whether the object has made a gesture. In addition, in some aspects, the wearable display device 120 may include a camera and / or other sensors to determine the user's gaze (e.g., where the user is looking). This eye tracking functionality may be incorporated into gestures and / or other user inputs.

[0041] In addition, the wearable display device 120 may include and / or be communicatively coupled to a motion sensor, such as one or more accelerometers, magnetometers, gyroscopes, global positioning system (GPS) receivers (and / or other satellite positioning system (SPS) receivers), altimeters, cameras, infrared sensors, etc., to sense the position (e.g., movement and / or orientation) of the user 110 and / or the user's head. For the purpose of displaying images as described herein, the determination of the user's field of view may be based on sensor information that provides (e.g.,) the detected head orientation / direction of travel of the user. The direction of travel may be determined by, for example, a compass (e.g., a magnetometer) and / or multiple positioning measurements (e.g., SPS position measurements). Altitude and / or contextual data can be used to determine whether the user is looking down or up because the path is tilted. For example, data from a gyroscope may be compared with altitude data to determine whether the head is tilted in the direction of the path tilt where the user is located.

[0042] Such as Figure 2A and Figure 2B A computing system such as computing system 200 of can be integrated into the wearable display device 120 and / or communicatively coupled (wired or wirelessly) to the wearable display device. Thus, the wearable display device 120 can execute software applications such as games, maps, videos, and / or other applications described herein that generate images displayed on the wearable display device 120. Additionally or alternatively, in some aspects, the wearable display device 120 may display software applications executed by one or more devices communicatively coupled to the wearable display device 120 via (for example) a wireless device-to-device protocol (e.g., Bluetooth or WI-FIDIRECT) and / or via a network (e.g., a WIFI network). For example, the wearable display device 120 can be communicatively coupled to one or more mobile devices such as mobile phones, personal media players, gaming systems, and / or other devices, and display information from the mobile devices.

[0043] One or more components of computing system 200 may be shared between different devices, such as a wearable display device, a smartphone, a tablet, a personal computer, or other computing device. In some embodiments, software and other applications may run on a separate device that is communicatively linked to wearable display device 120. In other embodiments, wearable display device 120 may have some or all of computing system 200 integrated therewith.

[0044] Computing system 200 may include one or more components and / or circuits for performing the various functions described herein. In some examples, one or more components of computing system 200 may be components of a system on a chip (SOC) or an integrated circuit (IC). Figure 2A and Figure 2B In the example shown, computing system 200 includes one or more central processing units (CPUs) 202, one or more graphics processing units (GPUs) 204, system memory 206, one or more input / output (I / O) devices 208, communication interface 210, and display driver 212. In some aspects, communication interface 210 includes a transceiver 214 (e.g., including a receiver 216 and a transmitter 218) to permit communication with a server (via a network, as described with respect to FIG. 3), a network, other computer systems, and / or any other electrical device / peripheral device. In some aspects, GPU 204 includes a reprojection engine 220 (described in more detail below).

[0045] In some aspects, the components of computing system 200 are communicatively connected to each other via one or more buses 150. It should be noted that although Figure 1 1 and 2. Some connections between components are shown in FIG. 1, but the connections between components may be different from the connections shown. For example, all components may be coupled to a shared bus 150, different components may be coupled to different buses, and so on.

[0046] The display driver 240 is configured to drive a display on the wearable display device 120. In particular, the GPU 204 may be configured to perform graphics processing. The display driver 240 may be configured to drive the display of frames processed by the GPU 204 and / or the CPU 202 on the wearable display device 120. In some examples, the display at the wearable display device 120 includes one or more of the following items: a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, a projection display device, a VR / AR / MR display device, a head-mounted display, or any other type of display device. When discussing displaying an image on the wearable display device 120, such display refers to displaying an image on a display of the wearable display device 120. In some aspects, the display driver 240 is configured to drive visual representations of virtual objects and real-world objects.

[0047] Memory external to GPU 204 and CPU 202, such as system memory 206 (described in more detail below), can be accessed by GPU 204 and CPU 202. For example, GPU 204 and CPU 202 can be configured to read from and / or write to external memory, such as system memory 206. GPU 204 and CPU 202 can be communicatively coupled to system memory 206 via a bus. In some examples, GPU 204 and CPU 202 can be communicatively coupled to each other via a bus or a different connection.

[0048] GPU 204 may be a general-purpose GPU or any other processing unit that may be configured to perform graphics processing. In some examples, GPU 204 may be integrated into a motherboard. In some examples, GPU 204 may be present on a graphics card installed in a port in a motherboard, or may be otherwise incorporated into a peripheral device configured to interoperate with computing system 200. GPU 204 may include one or more processors, such as one or more microprocessors, GPUs, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If these techniques are partially implemented in software, GPU 204 may store instructions for the software in a suitable, non-transitory computer-readable storage medium (e.g., system memory 206), and may use one or more processors to execute instructions in hardware to perform the techniques of the present disclosure. Any of the above (including hardware, software, a combination of hardware and software, etc.) may be considered to be one or more processors.

[0049] CPU 202 may be any processing unit configured to transmit instructions to GPU 204 and perform general computing processing (e.g., non-graphics processing). In some examples, CPU 202 may be integrated into a motherboard. CPU 202 may include one or more processors, such as one or more microprocessors, ASICs, FPGAs, ALUs, DSPs, video processors, discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If these techniques are partially implemented in software, CPU 202 may store instructions for the software in a suitable, non-transitory computer-readable storage medium (e.g., system memory 206), and may use one or more processors to execute instructions in hardware to perform the techniques of the present disclosure. Any of the above (including hardware, software, a combination of hardware and software, etc.) may be considered to be one or more processors.

[0050] In some aspects, the CPU 202 and / or the GPU 204 are configured to perform graphics processing. In particular, the CPU 202 and / or the GPU 204 may be configured to render an image on a display of the wearable display device 120. For example, the CPU 202 may determine the position of the wearable display device 120 (e.g., three degrees of freedom (3DoF) posture data, 6DoF posture data, etc.) and generate an application command for the GPU 204 to render an image based on the position information. For example, the command may instruct the GPU 204 to render a scene, such as for executing an application. In some aspects, the command may instruct the GPU 204 to render an image having both virtual objects and real objects.

[0051] In some aspects, the commands may instruct the GPU 204 to render an image with both virtual objects and real objects using an occlusion-based rendering technique. Occlusion-based rendering techniques involve identifying portions of a virtual object that should not be rendered and / or displayed to a user because the presence of the object in the physical world prevents the user from viewing where the virtual object will be perceived by the user. Appropriate occlusion-based rendering techniques may be important for achieving realism in AR applications and MR applications.

[0052] For example, according to certain aspects of the present disclosure, Figure 3A An example AR image 300A generated without using masking techniques is illustrated. Figure 3B An example AR image 300B generated using occlusion techniques is illustrated.

[0053] exist Figure 3A and Figure 3B In the example of FIG. 1 , GPU 204 may receive a command from CPU 202 to render an image having a virtual dog object 302 placed in a real-world scene. In the case of not using occlusion-based rendering techniques, such as Figure 3A As shown, the virtual dog object 302 may be placed as an overlay in the real world scene, making it appear as if the virtual dog object 302 is "pasted" on the person 304 in the real world scene. This disrupts natural perception and provides a less immersive experience for the user of the wearable display device displaying the rendered image.

[0054] On the other hand, in the case of using occlusion-based rendering techniques, such as Figure 3B As shown, the virtual dog object 302 can be placed in the real-world scene following the physical laws of light and depth. More specifically, because the virtual dog object 302 (e.g., the pixels of the virtual dog object 302) corresponds to a depth greater than the depth of the person 304 (e.g., the depth of each pixel of the person 304), the virtual dog object 302 can be placed behind the person 304 in the rendered image. In other words, the virtual dog object 302 can be obscured by the person 304 in the real-world scene, thereby providing a more seamless interaction between the real object and the virtual object in the rendered image.

[0055] In some aspects, the GPU 204 is configured to perform a time warping technique as part of graphics processing. Time warping an image refers to changing the rendered image to be displayed on the wearable display device 120 due to the changed position of the wearable display device 120 after the image has been rendered, such as by the GPU 204. For example, the GPU 204 can render the image based on the position of the wearable display device 120. However, such position information is determined at a certain time interval before the GPU 204 actually renders the image (e.g., due to the latency of the CPU, GPU, etc. processing and memory reading and writing). Therefore, the position of the wearable display device 120 may further change between when the position is initially determined, when the image is rendered, and further when the rendered image is displayed on the display of the wearable display device 120. Therefore, after the image is rendered and before the image is displayed on the wearable display device 120, the time warp can be applied to the rendered image, which modifies the rendered image based on the updated position information for the wearable display device 120.

[0056] For example, at a first time, the user may be looking in a first direction. When the user is looking in the first direction, the CPU 202 may determine the position of the wearable display device 120 and request the GPU 204 to render an image based on the position information. The GPU 204 may generate a rendered image based on this position information. However, from the time of the request (e.g., the first time) to the current time (e.g., the second time), the position of the user's eyes or head may have changed. At a second time, the user may be looking in a second direction. Therefore, the GPU 204 may distort the rendered image to account for such changes in the position of the wearable display device 120, and in some cases, to account for changes in the position of the user's eyes from the first time to the second time.

[0057] In certain aspects, the GPU 204 is configured to perform lens warping techniques. Lens warping an image refers to altering a rendered image to be displayed on the wearable display device 120 to correct for any distortion introduced by a lens in the wearable display device 120.

[0058] In some aspects, the GPU 204 includes a reprojection engine 220. The reprojection engine 220 may be configured to perform graphic adjustments on the rendered image based on post-position information associated with the wearable display device 120 (e.g., the pose of the wearable display device 120 when the image is to be displayed, referred to herein as the display pose). The post-position information may be updated 3DoF pose data, 6DoF pose data, etc. for the wearable display device 120. In some aspects, such graphic adjustments include reprojecting the outline of the real-world object based on the post-position information. In some aspects, such graphic adjustments include updating the depth information for the real-world object based on the post-position information. In some aspects, such graphic adjustments include updating the depth information for the virtual object in the rendered image based on the post-position information. In some aspects, such graphic adjustments include distorting (e.g., time warping) the virtual object in the rendered image based on the post-position information. In some aspects, such graphic adjustments include reprojecting the outline of the virtual object in the rendered image based on the post-position information.

[0059] In addition, the reprojection engine 220 may be configured to perform occlusion rendering after performing such graphics adjustments. For example, the reprojection engine 220 may render a second image for display using reprojected outlines of real-world objects, updated depth information for real-world objects, updated depth information for virtual objects, distorted virtual objects, and / or reprojected outlines of real-world objects. A virtual object (e.g., a portion or all of the virtual object) may be hidden (e.g., occluded) by the real-world object in the rendered second image, wherein the updated depth information for the virtual object indicates that the virtual object is behind (e.g., farther than) the real-world object at a later position (e.g., based on the updated depth information for the real-world object). A virtual object (e.g., a portion or all of the virtual object) may not be occluded in the rendered second image, wherein the updated depth information for the virtual object indicates that the virtual object is in front of (e.g., closer than) the real-world object at a later position (e.g., based on the updated depth information for the real-world object). The following is relative to Figure 7 , Fig. 9 , Fig.11 , Fig.12 and Fig.13 The techniques performed by the re-projection engine 220 are described in more detail.

[0060] Re-reference Figure 2A and Figure 2B, system memory 206 may include one or more volatile or non-volatile memory or storage devices. In some examples, system memory 206 may include random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), erasable programmable ROM (EPRAM), EEPROM, flash memory, magnetic data media, or optical storage media, or any other type of memory.

[0061] According to some examples, system memory 206 may be a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or propagating signal. However, the term "non-transitory" should not be interpreted to mean that system memory 206 is non-removable or that its contents are static. As an example, system memory 206 may be removed from a device in which computing system 200 is integrated and moved to another device. As another example, system memory 206 may not be removable from a device in which computing system 200 is integrated.

[0062] One or more I / O devices 208 may include, but are not limited to, one or more GPS receivers (and / or receivers from any other satellite positioning system (SPS)), accelerometers, gyroscopes, magnetometers (and / or other orientation sensors), altimeters, cameras, and / or other sensors capable of detecting position, motion, orientation, light, sound, etc., which may be used as components for collecting position, movement, and other data. In addition, the sensor may be used together with the CPU 202 as a component for detecting movement of the wearable display device 120 on the user's head, as described herein. Other devices such as a touchpad, keyboard, microphone, etc. may also be included. One or more displays and / or other display components, as well as speakers and / or other devices may also be included.

[0063] As mentioned, the communication interface 210 may include a receiver 216 and a transmitter 218. The receiver 216 may be configured to perform any receiving functions described herein with respect to the computing system 200. In addition, the receiver 216 may be configured to receive information from another device or server. The transmitter 218 may be configured to perform any transmitting functions described herein with respect to the computing system 200. For example, the transmitter 218 may be configured to transmit information to another device or server. The receiver 216 and the transmitter 218 may be combined into a transceiver 214. In such examples, the transceiver 214 may be configured to perform any receiving functions and / or transmitting functions described herein with respect to the computing system 200.

[0064] In some aspects, the communication interface 210 may be configured to communicate with a server via a network, and more specifically, as Figure 2BAs shown in , the communication interface 210 can be configured to communicate with the rendering server 102 via the network 106. For example, in some aspects, the rendering of an image or frame can be performed at multiple locations and / or multiple devices, for example, in order to divide the rendering workload between different devices. The rendering workload can be split between the rendering server 102 and the computing system 200 (e.g., integrated into the wearable display device 120 and / or communicatively coupled with the wearable display device). This process can be referred to as "split rendering".

[0065] The rendering server 102 may be any device capable of performing graphics processing offloaded from the computing system 200. For example, the rendering server 102 may be a cloud server, an edge server, a personal computer, a smart phone, or any suitable device. Split rendering may distribute rendering graphics processing between the rendering server 102 and the computing system 200 in a variety of different ways (e.g., 70% of the processing at the rendering server 102 and 30% of the processing at the computing system 200, 50% of the processing at the rendering server 102 and 50% of the processing at the computing system 200, etc.).

[0066] In some aspects, split rendering is a method for providing content (e.g., virtual content) to a wearable display device 120 for display, wherein a portion of the computer or graphics processing for rendering such content is performed outside of the computing system 200 (e.g., at a rendering server 102). The content can be rendered on the rendering server 102 and then encoded and sent to the computing system 200. The computing system 200 can then decode the content and reproject and warp the content to the nearest user position (e.g., pose), and display the content at the wearable display device 120 via a display driver 212 on the computing system 200.

[0067] According to various aspects described herein, the computing system 200 may also reproject depth information for real-world objects, reproject outlines for real-world objects, and / or distort real-world objects (e.g., in an MR system) to help accurately display content at the wearable display device 120. For example, while split rendering can help offload portions of the graphics processing performed at the computing system 200 to the rendering server 102, split rendering can result in high latency, and more specifically, high motion-to-rendering-to-photon latency or M2R2P latency. Latency is the difference between action and reaction. In XR, M2R2P latency refers to the delay between movement of the wearable display device 120 (e.g., movement of the user's head) and changes in the display of the wearable display device 120 that reflect the user's movement. In other words, M2R2P latency is the amount of time it takes to render an image at the rendering server 102, transmit the rendered image over the network 106, and display the rendered image on the wearable display device 120 (e.g., from the time the user's head moves / the wearable display device 120 moves). The greater the delay (e.g., latency) between physical movement and display output, the less realistic the display of virtual content will appear. In some cases, when the rendered image is communicated from the rendering server 102 to the computing system 200 (e.g., of the wearable display device 120), the M2R2P latency may be even further increased due to frame duplication caused by network jitter and / or network irregularities.

[0068] For example, in the presence of high M2R2P latency due to split rendering, real-world objects in the real-world scene may have moved when the rendered image is to be displayed (e.g., from the time the user's head moves to the time the rendered image is to be displayed). As a result, depth values ​​and / or outlines of real-world objects in the real-world scene to be augmented with virtual objects rendered by the rendering server 102 may have changed. Therefore, when occluding virtual objects in the real-world scene, outdated depth values ​​and outlines of the real-world objects may be used, thus resulting in poor occlusion processing. More specifically, when performing occlusion, gaps and / or improper coverage between real objects and virtual objects may result in images being displayed to the user due to inaccurate depth estimates and outlines of the real-world objects.

[0069] Figure 44 is a timing diagram 400 illustrating an example M2R2P latency caused by split rendering according to certain aspects of the present disclosure. As shown in the timing diagram 400, at time 402, the position (e.g., posture) of the wearable display device 120 is determined (e.g., the user posture is sampled). In some aspects, the I / O device 208 on the computing system 200 determines the position of the wearable display device 120 and communicates the position to the rendering server 102. The rendering server 102 renders a frame with a virtual object based on the position of the wearable display device 120, and encodes the frame at time 404 for transmission to the computing system 200. At time 406, the computing system 200 receives the frame from the rendering server 102 and decodes it. The decoded frame can be displayed to the user of the wearable display device 120 via the display driver 212 at the computing device 120. Time 408 represents the time at which the frame is to be displayed to the user of the wearable display device 120. However, frame duplication may occur due to jitter and / or irregularities in the network 106. Thus, the frame may be displayed to the user at time 410 instead of time 408. Thus, such repeated frames may result in increased M2R2P latency. High M2R2P latency may result in frames being displayed to the user when the position of the wearable display device 120 has changed (and / or the real-world object has changed since the original pose was determined for rendering). Thus, high M2R2P latency may result in degradation of the image displayed to the user of the wearable display device 120.

[0070] For example, Figure 5 An example image degradation in a high latency split XR system according to certain aspects of the present disclosure is illustrated. Figure 5As shown, example rendered images 502, 504, and 506 may include virtual animal objects placed in a real-world scene. However, due to high M2R2P latency, the real-world scene and the virtual animal objects may be misaligned, thereby resulting in (1) improper overlap between the virtual animal objects and objects in the real-world scene, as shown in rendered image 502, and / or (2) gaps between the virtual animal objects and objects in the real-world scene, as shown in rendered images 504 and 506. For example, in some aspects, the position of the wearable display device 120 when the virtual animal objects in each of the images 502, 504, and 506 are rendered may be different from the position of the wearable display device 120 when the virtual animal objects are presented in the real-world scene for display to the user, thereby resulting in conflicts and / or gaps between the real-world objects and the virtual world objects (e.g., when such masking techniques are used). In some aspects, the position of the wearable display device 120 when the virtual animal object in each of the images 502, 504, and 506 is rendered may be the same as the position of the wearable display device when the virtual animal object is rendered; however, real-world objects in the real-world scene may have moved from the time the virtual object was rendered to the display of the virtual object, thereby causing conflicts and / or gaps between the real-world and virtual-world objects.

[0071] Therefore, techniques for mitigating M2R2P latency to provide more seamless masking between virtual objects and real objects displayed to a user are desired.

[0072] Example post-processing occlusion rendering for extended reality (XR)

[0073] Various aspects of the present disclosure provide techniques for post-occlusion rendering in extended reality (XR) applications. The post-occlusion rendering techniques described herein can be used to reproject depth information for real-world objects and virtual objects in a rendered image due to changing content of a real-world scene, where the virtual objects will be displayed as augmented objects in the real-world scene. In particular, in the case of rendering latency, the real-world scene may change between the time when the real-world scene is first captured and used to render the virtual object and the time when the rendered virtual object is displayed as an augmented object in the real-world scene. For example, the real-world scene may be updated (e.g., between these two times) due to the following reasons: (1) a change in the position of the wearable display device (e.g., due to head movement of a user of the wearable display device) and / or (2) one or more dynamic real-world objects in the real-world scene.

[0074] When modifying a previously rendered image, the re-projected real depth and virtual depth can be used to perform occluded rendering. In particular, the re-projected real depth and virtual depth can be taken into account to ensure that portions of virtual objects and / or real-world objects are correctly occluded relative to each other when modifying a rendered image. By occluding images in this manner, realistic depictions of virtual objects and real-world objects can be achieved even in the presence of high motion-to-rendering-to-photon (M2R2P) latency, such as when using split rendering techniques.

[0075] For example, an initial image with both real-world objects and virtual objects may be rendered for an XR application based on the position of the wearable display device. However, such position information is determined at some time interval before the image is rendered and displayed to the user of the wearable display device (e.g., due to processing latency, network jitter, and / or network irregularities present when split rendering is used). Therefore, the position of the wearable display device may further change between when the position is initially determined, when the image is rendered, and further when the rendered image is displayed on the display of the wearable display device. In addition, real-world objects in a real-world scene displayed by the wearable display device may change between when the position is initially determined, when the image is rendered, and further when the rendered image is displayed on the display of the wearable display device. Therefore, reprojection and masking techniques may be applied after the image is rendered and before the image is displayed on the wearable display device.

[0076] In addition, in some aspects, distortions may be applied to virtual objects (and / or real-world objects in mixed reality (MR) use cases) of a rendered image to change the virtual objects (and / or real-world) objects of the rendered image to be displayed in the wearable display device due to the changed position of the wearable display device. For example, reprojected depth information associated with the virtual object (and / or reprojected depth information associated with the real-world object in MR use cases) can be used to move the virtual object (and / or real-world object) in the image to give parallax based on the updated position. Applying post-reprojection, distortion, and / or masking techniques to the rendered image before displaying the image may allow the rendered image to be modified based on changes to the real-world scene between when the real-world scene is first captured and used to render the virtual object and when the rendered virtual object is displayed as an augmented object in the real-world scene (e.g., due to the updated position of the wearable display device and / or the updated position of the real-world object in the image).

[0077] Thus, the techniques described herein can help mitigate or eliminate image degradation caused by high M2R2P latency. In particular, the techniques described herein can be used to modify rendered images so that virtual objects and / or real-world objects in the rendered images are properly shaded to provide more seamless interactions between such objects (e.g., eliminating gaps and / or conflicts between virtual objects and / or real-world objects in the displayed image, such as with respect to Figure 5 A more seamless interaction between virtual objects and real-world objects can provide a more immersive and real-world experience for users of wearable display devices that display rendered images.

[0078] Figure 6 is a conceptual diagram 600 illustrating an example process for occlusion-based rendering of real and virtual objects using depth maps and / or depth textures, in accordance with certain aspects of the present disclosure.

[0079] like Figure 6 As shown, conceptual diagram 600 includes a server device (e.g., a smart phone, a remote server, an edge server, etc.) such as rendering server 102 shown in 2B and a server device such as Figure 1 , Figure 2A and Figure 2B The server device may include an XR device such as the wearable display device 120 shown, and a computer system such as the computer system 200 may be integrated into the server device. The wearable display device 120 is configured to communicate with the rendering server 102 via the network 106 (e.g., for split rendering). Figure 6 2 is described as being integrated within the wearable display device 120 , but in certain other embodiments, the computer system 200 may be communicatively coupled to the wearable display device 120 .

[0080] The wearable display device 120 (eg, computer system 200) includes a plurality of components, such as a perception service component 602, a semantic segmentation component 610, a depth combiner component 612, and a reprojection engine, such as with respect to Figure 2A and Figure 2B The reprojection engine 220 is depicted. Similarly, the rendering server 102 includes multiple components, such as an XR application 604 and a plug-in 606 (eg, a custom game plug-in) having a depth sampling component 608.

[0081] Based on the processing limitations of the wearable display device 120 (e.g., an XR device), the rendering operation is offloaded from the wearable display device 120 to a rendering server 102 connected to the wearable display device 120 via a network 106, such as Figure 6As shown. In particular, the perception service component 602 on the wearable display device 120 determines the position (e.g., posture) of the wearable display device 120 and communicates the position to the rendering server 102 via the network 106. The perception service component 602 may include a 6DoF posture tracker or a 3DoF posture tracker; therefore, the position may include head posture information and / or anchor posture information. An anchor point can be a specific posture in a physical environment that can be tracked. In some aspects, a controller is connected to the wearable display device 120 or the rendering server 102 for additionally or alternatively providing controller input to the rendering server 102.

[0082] The rendering server 102 executes the XR application 604 to render virtual content in the eye buffer and / or generate a depth buffer based on the position information (e.g., head gesture input, controller input, and / or anchor gesture input, etc.) received from the perception service component 602 and / or controller of the wearable display device 120.

[0083] The eye buffer includes a virtual object (or more than one virtual object) based on the received position information, and the generated virtual object is transmitted to the eye buffer encoder 618. The eye buffer encoder 618 on the rendering server 102 encodes the eye buffer and transmits the encoded eye buffer (e.g., as an encoded eye buffer of High Efficiency Video Coding (HEVC) / H.264) to the eye buffer decoder 620 on the wearable display device 120 via a wireless network (e.g., network 106).

[0084] The depth buffer determines a depth value for each pixel of the generated virtual object (or each of the generated virtual objects) and stores these depth values ​​in a depth texture or depth map. For example, the distance of each pixel of the virtual object from the wearable display device (e.g., at a location provided to the XR application 604) may be stored in a depth texture or depth map. In some aspects, the depth texture / depth map may be transmitted to a depth sampling component 608 of the plug-in 606. The plug-in 606 is a software component configured to perform a specific function. The depth sampling component 608 is configured to provide a sampled depth texture / depth map to the depth encoder 614. In some aspects, the depth sampling component 608 can help reduce the depth buffer size. In particular, the depth sampling component 608 can be used to sample the depth texture / depth map in order to reduce the transmission bandwidth (e.g., to the wearable display device 120). The depth sampling component 608 may be part of the depth encoder 614 in some aspects, or may be separate from the depth encoder 614 in other aspects, such as Figure 6The depth encoder 614 on the rendering server 102 encodes the depth texture / depth map and transmits the encoded depth texture / depth map to the depth decoder 616 on the wearable display device 120 via a wireless network (eg, the network 106 ).

[0085] In some cases, the XR application 604 may not submit a depth buffer, but as part of the plugin 606, a depth texture / depth map may be determined from the left and right stereo eye buffers.

[0086] The depth decoder 616 decodes the coded depth texture / depth map and provides the decoded depth texture / depth map to the reprojection engine 220. Similarly, the eye buffer decoder 620 decodes the coded eye buffer and provides the decoded eye buffer to the reprojection engine 220. The decoded eye buffer may be a rendered virtual object or a rendered image.

[0087] In addition to the perception service component 602 including a 6DoF / 3DoF pose tracker, the perception service component 602 may also include an image sensor, a three-dimensional (3D) reconstruction component, and / or a depth for stereo (DFS) component.

[0088] The image sensor is configured to continuously (or periodically) capture images of the real world (e.g., real images) and provide these real images to the semantic segmentation component 610. The semantic segmentation component 610 is configured to perform semantic segmentation or image segmentation in real time. Semantic segmentation is the process of clustering parts of an image belonging to the same object class together. Different instances of the same object class can be given the same label. For example, if there are two cats in the real image provided to the semantic segmentation component 610, the semantic segmentation component can give the same "cat" label to all pixels in the image corresponding to each of the two different cats. Therefore, the semantic segmentation component 610 is a real-time semantic segmenter for segmenting static objects and dynamic objects (e.g., referred to as real-world objects herein) in real-world images to generate segmentation maps. Such segmentation can help identify real-world objects of interest in real images. More specifically, such segmentation can provide pixel-level fragments (also referred to as "contours" herein) of real-world objects of interest (e.g., people, foreground objects, etc.) in real images. The semantic segmentation component 610 provides these contours of real-world objects to the reprojection engine 220.

[0089] As mentioned, in some aspects, the wearable display device 120 may include a camera and / or other sensors to determine the user's gaze (e.g., where the user is looking). Such eye tracking functionality can be used to identify and subsequently segment objects of interest or regions of interest in a real image. For example, real-world objects in portions of a real image where the eye is focused (e.g., within the fovea) may be identified as objects of interest and subsequently segmented, while real-world objects in portions of the real image where the eye is not focused (e.g., in the periphery) may not be segmented by the semantic segmentation component 610.

[0090] Although aspects herein are described with respect to using the semantic segmentation component 610, in certain other aspects, the instance segmentation component can instead be used to give a unique label to each instance of a particular object in a real image.

[0091] The DFS component (of the perception service component 602) is configured to perform stereoscopic imaging. Stereoscopic imaging is a technique for achieving a 3D effect, thereby adding an illusion of depth to a flat image. In other words, stereoscopic imaging can use this illusion of depth to create a single 3D image, referred to as a stereoscopic image. A stereoscopic image can capture the depth of both static real-world objects and dynamic real-world objects; however, the depth information may not be reliable (e.g., due to noise).

[0092] The 3D reconstruction component (of the perception service component 602) is configured to create a 3D mesh of the real-world scene by integrating several depth maps obtained from stereo images. Although the 3D mesh created by the 3D reconstruction component can provide reliable information about the depth of static objects in the real world (e.g., less noise than the depth map created by the DFS component), the 3D reconstruction component may not be able to capture depth information for dynamic objects.

[0093] Therefore, in some aspects (such as Figure 6 ), the depth combiner component 612 is implemented to combine the outputs from the DFS component and the 3D reconstruction component (e.g., in Figure 6 The depth combiner component 612 is configured to combine the depth map from the 3D reconstruction component and the stereo image from the DFS component to produce a clearer, more reliable and / or more detailed depth map for real-world objects. The depth combiner component 612 provides the depth map for the real-world object to the re-projection engine 220.

[0094] Although aspects herein are described with respect to using a depth combiner component 612 to combine outputs from a DFS component and a 3D reconstruction component (of the perception service component 602) to produce depth information for real-world objects, in certain other aspects, the perception service component 602 may include only a DFS component (e.g., may not have a 3D reconstruction component); thus, stereo images only from the DFS component can be used to create depth maps for real-world objects (although such depth information in the depth maps may be noisy).

[0095] The reprojection engine 220 may use as input the decoded eye buffer (e.g., rendered image), the decoded depth information for the virtual object, the outline of the real-world object, and the depth information for the real-world object (e.g., depth map) to graphically adjust the rendered image based on the updated position of the wearable display device 120. The updated position may be referred to herein as the display pose P. D , the display posture represents the posture of the wearable display device 120 when the image is to be displayed (e.g., as described above, after taking into account the M2R2P latency present in split rendering). In some aspects, graphically adjusting the rendered image occurs immediately before displaying the adjusted rendered image.

[0096] As compared to Figure 7 In more detail, graphically adjusting the rendered image may include: (1) reprojecting an outline of a real-world object based on a display pose; (2) updating depth information for the real-world object based on the display pose; (3) updating depth information for the virtual object based on the display pose; and (4) warping (e.g., time warping) the virtual object based on the display pose. In some aspects, such as with respect to Fig.11As described, graphically adjusting the rendered image may also include (5) reprojecting the outline of the virtual object (e.g., wherein the outline of the virtual object is provided to the reprojection engine 220). For example, the reprojection engine 220 may use the reprojected outline of the real-world object, the updated depth information for the real-world object, the updated depth information for the virtual object, the distorted virtual object, and / or the reprojected outline of the real-world object to graphically adjust the rendered image for display at the display posture of the wearable display device 120. The virtual object (e.g., a portion or all of the virtual object) may be hidden (e.g., obscured) by the real-world object in the graphically adjusted image, wherein the updated depth information for the virtual object indicates that the virtual object is behind the real-world object at a later position (e.g., farther than the real-world object) (e.g., based on the updated depth information for the real-world object). The virtual object (e.g., a portion or all of the virtual object) may not be obscured in the rendered second, wherein the updated depth information for the virtual object indicates that the virtual object is in front of the real-world object at a later position (e.g., closer than the real-world object) (e.g., based on the updated depth information for the real-world object). The graphically adjusted image with obscured real-world objects and virtual objects may be displayed to a user of the wearable display device 120 .

[0097] Figure 7 is a conceptual diagram 700 illustrating an example process for re-projection in occlusion-based rendering using a depth map and / or a depth texture, in accordance with certain aspects of the present disclosure. Figure 7 The example process illustrated in may occur at a reprojection engine, such as Figure 2A , Figure 2B and Figure 6 A reprojection engine 220 is shown. Figure 7 The example process shown describes in more detail the Figure 6 The operations described by the re-projection engine 220 in FIG.

[0098] like Figure 7 As shown, the reprojection engine 220 and more specifically the reprojection module 702 of the reprojection engine 220 are Figure 6 The semantic segmentation component 610 shown receives the contours of real-world objects as input. The reprojection module 702 Figure 6 The depth combiner component 612 shown receives depth information (e.g., a depth map) for real-world objects as input. The reprojection module 702 Figure 6 The depth decoder 616 shown receives as input the decoded depth information for the virtual object (or multiple virtual objects). The re-projection module 702 receives the decoded depth information for the virtual object (or multiple virtual objects). Figure 6The eye buffer decoder 620 shown receives the decoded eye buffer (e.g., rendered image) as input. In addition, the re-projection module 702 receives the updated position (e.g., display pose P) of the wearable display device 120 from the perception service component 602. D ) as input. The updated position (e.g., display pose P D ) may be different from the location initially determined by the perception service component 602 on the wearable display device 120 and communicated to the rendering server 102 for the purpose of rendering the virtual object. The location may be different due to: (1) a change in the position of the wearable display device (e.g., due to head movement of a user of the wearable display device) and / or (2) one or more dynamic real-world objects moving in the real-world scene.

[0099] As a first step (in Figure 7 At 1), the reprojection module 702 reprojects the received depth information. Specifically, the reprojection module 702 performs the following operations: (1) updates depth information (e.g., depth map / depth texture) for real-world objects based on the display pose; and (2) updates depth information (e.g., depth map / depth texture) for virtual objects based on the display pose. The depth information may need to be updated because the depth information provided to the reprojection module 702 represents depth information for real-world objects and virtual objects in a rendered pose (as opposed to a displayed pose).

[0100] As a second step (in Figure 7 2), the reprojection module 702 reprojects the contour of the real-world object. The reprojection module 702 performs the reprojection using the contour of the real-world object, the display pose, and the updated depth information for the real-world object (e.g., determined at the first step). The reprojection module 702 performs the reprojection using depth-based reprojection.

[0101] As the third step (in Figure 7 3), the reprojection module 702 distorts the image content contained in the virtual eye buffer (e.g., distorts the virtual objects in the rendered image). The reprojection module 702 performs the distortion using the virtual eye buffer, the display pose, and the updated depth information for and for the virtual objects (e.g., determined at the first step).

[0102] although Figure 7 The example illustrates image distortion occurring after reprojection of the outline of the real-world object has occurred (e.g., at the third step), but in some aspects, the third step may be performed before the second step, and in some other aspects, the second and third steps may be performed simultaneously.

[0103] The image to be displayed at the display of the wearable display device 120 can be rendered based on: (1) updated depth information for real-world objects and virtual objects (e.g., determined at the first step); (2) reprojected outlines for the real-world objects at the display pose; and (3) a warped eye buffer including distorted content (e.g., distorted virtual objects) at the display pose. A compositor module such as the compositor module 704 of the reprojection engine 220 can be configured to render an image to be displayed at the display of the wearable display device 120.

[0104] The compositor module 704 may compare the reprojected depth of the virtual object with the reprojected depth of the real-world object to ensure that the virtual object and / or the real-world object are correctly occluded relative to each other when rendering an image for display. For example, the reprojected outline for the real-world object may include a plurality of first pixels, and the warped eye buffer (e.g., the virtual object) may include a plurality of second pixels. The compositor module 704 may determine which second pixels overlap with which first pixels, and for these overlapping pixels, perform a depth comparison to determine how to accurately occlude the virtual object and / or the real-world object. For example, in the case where a second pixel of the warped eye buffer overlaps with a first pixel of the real-world object, the compositor module 704 may determine an updated depth of the second pixel and an updated depth of the first pixel. Given that a pixel with a smaller depth is a pixel closer to the wearable display device 120 in a display posture, the smaller depth pixel can be used to render the image. The compositor module may perform this process for a plurality of pixels until the image is rendered. In other words, the virtual object is rendered without occlusion at the rendering server 102 and transmitted to the reprojection engine 220. The reprojection engine 220, and more specifically, the reprojection module 702 and the compositor module 704 may be configured to modify rendered virtual objects to account for occlusions at a display pose (e.g., an update pose) of the wearable display device 120.

[0105] In some aspects, the real-world object captured in the rendered image is a moving object. Therefore, in order to accurately adjust the rendered image based on the updated position of the wearable display device 120 (e.g., for display at the wearable display device 120), a motion vector (e.g., an optical flow vector) corresponding to the moving object may be determined and provided to the reprojection engine 220. For example, the motion vector may be determined by comparing a current image (e.g., a current frame, a current rectangular image, etc.) with one or more past images (e.g., a previous frame, a previous rectangular image, etc.). In some aspects, the images used to determine the motion vector may be taken at different times (from the same lens, sensor, camera, etc.). The motion vector may indicate the amount of movement between images (e.g., a current frame and a previous frame). For example, the motion vector may indicate the distance (e.g., in pixels) that the object has moved between images. Additionally or alternatively, the motion vector may be based on motion sensor (e.g., accelerometer) data. For example, the motion vector may indicate the amount of movement (e.g., rotation, translation, etc.) of the real-world object of interest between images. The motion vector may indicate the speed at which the real-world object of interest is moving (e.g., pixels per frame, inches per second, etc.). The reprojection engine 220 may determine the position of the real-world object at the updated position based on the velocity, and more specifically, determine whether the real object overlaps with the virtual object at the updated position (e.g., for occlusion purposes) based on the determined position of the real-world object. In addition, the reprojection engine 220 may use the motion vector corresponding to the real-world object to: (1) update the depth information for the real-world object (e.g., at Figure 7 ); and (2) reprojecting the contours of the real-world object (e.g., Figure 7 exemplified in the second step in ).

[0106] Figure 8 is a conceptual diagram 800 illustrating an example process for occlusion-based rendering of real and virtual objects using approximate depth values, in accordance with certain aspects of the present disclosure. Figure 8 The example process shown is similar to Figure 6 However, instead of using Figure 6 The depth sampling component 608 in the embodiment is used to obtain a depth map / depth texture for the virtual object, such as Figure 8The plane modeling component 802 shown can be used to generate 3D plane data for a virtual object, where the 3D plane data is a representation of the depth buffer data for the virtual object. For example, the entire visible surface of the virtual object can be approximated as a single plane in 3D space, i.e., using a plane parallel to the image plane of the rendering camera. By calculating the optimal (scene-dependent) depth as an approximation of the various depths of the scene, the complete depth information for the virtual object (e.g., the depth map / depth texture for the virtual object) may not need to be sent to the reprojection engine 220 on the wearable display device 120. In other words, the rendering server 102 can approximate the depth of a virtual scene with multiple planes to reduce the bandwidth required when transmitting this depth information to the wearable display device 120 via the network 106. Therefore, when using Figure 6 The depth sampling component 608 or Figure 8 When modeling the plane component 802 as shown, there may be a tradeoff between the bandwidth used and the depth accuracy.

[0107] In addition, if Figure 8 As shown, the plane modeling component 804 can be used instead of Figure 6 The depth combiner component 612 is shown. In particular, instead of determining a depth map / depth texture for a real-world object, the real-world object can be modeled as one or more planes. Therefore, the depth information for the virtual object (or multiple virtual objects) and the real-world object provided to the reprojection engine 220 can be planar data (e.g., approximate depth information), which is contrary to the depth map / depth texture for the virtual object and the real-world object.

[0108] In some cases, the XR application 604 may not submit a depth buffer, but as part of the plugin 606 , plane data (eg, approximate depth information) may be determined from the left and right stereo eye buffers.

[0109] Fig. 9 is a conceptual diagram 900 illustrating an example process for reprojection in occlusion-based rendering using approximate depth values, in accordance with certain aspects of the present disclosure. Fig. 9 The example process shown is similar to Figure 7 However, instead of the reprojection engine receiving a depth map / depth texture for a virtual real-world object, the reprojection engine 220 receives planes (e.g., virtual planes and real planes) for the virtual object and the real-world object. Fig. 9 1), the reprojection module 702 of the reprojection engine 220 reprojects the received virtual plane and real plane instead of the information included in the depth map / depth texture corresponding to the virtual object and the real world object. Fig. 9 At step 2), the reprojection module 702 reprojects the real-world object's outline using the real-world object's outline, the display pose, and the reprojected real-world object (e.g., determined at step 1). Fig. 9 At 3), the reprojection module 702 performs reprojection using the virtual eye buffer, the display pose, and the reprojection plane for the virtual object (eg, determined at the first step).

[0110] The image to be displayed at the display of the wearable display device 120 may be rendered based on: (1) a reprojected plane for real-world objects and virtual objects (e.g., determined in the first step); (2) a reprojected outline for the real-world objects at the display pose; and (3) a distorted eye buffer including distorted content (e.g., distorted virtual objects) at the display pose.

[0111] A compositor module, such as the compositor module 704 of the reprojection engine 220, may be configured to render an image to be displayed at a display of the wearable display device 120. More specifically, the compositor module 704 may compare the reprojected plane of the virtual object to the reprojected plane of the real-world object to ensure that the virtual object and / or the real-world object are correctly occluded relative to each other when rendering the image for display.

[0112] In some aspects, when rendering an image at the display pose, the compositor module 704 does not draw pixels for areas outside the plane boundary for the virtual object (e.g., real-world pixels may be displayed instead). In some aspects, when rendering an image at the display pose, the compositor module 704 does not draw pixels for areas outside the plane boundary for the real-world object (e.g., virtual-object pixels may be displayed instead), given that the real-world object may be a background object and / or an object that may not be of interest. In some aspects, the depth of a pixel located on the outline of the real-world object may be obtained by ray tracing onto a re-projected plane of the real-world object that approximates the re-projected outline for the real-world object at the display pose.

[0113] Fig.10 is a conceptual diagram 1000 illustrating an example process for occlusion-based rendering of real and virtual objects using approximate depth values ​​and outlines of virtual objects, in accordance with certain aspects of the present disclosure. Fig.10 The example process shown is similar to Figure 8 However, except for Figure 8 In addition to the components listed in Fig.10, a semantic segmentation component 1002 is added to the rendering server 102. Similar to the semantic segmentation component 610 on the wearable display device 120, the semantic segmentation component 1002 on the rendering server 102 is configured to perform semantic segmentation or image segmentation in real time to cluster parts of the virtual scene belonging to the same object class. In other words, the semantic segmentation component 1002 can be used to obtain segments (e.g., outlines) of multiple virtual objects in the scene. The semantic segmentation performed by the semantic segmentation component 1002 can be based on machine learning (ML).

[0114] For example, in the case where there are multiple virtual objects in the scene (e.g., the virtual objects in the scene are not sparse), the introduction of the semantic segmentation component 1002 on the rendering server 102 can be used. The outlines of the multiple virtual objects determined by the semantic segmentation component 1002 can be transmitted to the reprojection engine 220, so that the reprojection engine can perform occlusion relative to the virtual objects in the virtual scene when rendering the image at the display posture.

[0115] As an illustrative example, one virtual object may overlap with another virtual object in an image rendered at rendering server 102. Therefore, due to the high M2R2P latency present in the split rendering operation (e.g., Fig.10 ), gaps and / or conflicts may result between the virtual objects themselves (and not just the virtual objects and the real-world objects) when the rendered image is displayed at the display of the wearable display device 120. Therefore, by obtaining the outlines of multiple virtual objects in the virtual scene at the rendering server 102, the reprojection engine 122 may be configured to reproject the outlines of the virtual objects and use the reprojected outlines of the virtual objects to accurately mask the virtual objects relative to each other in the image rendered for display.

[0116] like Fig.10 As shown, the outline of the virtual object determined by the semantic segmentation component 1002 can be transmitted to the outline encoder 1004 on the rendering server 102. The outline encoder 1004 on the rendering server 102 encodes the outline and transmits the encoded outline to the outline decoder 1006 on the wearable display device 120 via a wireless network (e.g., network 106). The outline decoder 1006 decodes the outline for the virtual object and provides the decoded outline for the virtual object to the reprojection engine 220 so that the reprojection engine can accurately mask the virtual objects relative to each other in the display posture.

[0117] In addition, if Fig.10As shown, the outline of the virtual object determined by the semantic segmentation component 1002 can also be provided to the plane modeling component 802. The plane modeling component 802 can use the provided outline to determine the plane boundary of the virtual object defined by the outline.

[0118] Fig.11 is a conceptual diagram 1100 illustrating an example process for reprojection in occlusion-based rendering using approximate depth values ​​and silhouettes of virtual objects, in accordance with certain aspects of the present disclosure. Fig.11 The example process shown is similar to Fig. 9 However, instead of receiving only outlines for real-world objects, the reprojection engine 220 also receives outlines for virtual objects. Fig. 9 At the position marked as 2), the reprojection module 702 (1) performs the following operations: reprojects the contour of the real-world object using the contour, display posture, and reprojection plane for the real-world object (e.g., determined in the first step); and (2) reprojects the contour of the virtual object using the contour, display posture, and reprojection plane for the virtual object (e.g., determined in the first step).

[0119] The image to be displayed at the display of the wearable display device 120 may be rendered based on: (1) a reprojected plane for real-world objects and virtual objects (e.g., determined in the first step); (2) a reprojected outline for the real-world objects and virtual objects at the display pose; and (3) a distorted eye buffer including distorted content (e.g., distorted virtual objects) at the display pose.

[0120] Although not illustrated, in some aspects, occlusion-based rendering of real and virtual objects may be performed using depth maps and / or depth textures (as opposed to approximated depth values) and outlines of virtual objects. Fig.10 The semantic segmentation component 1002 introduced in can be similarly introduced Figure 6 In the embodiment of the present invention, a depth map / depth texture is used for occlusion-based rendering. Specifically, a semantic segmentation component 1002, a contour encoder 1004, and a contour decoder 1006 can be added to the conceptual graph 600 to allow occlusion of virtual objects with other virtual objects (e.g., in the case where there are multiple virtual objects in the virtual scene).

[0121] exist Figure 7 , Fig. 9 and Fig.11The reprojection engine 220 shown performs operations to render images in a display pose with an optical see-through (OST) display technology (e.g., in an AR use case), which allows the real world to be seen in full detail where augmentation is not present. A video see-through (VST) display can provide an alternative to the OST display technology by using live video (e.g., real images) to represent the real world. VST displays can be used for MR use cases. Therefore, in MR situations considering high M2R2P latency when split rendering is used, real images of the real world (e.g., real world objects) may need to be distorted in addition to virtual objects rendered by the rendering server 102.

[0122] Fig.12 is a conceptual diagram 1200 illustrating an example process for re-projection in occlusion-based rendering using a depth map and / or depth texture for MR applications, in accordance with certain aspects of the present disclosure. Fig.12 The example process shown is similar to Figure 7 However, in addition to distorting the virtual eye buffer at the third step (in Fig.12 In addition to the image content contained in the real image (e.g., real-world object) received from the perception service component 602, the reprojection module 702 also distorts the image content contained in the real image (e.g., real-world object) received from the perception service component 602. The reprojection module 702 uses the real image (e.g., real-world object), the display pose, and the updated depth information for the real-world object (e.g., determined at the first step) to perform the reprojection. In some aspects, the real-world object provided to the reprojection module 702 is rendered independently of the virtual object (e.g., where both the real-world object and the virtual object are rendered before the object is provided to the reprojection module 702). For example, the real-world object may be rendered on a computer system 200 that is integrated into the wearable display device 120 and / or communicatively coupled to the wearable display device, while the virtual object may be rendered at the rendering server 102.

[0123] exist Fig.12 In the embodiment of the present invention, both the warped eye buffer and the warped real-world object are provided to the compositor module 704 for occlusion-based rendering. The compositor module 704 can select the real-world object pixels or the virtual object pixels to be displayed based on the re-projected depth for each of these pixels (e.g., where such pixels overlap). For virtual object pixels that do not overlap with real-world object pixels, the compositor module 704 can select the virtual object pixels to be displayed.

[0124] Fig.13 is a conceptual diagram 1300 illustrating an example process for reprojection in occlusion-based rendering using approximate depth values ​​for MR applications, in accordance with certain aspects of the present disclosure. Fig.13The example process shown is similar to Fig. 9 and Fig.11 However, in addition to distorting the virtual eye buffer at the third step (in Fig.13 In addition to the above, the reprojection module 702 also distorts the image content contained in the real image (e.g., the real-world object) received from the perception service component 602. The reprojection module 702 performs the reprojection using the real image (e.g., the real-world object), the display pose, and the updated depth information for the real-world object (e.g., determined at the first step).

[0125] exist Fig.13 In the embodiment of the present invention, both the warped eye buffer and the warped real-world object are provided to the compositor module 704 for occlusion-based rendering. The compositor module 704 can select the real-world object pixels or virtual object pixels to be displayed based on the re-projected plane for each of these pixels (e.g., where such pixels overlap). For virtual object pixels that do not overlap with real-world object pixels, the compositor module 704 can select the virtual object pixels to be displayed.

[0126] As mentioned, in certain aspects, the wearable display device 120 is operable to receive audio and / or gesture input from a user. Such gestures or audio input may be spoken voice commands or identified user gestures. Gestures may be received from one or more motion sensors and / or cameras integrated into and / or communicatively coupled to the wearable display device 120 to allow the wearable display device to track one or more objects, such as a user's hands, and determine whether an object has made a gesture. In certain aspects where the wearable display device 120 is a VST display (e.g., in an MR case), gestures received from one or more motion sensors and / or cameras integrated into and / or communicatively coupled to the wearable display device 120 may be used as input provided to the semantic segmentation component 610 (e.g., as relative to Figure 6 , Figure 8 and Fig.10 ). In particular, the user can optionally select an object of interest (e.g., using a gesture to point to and / or pick an object of interest in the real world), and the selection can be provided to the semantic segmentation component 610. The semantic segmentation component 610 can treat these selected objects of interest as primary foreground objects. The semantic segmentation component 610 can segment the selected objects of interest in the image (e.g., using a gesture), while other real-world objects in the image may not be segmented by the semantic segmentation component 610 (but remain in the background / far plane).

[0127] Example operation for post-occlusion rendering

[0128] Fig.14 1400 is a flow chart illustrating example operations 1400 for graphics processing according to certain aspects of the present disclosure. Operations 1400 may be performed, for example, by integrating into a wearable display device (e.g., Figure 1 , Figure 2A and Figure 2B A computing system (e.g., a wearable display device 120) in and / or communicatively coupled to the wearable display device Figure 2A and Figure 2B The computing system 200 shown is executed.

[0129] Operations 1400 begin at a first block 1402 by rendering a first image based on a position of a wearable display device and contour and depth information for one or more real-world objects associated with the position, the rendered first image including at least one or more virtual objects.

[0130] Operation 1400 continues at a second box 1404 by reprojecting the first image based on an updated position of the wearable display device that is different from the position, wherein the reprojection includes: reprojecting the outline of one or more real-world objects based on the updated position; updating depth information for the one or more real-world objects based on the updated position; updating depth information for one or more virtual objects in the first image based on the updated position; and distorting one or more virtual objects in the first image based on the updated position.

[0131] The operation 1400 continues at a third block 1406 by rendering a second image using the reprojected outlines of the one or more real-world objects, the updated depth information for the one or more real-world objects, the updated depth information for the one or more virtual objects, and the warped one or more virtual objects.

[0132] Operation 1400 continues by displaying the second image on the display of the wearable display device at a fourth block 1408. In some aspects, rendering the second image at the third block 1406 occurs immediately in time before displaying the second image on the display of the wearable display device at the fourth block 1408.

[0133] Sample Clauses

[0134] Specific implementation examples are described in the following numbered clauses:

[0135] Item 1: A method for performing graphics processing at a wearable display device, the method comprising: rendering a first image based on a position of the wearable display device and contours and depth information for one or more real-world objects associated with the position, the rendered first image comprising at least one or more virtual objects; reprojecting the first image based on an updated position of the wearable display device that is different from the position, wherein the reprojection comprises: reprojecting the contours of the one or more real-world objects based on the updated position; updating the depth information for the one or more real-world objects based on the updated position; updating the depth information for the one or more virtual objects in the first image based on the updated position; and distorting the one or more virtual objects in the first image based on the updated position; rendering a second image using the reprojected contours of the one or more real-world objects, the updated depth information for the one or more real-world objects, the updated depth information for the one or more virtual objects, and the distorted one or more virtual objects; and displaying the second image on a display of the wearable display device.

[0136] Clause 2: A method according to clause 1, wherein: the reprojected contour for the one or more real-world objects includes multiple first pixels; the distorted one or more virtual objects include multiple second pixels; and rendering the second image using the reprojected contour for the one or more real-world objects, the updated depth information for the one or more real-world objects, the updated depth information for the one or more virtual objects, and the distorted one or more virtual objects includes: determining that at least a portion of the second pixel overlaps with a portion of the first pixel; determining the depth of each pixel of the portion of the first pixel based on the updated depth information for the one or more virtual objects; determining the depth of each pixel of the portion of the second pixel based on the updated depth information for the one or more virtual objects; and for each pixel of the portion of the second pixel that overlaps with the portion of the first pixel, selecting a pixel to be rendered in the second image based on the depth of each pixel of the portion of the first pixel and the depth of each pixel of the portion of the second pixel.

[0137] Clause 3: The method of any of clauses 1 or 2, wherein re-projecting the outline of the one or more real-world objects is further based on updated depth information for the one or more real-world objects.

[0138] Clause 4: The method according to clause 3, further comprising: calculating a motion vector for at least one of the one or more real-world objects, wherein reprojecting the contours of the one or more real-world objects is further based on the motion vector.

[0139] Clause 5: The method of any one of clauses 1 to 4, wherein warping the one or more virtual objects in the first image is further based on updated depth information for the one or more virtual objects in the first image.

[0140] Clause 6: A method according to any one of clauses 1 to 5, the method further comprising: capturing an image from the perspective of the wearable display device when the wearable display device is in the position; processing the image to determine: the outline of the one or more real-world objects in the image; and the depth information for the one or more real-world objects; receiving the one or more virtual objects from a server; and receiving the depth information for the one or more virtual objects from the server based on the position.

[0141] Clause 7: A method according to clause 6, wherein: the one or more real-world objects include static objects, dynamic objects, or a combination thereof; and processing the image to determine the contours of the one or more real-world objects in the image includes running a semantic segmentation algorithm to identify the contours of the static objects, the dynamic objects, or the combination thereof.

[0142] Clause 8: A method according to any one of clauses 6 or 7, wherein the depth information for the one or more real-world objects comprises one or more of a three-dimensional (3D) reconstruction and understanding (3DRU) mesh or a stereo depth (DFS) map.

[0143] Clause 9: According to the method described in any one of clauses 6 to 8, the method also includes: sending the position of the wearable display device to the server; and receiving the one or more virtual objects and the depth information for the one or more virtual objects from the server in response to sending the position.

[0144] Clause 10: The method of clause 9, further comprising: receiving outlines of the one or more virtual objects from the server.

[0145] Clause 11: A method according to clause 10, wherein reprojecting the first image based on the updated position of the wearable display device that is different from the position further includes: reprojecting the outlines of the one or more virtual objects based on the updated position, wherein the second image is further rendered using the reprojected outlines of the one or more virtual objects.

[0146] Clause 12: A method according to any one of clauses 1 to 11, wherein the depth information for the one or more virtual objects comprises: a depth map, a depth texture, or one or more planes.

[0147] Clause 13: A method according to any one of clauses 1 to 12, wherein: the rendered first image also includes the one or more real-world objects; and re-projecting the first image based on an updated position of the wearable display device that is different from the position also includes distorting the one or more real-world objects in the first image based on the updated position.

[0148] Clause 14: The method of clause 13, wherein the second image is further rendered using the distorted one or more real-world objects.

[0149] Clause 15: A method as described in any one of clauses 1 to 14, further comprising: selecting the one or more real-world objects based on eye tracking data.

[0150] Clause 16: A method as described in any of clauses 1 to 15, wherein rendering the second image occurs immediately before displaying the second image on the display of the wearable display device.

[0151] Clause 17: An apparatus comprising means for performing the method according to any one of clauses 1 to 16.

[0152] Clause 18: A non-transitory computer readable medium comprising executable instructions which, when executed by a processor of an apparatus, cause the apparatus to perform the method of any one of clauses 1 to 16.

[0153] Clause 19: A computer program product embodied on a computer readable storage medium, the computer readable storage medium comprising code for performing the method according to any one of clauses 1 to 16.

[0154] Additional considerations

[0155] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. The examples discussed herein do not limit the scope, applicability, or aspects set forth in the claims. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. For example, without departing from the scope of the present disclosure, the functions and arrangements of the elements discussed may be changed. Various examples may omit, replace, or add various processes or components as appropriate. For example, the described methods may be performed in a different order than the described order, and various actions may be added, omitted, or combined. In addition, the features described for some examples may be combined in some other examples. For example, any number of aspects set forth herein may be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such devices or methods practiced using other structures, functionality, or structures and functionality other than the various aspects of the present disclosure set forth herein or different from the various aspects of the present disclosure set forth herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of the claims.

[0156] The various illustrative logical blocks, modules, and circuits described in conjunction with the present disclosure may be implemented or executed using a general purpose processor, a digital signal processor (DSP), an ASIC, a field programmable gate array (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof, designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any commercially available processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, a system on a chip (SoC), or any other such configuration.

[0157] As used herein, a phrase referring to "at least one of" a list of items refers to any combination of those items (including single members). As an example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c).

[0158] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, searching (e.g., searching in a table, a database, or another data structure), ascertaining, and the like. Also, "determining" may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. Additionally, "determining" may include resolving, selecting, choosing, establishing, and the like.

[0159] The method disclosed herein includes one or more actions for implementing the method. The method actions can be interchangeable with each other without departing from the scope of the claims. In other words, unless a specific order of actions is specified, the order and / or use of specific actions can be modified without departing from the scope of the claims. In addition, the various operations of the method described above can be performed by any suitable component that can perform the corresponding function. The component may include various hardware and / or software components and / or modules, including but not limited to circuits, application specific integrated circuits (ASICs) or processors.

[0160] The following claims are not intended to be limited to the various aspects shown herein, but should be given the full scope consistent with the language of the claims. Within the claims, unless explicitly stated, reference to an element in the singular form is not intended to mean "one and only one", but "one or more". Unless otherwise specified, the term "some" refers to one or more. Any claim element is not interpreted according to the provisions of 35 U.S.C. § 112 (f) unless the element is explicitly stated using the phrase "member for...". All structural and functional equivalents of the elements of the various aspects described throughout this disclosure that are known or will be known later to a person of ordinary skill in the art are expressly incorporated herein by reference and are intended to be covered by the claims. In addition, regardless of whether such disclosure is explicitly stated in the claims, anything disclosed herein is not intended to be disclosed to the public.

Claims

1. A method for performing graphics processing at a wearable display device, the method comprising: Rendering a first image based on the position of the wearable display device and contour and depth information for one or more real-world objects associated with the position, the rendered first image comprising at least one or more virtual objects; re-projecting the first image based on an updated position of the wearable display device that is different from the position, wherein the re-projecting comprises: reprojecting outlines of the one or more real-world objects based on the updated positions; updating depth information for the one or more real-world objects based on the updated positions; updating depth information for the one or more virtual objects in the first image based on the updated positions; and warping the one or more virtual objects in the first image based on the updated positions; rendering a second image using the reprojected outlines of the one or more real-world objects, the updated depth information for the one or more real-world objects, the updated depth information for the one or more virtual objects, and the warped one or more virtual objects; and The second image is displayed on a display of the wearable display device.

2. The method of claim 1, wherein rendering the second image occurs immediately prior to displaying the second image on the display of the wearable display device.

3. The method according to claim 1, wherein: The reprojected contour for the one or more real-world objects comprises a plurality of first pixels; The distorted one or more virtual objects include a plurality of second pixels; and Rendering the second image using the reprojected outlines for the one or more real-world objects, the updated depth information for the one or more real-world objects, the updated depth information for the one or more virtual objects, and the warped one or more virtual objects includes: determining that at least a portion of the second pixel overlaps a portion of the first pixel; determining a depth of each pixel of the portion of the first pixels based on the updated depth information for the one or more virtual objects; determining a depth of each pixel of the portion of the second pixels based on the updated depth information for the one or more virtual objects; and For each pixel of the portion of the second pixels that overlaps the portion of the first pixels, a pixel to be rendered in the second image is selected based on the depth of each pixel of the portion of the first pixels and the depth of each pixel of the portion of the second pixels. 4 . The method of claim 1 , wherein re-projecting the outlines of the one or more real-world objects is further based on updated depth information for the one or more real-world objects.

5. The method according to claim 4, further comprising: A motion vector is calculated for at least one of the one or more real-world objects, wherein reprojecting the contours of the one or more real-world objects is further based on the motion vector. 6 . The method of claim 1 , wherein warping the one or more virtual objects in the first image is further based on updated depth information for the one or more virtual objects in the first image.

7. The method according to claim 1, further comprising: When the wearable display device is in the position, capturing an image from a perspective of the wearable display device; The image is processed to determine: said contours of said one or more real-world objects in said image; and said depth information for said one or more real-world objects; receiving the one or more virtual objects from a server; as well as The depth information for the one or more virtual objects is received from the server based on the position.

8. The method according to claim 7, wherein: The one or more real-world objects include static objects, dynamic objects, or a combination thereof; and Processing the image to determine the outlines of the one or more real-world objects in the image includes running a semantic segmentation algorithm to identify the outlines of the static objects, the dynamic objects, or the combination thereof.

9. The method of claim 7, wherein the depth information for the one or more real-world objects comprises one or more of a three-dimensional (3D) reconstruction and understanding (3DRU) mesh or a depth-for-stereo (DFS) map.

10. The method according to claim 7, further comprising: Sending the location of the wearable display device to the server; as well as The one or more virtual objects and the depth information for the one or more virtual objects are received from the server in response to sending the position.

11. The method according to claim 10, further comprising: Outlines of the one or more virtual objects are received from the server.

12. The method of claim 11, wherein re-projecting the first image based on the updated position of the wearable display device that is different from the position further comprises: Outlines of the one or more virtual objects are reprojected based on the updated positions, wherein the second image is further rendered using the reprojected outlines of the one or more virtual objects.

13. The method of claim 1 , wherein the depth information for the one or more virtual objects comprises: Depth map, Depth Texture or One or more planes.

14. The method of claim 1, wherein: the rendered first image further comprising the one or more real-world objects rendered independently of the one or more virtual objects in the rendered first image; and Re-projecting the first image based on an updated position of the wearable display device that is different from the position also includes distorting the one or more real-world objects in the first image based on the updated position.

15. The method of claim 14, wherein the second image is further rendered using the distorted one or more real-world objects.

16. The method according to claim 1, further comprising: The one or more real-world objects are selected based on the eye tracking data.

17. A device configured for graphics processing, the device comprising: a memory including computer executable instructions; and a processor configured to execute the computer executable instructions and cause the apparatus to: Rendering a first image based on a position of the wearable display device and contour and depth information for one or more real-world objects associated with the position, the rendered first image comprising at least one or more virtual objects; re-projecting the first image based on an updated position of the wearable display device that is different from the position, wherein the re-projecting comprises: reprojecting outlines of the one or more real-world objects based on the updated positions; updating depth information for the one or more real-world objects based on the updated positions; updating depth information for the one or more virtual objects in the first image based on the updated positions; and warping the one or more virtual objects in the first image based on the updated positions; rendering a second image using the reprojected outlines of the one or more real-world objects, the updated depth information for the one or more real-world objects, the updated depth information for the one or more virtual objects, and the warped one or more virtual objects; and The second image is displayed on a display of the wearable display device.

18. The apparatus of claim 17, wherein the processor is configured to execute the computer-executable instructions and cause the apparatus to render the second image immediately prior to displaying the second image on the display of the wearable display device.

19. The apparatus of claim 17, wherein: The reprojected contour for the one or more real-world objects comprises a plurality of first pixels; The distorted one or more virtual objects include a plurality of second pixels; and The processor is configured to execute the computer-executable instructions and cause the apparatus to render the second image using the reprojected contours for the one or more real-world objects, the updated depth information for the one or more real-world objects, the updated depth information for the one or more virtual objects, and the warped one or more virtual objects by: determining that at least a portion of the second pixel overlaps a portion of the first pixel; determining a depth of each pixel of the portion of the first pixels based on the updated depth information for the one or more virtual objects; determining a depth of each pixel of the portion of the second pixels based on the updated depth information for the one or more virtual objects; as well as For each pixel of the portion of the second pixels that overlaps the portion of the first pixels, a pixel to be rendered in the second image is selected based on the depth of each pixel of the portion of the first pixels and the depth of each pixel of the portion of the second pixels.

20. The apparatus of claim 17, wherein the processor is configured to execute the computer-executable instructions and cause the apparatus to further reproject the outlines of the one or more real-world objects based on the updated depth information for the one or more real-world objects.

21. The apparatus of claim 20, wherein the processor is further configured to execute the computer executable instructions and cause the apparatus to: A motion vector is calculated for at least one of the one or more real-world objects, wherein reprojecting the contours of the one or more real-world objects is further based on the motion vector.

22. The apparatus of claim 17, wherein the processor is configured to execute the computer-executable instructions and cause the apparatus to further warp the one or more virtual objects in the first image based on the updated depth information for the one or more virtual objects in the first image.

23. The apparatus of claim 17, wherein the processor is further configured to execute the computer executable instructions and cause the apparatus to: When the wearable display device is in the position, capturing an image from a perspective of the wearable display device; The image is processed to determine: said contours of said one or more real-world objects in said image; and said depth information for said one or more real-world objects; receiving the one or more virtual objects from a server; as well as The depth information for the one or more virtual objects is received from the server based on the position.

24. The apparatus of claim 23, wherein: The one or more real-world objects include static objects, dynamic objects, or a combination thereof; and The processor is configured to execute the computer-executable instructions and cause the apparatus to process the image to determine the outlines of the one or more real-world objects in the image by running a semantic segmentation algorithm to identify the outlines of the static objects, the dynamic objects, or the combination thereof.

25. The apparatus of claim 23, wherein the depth information for the one or more real-world objects comprises one or more of a three-dimensional (3D) reconstruction and understanding (3DRU) mesh or a depth-for-stereo (DFS) map.

26. The apparatus of claim 23, wherein the processor is further configured to execute the computer executable instructions and cause the apparatus to: sending the location of the wearable display device to the server; and The one or more virtual objects and the depth information for the one or more virtual objects are received from the server in response to sending the position.

27. The apparatus of claim 26, wherein the processor is further configured to execute the computer executable instructions and cause the apparatus to: Outlines of the one or more virtual objects are received from the server.

28. The apparatus of claim 27, wherein the processor is configured to execute the computer executable instructions and cause the apparatus to reproject the first image based on the updated position of the wearable display device that is different from the position by further: Outlines of the one or more virtual objects are reprojected based on the updated positions, wherein the second image is further rendered using the reprojected outlines of the one or more virtual objects.

29. A device for graphics processing, the device comprising: means for rendering a first image based on a position of the wearable display device and contour and depth information associated with the position for one or more real-world objects, the rendered first image comprising at least one or more virtual objects; Means for reprojecting the first image based on an updated position of the wearable display device that is different from the position, wherein the means for reprojecting comprises: means for reprojecting outlines of the one or more real-world objects based on the updated positions; means for updating depth information for the one or more real-world objects based on the updated positions; means for updating depth information for the one or more virtual objects in the first image based on the updated positions; and warping the one or more virtual objects in the first image based on the updated positions; means for rendering a second image using the reprojected outlines of the one or more real-world objects, the updated depth information for the one or more real-world objects, the updated depth information for the one or more virtual objects, and the warped one or more virtual objects; and Means for displaying the second image on a display of the wearable display device.

30. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by a processor of a wearable display device, cause the wearable display device to perform a method of graphics processing, the method comprising: Rendering a first image based on the position of the wearable display device and contour and depth information for one or more real-world objects associated with the position, the rendered first image comprising at least one or more virtual objects; re-projecting the first image based on an updated position of the wearable display device that is different from the position, wherein the re-projecting comprises: reprojecting outlines of the one or more real-world objects based on the updated positions; updating depth information for the one or more real-world objects based on the updated positions; updating depth information for the one or more virtual objects in the first image based on the updated positions; and warping the one or more virtual objects in the first image based on the updated positions; rendering a second image using the reprojected outlines of the one or more real-world objects, the updated depth information for the one or more real-world objects, the updated depth information for the one or more virtual objects, and the warped one or more virtual objects; and The second image is displayed on a display of the wearable display device.

Images