Displaying applications in 3D within augmented reality environment

Abstract

Various implementations disclosed herein include devices, systems, and methods of applying 3-dimensional (3D) effects to content for rendering. For example, a process may obtain content to be rendered within an augmented reality (XR) environment. The process may also generate a two-dimensional (2D) rendering of the content via the rendering framework. The rendering framework generates 3D information based on the content. The process may also generate a 3D effect for rendering the content based on the 3D information. The process may also determine a location of a display area of the content within the XR environment, and may present a view of the XR environment. The rendering of the content may be rendered in a 3D effect at a location in a view of the XR environment.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. Provisional Application Serial No. 63 / 541,039, filed on September 28, 2023, entitled “Displaying Applications in 3DWithin an Extended Reality Environment,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates in general to systems, methods, and apparatus for presenting 3D effects of content viewed via electronic devices, such as head-mounted displays (HMDs). Background Technology

[0004] Existing technologies used for presenting content via electronic devices may not be able to present 3D elements depicted in such content in the desired manner. Summary of the Invention

[0005] The various specific embodiments disclosed herein include devices, systems, and methods for receiving content to be rendered within an extended reality (XR) environment. The content for rendering may include a content scene. The content or content scene may include, in particular, 3D content, 2.5D content, etc. For example, the content may be associated with a rendered video game configured to present a 3D game environment on a 2D monitor / display of a desktop / laptop computer. The content scene may be configured to use a rendering framework, such as, in particular, a hardware-accelerated 3D graphics and computation shader application programming interface (API), to render the content in 2D.

[0006] Some specific implementations present 2D rendering of content at a location within a 3D environment (e.g., on a virtual screen positioned within a flat rectangular area within an XR environment rendered via an HMD). For example, a computer game application configured to present a 2D view of the game on a 2D monitor can be presented within an XR environment by presenting a 2D view of the game on a 2D virtual screen within the XR environment, which is surrounded by a rendering of the user's physical environment and / or virtual content. In another example, 2D photos can be presented within a flat 2D photo viewing portal within such an XR environment.

[0007] Some implementations utilize additional information available for source content configured for 2D display on a flat panel display or virtual flat panel screen (e.g., video games or other applications, photos, other scene content, etc.). This additional information can be used to enhance the appearance of the content by leveraging known information about the depth / 3D characteristics depicted within it. For example, a video game configured to provide a 2D view of a 3D environment can be associated with additional information about that 3D environment, which can be used to enhance the appearance of the content. For instance, such 3D information can be used to provide a 3D effect to a view of content presented within an XR environment.

[0008] In some implementations, a depth-enhanced view of the content is generated using pre-generated / available 3D information associated with the rendering framework. This pre-generated / available 3D information may include depth information or other 3D information used, for example, within a 3D rendering pipeline configured to generate a 2D view in 3D space. The pre-generated / available 3D information may include depth information associated with objects in 3D space relative to a specified viewing angle. Depth information may be retrieved from depth buffers, such as, in particular, a Z-buffer (e.g., a 2D array of floating-point values ​​between zero and one, used to help ensure proper occlusion in the 3D rendering pipeline). In some implementations, a depth-enhanced view of the content is additionally generated using pre-generated / available RGB color information retrieved from an RGB color buffer. In some implementations, a depth-enhanced view of the content is additionally generated using information retrieved from geometry buffers, such as, in particular, a G-buffer (e.g., a texture used to store lighting-related data).

[0009] A depth-enhanced view of content can be presented via a portal within an XR environment. The portal may include a virtual display area to present the content scene within a portion of the XR environment. The depth-enhanced view can be provided by changing some or all of the appearance of the content presented at the portal location to generate or hallucinate the appearance of depth. Such depth-enhanced views can be presented based on depth information, such as 3D information retrieved from depth buffers (e.g., Z-buffers), RGB buffers, geometry buffers (e.g., G-buffers). The system can be configured to present a portal such that portions of the content (e.g., parts of the content that may otherwise be positioned on a flat surface of the portal) are repositioned. For example, portions of the content may be moved out of plane, e.g., pushed into the surface of the portal and / or extended from the surface of the portal (relative to the Z-direction). This repositioning of virtual content can be employed so that virtual objects (e.g., cars, people, etc.) appear closer to the user's view relative to background portions of the content scene (e.g., trees, mountains, etc.).

[0010] In some implementations, depth-enhanced views of content presented via a portal can be created by providing views that offer changes based on one viewpoint or different viewpoints (e.g., left-eye and right-eye viewpoints provided via the HMD). Views that offer changes based on different viewpoints can provide parallax-type view effects, thus providing a depth (3D) effect.

[0011] In some implementations, a depth-enhanced view of the content presented via a portal can be created by reprojecting each frame of the content for each of the user's eyes using 3D information. For example, an application (such as a game) can generate frames over time, each frame presented sequentially, for example, one after another, and each such frame can be enhanced with appropriate depth enhancement.

[0012] In some embodiments, an HMD has a processor (e.g., one or more processors) that executes instructions stored in a non-transitory computer-readable medium to perform a method. The method performs one or more steps or procedures. In some embodiments, the HMD obtains content to be rendered within an XR environment. A 2D rendering of the content can be generated via a rendering framework. The rendering framework can generate 3D information based on the content or otherwise obtain 3D information associated with the content. In some embodiments, 3D effects can be generated for rendering the content scene based on the 3D information. In some embodiments, the location of the display area of ​​the content within the XR environment can be determined. A view of the XR environment can be rendered such that the content scene is rendered in 3D at the location within the view of the XR environment.

[0013] According to some embodiments, an apparatus includes one or more processors, non-transitory memory, and one or more programs; the one or more programs are stored in the non-transitory memory and configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing to perform any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors of the apparatus, cause the apparatus to perform or cause to perform any of the methods described herein. According to some embodiments, an apparatus includes: one or more processors, non-transitory memory, and components for performing or causing to perform any of the methods described herein. Attached Figure Description

[0014] To enable those skilled in the art to understand this disclosure, more detailed descriptions can be made with reference to aspects of some exemplary embodiments, some of which are shown in the accompanying drawings.

[0015] Figures 1A to 1B Exemplary electronic devices operating in a physical environment according to some specific implementations are illustrated.

[0016] Figure 2A This example illustrates a rendering framework that is enabled to generate 3D rendering content from 2D rendering content based on some specific implementation representations.

[0017] Figure 2B This illustrates a view of an XR environment that includes a portal and 2D rendered content, rendered in 3D, based on some specific implementations.

[0018] Figure 3A A second example is shown of a rendering framework that is enabled to generate 3D rendering of 2D rendered content based on some specific implementation representations.

[0019] Figure 3B Examples of left-eye and right-eye content scene versions presented in different views of a portal positioned within an XR environment, based on some specific implementations.

[0020] Figure 4A It is a flowchart representation of an exemplary method based on some specific implementations, which dynamically utilizes 3D information generated via a rendering framework to provide a depth-enhanced 3D view of the displayed content at a display portal location within an XR environment.

[0021] Figure 4B This is a flowchart representation of an exemplary method, based on some specific implementations, for using 3D information to provide a depth-enhanced view of image content at a portal location in an XR environment.

[0022] Figure 5 It is a block diagram based on some specific implementations of electronic devices.

[0023] As is customary practice, various features illustrated in the accompanying drawings may not be drawn to scale. Therefore, for clarity, the dimensions of various features may be arbitrarily expanded or reduced. Furthermore, some drawings may not depict all components of a given system, method, or apparatus. Finally, throughout the specification and drawings, the same reference numerals may be used to denote the same features. Detailed Implementation

[0024] Numerous details have been described to provide a thorough understanding of the exemplary embodiments illustrated in the accompanying drawings. However, the drawings illustrate only some exemplary aspects of this disclosure and should not be considered limiting. Those skilled in the art will understand that other effective aspects and / or variations do not include all the specific details described herein. Furthermore, well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure further relevant aspects of the exemplary embodiments described herein.

[0025] Figures 1A to 1BExemplary electronic devices 105 and 110 are illustrated in physical environment 100. Figures 1A to 1B In the example, physical environment 100 is a room including table 120. Electronic devices 105 and 110 may include one or more cameras, microphones, depth sensors, or other sensors that can be used to capture and evaluate information about physical environment 100 and objects within it, as well as information about user 102 of electronic devices 105 and 110. Information about physical environment 100 and / or user 102 can be used to provide visual and audio content, and / or identify the current location of physical environment 100 and / or the location of the user within physical environment 100.

[0026] In some implementations, a view of an extended reality (XR) environment may be provided to one or more participants (e.g., user 102 and / or other participants not shown) via electronic devices 105 (e.g., wearable devices such as HMDs) and / or 110 (e.g., handheld devices such as mobile devices, tablet computing devices, laptops, etc.). Such an XR environment may include a view of a 3D environment generated based on camera images and / or depth camera images of the physical environment 100, and a representation of user 102 based on camera images and / or depth camera images of user 102. Such an XR environment may include virtual content positioned at a 3D location relative to a 3D coordinate system associated with the XR environment, which may correspond to the 3D coordinate system of the physical environment 100.

[0027] In some implementations, an HMD (e.g., device 105) communicatively coupled to a server or other external device (e.g., a rendering framework) can be configured to receive content to be rendered within an XR environment. The content may include, in particular, 3D content or content scenes, 2.5D content or content scenes, etc. 2.5D content includes presentations associated with rendered movement within a virtual reality environment confined to a 2D plane, where the third dimension is virtually inaccessible in what appears to be a 3D rendered space. 3D content includes presentations associated with rendered movement within a virtual reality environment that simulate the appearance of 3D rendering. Content or content scenes may be associated with a rendered video game configured to render a 3D game on a 2D monitor / display of a computer. Alternatively, content scenes may be associated with a rendered video (e.g., a movie) configured to render on a 2D monitor / display, particularly a computer, television, etc.

[0028] In some implementations, a rendering framework can be enabled to generate 3D renderings of content or content scenes. The rendering framework can also be enabled to generate 3D information based on the content. 3D information may include depth information associated with objects in the content scene relative to a specified viewing angle. Depth information can be retrieved from depth buffers, such as, in particular, Z-buffers. For example, the graphics rendering engine of the rendering framework can generate depth buffers as a byproduct of rendering a 3D scene to a view frustum corresponding to a 2D display of the 3D scene. Similarly, depth buffers such as Z-buffers can be generated and utilized by such rendering engines to avoid overdrawing portions of the view frustum, where multiple surfaces in the 3D scene may exist at different depths, but only the surface with the closest depth is visible to the view frustum.

[0029] 3D information (i.e., byproducts of the graphics rendering engine, such as depth buffers) can be used to generate depth-enhanced 3D views of content or content scenes. In some specific implementations, depth-enhanced views of the content scene can be additionally generated using pre-generated / available RGB color information retrieved from an RGB color buffer. The RGB buffer includes color information that applies various colors to different parts of the depth-enhanced 3D view of the content, such that different parts of the depth-enhanced 3D view do not include a uniform color that can be viewed from all directions (relative to depth), thus producing an appearance that emphasizes depth from various viewpoints.

[0030] In some implementations, a depth-enhanced view of the content scene is additionally generated by using information retrieved from a geometry buffer (e.g., a texture used to store lighting-related data).

[0031] A depth-enhanced view of the content can be presented via a portal within the XR environment. The portal may include an area within the XR environment where the content is presented. Repositioning or otherwise altering the content displayed via the content portal can be used to generate or infer the appearance of depth based on 3D information. For example, a 3D effect can be generated based on additional 3D information, such as depth information retrieved from a depth buffer. For instance, the portal can be configured to allow specified pixels of the content to be pushed into the portal (i.e., repositioned behind the plane of the portal), such that a virtual object (e.g., a car) may appear closer to the user's view than the background portion depicted by the pushed-in pixels of the content (e.g., pixels depicting trees, mountains, etc.). In some implementations, a depth-enhanced view of the content can be formed by placing voxels at different depths relative to the front surface of the portal.

[0032] In some implementations, depth-enhanced views of content presented via a portal can be created by providing views that change the content based on one viewpoint or different viewpoints (e.g., left-eye and right-eye viewpoints provided via the HMD). Views that change the content scene based on different viewpoints can provide a parallax-type view effect, thus providing a depth (3D) effect.

[0033] In some implementations, a depth-enhanced view of the content scene presented via a portal can be created by reprojecting each frame of the content for each of the user's eyes using 3D information. For example, an application (such as a game) can generate frames over time, each frame presented sequentially, for example, one after another, and each such frame can be enhanced with appropriate depth enhancement.

[0034] In some implementations, an HMD (e.g., device 105) communicatively coupled to a server or other external device (e.g., a rendering framework) can be configured to receive 2D content to be rendered within an XR environment. 3D information can be used to generate the 2D content. For example, 2D content can be generated by using depth information associated with objects in 3D space relative to a specific viewpoint. The depth information can be obtained from a depth buffer (such as, in particular, a Z-buffer). Depth information can be generated for each frame of the 2D content. The 3D information can be used to provide a depth-enhanced view of the 2D content at a portal (display) location within the XR environment. For example, a depth-enhanced view of the 2D content can be generated by modifying the 2D content based on the user's viewpoint and 3D information using a parallax process. In some implementations, a user can attempt to view the depth-enhanced 2D content relative to an off-axis viewpoint (e.g., a viewpoint offset by 15 degrees relative to the central axis in front of the user), and thus, an off-axis mitigation process can be implemented to reduce off-axis viewing effects. For example, as the user's viewpoint moves away from the central axis, the parallax effect (e.g., the amount of depth effect) can be reduced. Similarly, the relighting process can be enabled (to mitigate off-axis viewing effects) using normal maps obtained, especially from geometric buffers such as G buffers and / or surface type labels associated with properties such as mirror, metal, etc.

[0035] Alternatively, depth-enhanced views of 2D content can be generated by creating stereoscopic parallax through reprojecting different views for the left and right eyes based on 3D information. For example, the rendered 2D content can be used as the initial rendering viewpoint for the user's first eye, and subsequently, the rendered 2D content can be reprojected to the user's second eye. Alternatively, the rendered 2D content can be used as a center point, and subsequently, the rendered 2D content can be independently reprojected to the user's right and left eyes. Reprojecting different views allows for modification of the projected content while the user's head remains in a static position.

[0036] During the process of reprojecting different views, low-resolution and low-frequency meshes (for depth modeling) can be generated. The low-resolution / low-frequency meshes are configured to fit the pixel depth information of the depth buffer. For example, a low-resolution / low-frequency mesh can be generated such that it uses a granularity that includes a lower resolution that fits a consistent portion of the Z-depth.

[0037] Generating left and right eye viewpoints (at different angles) produces projections including holes that can be filled to enhance the viewing of 2D content in a depth-enhanced (3D) view. Therefore, a process for filling the holes can be implemented such that texture edges are stretched at depth discontinuities associated with adjacent pixels. Similarly, holes can be filled using data obtained from multiple frames of the content. Alternatively, holes can be filled using a repair process relative to the background of the content.

[0038] Figure 2A An example 203 illustrates a rendering framework 205 that is enabled, according to some specific implementation, to generate 2D rendered content (or content scene) 202 and 3D rendering 202a. The rendering framework portion 205a (of the rendering framework 205) is configured to initially acquire content 201 for rendering, particularly within an XR environment. The initial input content 201 may be associated with a video game or software application configured to render a view of 3D video on a computer's 2D monitor / display. Content 201 may include a content scene. Content 201 may include, in particular, 2.5D content, 3D content, etc. Figure 2A The 2D rendering content 202 shown includes a 2D rendering of a person in the foreground 208, a 2D rendering of a landscape in the background 212, a 2D rendering of a mountain further away in the background 204, and a 2D rendering of a horizon further away in the background 211.

[0039] The process of 3D rendering 202a for generating 2D rendered content 202 may include a two-step process that enables the rendering frame portion 205a to initially acquire content 201 and generate 2D rendered content 202. During the process of generating 2D rendered content 202, information such as 3D information 207, RGB information 209, and / or geometry buffer information 210 may be generated based on the initial content. 3D information 207, RGB information 209, and geometry buffer information 210 may be initially generated to create realistic 2D content for rendering. 2D rendered content 202 may be a frustum of 3D content within the field of view relative to the desired camera viewpoint.

[0040] 3D information 207 may include information retrieved from a depth buffer. A depth buffer (such as, in particular, a Z-buffer) may include information for achieving an accurate and realistic view of a content scene, which includes hidden surfaces, such as objects within the view that may be located behind additional objects within the user's view (e.g., these additional objects are closer to the user's view). For example, during content scene rendering, each pixel of the content scene is associated with x, y, and z coordinates. A depth buffer (such as a Z-buffer) includes a two-dimensional array (x and y coordinates) that stores the Z value for each pixel, such that if multiple objects are to be rendered at the same pixel location, the Z-buffer can overwrite previous values ​​if it is determined that a subsequent pixel is closer to the camera / view. Therefore, the depth buffer (Z-buffer) is configured to compare the surface depth of the content scene relative to each pixel location on a projection plane (Z-plane) associated with the content scene.

[0041] RGB information 209 may include color information retrieved from the RGB buffer.

[0042] The geometry buffer information 210 may include information associated with the rendering process performed relative to the two sequential passes. During the first geometry pass, the scene is rendered once, and geometric information from objects within the render is retrieved and stored as a collection of textures within the geometry buffer. For example, the geometric information may include, in particular, positioning vectors, color vectors, normal vectors, specular values, etc. The geometric information stored in the G buffer can be used for subsequent lighting calculations during the second pass.

[0043] Subsequently, the rendering framework section 205b reuses the 3D information 207, RGB information 209, and / or geometry buffer information 210 (which have already been generated to create the 2D rendered content scene 202) to generate a 3D render 202a of the 2D rendered content scene 202. In a sense, the graphics processing unit (GPU) pipeline can be executed in the opposite direction (e.g., relative to typical operation) to reuse byproducts of the initial process used to generate the 2D rendered content scene 202 (e.g., 3D information 207, RGB information 209, and / or geometry buffer information 210), such that these byproducts are used to recreate the geometry used to generate the 3D effects for rendering. The 3D render 202a includes a depth-enhanced view of the content at the portal 215 within the XR environment by creating depth, for example, using a parallax view type effect (e.g., as described below). Figure 2B exemplified).

[0044] 3D rendering 202a can be generated using 3D information 207, RGB information 209, and / or geometry buffer information 210 to create depth (i.e., 3D effect) by extruding voxels 240 (e.g., 3D pixels) at different depths relative to the front / top surface 215a of portal 215. For example, 3D rendering 202a can be generated by placing voxels 240 in and / or at different depths extending from the front / top surface 215a of portal 215 into an XR environment.

[0045] 3D rendering 202a includes: 3D rendering of a person in the foreground at a first depth represented by extruded voxels 208b 208a, 3D rendering of a landscape in the background at a second depth represented by extruded voxels 212b 212a, 3D rendering of mountains in the background at a third depth represented by extruded voxels 212b 204a, and 3D rendering of the horizon in the background at a fourth depth represented by extruded voxels 211b 211a.

[0046] Figure 2B View 250 illustrates an XR environment according to some specific implementation, which includes a physical environment 200 and a virtual environment, the physical environment including a room 223 and a table 220, and the virtual environment including a portal 215 and, relative to... Figure 2A The illustrated 2D rendered content is rendered in 3D 202a. 3D rendering 202a includes a depth-enhanced view of the content at portal 215 within the XR environment by creating depth, for example, using parallax.

[0047] Portal 215 may be a user interface (UI), including a display formed by creating an opening (relative to Z-direction 232) within, for example, the rear wall 223a of room 223. Portal 215 is configured to render 3D rendering 202a within portal 215 to create depth (i.e., 3D effect) based on depth information retrieved from, for example, a depth buffer, an RGB buffer, a geometry buffer, etc. For example, portal 215 may be configured to allow content scenes (e.g., voxels) to be pushed into and / or extended from the surface 215a of portal 215 (in Z-direction 232), such that virtual objects (e.g., rendered 3D rendering 208a) may appear closer to the user view relative to the background portion of the content (e.g., rendered 3D rendering 204a).

[0048] Figure 3AExample 317 illustrates a rendering frame 305 that is enabled, according to some specific implementation, to generate 3D rendering 303 (including left-eye content version 302a and right-eye content version 302b) of 2D rendered content 302. The rendering frame portion 305a (of the rendering frame 305) is configured to initially acquire content 301 for rendering, particularly within an XR environment. Content 201 may be associated with a video game or software application configured to render a view of 3D video on a 2D monitor / display of a computer. Content may include content scenes. Content 301 may include, in particular, 2.5D content, 3D content, etc. Figure 3A The illustrated 2D rendering content 302 includes a 2D rendering of a person in the foreground 308 and a 2D rendering of a mountain in the background 304.

[0049] The process of 3D rendering 303 for generating 2D rendered content 302 may include a two-step process that enables the rendering frame portion 305a to initially acquire content 301 and generate 2D rendered content 302. During the process of generating 2D rendered content 302, information such as 3D information 307, RGB information 309, and / or geometry buffer information 310 may be generated based on the initial content 301. 3D information 307 (e.g., from a Z-buffer), RGB information 309, and geometry buffer information 310 may be initially generated to create realistic 2D content for rendering.

[0050] Subsequently, the rendering frame portion 305b reuses the 3D information 307, RGB information 309, and / or geometry buffer information 310 to generate a 3D render 303 of the 2D rendered content scene 302. The 3D render 303, for example, reprojects the 2D rendered content 302 and / or portal 315 for each eye using the 3D information 307, RGB information 309, and / or geometry buffer information 310 to generate left and right viewpoints, thus including a depth-enhanced view of the content within the portal 315 within the XR environment (e.g., as described below). Figure 3B exemplified).

[0051] The reprojection process may, in particular, involve altering the 2D rendered content 302 by providing (for viewing by a user via, for example, an HMD) a left-eye content version 302a and a right-eye content version 302b (2D rendered content 302). The left-eye content version 302a represents a view 308a (e.g., a first viewpoint) of the 2D rendered presentation 308 located at a first position, which differs from the original position 306 of the 2D rendered presentation 308 within the 2D rendered content 302 (e.g., horizontally offset in direction 312a). The right-eye content version 302b represents a view 308b (a second, different viewpoint) of the 2D rendered presentation 308 located at a second position, which differs from the original position 306 of the 2D rendered presentation 308 within the 2D rendered content scene 302 (e.g., horizontally offset in direction 212b). This first position represents the 2D rendered presentation 308 within the left-eye content version 302a at a position different from the second position within the right-eye content version 302b. Therefore, when viewed via HMD, the combination of left-eye content version 302a and right-eye content version 302b is presented (to the user) as merged content, which represents a 3D effect relative to 2D rendering 308.

[0052] Figure 3B Examples are shown of the different views presented within portal 315, located within XR environment 300, according to some specific implementations. Figure 3A The left-eye content scene version 302a and the right-eye left-eye content scene version 302a. The XR environment 300 includes a physical environment (e.g., physical environment 100 in Figure 1), which includes a table 320 and a portal 315.

[0053] exist Figure 3B In this context, by providing a left-eye view 305a and a right-eye view 305b (for viewing by the user via, for example, an HMD), 3D effects (generated based on 3D information retrieved from, for example, a depth buffer, an RGB buffer, a geometry buffer, etc.) are applied to the portal 315, including left-eye content version 302a and right-eye content version 302a. The left-eye view 305a represents a view of the portal 315 located at a first position (e.g., horizontally offset in direction 312a). The right-eye view 305b represents a view of the portal 315 located at a second position (e.g., horizontally offset in direction 312b), which differs from the first position of the portal 315 presented in the left-eye view 305a. The different positioning of the left-eye view 305a and the right-eye view 305b allows the user to view the combination of the left-eye view 305a and the right-eye view 305b as a portal view (when viewed via the HMD), representing a 3D effect relative to portal 315 and / or 2D rendering of 308. Relative to Figure 3A and Figure 3BEach process described can be performed independently or in combination to generate 3D effects.

[0054] Figure 4A This is a flowchart representation of an exemplary method 400 according to some specific implementations, which dynamically utilizes 3D information generated via a rendering framework to provide a depth-enhanced 3D view of displayed content at a display portal location within an XR environment. In some implementations, method 400 is performed by a device, such as a mobile device, desktop computer, laptop computer, HMD, or server device. In some implementations, the device has a screen for displaying images and / or a screen for viewing stereoscopic images, such as a head-mounted display (HMD, such as device 105 of Figure 1, for example). In some implementations, method 400 is performed by processing logic components, including hardware, firmware, software, or combinations thereof. In some implementations, method 400 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., memory). Each block in method 400 can be enabled and executed in any order.

[0055] At box 402, method 400 obtains the content to be rendered within the extended reality (XR) environment. The content may include, in particular, 3D content, 2.5D content, etc. For example, the content may be associated with a video game or software application configured to render a view of a 3D environment on a computer's 2D monitor / display.

[0056] At box 404, method 400 generates a 2D rendering of the content via a rendering framework. The rendering framework can be configured to generate 3D information based on the content. The 3D information may include depth information received from depth buffers, such as, in particular, a Z-buffer. The depth information may be used by the rendering framework to render the 3D scene to a view frustum corresponding to the 2D rendering. Similarly, the 3D information may include information obtained from, in particular, RGB buffers, geometry buffers, etc. The information obtained from the RGB buffer may include color information used by the rendering framework to apply various colors to different parts associated with different depth-enhanced views of the content rendered in 3D. The information obtained from the geometry buffer may include depth information used by the rendering framework to generate lighting effects within the 2D rendering of the content.

[0057] At box 406, method 400 generates 3D effects for rendering the content scene based on 2D rendering of 3D information and content.

[0058] At box 408, method 400 determines the location of the display area for the content. The display location may be within a portion of the XR environment. The display area may be smaller than the entire XR environment. The display area for the content may include a portal structure formed within a portion of the XR environment. The portal structure may include a rectangular, circular, or window-type display area within the XR environment.

[0059] At box 410, method 400 presents a view of the XR environment. In some implementations, content is rendered with a 3D effect at the location of the display area. In some implementations, the display area of ​​the content is a portal structure formed within that part of the XR environment. In some implementations, content with a 3D effect can be presented within the portal structure.

[0060] In some implementations, the portal structure can be formed by multiple portals, each positioned relative to a different viewpoint of the user, such that the portal structure includes non-planar structures, such as curved structures, tilted structures, etc. Therefore, the depth value from the depth buffer is configured to be manipulated relative to the non-planar baseline.

[0061] In some implementations, the portal structure can be formed by multiple portals within a simulated environment (such as a virtual environment, XR environment, etc.). For example, the portal structure can be formed by multiple portals, each placed in a different location within a simulated room. Similarly, the portal structure can be formed by multiple portals, each placed in a different location within a simulated vehicle associated with the driving simulation. For example, multiple portals may include, in particular, windshield portals, driver's window portals, passenger window portals, etc.

[0062] In some specific implementations, content scenes presented in 3D can be formed by placing voxels at different depths relative to the front surface of the portal structure.

[0063] In some specific implementations, content scenes presented in 3D can be formed by placing voxels at different depths extending from the front surface of the portal structure into the XR environment.

[0064] In some specific implementations, content scenes presented in 3D can be formed by placing voxels at different depths within the portal structure and extending from the front surface of the portal structure to different depths in the XR environment.

[0065] In some specific implementations, content scenes presented in 3D can be formed by reprojecting each frame of the content scene to each of the user's eyes using 3D information.

[0066] In some specific implementations, rendering content scenes in 3D can provide a depth-enhanced view of the content scene within the portal structure.

[0067] In some implementations, the 3D effect is provided by offering a parallax effect by providing a changing view of the image based on different viewpoints within the portal structure.

[0068] Figure 4B This is a flowchart representation of an exemplary method 420 for providing a depth-enhanced view of image content at a portal location in an XR environment using 3D information, according to some specific implementations. In some implementations, method 420 is performed by a device, such as a mobile device, desktop computer, laptop computer, HMD, or server device. In some implementations, the device has a screen for displaying images and / or a screen for viewing stereoscopic images, such as a head-mounted display (HMD, such as device 105 of Figure 1, for example). In some implementations, method 420 is performed by processing logic components, including hardware, firmware, software, or combinations thereof. In some implementations, method 420 is performed by a processor executing code stored in a non-transitory computer-readable medium (e.g., memory). Each block in method 420 can be enabled and executed in any order.

[0069] At box 422, method 420 obtains the content to be rendered within the extended reality (XR) environment. The content may include a two-dimensional (2D) image.

[0070] At box 424, method 420 obtains 3D information associated with the scene depicted in the 2D image. In some implementations, the 3D information can be obtained by extrapolating the depth buffer based on depth sensor data used to create the 2D image during capture. The depth sensor data can be stored within the metadata of the 2D image.

[0071] At box 426, method 420 generates a 3D effect from a 2D image based on 3D information.

[0072] At box 428, method 420 determines the location of the content display area within a portion of the XR environment. In some implementations, the content display area may include a portal structure formed within that portion of the XR environment. A 2D image can be rendered in 3D within the portal structure.

[0073] In some specific implementations, the portal structure can be formed by multiple portals, each positioned relative to a different viewpoint of the user, such that the portal structure includes non-planar display structures, such as curved display structures, tilted display structures, etc.

[0074] At box 430, method 420 presents a view of the XR environment. In some implementations, a 2D image is presented in a 3D effect at this location in the view of the XR environment.

[0075] In some specific implementations, a 2D image presented with a 3D effect can be formed by placing voxels within the portal structure at different depths relative to the front surface of the portal structure.

[0076] In some implementations, a 2D image presented with a 3D effect can be formed by reprojecting a 2D image to each of the user's eyes using 3D information.

[0077] In some implementations, 2D images presented with 3D effects can provide a depth-enhanced view of the 2D images within the portal structure.

[0078] Figure 5 This is a block diagram of example device 500. Device 500 illustrates an exemplary device configuration for electronic devices 105 and 110 of FIG1. ​​Although certain specific features are illustrated, those skilled in the art will understand from this disclosure that various other features are not illustrated for the sake of brevity and to avoid obscuring more relevant aspects of the specific embodiments disclosed herein. Therefore, as a non-limiting example, in some specific implementations, device 500 includes one or more processing units 502 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 504, one or more communication interfaces 508 (e.g., USB, FireWire, Thunderbolt, IEEE 802.3x, IEEE 802.11x, IEEE 802.14x, GSM, CDMA, TDMA, GPS, IR, Bluetooth, ZigBee, SPI, I2C and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 510, output devices (e.g., one or more displays) 512, one or more internal and / or external image sensor systems 514, memory 520, and one or more communication buses 504 for interconnecting these components and various other components.

[0079] In some embodiments, one or more communication buses 504 include circuitry for interconnecting system components and controlling communication between system components. In some embodiments, one or more I / O devices and sensors 506 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time-of-flight, etc.), one or more cameras (e.g., an inward-facing camera and an outward-facing camera of an HMD), one or more infrared sensors, one or more thermal sensors, etc.

[0080] In some embodiments, one or more displays 512 are configured to present a view of a physical environment, a graphical environment, an extended reality environment, etc., to a user. In some embodiments, one or more displays 512 are configured to present content to a user (determined based on the user / object's determined position within the physical environment). In some embodiments, one or more displays 512 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light-emitting field-effect transistor (OLET), organic light-emitting diode (OLED), surface-conducting electron emission display (SED), field emission display (FED), quantum dot light-emitting diode (QD-LED), microelectromechanical systems (MEMS), and / or similar display types. In some embodiments, one or more displays 512 correspond to waveguide displays such as diffraction, reflection, polarization, and holography. In one example, device 500 includes a single display. In another example, device 500 includes displays for each of the user's eyes.

[0081] In some embodiments, one or more image sensor systems 514 are configured to acquire image data corresponding to at least a portion of the physical environment 100. For example, one or more image sensor systems 514 may include one or more RGB cameras (e.g., having a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor), monochrome cameras, IR cameras, depth cameras, event-based cameras, etc. In various embodiments, one or more image sensor systems 514 may also include an illumination source emitting light, such as a flash. In various embodiments, one or more image sensor systems 514 may also include an on-camera image signal processor (ISP) configured to perform multiple processing operations on the image data.

[0082] In some embodiments, sensor data may be acquired by devices (e.g., devices 105 and 110 of Figure 1) during a scan of a room in a physical environment. The sensor data may include a 3D point cloud and a sequence of 2D images corresponding to views of the room captured during the scan. In some embodiments, the sensor data includes image data (e.g., from an RGB camera), depth data (e.g., depth images from a depth camera), ambient light sensor data (e.g., from an ambient light sensor), and / or motion data from one or more motion sensors (e.g., accelerometers, gyroscopes, IMUs, etc.). In some embodiments, the sensor data includes visual inertial odometry (VIO) data determined based on the image data. The 3D point cloud can provide semantic information about one or more elements of the room. The 3D point cloud can provide information about the location and appearance of surface portions within the physical environment. In some embodiments, the 3D point cloud is acquired over time (e.g., during a scan of the room) and can be updated, with updated versions of the 3D point cloud obtained over time. For example, when the 3D representation is updated / adjusted over time (e.g., when a user scans a room), the 3D representation can be obtained (and analyzed / processed).

[0083] In some embodiments, sensor data may be positioning information, and some embodiments include VIO (Vehicle Identification and Odometry) to determine equivalent odometry information to estimate travel distance using sequential camera images (e.g., light intensity image data) and motion data (e.g., acquired from an IMU / motion sensor). Alternatively, some embodiments of this disclosure may include a Simultaneous Localization and Mapping (SLAM) system (e.g., a positioning sensor). This SLAM system may include a GPS-independent, multi-dimensional (e.g., 3D) laser scanning and range measurement system that provides real-time simultaneous localization and mapping. This SLAM system can generate and manage highly accurate point cloud data produced by reflections from laser scans of objects in the environment. Accurately tracking the movement of any points in the point cloud over time allows the SLAM system to use points in the point cloud as reference points for its location, maintaining an accurate understanding of its position and orientation as it travels through the environment.

[0084] In some embodiments, device 500 includes an eye-tracking system for detecting eye positioning and eye movement (e.g., eye gaze detection). For example, the eye-tracking system may include one or more infrared (IR) light-emitting diodes (LEDs), an eye-tracking camera (e.g., a near-infrared (NIR) camera), and an illumination source (e.g., an NIR light source) that emits light (e.g., NIR light) towards the user's eyes. Furthermore, the illumination source of device 500 may emit NIR light to illuminate the user's eyes, and the NIR camera may capture images of the user's eyes. In some embodiments, the images captured by the eye-tracking system may be analyzed to detect the positioning and movement of the user's eyes, or to detect other information about the eyes such as pupil dilation or pupil diameter. Furthermore, the gaze point estimated from the eye-tracking images enables gaze-based interaction with content displayed on a near-eye display of device 500.

[0085] Memory 520 includes high-speed random access memory, such as DRAM, SRAM, DDR RAM, or other random access solid-state memory devices. In some embodiments, memory 520 includes non-volatile memory, such as one or more disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. Memory 520 may optionally include one or more storage devices remotely located to one or more processing units 502. Memory 520 includes a non-transitory computer-readable storage medium.

[0086] In some embodiments, memory 520 or a non-transitory computer-readable storage medium of memory 520 stores an optional operating system 530 and one or more instruction sets 540. Operating system 530 includes procedures for handling various basic system services and for performing hardware-related tasks. In some embodiments, instruction set 540 includes executable software defined by binary information stored in charge form. In some embodiments, instruction set 540 is software executable by one or more processing units 502 to perform one or more of the techniques described herein.

[0087] Instruction set 540 includes 3D effects application instruction set 542 and 3D effects rendering instruction set 544. Instruction set 540 can be represented as a single software executable file or multiple software executable files.

[0088] The 3D effects application instruction set 542 is configured with instructions that can be executed by the processor to determine the generation and application of 3D effects for rendering content scenes (such as video game or movie scenes) based on 3D information retrieved from a depth buffer (e.g., a Z buffer).

[0089] The 3D rendering instruction set 544 is configured with instructions that can be executed by the processor to render a view of the XR environment, so that the content scene is rendered in 3D / depth effect within the portal location in the view of the XR environment.

[0090] Although instruction set 540 is shown as residing on a single device, it should be understood that in other specific implementations, any combination of elements may reside on separate computing devices. Furthermore, Figure 5 This is intended more as a functional description of various features present in a particular specific implementation than as a structural diagram of the specific implementation described herein. As will be appreciated by those skilled in the art, the items shown individually can be combined, and some items can be separated. The actual number of instruction sets and how features are allocated therein will vary depending on the specific implementation and may depend in part on the specific combination of hardware, software, and / or firmware chosen for that particular implementation.

[0091] Those skilled in the art will understand that well-known systems, methods, components, devices, and circuits have not been described exhaustively so as not to obscure more relevant aspects of the specific embodiments of the examples described herein. Furthermore, other effective aspects and / or variations do not include all the details in the specific details described herein. Therefore, several details are described to provide a thorough understanding of the exemplary aspects illustrated in the accompanying drawings. Moreover, the drawings only illustrate some exemplary embodiments of this disclosure and should not be considered limiting.

[0092] While this specification contains numerous specific implementation details, these details should not be construed as limiting the scope of any invention or potentially claimed content, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the context of different embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, while certain features may be described above as functioning in certain combinations and even initially claimed in this manner, one or more features of a claimed combination may be removed from that combination in some cases, and the claimed combination may involve sub-combinations or variations thereof.

[0093] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in a sequential order or the specific order shown, or requiring all illustrated operations to achieve the desired result. In some situations, multitasking and parallel processing may be advantageous. Furthermore, the partitioning of the various system components in the above embodiments should not be construed as requiring such partitioning in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged into multiple software products.

[0094] Therefore, specific embodiments of the subject matter have been described. Other embodiments are also within the scope of the following claims. In some cases, the actions described in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

[0095] The embodiments of the subject matter and operation described in this specification may be implemented in digital electronic circuits or in computer software, firmware, or hardware (including the structures disclosed in this specification and their structural equivalents) or in a combination thereof. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, such as one or more modules of computer program instructions encoded on a computer storage medium for execution by or control of the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium may be or be included in a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination thereof. Furthermore, although the computer storage medium is not a propagating signal, it may be a source or destination of computer program instructions encoded in artificially generated propagating signals. The computer storage medium may also be or be included in one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices).

[0096] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines for processing data, including programmable processors, computers, systems-on-a-chip, or many or combinations of the foregoing. The apparatus may include special-purpose logic circuitry (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)). In addition to hardware, the apparatus may include code that creates an execution environment for the computer program under consideration, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The apparatus and execution environment can implement a variety of different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures. Unless otherwise specifically stated, it should be understood that throughout this specification, discussions using terms such as "processing," "computing," "calculating," "determining," and "identifying" refer to the actions or processes of computing devices, such as one or more computers or similar electronic computing devices, that manipulate or convert data represented as physical electronic or magnetic quantities within the memory, registers, or other information storage devices, transmission devices, or display devices of a computing platform.

[0097] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include computer systems based on multi-purpose microprocessors that access stored software that programs or configures the computing system from a general-purpose computing device to a special-purpose computing device that implements one or more specific implementations of this subject. The teachings contained herein may be implemented in the software used for programming or configuring the computing device using any suitable programming, scripting, or other type of language or combination of languages.

[0098] Specific implementations of the methods disclosed herein can be performed in the operation of such computing devices. The order of the boxes presented in the examples above can be varied; for example, the boxes can be reordered, combined, and / or divided into sub-blocks. Some boxes or processes can be executed in parallel. The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0099] The use of "applies to" or "configured to" in this document implies open and inclusive language, which does not exclude applicability to or configuration for performing additional tasks or steps. Additionally, the use of "based on" implies openness and inclusivity, as processes, steps, calculations, or other actions "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbering included herein are for illustrative purposes only and are not intended to be restrictive.

[0100] It will also be understood that while terms such as "first," "second," etc., may be used in this document to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first node may be called a second node, and similarly, a second node may be called a first node, changing the meaning of the description, provided that all occurrences of "first node" are consistently renamed and all occurrences of "second node" are consistently renamed. First nodes and second nodes are both nodes, but they are not the same node.

[0101] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the claims. As used in the description of these embodiments and in the appended claims, the singular forms “a,” “an,” and “the” are intended to also cover the plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. It will be further understood that the term “comprising,” when used in this specification, specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0102] As used herein, the term "if" can be interpreted as meaning "when the prerequisite is true" or "when the prerequisite is true" or "in response to determination" or "according to determination" or "in response to detection" that the prerequisite is true, depending on the context. Similarly, the phrases "if it is determined [the prerequisite is true]" or "if [the prerequisite is true]" or "when [the prerequisite is true]" can be interpreted as meaning "when it is determined that the prerequisite is true" or "in response to determination" or "according to determination" that the prerequisite is true or "when it is detected that the prerequisite is true" or "in response to detection" that the prerequisite is true, depending on the context.

Claims

1. A method, the method comprising: At head-mounted displays (HMDs) that have a processor and a display: Obtain the content to be rendered within the extended reality (XR) environment; A two-dimensional (2D) rendering of the content is generated via a rendering framework, wherein the rendering framework generates three-dimensional (3D) information based on the content; The 3D information is used to generate the 3D effect for rendering the content; Determine the location of the display area of ​​the content within the XR environment; as well as A view of the XR environment is presented, wherein the content is rendered with the 3D effect at the location in the view of the XR environment.

2. The method according to claim 1, wherein the content is 3D content.

3. The method according to claim 1, wherein the content is 2.5D content.

4. The method according to any one of claims 1 to 3, wherein the 3D information includes depth information obtained from a depth buffer, the depth information being used by the rendering framework to render the 3D scene to a view frustum corresponding to the 2D rendering.

5. The method according to any one of claims 1 to 4, wherein the 3D information includes color information obtained from an RGB buffer, the color information being used by the rendering framework to apply various colors to different portions associated with different depth-enhanced views of the content rendered with the 3D effect.

6. The method according to any one of claims 1 to 5, wherein the 3D information includes depth information obtained from a geometry buffer, the depth information being used by the rendering framework to generate lighting effects within the 2D rendering of the content.

7. The method according to any one of claims 1 to 6, wherein the display area of ​​the content includes a portal structure formed within a portion of the XR environment, and wherein the rendering of the content presented with the 3D effect is presented within the portal structure.

8. The method of claim 7, wherein the portal structure is formed by a plurality of portals, each portal being positioned relative to a different viewpoint of the user such that the portal structure comprises a non-planar structure.

9. The method of claim 7, wherein the content scene presented with the 3D effect is formed by placing voxels within the portal structure at different depths relative to the front surface of the portal structure.

10. The method of claim 7, wherein the content presented with the 3D effect is formed by placing voxels at different depths extending from the front surface of the portal structure into the XR environment.

11. The method of claim 7, wherein the content scene presented with the 3D effect is formed by placing voxels at different depths within the portal structure and extending from the front surface of the portal structure to different depths in the XR environment.

12. The method of claim 7, wherein the content scene presented with the 3D effect is formed by reprojecting each frame of the content for each of the user's eyes using the 3D information.

13. The method of claim 7, wherein the rendering of the content presented with the 3D effect provides a depth-enhanced view of the content within the portal structure.

14. The method of claim 13, wherein the 3D effect is provided by providing a parallax effect by offering a changing view of the image based on different viewpoints within the portal structure.

15. A head-mounted device (HMD), comprising: Non-transitory computer-readable storage medium; and One or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the HMD device to perform operations including: Obtain the content to be rendered within the extended reality (XR) environment; A two-dimensional (2D) rendering of the content is generated via a rendering framework, wherein the rendering framework generates three-dimensional (3D) information based on the content; The 3D information is used to generate the 3D effect for rendering the content; Determine the location of the display area of ​​the content within the XR environment; as well as A view of the XR environment is presented, wherein the rendering of the content scene is presented in 3D at the location in the view of the XR environment.

16. The HMD of claim 15, wherein the content is 3D content.

17. The HMD of claim 15, wherein the content is 2.5D content.

18. The HMD according to any one of claims 15 to 17, wherein the 3D information includes depth information obtained from a depth buffer, the depth information being used by the rendering framework to render the 3D scene to a view frustum corresponding to the 2D rendering.

19. The HMD of any one of claims 15 to 18, wherein the 3D information includes color information obtained from an RGB buffer, the color information being used by the rendering framework to apply various colors to different portions associated with different depth-enhanced views of the content rendered with the 3D effect.

20. The HMD according to any one of claims 15 to 19, wherein the 3D information includes depth information obtained from a geometry buffer, the depth information being used by the rendering framework to generate lighting effects within the 2D rendering of the content.

21. The HMD according to any one of claims 15 to 20, wherein the display area of ​​the content includes a portal structure formed within a portion of the XR environment, and wherein the rendering of the content presented with the 3D effect is presented within the portal structure.

22. The HMD of claim 21, wherein the portal structure is formed by a plurality of portals, each portal being positioned relative to a different viewpoint of the user such that the portal structure comprises a non-planar structure.

23. The HMD of claim 21, wherein the content presented with the 3D effect is formed by placing voxels within the portal structure at different depths relative to the front surface of the portal structure.

24. The HMD of claim 21, wherein the content presented with the 3D effect is formed by placing voxels at different depths extending from the front surface of the portal structure into the XR environment.

25. The HMD of claim 21, wherein the content presented with the 3D effect is formed by placing voxels at different depths within the portal structure and extending from the front surface of the portal structure to different depths in the XR environment.

26. The HMD of claim 22, wherein the content presented with the 3D effect is formed by reprojecting each frame of the content for each of the user's eyes using the 3D information.

27. The HMD of claim 2, wherein the rendering of the content presented with the 3D effect provides a depth-enhanced view of the content within the portal structure.

28. The HMD of claim 27, wherein the 3D effect is provided by providing a parallax effect by offering a changing view of the image based on different viewpoints within the portal structure.

29. A non-transitory computer-readable storage medium storing program instructions executable via one or more processors of a head-mounted device (HMD) to perform operations including: Obtain the content to be rendered within the extended reality (XR) environment; A two-dimensional (2D) rendering of the content is generated via a rendering framework, wherein the rendering framework generates three-dimensional (3D) information based on the content; The 3D information is used to generate the 3D effect for rendering the content; Determine the location of the display area of ​​the content within the XR environment; as well as A view of the XR environment is presented, wherein the rendering of the content scene is presented in 3D at the location in the view of the XR environment.

30. The non-transitory computer-readable storage medium of claim 29, wherein the content is 3D content.

31. The non-transitory computer-readable storage medium of claim 29, wherein the content is 2.5D content.

32. The non-transitory computer-readable storage medium according to any one of claims 29 to 31, wherein the 3D information includes depth information obtained from a depth buffer, the depth information being used by the rendering framework to render the 3D scene to a view frustum corresponding to the 2D rendering.

33. The non-transitory computer-readable storage medium according to any one of claims 29 to 32, wherein the 3D information includes color information obtained from an RGB buffer, the color information being used by the rendering framework to apply various colors to different portions associated with different depth-enhanced views of the content rendered with the 3D effect.

34. The non-transitory computer-readable storage medium according to any one of claims 29 to 33, wherein the 3D information includes depth information obtained from a geometry buffer, the depth information being used by the rendering framework to generate lighting effects within the 2D rendering of the content.

35. The non-transitory computer-readable storage medium according to any one of claims 29 to 34, wherein the display area of ​​the content includes a portal structure formed within a portion of the XR environment, and wherein the rendering of the content presented with the 3D effect is presented within the portal structure.

36. The non-transitory computer-readable storage medium of claim 35, wherein the portal structure is formed by a plurality of portals, each portal being positioned relative to a different viewpoint of the user such that the portal structure includes a non-planar structure.

37. The non-transitory computer-readable storage medium of claim 35, wherein the content presented with the 3D effect is formed by placing voxels within the portal structure at different depths relative to the front surface of the portal structure.

38. The non-transitory computer-readable storage medium of claim 35, wherein the content presented with the 3D effect is formed by placing voxels at different depths extending from the front surface of the portal structure into the XR environment.

39. The non-transitory computer-readable storage medium of claim 35, wherein the content presented with the 3D effect is formed by placing voxels at different depths within the portal structure and extending from the front surface of the portal structure to different depths in the XR environment.

40. The non-transitory computer-readable storage medium of claim 35, wherein the content presented with the 3D effect is formed by reprojecting each frame of the content for each of the user's eyes using the 3D information.

41. The non-transitory computer-readable storage medium of claim 35, wherein the rendering of the content presented in the 3D effect provides a depth-enhanced view of the content within the portal structure.

42. The non-transitory computer-readable storage medium of claim 41, wherein the 3D effect is provided by providing a parallax effect by providing a changing view of the image based on different viewpoints within the portal structure.

Images