Unified rendering

Abstract

To provide a system and a method for visually rendering graphical data on a display.SOLUTION: In a computer system 200, an application 210 makes a renderer 250 render an object 220 for presentation on a display 290 and writes data to a scene graph 240. An object includes several polygons with 3D representations of two human hands 232 and 236. For relative positioning of the hand, a viewer predicts that some parts of the hand 232 occlude a part of the hand 236 and some polygons of the hand 236 occlude a part of the hand 232, with respect to a visual starting point and a visual axis. The application also writes a description of spatial relationships between objects or the like into the scene graph. The renderer outputs a subset of the object or the hand 232 only, a non-occluded part of the hand 232 or the hand 236 only, or the like to the display.SELECTED DRAWING: Figure 2A

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of U.S. Patent Application No. 16 / 011,413, filed on Jun. 18, 2018, the content of which is incorporated herein by reference in its entirety for all purposes.

[0002] The present disclosure generally relates to systems and methods for visually rendering graphical data on a display, and more particularly, to systems and methods for visually rendering data from multiple computer applications on a single display.

Background Art

[0003] Various techniques exist for rendering graphical data of computer applications to a display. It may be desirable for these techniques to realistically render the graphical data, that is, in a manner consistent with and efficient for a viewer's expectations based on the physical world. Also, it may be desirable for the rendering techniques to adapt to computer systems of various topologies, including, for example, computer systems where multiple applications contribute to graphical data to be displayed on a single display.

[0004] Conventional systems often cannot realistically and efficiently render content in such multi-application systems. For example, in some such systems, rendering graphical data from multiple applications on a single display results in the data being incorrectly sorted on the display, producing unexpected visual results that compromise the realism of the display. Further, graphical data from one application may not realistically interact with graphical data from another application, such as via lighting and shading effects or via shaders. Additionally, some such systems are limited in their ability to use rendering optimizations such as culling of invisible surfaces to increase computational efficiency.

[0005] Systems with augmented reality (AR) or "mixed reality" in particular require better solutions to the problem of rendering graphical data from multiple applications onto a single display. For example, an AR system has the potential for multiple users to interact with virtual content from all users that is rendered on a single display within a shared virtual space. Such interactions need to be realistic, meaningful to the user, require that the graphical output of the AR system be persuasive and consistent with the user's visual expectations, and be flexible enough to adapt to different types and numbers of users, user hardware, and user software, as well as different ways in which users may wish to engage with the system, sustain continuous operation at a high frame rate, and be efficient enough to maximize battery life on mobile devices. Further, it may be desirable for the applications and application data associated with individual users in an AR system to remain independent from other users both to provide security (which can be compromised by data access between untrusted users) and, in particular, to maintain scalability as the number of users of the system grows. Additionally, such systems can benefit from minimizing technical constraints on users and user applications. For example, limiting the hardware requirements for users to participate in an AR system encourages more users to participate. This can be achieved, for example, by limiting the extent to which individual users or applications running on the user's hardware need to perform complex rendering operations, such as by offloading such operations to a shared system such as a server-side host application that runs on dedicated hardware. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0006] Embodiments of the present disclosure describe a computer system in which multiple applications contribute to graphical data to be displayed on a single display. Embodiments of the present disclosure can be used to realistically render graphical data, i.e., in a manner consistent with and efficient for a viewer's expectations based on the physical world. According to embodiments of the present disclosure, the first graphical data may be received from a first client application, and the second graphical data may be received from a second independent client application. The first and second graphical data may be combined into a "unified" data structure, such as a scene graph, that can be used to describe the relationships between the nodes represented by the first and second graphical data. The unified data structure can, therefore, be used to render a scene that reflects the first and second graphical data in a realistic and efficient manner on a display. The present invention provides, for example, the following. (Item 1) A method comprising: receiving first graphical data including a plurality of first nodes from a first client application of a computer system; receiving second graphical data including a plurality of second nodes from a second client application of the computer system; generating a scene graph; wherein the scene graph describes an occlusion relationship between at least one first node of the plurality of first nodes and at least one second node of the plurality of second nodes; and the scene graph is configured to create a scene to be rendered based on the occlusion relationship, and at least one second node occludes at least one first node. (Item 2) The method according to item 1, further comprising traversing the scene graph by a processor of the computer system.​ (Item 3) The computer system is configured to communicate with a display, and the method further includes displaying an output on the display, the method according to item 2. (Item 4) Displaying the output includes displaying at least one first node among the plurality of first nodes and at least one second node among the plurality of second nodes, the method according to item 3. (Item 5) Displaying the output consists of displaying at least one first node, the method according to item 4. (Item 6) The method according to item 2, further including applying optimization to the output in the computer system. (Item 7) Applying the optimization includes culling a surface, the method according to item 6. (Item 8) The method according to item 2, further including applying a visual effect to the output in the computer system. (Item 9) Applying the visual effect includes calculating a light quantity value, the method according to item 8. (Item 10) Applying the visual effect includes executing a shader, the method according to item 8. (Item 11) The method according to item 2, further including applying a physical effect to the output in the computer system. (Item 12) Applying the physical effect includes detecting a collision, the method according to item 11. (Item 13) The first client application is a first application executed on the computer system, the second client application is a second application executed on the computer system, and the first client application is sandboxed on the computer system with respect to the second client application, the method according to item 1. (Item 14) The first graphical data corresponds to a first client scene graph associated with the first client application, The second graphical data corresponds to a second client scene graph associated with the second client application, The first client scene graph is sandboxed on the computer system with respect to the second client scene graph, The first client scene graph is sandboxed on the computer system with respect to the scene graph, The second client scene graph is sandboxed on the computer system with respect to the scene graph, the method according to item 1. (Item 15) The scene graph corresponds to a version of a versioned scene graph, the method according to item 1. (Item 16) The first graphical data is communicated to the scene graph using a first processing thread of the computer system, and the second graphical data is communicated to the scene graph using a second processing thread of the computer system independent of the first processing thread, the method according to item 1. (Item 17) A system comprising one or more processors, the one or more processors being configured to implement a method, the method comprising Receiving, from a first client application of a computer system, first graphical data comprising a plurality of first nodes; Receiving, from a second client application of the computer system, second graphical data comprising a plurality of second nodes; Generating a scene graph; including; wherein the scene graph describes an occlusion relationship between at least one first node of the plurality of first nodes and at least one second node of the plurality of second nodes; wherein the scene graph is configured to create a scene to be rendered based on the occlusion relationship, and at least one second node occludes at least one first node, the system. (Item 18) The method further includes traversing, by a processor of the computer system, the scene graph, the system according to item 17. (Item 19) The computer system is configured to communicate with a display, and the method further includes causing the computer system to present an output on the display, the system according to item 18. (Item 20) Displaying the output includes displaying at least one first node of the plurality of first nodes and at least one second node of the plurality of second nodes, the system according to item 19. (Item 21) Displaying the output consists of displaying at least one first node, the system according to item 20. (Item 22) The method further includes causing the computer system to apply an optimization to the output, the system according to item 18. (Item 23) Applying the optimization includes culling surfaces, the system according to item 22. (Item 24) The method further includes causing the computer system to apply a visual effect to the output, the system according to item 18. (Item 25) Applying the visual effect includes calculating a light quantity value, the system according to item 24. (Item 26) Applying the visual effect includes executing a shader, the system according to item 24. (Item 27) The method further includes causing the computer system to apply a physical effect to the output, the system according to item 18. (Item 28) Applying the physical effect includes detecting a collision, the system according to item 27. (Item 29) The first client application is a first application executed on the computer system, the second client application is a second application executed on the computer system, and the first client application is sandboxed on the computer system with respect to the second client application, the system according to item 17. (Item 30) The first graphical data corresponds to a first client scene graph associated with the first client application, The second graphical data corresponds to a second client scene graph associated with the second client application, The first client scene graph is sandboxed on the computer system with respect to the second client scene graph, The first client scene graph is sandboxed on the computer system with respect to the scene graph, The second client scene graph is the system according to item 17, which is sandboxed on the computer system with respect to the scene graph. (Item 31) The scene graph is the system according to item 17, which corresponds to a version of the versioned scene graph. (Item 32) The first graphical data is communicated to the scene graph using a first processing thread of the computer system, and the second graphical data is communicated to the scene graph using a second processing thread of the computer system that is independent of the first processing thread, the system according to item 17. (Item 33) The system includes the computer system, the system according to item 17. (Item 34) The first client application is a first application executed via the one or more processors, the second client application is a second application executed via the one or more processors, and the first client application is sandboxed on the system with respect to the second client application, the system according to item 17.

Brief Description of the Drawings

[0007]

Figure 1

[0008]

Figure 2A

[0009]

Figure 2B

[0010]

Figure 2C

[0011]

Figure 2D

[0012]

Figure 3A

[0013]

Figure 3B

[0014]

Figure 3C

[0015]

Figure 3D

[0016]

Figure 3E

[0017]

Figure 4

DETAILED DESCRIPTION OF THE INVENTION

[0018] In the following description of embodiments, reference is made to the accompanying drawings that form a part hereof and in which are shown, by way of illustration, specific embodiments that may be practiced. It is to be understood that other embodiments may be used and structural changes may be made without departing from the scope of the embodiments disclosed.

[0019] FIGS. 1A-1E illustrate various exemplary computer systems with displays. FIG. 1A shows an exemplary desktop computer connected to an external monitor. FIG. 1B shows an exemplary laptop including a display. FIG. 1C shows an exemplary mobile device including an integrated display. FIG. 1D shows an exemplary television including a display. FIG. 1E shows an exemplary computer system including a head-mounted display. The present disclosure is not limited to any particular type of computer system, any particular type of display, or any particular means of connecting a computer system to a display. The present disclosure is further not limited to 2D displays, and in particular, 3D displays such as stereoscopic displays are contemplated.

[0020] In some exemplary computer systems, the data to be presented graphically (as a "scene to be rendered") on a display includes data representing objects (such as 2D or 3D geometric primitives including polygons) in a three-dimensional space ("3D data"), and presenting the 3D data on the display includes presenting an image ("scene to be displayed") corresponding to the objects in the three-dimensional space as viewed from a viewing origin oriented along the viewing axis. For example, in a software application (such as a video game using a 3D engine) launched on a computer system, the 3D data may include the spatial coordinates, orientation, and / or visual properties of the objects in the three-dimensional game world, as well as data describing the viewing origin and viewing axis in the game world. The 3D data may also include data related to textures associated with the objects to be rendered, shader parameters associated with the objects, and other information that affects how the objects can be displayed. The game may, for example, instruct a software and / or hardware "pipeline" to create a scene to be rendered for presentation on the display as the scene to be displayed during, for example, the "rendering" or "drawing" stage. In such embodiments, generally, it is desirable for the resulting image to reflect the user's expectations about the visual world. In particular, generally, it is desirable for a first opaque object closer to the viewing origin to occlude a second object behind the first object. Objects that are not correctly occluded can confuse the user and may not clearly present where the objects are located in the three-dimensional space. In some exemplary computer systems, occlusion is achieved through sorting, where objects closer to the viewing origin are sorted or drawn on top of objects farther from the viewing origin.

[0021] Sorting a plurality of objects for presentation on a display such that one object realistically occludes another requires information about the relationships between the objects, such as the spatial relationships between objects in three-dimensional space. Some exemplary computer systems utilize a scene graph to represent relationships (e.g., hierarchical relationships) between one or more objects, such as objects to be rendered as a scene. As used herein, a scene graph is any data structure that represents such relationships. For example, in a scene graph, the objects to be rendered for presentation may be represented as nodes in the graph, and the relationships between the nodes represent the logical or spatial relationships between the objects. The renderer can then traverse the scene graph according to techniques known in the art and render or prepare at least one of the objects for display in a manner that will achieve appropriate occlusion. In other words, the renderer can create a scene of objects with nodes, but the presentation on the display may only be a subset of the objects to be rendered such that objects occluded by another object within the renderer will only be partially presented within the resulting displayed scene (the output in such embodiments being the non-occluded portions of the objects). Such selective presentation can be beneficial for obscuring the content embodied by a first object launched from a first application when the user desires only the content embodied by a second object launched from a second application to be visible within a given time period. In some examples, the scene graph is an intermediate data structure positioned between an application that includes 3D data and a renderer for rendering that 3D data for presentation on a screen, and in some examples, the application writes scene information to the scene graph, and the scene graph may later be used by the renderer to render the scene or output the scene to be displayed.

[0022] FIG. 2A shows an exemplary flow of data in an exemplary computer system 200. In system 200, a single application 210 can write data to a scene graph 240 that a renderer 250 can use to render an object 220 for presentation on a display 290. For example, object 220 may include several objects (e.g., polygons) that together provide a 3D representation of two human hands, namely, hand 232 and hand 236, and the application may direct object 220 to be presented on the display from a viewing origin viewpoint oriented along the line of sight, for example, during a rendering or drawing stage. In an embodiment, hands 232 and 236 are interlocked in a handshake, and due to the relative positioning of the hands, a viewer expects that some portions of hand 232 occlude portions of hand 236 and that some polygons making up hand 236 occlude portions of hand 232 with respect to the viewing origin and the line of sight. Application 210 can write information describing relationships between objects 220, such as spatial relationships between polygons making up object 220, which can be used to identify polygons that should occlude other polygons, i.e., polygons that should be sorted to be displayed on top of others, to the scene graph 240. For example, scene graph 240 may reflect that polygon 234 (belonging to hand 232) is positioned between the viewing origin and the polygons making up hand 236 and should thus occlude those polygons in hand 236, and that polygon 238 (belonging to hand 236) is positioned between the viewing origin and the polygons comprising hand 232 and should thus occlude those polygons in hand 232. Renderer 250 may then output a subset such as object 220 or only hand 232 or the non-occluded portions of hand 232 or only hand 236, etc., as output to display 290 consistent with the desired occlusion for presentation.

[0023] FIG. 2B shows an exemplary output of the renderer 250 of the exemplary computer system 200 shown in FIG. 2A. In the embodiment described above with respect to FIG. 2A, based on the relative positions of the objects 220, the viewer expects that some of the objects (such as polygon 234) that make up the hand 232 occlude the hand 236, and that some of the objects (such as polygon 238) that make up the hand 236 occlude the hand 232. The exemplary output shown in FIG. 2B is consistent with the expected occlusion. That is, the object 220 in FIG. 2A is correctly displayed to present a handshake on the display 290 that is consistent with the viewer's expectations.

[0024] In the exemplary computer systems 200 shown in FIGS. 2A and 2B, the scene graph 240 is written directly into the application 210 by only a single application. The renderer 250 then traverses the scene graph 240 and renders the hands 232 and 236 with appropriate occlusion. Conventional systems that use a scene graph as part of a rendering process, such as the exemplary computer system 200, may not correctly occlude objects when the scene graph (e.g., scene graph 240) receives input directly from multiple independent applications. In these situations, unlike the exemplary computer system 200, there may not be an application that can provide a scene graph with all of the object relationship data that may be required to correctly sort the objects on the display.

[0025] Figure 2C shows an exemplary flow of data in an exemplary computer system 201 that uses two independent applications and illustrates the occlusion problem described above. Unlike the exemplary computer system 200 described above, the exemplary computer system 201 includes two independent applications, namely, application 212 and application 214. In the exemplary computer system 201, both application 212 and application 214 write data to the scene graph 240 and render their individual 3D data to a single display 290. In Figure 2C, application 212 attempts to render and present object 222 (including the objects that make up hand 232), and application 214 attempts to render and present object 224 (including the objects that make up hand 236). In this embodiment, as in the embodiments described in Figures 2A and 2B, hands 232 and 236 will be interlocked in a handshake such that when displayed in parallel within the same 3D environment, a viewer would expect that portions of hand 232 would occlude hand 236 and portions of hand 236 would occlude hand 232.

[0026] The embodiment shown in FIG. 2C may have difficulty achieving realistic occlusion of the objects to be rendered. In the embodiment, application 212 can write data corresponding to object 222 (including hand 232) to scene graph 240, and application 214 can write data corresponding to object 224 (including hand 236) to the same scene graph 240. However, in exemplary computer system 201, if application 212 and application 214 are independent ( "sandboxed") applications, application 212 cannot access data related to object 224 (including hand 236 and its constituent objects) of application 214, and similarly, application 214 cannot access data related to object 222 (including hand 232 and its constituent objects) of application 212. That is, in some embodiments, neither application 212 nor application 214 can fully identify the relationship between object 222 and object 224. Therefore, neither application 212 nor application 214 can write to scene graph 240 information that may be necessary to identify an object that occludes other objects or the order in which objects should be sorted on the display.

[0027] Figure 2D shows an exemplary output of the renderer 250 of the exemplary computer system 201 shown in Figure 2C. In the embodiment described above with respect to Figure 2C, based on the relative positioning of object 222 and object 224, a viewer would expect that some portions of hand 232 occlude hand 236 and some portions of hand 236 occlude hand 232. However, unlike Figures 2A and 2B, the scene graph 240 of Figure C, as described above, cannot correctly sort objects 222 and 224 to produce the desired occlusion. Instead, in the embodiment shown, all of object 222 is sorted on top of all of object 224. The exemplary output shown in Figure 2D is thus inconsistent with the expected occlusion. The resulting handshake image thus does not accurately reflect the objects in applications 212 and 214, and the image can additionally look unnatural and be confusing to the viewer.

[0028] Other disadvantages of conventional scene graphs such as the scene graph 240 of FIGS. 2A-2D are evident when used in conjunction with multiple independent applications such as in the exemplary computer system 201. For example, rendering efficiency can be achieved using data corresponding to the entire scene to be rendered, such as in the application 210 of FIG. 2A. For example, by determining surfaces that will be occluded, the exemplary computer system 200 of FIG. 2A can instruct the system 200 to cull those surfaces, thereby avoiding unnecessary expenditure of computational resources. This culling may not be possible in a multi-application system such as the exemplary computer system 201 of FIG. 2C because each application may not possess the scene knowledge to determine the surfaces to be culled. Further, in some embodiments involving only a single application such as the exemplary computer system 200, beneficial effects may be applied to an object based on the presence of other objects within the scene. For example, applying realistic lighting and shading effects to an object may require data corresponding to nearby objects. Further, some shader effects benefit from such data. Similarly, effects generated by a particle system or a collision detection system may benefit from such data. Such effects may be limited or impossible in a system where 3D data is provided by multiple independent applications because there may not be any application that can provide all of the node relationship information necessary to apply such effects.

[0029] The present disclosure presents systems and methods that address the above disadvantages of systems that use a unified scene graph to render 3D data from multiple independent applications. The unified scene graph can be used in place of a conventional scene graph, such as scene graph 240 of FIG. 2C, in a system (such as exemplary computer system 201 of FIG. 2C) that provides 3D data to be rendered by multiple independent applications. As described herein, in some embodiments, the unified scene graph receives 3D data from multiple individual input sources, writes information corresponding to that 3D data to a central location, and maintains that information for access by a renderer that creates a rendered scene comprising objects based on that 3D data. The rendered scene may be used to generate an output (such as a graphical output) that reflects realistic object occlusion, computational efficiency, visual effects (such as lighting and shadowing), or physical effects (such as collision detection) or partial representation of occluded objects that would be difficult or impossible to achieve in a system that does not utilize a unified scene graph otherwise.

[0030] In some embodiments, an exemplary computer system includes a plurality of applications, each including 3D data representing one or more objects within a common 3D environment. The plurality of applications may each exist within a “sandboxed” environment such that it remains independent of other applications; for example, the data for each individual application may be independent from the data of each other application, each application may not have access to the data of each other application, while the respective 3D data of the applications may correspond to the same 3D environment, and each application maintains its own instance of the 3D environment. For example, each application may represent a player within an online multiplayer video game where each player exists within an instance of the same game world or 3D environment but lacks direct access to the data of other players. In such embodiments, it may be desirable for all players to be rendered simultaneously within a single instance of the game world, but it may not be desirable (or computationally infeasible) for each player to maintain the information necessary to render the 3D data of each other client participant. Further, for security purposes, it may be desirable to limit the information about a player that is available to other players.

[0031] In some embodiments, each of the plurality of sandboxed applications can independently write information corresponding to its 3D data to a local scene graph, and that information is later written to a common unified scene graph. The unified scene graph can then be traversed by a renderer to render a scene for presentation on a display as an image, based on the collective 3D data provided by each application. By communicating the 3D data from each of the plurality of sandboxed applications to a single unified scene graph, the renderer can apply beneficial techniques such as occlusion, lighting effects, and rendering optimizations (such as surface culling), which require or benefit from simultaneous knowledge of the 3D data of all applications. These benefits are achieved while limiting the required computational overhead of each sandboxed application, and from the perspective of a single application, all that the application needs to do is update a single scene graph to reflect its 3D data, and other operations are performed by other components of the system. Additionally, security benefits can be obtained by maintaining separation between the sandboxed applications.

[0032] FIG. 3A illustrates components of an exemplary computer system 300 that can render 3D data from multiple independent applications for display using a unified scene graph. The illustrated embodiment utilizes a client - server topology, however, the present disclosure is not limited to client - server embodiments. In the exemplary computer system 300, a first client application 310 and a second client application 320 each communicate 3D data (in some embodiments, via a network) to a client - server interface 330. In some embodiments, client applications 310 and 320 are “sandboxed” applications that operate independently of each other and independently communicate their 3D data to the client - server interface 330. The client - server interface 330 receives updated 3D data from client applications 310 and 320 and can communicate that 3D data (in some embodiments, via a network) to a server - side host application 340. In some embodiments, the client - server interface 330 uses multi - threading techniques and uses multiple processor threads to receive, process, and / or communicate 3D data to the host application 340. In some embodiments, the client - server interface includes logic for controlling (such as by throttling) the rate at which 3D data is communicated to the host application 340. The host application 340 uses the 3D data received from the client - server interface to update a unified scene graph 350 such that the unified scene graph 350 reflects the 3D data received from client applications 310 and 320. In some embodiments, the unified scene graph 350 comprises multiple versions of the scene graph, and known versioning techniques are used to enable updates to the unified scene graph 350 to occur in parallel.The renderer 360 then traverses the unified scene graph 350, applies optimizations and effects as appropriate, and generates an output (e.g., data of at least one of the client applications 310 and 320, and in some embodiments, a graphical output comprising only the occluded portion of one client application without occluded application data) to be displayed on a display 370 such as a computer monitor.

[0033] Figure 3B illustrates aspects of an exemplary client application 310 with respect to the exemplary computer system 300 shown in Figure 3A. In the illustrated embodiment, the 3D data 312 represents graphical objects (geometric primitives, e.g., polygons, etc.) within a 3D environment to be presented on the display 370. The 3D data 312 may be updated (314) by the client application 310. For example, if the client application 310 is an application with a rendering loop that iterates 60 times per second, the client application 310 may update the 3D data 312 60 times per second and reflect the changes in that data during the course of the application's operation that should be reflected in the rendering output. In some embodiments, the 3D data 312 may be represented as a local scene graph 316 that may be local to each client application 310. In some embodiments, the local scene graph 316 may include data (nodes, etc.) corresponding to data within the unified scene graph 350. When the 3D data 312 is updated (314), the client application 310 may update the local scene graph 316 to reflect the most recent version of the 3D data 312. When the local scene graph 316 is updated, this can be used by the client application 310 to generate (317) client data 318. In some embodiments, the client data 318 may represent the local scene graph 316 as a whole. In some embodiments, the client data 318 may represent changes made to the local scene graph 316 since the previous client data 318 was sent to the client - server interface 330. For example, the client data 318 may include nodes added to or removed from the local scene graph 316, changes to the relationships between nodes within the local scene graph 316, or changes to the properties of nodes within the local scene graph 316.In some embodiments, the client data 318 may use an identifier such as an identification number corresponding to a scene graph node to identify the relationship between data from the local scene graph 316 and corresponding data on the unified scene graph 350. The client data 318 can then be communicated to the client-server interface 330 for ultimate communication to the host application 340. In some embodiments, the communication of the client data 318 to the client-server interface 330 may occur via a network. In some embodiments, a client helper application may be used in conjunction with the client application 310 to generate the client data 318 from the local scene graph 316 or from the 3D data 312.

[0034] Aspects described with respect to client application 310 may similarly describe client application 320, or other client applications (along with client application 310) that comprise exemplary computer system 300. The systems and methods described herein may be extended to include any number of client applications and client data, and the present disclosure is not limited to any such number. Further, those skilled in the art will understand that some benefits (e.g., improved computational efficiency) may become more apparent with the use of an increasing number of client applications. As described above, client applications 310 and 320 may be sandboxed applications that do not share data or functionality. For example, in exemplary computer system 300, client application 320 may have its own 3D data and local scene graph that is independent of 3D data 312 and local scene graph 316 belonging to client application 310. However, in some embodiments, including exemplary computer system 300, a single client-server interface 300 is shared by multiple client applications such as client applications 310 and 320.

[0035] Figure 3C illustrates a side view of an exemplary client-server interface 330 for the exemplary computer system 300 shown in FIGS. 3A and 3B. In an embodiment, client data 318 and client data 328 are client data that are communicated to, or updated by, individual client applications 310 and 320 as described above with respect to FIG. 3B. In some embodiments, client data 318 and 328 may be updated on client-server interface 330 at different rates. This can occur, for example, when one client application is running on computing hardware that is less capable than another client application (causing that client application to update its client data less frequently), when one client application communicates with client-server interface 330 via a network with a lower bandwidth than another client application, or when the client data associated with one client application is more complex (and requires more processing time to generate) than the client data associated with another client application. Different rates of updating client data on client-server interface 330 can also be the result of a temporary change in operating conditions, such as when a network failure temporarily takes a client application offline. For example, it may be desirable for computer system 300 to tolerate different rates of updating client data, such that a network failure affecting one client application does not adversely affect the rate at which the unified scene graph 350 is updated using client data from other client applications, or the rate at which the scene is rendered.Also, when updating the unified scene graph 350, since this can cause destabilization or asynchronization of the unified scene graph or the rendered display for the client application, it may be desirable to ensure that client data from one client application does not lag excessively behind or lead excessively ahead of client data from other client applications.

[0036] In some embodiments, the role of the client-server interface 330 is to handle differences or increases and decreases in the rate at which client data is updated. Referring to FIG. 3C, an exemplary client-server interface 330 may receive client data 318 and 328 via independent processing threads 332 and 334, respectively, and may include a thread manager 336 for handling the threads. Utilizing multiple threads and updating client data from different sources such as client application 310 and client application 320 can prevent problems with one source from blocking or otherwise adversely affecting data from other sources. In the illustrated embodiment, the thread manager 336 may input client data 318 and 328 from client applications 310 and 320 using threads 332 and 334, respectively, and output host data 319 and 329 (corresponding to client data 318 and 328, client applications 310 and 320, and threads 332 and 334, respectively) to host application 340. The thread manager 336 may include logic for identifying and handling throughput problems or other problems associated with threads 332 and 334 in order to process threads 332 and 334 and / or for controlling the output of host data 319 and 329. For example, if client data 318 and client data 328 are being updated at approximately the same rate (via threads 332 and 334, respectively), the thread manager 336 may simply update host data 319 and 329 (corresponding to client data 318 and 319, respectively) at approximately the same rate. However, if client data 318 is being updated at a much faster rate than client data 328, the thread manager 336 may throttle client data 318 (e.g., by communicating this to host application 340 at a good low frequency) to prevent this from far exceeding client data 328.The thread manager 336 may also control the overall rate at which the host data is updated. For example, the thread manager 336 may suppress the rate at which the host data 319 and / or 329 is updated, preventing the host data from being updated faster than the host application 340 can process it, which could lead to undesirable asynchronization of the output to the client applications 310 and / or 320, the unified scene graph 350, and / or the display 370.

[0037] FIG. 3D illustrates a side view of an exemplary host application 340 with respect to the exemplary computer system 300 shown in FIGS. 3A - 3C. What is described herein are operations that are executed within a thread 341 within the host application 340, where the thread 341 can execute in parallel with additional threads within the host application 340. In some embodiments, multithreading within the host application 340 can have the advantage of allowing multiple client applications or multiple sets of host data to simultaneously update the same unified scene graph 350 (in some embodiments, by updating different versions of the unified scene graph). This can, in turn, increase the overall throughput of client data to the rendered scene for presentation on the display. In some embodiments, multithreading may require that locks be placed on the unified scene graph data, for example, to prevent threads from inadvertently writing to the same data. However, in some embodiments, one or more of the operations described may not be executed within a thread.

[0038] In the embodiment shown in FIG. 3D, host data 319 (corresponding to client application 310 and client data 318) is updated (342) as described above with respect to FIG. 3C. Host application 340 may then identify changes that host data 319 may make to a previous version of the unified scene graph 350. For example, host application 340 may identify, with respect to unified scene graph 350, that host data 319 will add a node, delete a node, change the relationship between two nodes, or change the properties of a node. (In some embodiments, such as the embodiment shown in FIG. 3D, host application 340 may use host data handler 344 to perform these or other operations). Host application 340 may identify the version of unified scene graph 350 to be created or updated according to host data 319 (352). In some embodiments, prior to writing to version 352 of unified scene graph 350, host application 340 may lock that version to prevent other processes from modifying it in parallel. Host application 340 may make changes to version 352 (such as by adding or deleting scene graph nodes within version 352 to correspond to host data 319) to reflect host data 319. In some embodiments, host application 340 may then unlock version 352 and update the value of the version number corresponding to version 352 (356). Host application 340 may then update the host data (342) and repeat the process shown in FIG. 3D. When the unified scene graph 350 is updated to reflect individual host data derived from individual client applications, the unified scene graph 350 will reflect the collective host data from multiple client applications, even if the individual client applications are "sandboxed" and may be independent of each other.

[0039] Figure 3E illustrates a side view of an exemplary renderer 360 with respect to the exemplary computer system 300 shown in FIGS. 3A - 3D. In some embodiments, the renderer 360 comprises part of the host application 340. In some embodiments, the renderer 360 may be part of another component of the exemplary computer system 300, or a separate component or application. In some embodiments, the renderer 360 may be implemented in physical hardware different from one or more of the components of the exemplary computer system 300 and may communicate with one or more of those components via a network.

[0040] In the example shown in FIG. 3E, the renderer 360 acts on version 352 of the unified scene graph 350. In an example, the role of the renderer is to create a rendered scene comprising data such as an output for presentation on the display 370 or a graphical output, based on version 352 of the unified scene graph 350. As part of this process, the renderer 360 may traverse version 352 using known scene graph traversal techniques (362). During or after traversal 362, the renderer 360 may optionally update the unified scene graph 350 (364) to reflect the results of the traversal. For example, as part of traversal 362, the renderer 360 may identify discarded nodes that should be removed from the unified scene graph 350. Following traversal 362 and / or update 364, the renderer 360 may apply various optimizations 366 to the scene. For example, the renderer 360 may cull unclear or invisible surfaces to avoid consuming unnecessary computational resources. Following traversal 362 and / or update 364, the renderer 360 may apply one or more visual effects 367 to the scene. For example, in some embodiments, the renderer 360 may apply one or more shaders that apply lighting effects or shadow effects, apply particle effects, and / or apply physical effects. Finally, the renderer 360 can output data to a graphical output pipeline, the result of which can be displayed as an output on the display 370.

[0041] The above exemplary process of the computer system may be provided by any suitable logic circuit. The suitable logic circuit may include one or more computer processors (e.g., CPU, GPU, etc.) that execute instructions implemented in a software program to implement the process. Additionally, such a process can also be provided via a corresponding logic design implemented in a hardware logic circuit such as programmable logic (e.g., PLD, FPGA, etc.) or customized logic (e.g., ASIC, etc.) that implements the logic design for providing the process. Further, such a process can be provided via an implementation that combines one or more processors of both software and hardware logic circuits to activate them.

[0042] FIG. 4 illustrates an exemplary system 400 that may be used to implement any or all of the above embodiments. The above embodiments (in whole or in part) may be embodied in any portable device (including wearable devices) or non-portable device, such as a communication device (e.g., mobile phone, smartphone), a multimedia device (e.g., MP3 player, TV, radio), a portable or handheld computer (e.g., tablet, netbook, laptop), a desktop computer, an all-in-one desktop, a peripheral device, a head-mounted device (which may include an integrated display), or any other system or device compatible with the inclusion of an exemplary system architecture 400, including combinations of two or more of these types of devices. The above embodiments may be embodied in two or more physically separate devices, such as two or more computers communicating via a wireless network. The above embodiments may be embodied in two or more physically different devices, such as communicating data to and / or from a head-mounted display, a belt pack, etc. FIG. 4 is a block diagram of one embodiment of system 400 generally including one or more computer-readable media 401, a processing system 404, an I / O subsystem 406, a radio frequency (RF) circuit 408, an audio circuit 410, and a sensor circuit 411. These components may be coupled by one or more communication buses or signal lines 403.

[0043] The architecture shown in FIG. 4 is merely an exemplary architecture of system 400, and it should be apparent that system 400 may have more or fewer components or a different configuration of components than shown. The various components shown in FIG. 4 may be implemented in hardware, software, firmware, or any combination thereof, including one or more signal processing and / or application specific integrated circuits.

[0044] Referring to the exemplary system architecture 400 of FIG. 4, the RF circuit 408 can be used to transmit and receive information via a wireless link or network to one or more other devices and includes well-known circuits for implementing this function. The RF circuit 408 and the audio circuit 410 can be coupled to the processing system 404 via the peripheral device interface 416. The interface 416 can include various known components for establishing and maintaining communication between the peripheral devices and the processing system 404. The audio circuit 410 can be coupled to the audio speaker 450 and the microphone 452 and can include well-known circuits for processing the audio signals received from the interface 416 and enabling the user to communicate with other users in real time. In some embodiments, the audio circuit 410 can include a headphone jack (not shown).

[0045] The sensor circuit 411 can be coupled to various sensors including, but not limited to, one or more light emitting diodes (LEDs) or other light emitters, one or more photodiodes or other light sensors, one or more thermopile sensors, magnetometers, accelerometers, gyroscopes, barometers, compasses, proximity sensors, cameras, ambient light sensors, thermometers, GPS sensors, electrooculogram (EOG) sensors, and various system sensors capable of sensing remaining battery life, power consumption, processor speed, CPU load, and the like. In embodiments involving a head-mounted device, one or more sensors may be employed in connection with functionality related to the user's eyes such as tracking the user's eye movements or identifying the user based on an image of their eye.

[0046] The peripheral device interface 416 can couple the input and output peripheral devices of the present system to the processor 418 and the computer-readable medium 401. One or more processors 418 may communicate with one or more computer-readable media 401 via the controller 44. The computer-readable medium 401 can be any device or medium (excluding signals) that can store code and / or data for use by one or more processors 418. In some embodiments, the medium 401 can be a non-transitory computer-readable storage medium. The medium 401 can include a memory hierarchy including, but not limited to, cache, main memory, and secondary memory. The memory hierarchy can be implemented using any combination of magnetic and / or optical storage devices such as RAM (e.g., SRAM, DRAM, DDRAM), ROM, FLASH (registered trademark), disk drives, magnetic tapes, CDs (compact discs), and DVDs (digital video discs). The medium 401 may also include a transmission medium for carrying an information-carrying signal (excluding signals and excluding the carrier wave on which the signal is modulated) that represents a computer instruction or data. For example, the transmission medium may include communication networks including, but not limited to, the Internet (also referred to as the World Wide Web), intranets, local area networks (LANs), wide local area networks (WLANs), storage area networks (SANs), metropolitan area networks (MANs), and equivalents.

[0047] One or more processors 418 can launch various software components stored in the medium 401 and perform various functions for the system 400. In some embodiments, the software components can include an operating system 422, a communication module (or set of instructions) 424, an I / O processing module (or set of instructions) 426, a graphics module (or set of instructions) 428, and one or more applications (or set of instructions) 430. Each of these modules and the above applications can correspond to a set of instructions for performing one or more of the functions described above and the methods described in this application (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus, various subsets of these modules may be combined or otherwise rearranged in various embodiments. In some embodiments, the medium 401 may store a subset of the modules and data structures identified above. Additionally, the medium 401 may store additional modules and data structures not described above.

[0048] The operating system 422 can include various procedures, sets of instructions, software components, and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware and software components.

[0049] The communication module 424 can facilitate communication with other devices via one or more external ports 436 or via the RF circuit 408 and can include various software components for handling data received from the RF circuit 408 and / or the external port 436.

[0050] The graphics module 428 can include various known software components for rendering, animating, and displaying graphical objects on one or more display surfaces. The display surface may include a 2D or 3D display. The display surface may be directly or indirectly coupled to one or more components of the exemplary system 400. In embodiments with a touch-sensitive display (e.g., a touch screen), the graphics module 428 can include components for rendering, displaying, and animating objects on the touch-sensitive display. In some embodiments, the graphics module 428 can include components for rendering to a remote display. In some embodiments, such as those incorporating a camera, the graphics module 428 can include components for creating and / or displaying an image formed by synthesizing a rendered graphical object with camera data (such as that captured from a head-mounted camera) or photo data (such as an image captured by a satellite). In some embodiments, the graphics module can include components for rendering an image to a head-mounted display device. In some embodiments, the image may include a view of elements of virtual content (e.g., an object within a three-dimensional virtual environment) and / or a view of the physical world (e.g., camera input showing the user's physical surroundings). In some embodiments, the display may present a composite image of virtual content and a view of the physical world. In some embodiments, the view of the physical world may be a rendered image, and in some embodiments, the view of the physical world may be an image from a camera.

[0051] One or more applications 430 can include any application installed on system 400, including but not limited to browsers, address books, contact lists, email, instant messaging, document processing, keyboard emulation, widgets, JAVA (registered trademark)-compatible applications, encryption, digital rights management, voice recognition, voice replication, location determination capabilities (such as those provided by the Global Positioning System (GPS)), music players, and the like.

[0052] The I / O subsystem 406 can be coupled to the eye I / O device 412 and one or more other I / O devices 414 for controlling or performing various functions. For example, the eye I / O device 412 can communicate with the processing system 404 via an eye I / O device controller 432 that can include various components for processing eye inputs (e.g., sensors for eye tracking) or user gesture inputs (e.g., optical sensors). One or more other input controllers 434 can send and receive electrical signals to and from the other I / O devices 414. The other I / O devices 414 may include physical buttons, dials, slider switches, sticks, keyboards, touch pads, additional display screens, or any combination thereof.

[0053] The I / O processing module 426 can include various software components for performing various tasks associated with the eye I / O device 412 and / or other I / O devices 414, including receiving and processing inputs received from the eye I / O device 412 via the eye I / O device controller 432, or from other I / O devices 414 via the I / O controller 434. In some embodiments, the I / O device 414 and / or the I / O processing module 426 may perform various tasks associated with gesture input, which may be provided by tactile or non-tactile means. In some embodiments, the gesture input may be provided, for example, by a camera or another sensor for detecting movement of the user's eyes, arms, hands, and / or fingers. In some embodiments, the I / O device 414 and / or the I / O processing module 426 may be configured to identify objects on a display with which the user desires to interact, such as GUI elements pointed to by the user. In some embodiments, the eye I / O device 412 and / or the I / O processing module 426 may be configured to perform eye tracking tasks such as identifying an object or region on a display that the user is looking at (such as with the aid of an optical or EOG sensor). In some embodiments, a device (such as a hardware “beacon”) may be worn or held by the user to assist with gesture-related tasks of the touch I / O device 412 and / or the I / O processing module 426, such as identifying the location of the user's hand in a 2D or 3D environment. In some embodiments, the eye I / O device 412 and / or the I / O processing module 426 may be configured to identify the user based on sensor inputs such as data from a camera sensor associated with the user's eyes.

[0054] In some embodiments, the graphics module 428 can display visual output to the user within the GUI. The visual output may include text, graphics, video, and any combination thereof. Some or all of the visual output may correspond to user interface objects. In some embodiments, the I / O devices 412 and / or 414 and / or the controllers 432 and / or 434 (along with any associated modules and / or sets of instructions within the medium 401) can detect and track gestures and / or eye movements and can convert the detected gestures and / or eye movements into interactions with graphical objects such as one or more user interface objects. In embodiments where the eye I / O device 412 and / or the eye I / O device controller 432 is configured to track the user's eye movements, the user can interact directly with graphical objects by looking at them.

[0055] Feedback may be provided by the eye I / O device 412 or another I / O device 414, etc., based on the content being displayed and / or the state or states of the computing system. The feedback may be transmitted optically (e.g., an optical signal or a displayed image), mechanically (e.g., tactile feedback, touch feedback, force feedback, or the like), electrically (e.g., electrical stimulation), olfactorily, acoustically (e.g., a beep or the like), or the like, or any combination thereof, and in a variable or non-variable manner.

[0056] The system 400 can also include a power system 444 for powering the various hardware components, which can include a power management system, one or more power sources, a recharge system, a power outage detection circuit, a power converter or inverter, a power status indicator, and typically any other components associated with the generation, management, and distribution of power in a portable device.

[0057] In some embodiments, the peripheral device interface 416, the one or more processors 418, and the memory controller 420 may be implemented on a single chip such as the processing system 404. In some other embodiments, they may be implemented on separate chips.

[0058] In some embodiments, a method is disclosed. The method includes receiving, from a first client application of a computer system, first graphical data comprising a plurality of first nodes; receiving, from a second client application of the computer system, second graphical data comprising a plurality of second nodes; and generating a scene graph, wherein the scene graph describes relationships such as occlusion relationships between the first nodes and the second nodes, and the scene graph is configured to render a scene comprising the nodes when traversed by a processor of the computer system. In addition to or as an alternative to one or more of the above embodiments, the method may further include traversing the scene graph by a processor of the computer system. In addition to or as an alternative to one or more of the above embodiments, the computer system may be configured to communicate with a display, and the method may further include presenting an output comprising at least one node that is not occluded by another node in a scene rendered on the display. In some embodiments, occlusion is a visual obstruction of one node by another when viewing a scene in which an object is rendered from a given perspective. In addition to or as an alternative to one or more of the above embodiments, the computer system may be configured to communicate with a display, and the method may further include displaying an output on the display, such as by displaying a scene rendered to exist on the scene graph or only those of the first or second plurality of nodes that are not occluded while not displaying other nodes. For example, if the second plurality of nodes occlude a portion of the first plurality of nodes, the output displayed may be only the first plurality of non-occluded nodes that do not display any of the second plurality of nodes. In addition to or as an alternative to one or more of the above embodiments, the method may further include applying optimization to the output in the computer system.In addition to, or as an alternative to, one or more of the above embodiments, applying optimization may include culling the surface. In addition to, or as an alternative to, one or more of the above embodiments, the method may further include applying a visual effect to the output in a computer system. In addition to, or as an alternative to, one or more of the above embodiments, applying a visual effect may include calculating a light quantity value. In addition to, or as an alternative to, one or more of the above embodiments, applying a visual effect may include executing a shader. In addition to, or as an alternative to, one or more of the above embodiments, the method may further include applying a physical effect to the output in a computer system. In addition to, or as an alternative to, one or more of the above embodiments, applying a physical effect may include detecting a collision. In addition to, or as an alternative to, one or more of the above embodiments, the first client application may be a first application executed on a computer system, the second client application may be a second application executed on the computer system, and the first client application may be sandboxed on the computer system with respect to the second client application. In addition to, or as an alternative to, one or more of the above embodiments, the first graphical data may correspond to a first client scene graph associated with the first client application, the second graphical data may correspond to a second client scene graph associated with the second client application, the first client scene graph may be sandboxed on the computer system with respect to the second client scene graph, the first client scene graph may be sandboxed on the computer system with respect to the scene graph, and the second client scene graph may be sandboxed on the computer system with respect to the scene graph.In addition to, or alternatively to, one or more of the above-described embodiments, the scene graph may correspond to a version of the versioned scene graph. In addition to, or alternatively to, one or more of the above-described embodiments, the first graphical data may be communicated to the scene graph using a first processing thread of the computer system, and the second graphical data may be communicated to the scene graph using a second processing thread of the computer system independent of the first processing thread.

[0059] In some embodiments, a method is disclosed. The method is to traverse a scene graph of a computer system with a display, where the scene graph comprises first 3D data associated with a first application, the first 3D data comprises one or more nodes, the scene graph comprises second 3D data associated with a second application, the second 3D data comprises one or more nodes, the first application is sandboxed on the computer system with respect to the second application, and the scene graph comprises a relationship between the nodes of the first 3D data and the nodes of the second 3D data, and to display an image corresponding to the scene graph on the display, where the image corresponds to the output of traversing the scene graph and the image reflects either a partial or a complete display of the data. In addition to or instead of one or more of the above embodiments, the relationship may be a spatial relationship. In addition to or instead of one or more of the above embodiments, the method may further include applying optimization to the output of traversing the scene graph in the computer system. In addition to or instead of one or more of the above embodiments, applying the optimization may include culling surfaces. In addition to or instead of one or more of the above embodiments, the method may further include applying visual effects to the output of traversing the scene graph in the computer system. In addition to or instead of one or more of the above embodiments, applying the visual effects may include calculating light intensity values. In addition to or instead of one or more of the above embodiments, applying the visual effects may include executing shaders. In addition to or instead of one or more of the above embodiments, the method may further include applying physical effects to the output of traversing the scene graph in the computer system. In addition to or instead of one or more of the above embodiments, applying the physical effects may include detecting collisions.In addition to, or as an alternative to, one or more of the above-described embodiments, the scene graph may correspond to a version of the versioned scene graph. In addition to, or as an alternative to, one or more of the above-described embodiments, the graphical data corresponding to the first 3D data may be communicated to the scene graph by a host application running on the computer system. In addition to, or as an alternative to, one or more of the above-described embodiments, the graphical data corresponding to the first 3D data may be communicated to the host application by a client of the host application. In addition to, or as an alternative to, one or more of the above-described embodiments, the first graphical data corresponding to the first 3D data may be communicated to the scene graph by a host application using a first processing thread, and the second graphical data corresponding to the second 3D data may be communicated to the scene graph by a host application using a second processing thread independent of the first processing thread.

[0060] In some embodiments, a computer system is disclosed. The system may comprise one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform one or more of the methods described above.

[0061] In some embodiments, a non-transitory computer-readable storage medium is disclosed. When executed by one or more processors, the non-transitory computer-readable storage medium causes the one or more processors to receive, from a first client application of a computer system, first graphical data including a plurality of first nodes; receive, from a second client application of the computer system, second graphical data including a plurality of second nodes; and generate a scene graph, the scene graph describing a relationship between the first nodes and the second nodes, the scene graph being configured to render a scene based on an occlusion relationship when traversed by a processor of the computer system, one or more of the first or second nodes within the plurality of first or second nodes occluding others, and may store instructions for implementing a method. In some embodiments, occlusion is a visual blockage of one node by another when viewing a scene in which an object is rendered from a given perspective. In addition to, or as an alternative to, one or more of the above embodiments, the method may further include traversing the scene graph by a processor of the computer system. In addition to, or as an alternative to, one or more of the above embodiments, the computer system may be configured to communicate with a display, and the method may further include displaying an output on the display, such as by rendering a scene to be present on the scene graph, or only displaying those nodes of the plurality of first or second nodes that are not occluded while not displaying other nodes. For example, if the plurality of second nodes occlude a portion of the plurality of first nodes, the output to be displayed may be only the plurality of first non-occluded nodes that do not display any of the plurality of second nodes. In addition to, or as an alternative to, one or more of the above embodiments, the method may further include applying an optimization to the output in the computer system.In addition to one or more of the above-described embodiments, or as an alternative thereto, applying optimization may include chamfering the surface. In addition to one or more of the above-described embodiments, or as an alternative thereto, the method may further include applying a visual effect to the output in a computer system. In addition to one or more of the above-described embodiments, or as an alternative thereto, applying a visual effect may include calculating a light quantity value. In addition to one or more of the above-described embodiments, or as an alternative thereto, applying a visual effect may include executing a shader. In addition to one or more of the above-described embodiments, or as an alternative thereto, the method may further include applying a physical effect to the output in a computer system. In addition to one or more of the above-described embodiments, or as an alternative thereto, applying a physical effect may include detecting a collision. In addition to one or more of the above-described embodiments, or as an alternative thereto, the first client application may be a first application executed on a computer system, the second client application may be a second application executed on the computer system, and the first client application may be sandboxed on the computer system with respect to the second client application. In addition to one or more of the above-described embodiments, or as an alternative thereto, the first graphical data may correspond to a first client scene graph associated with the first client application, the second graphical data may correspond to a second client scene graph associated with the second client application, the first client scene graph may be sandboxed on the computer system with respect to the second client scene graph, the first client scene graph may be sandboxed on the computer system with respect to the scene graph, and the second client scene graph may be sandboxed on the computer system with respect to the scene graph.In addition to, or as an alternative to, one or more of the above-described embodiments, the scene graph may correspond to a version of the versioned scene graph. In addition to, or as an alternative to, one or more of the above-described embodiments, the first graphical data may be communicated to the scene graph using a first processing thread of the computer system, and the second graphical data may be communicated to the scene graph using a second processing thread of the computer system independent of the first processing thread.

[0062] In some embodiments, a non-transitory computer-readable storage medium is disclosed. When executed by one or more processors, the non-transitory computer-readable storage medium causes the one or more processors to traverse a scene graph of a computer system with a display, the scene graph comprising first 3D data associated with a first application, the first 3D data comprising one or more nodes, the scene graph comprising second 3D data associated with a second application, the second 3D data comprising one or more nodes, the first application being sandboxed on the computer system with respect to the second application, the scene graph comprising relationships between the nodes of the first 3D data and the nodes of the second 3D data, and to display an image corresponding to the scene graph on the display, the image corresponding to the output of traversing the scene graph and the image reflecting the relationships. The non-transitory computer-readable storage medium may store instructions for implementing the method. In addition to or as an alternative to one or more of the above embodiments, the relationships may be spatial relationships. In addition to or as an alternative to one or more of the above embodiments, the method may further comprise applying optimizations in the computer system to the output of traversing the scene graph. In addition to or as an alternative to one or more of the above embodiments, applying the optimizations may comprise culling surfaces. In addition to or as an alternative to one or more of the above embodiments, the method may further comprise applying visual effects in the computer system to the output of traversing the scene graph. In addition to or as an alternative to one or more of the above embodiments, applying the visual effects may comprise calculating light intensity values. In addition to or as an alternative to one or more of the above embodiments, applying the visual effects may comprise executing shaders. In addition to or as an alternative to one or more of the above embodiments, the method may further comprise applying physical effects in the computer system to the output of traversing the scene graph.In addition to, or as an alternative to, one or more of the above embodiments, applying a physical effect may include detecting a collision. In addition to, or as an alternative to, one or more of the above embodiments, the scene graph may correspond to a version of a versioned scene graph. In addition to, or as an alternative to, one or more of the above embodiments, graphical data corresponding to the first 3D data may be communicated to the scene graph by a host application running on a computer system. In addition to, or as an alternative to, one or more of the above embodiments, graphical data corresponding to the first 3D data may be communicated to the host application by a client of the host application. In addition to, or as an alternative to, one or more of the above embodiments, the first graphical data corresponding to the first 3D data may be communicated to the scene graph by a host application using a first processing thread, and the second graphical data corresponding to the second 3D data may be communicated to the scene graph by a host application using a second processing thread independent of the first processing thread.

[0063] In some embodiments, a computer system is disclosed. The system includes one or more processors, a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform a method including receiving, in the computer system, first scene data from a first client application and receiving, in the computer system, second scene data from a second client application, generating, by the one or more processors based on the first scene data and the second scene data, a graphical data structure that is configured to result in an output corresponding to an image on a display when provided as an input to a rendering operation executed by the one or more processors. In addition to or instead of one or more of the above embodiments, the graphical data structure may be at least one of a display list and a display tree. In addition to or instead of one or more of the above embodiments, the method may further include executing a rendering operation using the graphical data structure as an input. In addition to or instead of one or more of the above embodiments, the computer system may further include a display, and the method may further include displaying an image on the display. In addition to or instead of one or more of the above embodiments, the first client application may be a first application executed by one or more processors of a first device, and the second client application may be a second application executed by one or more processors of the first device. In addition to or instead of one or more of the above embodiments, the first client application may be a first application executed by one or more processors of a first device, and the second client application may be a second application executed by one or more processors of a second device.In addition to, or as an alternative to, one or more of the above-described embodiments, the storage device may further be configured to receive third scene data from a third client application. In addition to, or as an alternative to, one or more of the above-described embodiments, the method may further include deleting the first scene data from the storage device. In addition to, or as an alternative to, one or more of the above-described embodiments, the graphical data structure may include first data and second data, and the method may further include determining whether the first data corresponds to an occluded view or a non-occluded view, rendering an image including the non-occluded view based on the first data in response to determining that the first data corresponds to the non-occluded view, and rendering an image not including the occluded view in response to determining that the first data corresponds to the occluded view. In addition to, or as an alternative to, one or more of the above-described embodiments, the storage device may further be configured to store the first scene data as a first version in a version control system in response to receiving the first scene data. In addition to, or as an alternative to, one or more of the above-described embodiments, the storage device may further be configured to receive third scene data from the first client application and store the third scene data as a second version in the version control system. In addition to, or as an alternative to, one or more of the above-described embodiments, the method may further include deleting the first version from the storage device in response to generating the graphical data structure. In addition to, or as an alternative to, one or more of the above-described embodiments, the method may be performed in parallel with the storage device receiving the third scene data. In addition to, or as an alternative to, one or more of the above-described embodiments, the storage device may be configured to receive the first scene data in parallel with receiving the second scene data.In addition to, or alternatively to, one or more of the above-described embodiments, the memory device may be configured to receive third scene data at a first interval corresponding to a first data rate, and the method may further include adjusting the length of the first interval to correspond to a second data rate. In addition to, or alternatively to, one or more of the above-described embodiments, the first scene data may comprise at least one of new data, deleted data, and a change in the relationship between data.

[0064] In some embodiments, a computer system is disclosed. The computer system may comprise a server, server data, a first client application, and a second client application. At the server, receive first unprocessed scene data from the first client application, at the server, receive second unprocessed scene data from the second client application, at the server, incorporate the first unprocessed scene data from the first client application, the second unprocessed scene data from the second client application, and the server data into a unified scene data structure, at the server, execute at least a portion of the data contained within the unified scene data structure, and be configured to create a graphical data structure based on the data executed within the unified scene data structure. In addition to or alternatively to one or more of the above embodiments, the graphical data structure may be a display list or a display tree. In addition to or alternatively to one or more of the above embodiments, the computer system may further comprise a rendering engine configured to render the graphical data structure into a processed image. In addition to or alternatively to one or more of the above embodiments, the computer system may further comprise a display configured to display the processed image. In addition to or alternatively to one or more of the above embodiments, the display may be capable of displaying virtual content while maintaining at least a partial view of the physical world. In addition to or alternatively to one or more of the above embodiments, the first client application and the second client application may be two different applications launched on a single physical device. In addition to or alternatively to one or more of the above embodiments, the first client application and the second client application may be two different applications launched on separate physical devices, respectively.In addition to, or as an alternative to, one or more of the above embodiments, the server may be configured to receive third unprocessed scene data from a third client application. In addition to, or as an alternative to, one or more of the above embodiments, the server may be configured to delete unprocessed scene data from the first client application after execution of the unprocessed scene data from the first client application. In addition to, or as an alternative to, one or more of the above embodiments, the rendering engine may further comprise an occlusion module configured to separate data within a graphical data structure into a first occluded category and a second non-occluded category and display the second non-occluded category. In addition to, or as an alternative to, one or more of the above embodiments, the server may be configured to store the first unprocessed scene data from the first client application as a first version. In addition to, or as an alternative to, one or more of the above embodiments, the server may be configured to store the third unprocessed scene data from the first client application as a second version. In addition to, or as an alternative to, one or more of the above embodiments, the computer system may be configured to store the first version of the first unprocessed scene data from the first client application from the time the first version of the first unprocessed scene data from the first client application is received by the server until the first unprocessed scene data from the first client application is read and executed. In addition to, or as an alternative to, one or more of the above embodiments, the server may be configured to receive the first unprocessed scene data from the first client at the same time the server receives the second unprocessed scene data from the second client. In addition to, or as an alternative to, one or more of the above embodiments, the server may be configured to reduce the rate at which the first client application transmits unprocessed scene data to the server.In addition to, or alternatively to, one or more of the above-described embodiments, the data received from the first and second client applications may be at least one selected from the group consisting of new data, deleted data, changes in the relationships between previously transferred data, and modified data.

[0065] The disclosed embodiments have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. For example, the elements of one or more implementations may be combined, deleted, modified, or supplemented to form further implementations. Such changes and modifications are understood to be included within the scope of the disclosed embodiments as defined by the appended claims.

Claims

1. 1. A method comprising: communicating first graphical data comprising a first plurality of nodes from a first client application to a computer system, the computer system being in communication with a second client application; receiving a scene graph from the computer system, the scene graph describes a first occlusion relationship between at least one node of the first plurality of nodes and at least one node of a second plurality of nodes of a second graphical data of the second client application; the scene graph further describes a second occlusion relationship between at least one node of the first plurality of nodes and at least one node of the second plurality of nodes; the first occlusion relationship is configured such that at least a first node of the second plurality of nodes occludes at least a first node of the first plurality of nodes; the second occlusion relationship is configured such that at least a second node of the first plurality of nodes occludes at least a second node of the second plurality of nodes; creating a rendered scene based on the first occlusion relations, the rendered scene comprising a subset of the first plurality of nodes; Including, The method, wherein the scene graph corresponds to a version of a versioned scene graph.

2. The method of claim 1 , wherein creating the rendered scene includes traversing the scene graph.

3. 2. The method of claim 1 , further comprising: displaying the rendered scene, wherein displaying the rendered scene comprises displaying the at least a first node of the second plurality of nodes while refraining from displaying the at least a first node of the first plurality of nodes.

4. The method of claim 1 , wherein the first client application comprises a first application executing on the computer system, and the second client application comprises a second application executing on the computer system.

5. The method of claim 4 , wherein the first client application is sandboxed on the computer system relative to the second client application.

6. 2. The method of claim 1 , wherein the first client application comprises an application executing on a second computer system, the second computer system being in communication with the computer system over a network, and receiving the scene graph from the computer system comprises receiving the scene graph at the second computer system from the computer system over the network.

7. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising: communicating first graphical data comprising a plurality of first nodes from a first client application to a computer system, the computer system being in communication with a second client application; receiving a scene graph from the computer system, the scene graph describes a first occlusion relationship between at least one node of the first plurality of nodes and at least one node of a second plurality of nodes of a second graphical data of the second client application; the scene graph further describes a second occlusion relationship between at least one node of the first plurality of nodes and at least one node of the second plurality of nodes; the first occlusion relationship is configured such that at least a first node of the second plurality of nodes occludes at least a first node of the first plurality of nodes; the second occlusion relationship is configured such that at least a second node of the first plurality of nodes occludes at least a second node of the second plurality of nodes; creating a rendered scene based on the first occlusion relations, the rendered scene comprising a subset of the first plurality of nodes; Including, The non-transitory computer readable medium, wherein the scene graph corresponds to a version of a versioned scene graph.

8. The non-transitory computer-readable medium of claim 7 , wherein creating the rendered scene includes traversing the scene graph.

9. 8. The non-transitory computer-readable medium of claim 7, the method further comprising displaying the rendered scene, wherein displaying the rendered scene comprises displaying the at least a first node of the second plurality of nodes while refraining from displaying the at least a first node of the first plurality of nodes.

10. 8. The non-transitory computer-readable medium of claim 7, wherein the first client application comprises a first application executing on the computer system and the second client application comprises a second application executing on the computer system.

11. 11. The non-transitory computer-readable medium of claim 10, wherein the first client application is sandboxed on the computer system relative to the second client application.

12. 8. The non-transitory computer-readable medium of claim 7, wherein the first client application comprises an application executing on a second computer system, the second computer system being in communication with the computer system over a network, and receiving the scene graph from the computer system comprises receiving the scene graph at the second computer system from the computer system over the network.

13. 1. A system comprising one or more processors configured to perform a method, the method comprising: communicating first graphical data comprising a plurality of first nodes from a first client application to a computer system, the computer system being in communication with a second client application; receiving a scene graph from the computer system, the scene graph describes a first occlusion relationship between at least one node of the first plurality of nodes and at least one node of a second plurality of nodes of a second graphical data of the second client application; the scene graph further describes a second occlusion relationship between at least one node of the first plurality of nodes and at least one node of the second plurality of nodes; the first occlusion relationship is configured such that at least a first node of the second plurality of nodes occludes at least a first node of the first plurality of nodes; the second occlusion relationship is configured such that at least a second node of the first plurality of nodes occludes at least a second node of the second plurality of nodes; creating a rendered scene based on the first occlusion relations, the rendered scene comprising a subset of the first plurality of nodes; Including, The system, wherein the scene graph corresponds to a version of a versioned scene graph.

14. The system of claim 13 , wherein creating the rendered scene includes traversing the scene graph.

15. 14. The system of claim 13, wherein the method further includes displaying the rendered scene, and wherein displaying the rendered scene includes displaying the at least a first node of the second plurality of nodes while refraining from displaying the at least a first node of the first plurality of nodes.

16. 14. The system of claim 13, wherein the first client application comprises a first application executing on the computer system and the second client application comprises a second application executing on the computer system.

17. 20. The system of claim 16, wherein the first client application is sandboxed on the computer system relative to the second client application.

18. 14. The system of claim 13, wherein the first client application comprises an application running on a second computer system, the second computer system in communication with the computer system over a network, and receiving the scene graph from the computer system comprises receiving the scene graph at the second computer system from the computer system over the network.

Images