Centralized rendering
Through the centralized scene graph structure and client-server architecture, the realistic and efficient graphics data rendering problems in multi-application systems are solved, and efficient rendering and security in augmented reality systems are achieved.
Patent Information
- Application Number
- CN202411112839.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-06-18
- Filing Date
- 2019-06-18
- Publication Date
- 2024-12-06
AI Technical Summary
Existing systems are difficult to achieve realistic occlusion and efficient rendering when rendering graphics data from multiple computer applications onto a single display, especially in augmented or mixed reality systems, resulting in poor user experience and difficult Adapt to the needs of different users and hardware.
Using a centralized scene graph structure, 3D data from multiple independent applications is integrated in one central location, traversed through the structure through the renderer to achieve realistic occlusion and visual effects, and separating applications with client-server architecture to improve computing efficiency and security.
Realize realistic graphics data rendering on a single display, improve rendering efficiency and user experience, adapt to different users and hardware needs, while providing security and scalability.
Smart Images

Figure CN119091021A_ABST
Abstract
Description
[0001] This application is a divisional application of the Chinese patent application “Centralized Rendering” with application number 201980051078.7 (filing date is June 18, 2019).
[0002] CROSS-REFERENCE TO RELATED APPLICATIONS
[0003] This application claims the benefit of U.S. Patent Application No. 16 / 011,431, filed June 18, 2018, which is hereby incorporated by reference in its entirety for all purposes. Technical Field
[0004] The present disclosure relates generally to systems and methods for visually rendering graphical data on a display, and more particularly, to systems and methods for visually rendering data from multiple computer applications on a single display. Background Art
[0005] Various techniques exist for rendering graphics data for computer applications to displays. It is desirable for these techniques to render graphics data realistically (i.e., consistent with a viewer's expectations based on the physical world) and efficiently. It is also desirable for rendering techniques to accommodate computer systems having various topologies, including, for example, computer systems in which multiple applications contribute graphics data to be displayed on a single display.
[0006] Conventional systems are often unable to realistically and efficiently render content in such multi-application systems. For example, in some such systems, rendering graphics data from multiple applications onto a single display can cause the data to be incorrectly sorted on the display, producing unexpected visual results that compromise the realism of the display. Furthermore, graphics data from one application may not realistically interact with graphics data from another application (e.g., through lighting and shadowcasting effects or through shaders). In addition, some such systems are limited in their ability to use rendering optimizations (such as culling invisible surfaces) to improve computational efficiency.
[0007] Systems involving augmented reality (AR) or "mixed reality" are particularly in need of better solutions for the problem of rendering graphical data from multiple applications onto a single display. For example, AR systems have the potential for multiple users to interact in a shared virtual space, with virtual content from all users rendered onto a single display. For such interactions to be credible and meaningful to users, this may require the AR system's graphical output to be convincing and consistent with the user's visual expectations; flexible enough to accommodate different types and numbers of users, user hardware and software, and the different ways in which users wish to interact with the system; and efficient enough to maintain continuous operation at high frame rates and maximize the battery life of mobile devices. Furthermore, it is desirable for applications and application data associated with a single user in an AR system to remain isolated from other users to provide security (which can be compromised by data access between untrusted users) and maintain scalability, especially as the number of system users grows. Furthermore, such systems may benefit from minimizing technical constraints on users and their applications; for example, limiting the hardware requirements for users to participate in the AR system would encourage greater user participation. This can be achieved by limiting the extent to which a single user or an application running on the user's hardware needs to perform complex rendering operations, for example by offloading such operations to a shared system, such as a server-side host application running on dedicated hardware. Summary of the Invention
[0008] Examples of the present disclosure describe a computer system in which multiple applications contribute graphics data to be displayed on a single display. Examples of the present disclosure can be used to render graphics data realistically (i.e., consistent with a viewer's expectations based on the physical world) and efficiently. According to examples of the present disclosure, first graphics data can be received from a first client application, and second graphics data can be received from a second independent client application. The first graphics data and the second graphics data can be combined into a "centralized" data structure such as a scene graph, which can be used to describe the relationship between the nodes represented by the first graphics data and the second graphics data. Therefore, the centralized data structure can be used to render a scene reflecting the first graphics data and the second graphics data to a display in a realistic and efficient manner. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Figures 1A-1E An example computer system including a graphics display according to examples of the present disclosure is shown.
[0010] Figure 2A An example data flow in an example computer system according to examples of the present disclosure is shown.
[0011] Figure 2B Example renderer output corresponding to an example computer system according to examples of the present disclosure is shown.
[0012] Figure 2C An example data flow in an example computer system including multiple independent applications according to examples of the present disclosure is shown.
[0013] Figure 2D Example renderer output corresponding to an example computer system including multiple independent applications according to examples of the present disclosure is shown.
[0014] Figure 3A Components of an example computer system that can render 3D data from multiple independent applications to a display using a centralized scene graph are shown according to examples of the present disclosure.
[0015] Figure 3B Aspects of example client applications are shown relative to an example computer system including multiple independent client applications according to examples of the present disclosure.
[0016] Figure 3C Aspects of an example client-server interface according to examples of the present disclosure are shown with respect to an example computer system including multiple independent applications.
[0017] Figure 3D Aspects of an example host application 340 are shown relative to an example computer system including multiple independent client applications according to examples of the present disclosure.
[0018] Figure 3E Aspects of an example renderer 360 are shown relative to an example computer system including multiple independent applications according to examples of the present disclosure.
[0019] Figure 4 An example of a system architecture according to examples of the present disclosure that may be embodied within any portable or non-portable device is shown. DETAILED DESCRIPTION
[0020] In the following description of the examples, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific examples that may be practiced. It will be understood that other examples may be used and structural changes may be made without departing from the scope of the disclosed examples.
[0021] Figures 1A to 1E Various example computer systems with displays are shown. Figure 1A An example desktop computer connected to an external monitor is shown. Figure 1B An example laptop computer including a display is shown. Figure 1C An example mobile device including an integrated display is shown. Figure 1D An example television including a display is shown. Figure 1EAn example computer system including a head-mounted display is shown. The present disclosure is not limited to any particular type of computer system, any particular type of display, or any particular means of connecting the computer system to the display. The present disclosure is further not limited to two-dimensional displays; in particular, three-dimensional displays, such as stereoscopic displays, are contemplated.
[0022] In some example computer systems, data to be graphically presented on a display (as a "rendered scene") includes data representing objects in three-dimensional space (e.g., 2D or 3D geometric primitives, including polygons) ("3D data"), and presenting the 3D data on the display includes presenting an image corresponding to the objects in the three-dimensional space as viewed from a view origin oriented along a view axis ("display scene"). For example, in a software application running on the computer system (e.g., a video game using a 3D engine), the 3D data may include spatial coordinates, orientations, and / or visual attributes of objects in the three-dimensional game world, as well as data describing the view origin and view axes in the game world. The 3D data may also include data related to textures associated with the objects to be rendered, shader parameters related to the objects, and other information that affects how the objects are displayed. For example, during a "rendering" or "drawing" phase, the game may direct the software and / or hardware "pipeline" to create a rendered scene for presentation on the display as a display scene. In such examples, it is often desirable that the resulting image reflect the user's expectations of the visual world. In particular, it is often desirable for a first opaque object that is closer to the view origin to occlude a second object that is behind the first object. Objects that are not properly occluded may confuse the user and may not clearly convey the object's location in three-dimensional space. In some example computer systems, occlusion is implemented through sorting, where objects closer to the view origin are sorted or drawn above objects farther from the view origin.
[0023] Sorting multiple objects for presentation on a display so that one object realistically obscures another requires information about the relationships between objects, such as the spatial relationships between objects in three-dimensional space. Some example computer systems utilize scene graphs to represent the relationships (e.g., hierarchical relationships) between one or more objects (such as objects to be rendered as a scene). As used herein, a scene graph is any data structure representing such a relationship. For example, in a scene graph, the rendered objects to be presented can be represented as nodes in a graph, wherein the relationships between the nodes represent the logical or spatial relationships between the objects. The renderer can then traverse the scene graph according to techniques known in the art to render or prepare to display at least one object in a manner that will achieve appropriate obscuration. In other words, the renderer can create a scene of objects with nodes, but the presentation on the display may only be a subset of the rendered objects, so that an object obscured by another object in the renderer will only be partially presented in the resulting display scene (in such an embodiment, the output is the non-obscured portion of the object). If a user only wants content represented by a second object running from a second application to be visible for a given period of time, such selective display may be beneficial for obscuring content represented by a first object running from a first application. In some examples, a scene graph is an intermediate data structure between an application that includes 3D data and a renderer that renders the 3D data for presentation to a screen: in some examples, the application writes scene information to a scene graph that can later be used by the renderer to render a scene or output a display scene.
[0024] Figure 2AAn example data flow in an example computer system 200 is shown. In system 200, a single application 210 can write data to a scene graph 240, which a renderer 250 can use to render an object 220 for presentation on a display 290. For example, object 220 can include multiple objects (e.g., polygons) that together comprise a 3D representation of two human hands (hand 232 and hand 236); and the application can direct object 220 to be presented on the display, for example, from the perspective of a view origin oriented along a view axis during a rendering or drawing phase. In this example, hands 232 and 236 are interlocked in a handshake; due to the relative positioning of the hands, a viewer would expect some portions of hand 232 to occlude portions of hand 236, while some polygons comprising hand 236 would occlude portions of hand 232, relative to the view origin and the view axis. Application 210 can write information describing the relationships between objects 220 (e.g., spatial relationships between polygons comprising objects 220) into scene graph 240, which can be used to identify which polygons should occlude other polygons, i.e., which polygons should be ordered to be displayed above other polygons. For example, scene graph 240 can reflect that polygon 234 (which belongs to hand 232) is between the view origin and the polygon comprising hand 236 and, therefore, should occlude those polygons in hand 236; and polygon 238 (which belongs to hand 236) is between the view origin and the polygon comprising hand 232 and, therefore, should occlude those polygons in hand 232. Renderer 250 can then output objects 220 or a subset (e.g., only hand 232, or a non-occluded portion of hand 232, or only hand 236) consistent with the desired occlusion for presentation to display 290.
[0025] Figure 2B Shown Figure 2A Example output of the renderer 250 of the example computer system 200 is shown. Figure 2A In the depicted example, based on the relative positions of objects 220 , the viewer expects some objects (eg, polygon 234 ) including hand 232 to occlude hand 236 , and expects some objects (eg, polygon 238 ) including hand 236 to occlude hand 232 . Figure 2B The example output shown is consistent with the expected occlusion. That is, Figure 2A The object 220 in FIG. 2 is correctly displayed to present a handshake on the display 290 that is consistent with the viewer's expectations.
[0026] exist Figure 2A and 2BIn the example computer system 200 shown in FIG, scene graph 240 is written directly by only a single application (application 210). Renderer 250 then traverses scene graph 240 to render hand 232 and hand 236 with appropriate occlusion. When a scene graph (such as scene graph 240) receives direct input from multiple independent applications, conventional systems (such as example computer system 200) that use a scene graph as part of the rendering process may not be able to occlude objects correctly. In these cases, unlike example computer system 200, no single application can provide the scene graph with all the object relationship data that may be needed to correctly order objects on the display.
[0027] Figure 2C An example data flow in an example computer system 201 using two independent applications is shown, and the occlusion problem described above is illustrated. Unlike the example computer system 200 described above, the example computer system 201 includes two independent applications, namely, application 212 and application 214. In the example computer system 201, both application 212 and application 214 write data to the scene graph 240 to render their respective 3D data to a single display 290. Figure 2C In this example, application 212 attempts to render and present object 222 (which includes an object containing hand 232); and application 214 attempts to render and present object 224 (which includes an object containing hand 236). Figure 2A and 2B As in the example described in , if hand 232 and hand 236 are displayed simultaneously in the same 3D environment, they will interlock in a handshake such that the viewer expects that portions of hand 232 occlude hand 236 , and vice versa.
[0028] Figure 2CThe example shown in may have difficulty achieving realistic occlusion of objects being rendered. In this example, application 212 may write data corresponding to object 222 (including hand 232) to scene graph 240, and application 214 may write data corresponding to object 224 (including hand 236) to the same scene graph 240. However, in example computer system 201, if application 212 and application 214 are independent ("sandboxed") applications, application 212 cannot access data related to application 214's object 224 (including hand 236 and its constituent objects), and similarly, application 214 cannot access data related to application 212's object 222 (including hand 232 and its constituent objects). That is, in some examples, neither application 212 nor application 214 can fully identify the relationship between object 222 and object 224. Consequently, neither application 212 nor application 214 can write to scene graph 240 the information needed to identify which objects occlude other objects or the order in which objects should be sorted on the display screen.
[0029] Figure 2D Shown Figure 2C Example output of the renderer 250 of the example computer system 201 is shown. Figure 2C In the depicted example, based on the relative positioning of objects 222 and 224, the viewer would expect some portions of hand 232 to occlude hand 236, and some portions of hand 236 to occlude hand 232. Figure 2A and Figure 2B Unlike the scene graph 240 of FIG. C , the objects 222 and 224 are not properly sorted to produce the desired occlusion as described above. Instead, in the example shown, the entirety of object 222 is sorted above the entirety of object 224. Figure 2D The example output shown is inconsistent with the expected occlusion. Therefore, the resulting handshake image may not accurately reflect the objects in application 212 and application 214; moreover, the image may appear unnatural and may be confusing to the viewer.
[0030] When used with multiple independent applications such as in the example computer system 201, a traditional scene graph (e.g. Figures 2A to 2D Other disadvantages of the scene graph 240 in FIG. 240 are obvious. For example, it is possible to use Figure 2A Rendering efficiency is achieved by using data corresponding to the entire scene rendered in the application 210. By knowing which surfaces will be occluded, e.g. Figure 2A The example computer system 200 in FIG. 200 can instruct the system 200 to cull those surfaces, thereby avoiding unnecessary consumption of computing resources. Figure 2CIn the example computer system 201 in ), such culling may not be possible because each application may not have the scene knowledge to determine which surfaces should be culled. In addition, in some examples involving only a single application (such as the example computer system 200), beneficial effects can be applied to objects based on the presence of other objects in the scene. For example, applying realistic lighting and shadow effects to an object may require data corresponding to nearby objects. In addition, some shader effects will benefit from such data. Similarly, effects generated by particle systems or collision detection systems may benefit from such data. In systems where multiple independent applications provide 3D data, such effects may be limited or impossible because no single application may be able to provide all the node relationship information required to apply such an effect.
[0031] The present disclosure proposes systems and methods that use a centralized scene graph to address the above-mentioned shortcomings of systems that render 3D data from multiple independent applications. Figure 2C In the example computer system 201 in FIG. 1 , a centralized scene graph may be used instead of a traditional scene graph, such as Figure 2C 240 in a centralized scene graph. As described herein, in some examples, a centralized scene graph can include a system that receives 3D data from multiple separate input sources; writes information corresponding to the 3D data to a central location; and maintains the information for access by a renderer that creates a rendered scene including objects based on the 3D data. The rendered scene can be used to generate output (such as graphical output) that reflects realistic object occlusion; computational efficiency; visual effects (e.g., lighting and shadow casting); or physical effects (e.g., collision detection), or partial display of occluded objects, which would be difficult or impossible to achieve in a system that does not utilize a centralized scene graph.
[0032] In some examples, an example computer system includes multiple applications, each of which includes 3D data representing one or more objects in a common 3D environment. Each of the multiple applications can exist in a "sandboxed" environment, making it agnostic to the other applications: for example, each respective application's data can be independent of each other application's data; each application may not be able to access each other application's data; and, although each application's 3D data may correspond to the same 3D environment, each application maintains its own instance of the 3D environment. For example, each application can represent a player in an online multiplayer video game, where each player exists in the same instance of the game world or 3D environment but cannot directly access the data of other players. In such an example, it may be desirable to render all players simultaneously in a single instance of the game world, but it may be undesirable (or computationally prohibitive) for each player to maintain the information required to render the 3D data of each other client participant. Furthermore, for security purposes, it may be desirable to limit the player's information available to other players.
[0033] In some examples, each of multiple sandboxed applications can independently write information corresponding to its 3D data to a local scene graph, which is then written to a common centralized scene graph. The centralized scene graph can then be traversed by a renderer to render a scene based on the collective 3D data provided by each application for presentation as an image on a display. By transferring the 3D data from each of the multiple sandboxed applications to a single centralized scene graph, the renderer can apply beneficial techniques that require or benefit from knowledge of the 3D data of all applications, such as occlusion, lighting effects, and rendering optimizations (e.g., surface culling). These benefits are achieved while limiting the computational overhead required by each sandboxed application: from the perspective of an individual application, all the application needs to do is update a single scene graph to reflect its 3D data, with other operations performed by another component of the system. In addition, security benefits can be obtained by maintaining separation between sandboxed applications.
[0034] Figure 3AComponents of an example computer system 300 are shown, which can render 3D data from multiple independent applications to a display using a centralized scene graph. The example shown utilizes a client-server topology; however, the present disclosure is not limited to the client-server example. In the example computer system 300, a first client application 310 and a second client application 320 each transmit 3D data (in some examples, over a network) to a client-server interface 330. In some examples, client applications 310 and 320 are "sandboxed" applications that operate independently of each other and independently transmit their 3D data to the client-server interface 330. The client-server interface 330 can receive updated 3D data from the client applications 310 and 320 and transmit the 3D data (in some examples, over a network) to a server-side host application 340. In some examples, the client-server interface 330 uses multithreading techniques to receive 3D data, process the 3D data, and / or transmit the 3D data to the host application 340 using multiple processor threads. In some examples, the client-server interface includes logic to control (e.g., by throttling) the rate at which 3D data is transmitted to the host application 340. The host application 340 can use the 3D data received from the client-server interface to update the centralized scene graph 350 so that the centralized scene graph 350 reflects the 3D data received from the client applications 310 and 320. In some examples, the centralized scene graph 350 includes multiple versions of the scene graph, and known versioning techniques are used to allow updates to the centralized scene graph 350 to occur in parallel. The renderer 360 can then traverse the centralized scene graph 350, apply optimizations and effects as appropriate, and generate output to be displayed on a display 370, such as a computer monitor (e.g., graphical output including data of at least one of the client applications 310 and 320, and in some embodiments, including only the occluded portions of one client application without the occluding application data).
[0035] Figure 3B Shows relative to Figure 3AAspects of an example client application 310 of an example computer system 300 are shown. In the example shown, 3D data 312 represents graphical objects (e.g., geometric primitives, such as polygons) in a 3D environment to be rendered on a display 370. 3D data 312 can be updated 314 by client application 310. For example, if client application 310 is an application with a rendering loop that iterates sixty times per second, client application 310 can update 3D data 312 sixty times per second to reflect changes to the data during the application's operation that should be reflected in the rendered output. In some examples, 3D data 312 is represented as a local scene graph 316 that can be local to each client application 310. In some examples, local scene graph 316 can include data (e.g., nodes) corresponding to data in centralized scene graph 350. As 3D data 312 is updated 314, client application 310 can update local scene graph 316 to reflect the latest version of 3D data 312. Because the local scene graph 316 is updated, the client application 310 can use it to generate (317) client data 318. In some examples, the client data 318 can fully represent the local scene graph 316. In some examples, the client data 318 can represent changes made to the local scene graph 316 since the previous client data 318 was sent to the client-server interface 330. For example, the client data 318 can include: nodes added to or deleted from the local scene graph 316; changes to relationships between nodes in the local scene graph 316; or changes to properties of nodes in the local scene graph 316. In some examples, the client data 318 can use identifiers (e.g., identification numbers corresponding to scene graph nodes) to identify relationships between data from the local scene graph 316 and corresponding data on the centralized scene graph 350. The client data 318 can then be transmitted to the client-server interface 330 for ultimate transmission to the host application 340. In some examples, the communication of the client data 318 to the client-server interface 330 can occur over a network. In some examples, a client helper application can be used in conjunction with client application 310 to generate client data 318 from local scene graph 316 or 3D data 312 .
[0036] The aspects described with respect to client application 310 may be similarly described with respect to client application 320 or other client applications comprising example computer system 300 (along with client application 310). Those skilled in the art will appreciate that the systems and methods described herein can be extended to include any number of client applications and client data, and that the present disclosure is not limited to any such number; furthermore, as the number of client applications increases, some benefits (e.g., improved computational efficiency) may become more pronounced. As described above, client applications 310 and 320 may be sandboxed applications that do not share data or functionality. For example, in example computer system 300, client application 320 may have its own 3D data and local scene graph, separate from 3D data 312 and local scene graph 316 belonging to client application 310. However, in some examples comprising example computer system 300, a single client-server interface 300 is shared by multiple client applications (e.g., client applications 310 and 320).
[0037] Figure 3C Shows relative to Figure 3A and Figure 3B Aspects of the example client-server interface 330 of the example computer system 300 are shown. In this example, client data 318 and client data 328 are as described above with respect to Figure 3BThe client data is transmitted to or updated by the respective client applications 310 and 320. In some examples, the client data 318 and 328 may be updated at different rates on the client-server interface 330. This may occur, for example, if one client application executes on less powerful computing hardware than another client application (causing the client application to update its client data less frequently); if one client application communicates with the client-server interface 330 over a lower bandwidth network than the other client application; or if the client data associated with one client application is more complex (and requires more processing time to generate) than the client data associated with the other client application. The different rates at which client data is updated on the client-server interface 330 may also be the result of a temporary spike in operating conditions, such as if a network failure causes a client application to be temporarily offline. It may be desirable, for example, that the computer system 300 tolerate different rates of updating client data; for example, it may be desirable that a network failure affecting one client application not negatively impact the rate at which the centralized scene graph 350 is updated using client data from other client applications or the rate at which the scene is rendered. It may also be desirable to ensure that client data from one client application does not lag too far behind or get too far ahead of client data from other client applications when updating the centralized scene graph 350, as this could cause instability or desynchronization of the centralized scene graph or displays rendered by the client applications.
[0038] In some examples, the client-server interface 330 is used to handle differences or fluctuations in the rate at which client data is updated. Figure 3C, the example client-server interface 330 can receive client data 318 and 328 via independent processing threads 332 and 334, respectively, and can include a thread manager 336 to handle the threads. Utilizing multiple threads to update client data from different sources (e.g., client application 310 and client application 320) can prevent issues where one source blocks or otherwise negatively impacts data from other sources. In the example shown, the thread manager 336 can input client data 318 and 328 from client applications 310 and 320, respectively, using threads 332 and 334, and output host data 319 and 329 (corresponding to client data 318 and 328, client applications 310 and 320, and threads 332 and 334, respectively) to the host application 340. The thread manager 336 can include logic to handle threads 332 and 334, to identify and resolve throughput issues or other issues related to threads 332 and 334, and / or to control the output of host data 319 and 329. For example, if client data 318 and client data 328 (by threads 332 and 334, respectively) are updated at approximately the same rate, thread manager 336 can simply update host data 319 and 329 (corresponding to client data 318 and 319, respectively) at approximately the same rate. However, if client data 318 is updated at a much faster rate than client data 328, thread manager 336 can throttle client data 318 (e.g., by transmitting it to host application 340 less frequently) to prevent it from overtaking client data 328. Thread manager 336 can also control the overall rate at which host data is updated. For example, thread manager 336 can throttle the rate at which host data 319 and / or 329 are updated to prevent the host data from being updated faster than host application 340 can process it (which could result in undesirable desynchronization of client applications 310 and / or 320, centralized scene graph 350, and / or output to display 370).
[0039] Figure 3D Shows relative to Figures 3A-3C340 . Described herein are operations performed within a thread 341 within the host application 340, where the thread 341 can execute concurrently with additional threads within the host application 340. In some examples, multithreading within the host application 340 can have the advantage of allowing multiple client applications or multiple sets of host data to update the same centralized scene graph 350 simultaneously (in some examples, by updating different versions of the centralized scene graph). This, in turn, can increase the overall throughput of the rendered scene from client data to a display. In some examples, multiple threads may need to place locks on the centralized scene graph data, for example, to prevent threads from inadvertently writing to the same data. However, in some examples, one or more of the operations described may not be performed within a thread.
[0040] exist Figure 3D In the example shown above, Figure 3C As described above, host data 319 (corresponding to client application 310 and client data 318) is updated (342). Host application 340 can then identify changes that host data 319 may have made to the previous version of centralized scene graph 350. For example, host application 340 can identify that host data 319 will add a node, delete a node, change the relationship between two nodes, or change the properties of a node relative to centralized scene graph 350. (In some examples (e.g., Figure 3D ), the host application 340 may perform these operations or other operations using a host data handler 344. ) The host application 340 may identify a version (352) of the centralized scene graph 350 to be created or updated based on the host data 319. In some examples, before writing version 352 of the centralized scene graph 350, the host application 340 may lock the version to prevent other processes from modifying it simultaneously. The host application 340 may make changes to version 352 to reflect the host data 319 (e.g., by adding or removing scene graph nodes in version 352 to correspond to the host data 319). In some examples, the host application 340 may then unlock version 352 and update the value of the version number corresponding to version 352 (356). The host application 340 may then update the host data (342), and repeat Figure 3D Because the centralized scene graph 350 is updated to reflect the individual host data exported from the individual client applications, the centralized scene graph 350 will still reflect the aggregate host data from the multiple client applications, even though the individual client applications may be "sandboxed" and independent of each other.
[0041] Figure 3E Shows relative to Figures 3A-3D3. Aspects of an example renderer 360 of the example computer system 300 are shown. In some examples, the renderer 360 comprises part of the host application 340. In some examples, the renderer 360 can be part of another component of the example computer system 300, or can be a separate component or application. In some examples, the renderer 360 can be implemented in physical hardware different from one or more components of the example computer system 300 and can communicate with one or more of those components over a network.
[0042] exist Figure 3E In the example shown, a renderer 360 operates on a version 352 of a centralized scene graph 350. In this example, the renderer's role is to create a rendered scene (such as output or graphical output) based on the version 352 of the centralized scene graph 350 for presentation on a display 370. As part of this process, the renderer 360 may traverse (362) the version 352 using known scene graph traversal techniques. During or after the traversal 362, the renderer 360 may update (364) the centralized scene graph 350 as appropriate to reflect the results of the traversal. For example, as part of the traversal 362, the renderer 360 may identify isolated nodes that should be deleted from the centralized scene graph 350. After the traversal 362 and / or update 364, the renderer 360 may apply various optimizations 366 to the scene. For example, the renderer 360 may cull occluded or invisible surfaces to avoid consuming unnecessary computing resources. After traversing 362 and / or updating 364, the renderer 360 may apply one or more visual effects 367 to the scene. For example, in some examples, the renderer 360 may apply lighting or shadow effects, apply one or more shaders, apply particle effects, and / or apply physics effects. Finally, the renderer 360 may output data to a graphics output pipeline, which may display output on a display 370.
[0043] The above example processes of a computer system can be provided by any suitable logic circuit. Suitable logic circuitry may include one or more computer processors (e.g., CPU, GPU, etc.), wherein when executing instructions implemented in a software program, the one or more computer processors perform the process. In addition, such processes may also be provided by corresponding logic designs implemented in hardware logic circuits (e.g., programmable logic (e.g., PLD, FPGA, etc.) or custom logic (e.g., ASIC, etc.) that implements the logic design providing the process. In addition, such processes are provided via the implementation of one or more processors that run software and hardware logic circuits in combination.
[0044] Figure 4An example system 400 is shown that can be used to implement any or all of the above examples. The above examples (in whole or in part) can be embodied in any portable device (including wearable devices) or non-portable device, such as a communication device (e.g., a mobile phone, a smartphone), a multimedia device (e.g., an MP3 player, a television, a radio), a portable or handheld computer (e.g., a tablet, a netbook, a laptop), a desktop computer, an all-in-one desktop, a peripheral device, a head-mounted device (which may include, for example, an integrated display), or any other system or device suitable for including the example system architecture 400, including combinations of two or more of these types of devices. The above examples can be embodied in two or more physically separate devices, such as two or more computers communicating via a wireless network. The above examples can be embodied in two or more physically distinct devices, such as a belt pack that transmits data to and / or communicates data from a head-mounted display. Figure 4 4 is a block diagram of an example of a system 400 that generally includes one or more computer-readable media 401, a processing system 404, an I / O subsystem 406, radio frequency (RF) circuitry 408, audio circuitry 410, and sensor circuitry 411. These components may be coupled via one or more communication buses or signal lines 403.
[0045] It is obvious that Figure 4 The architecture shown in FIG. 4 is only one example architecture of the system 400 , and the system 400 may have more or fewer components or a different configuration of components than shown. Figure 4 The various components shown in the may be implemented in hardware, software, firmware, or any combination thereof, including one or more signal processing and / or application specific integrated circuits.
[0046] refer to Figure 4 In the example system architecture 400 in FIG. 4 , RF circuitry 408 may be used to send and receive information to one or more other devices via a wireless link or network and includes well-known circuitry for performing this function. RF circuitry 408 and audio circuitry 410 may be coupled to processing system 404 via peripheral device interface 416. Interface 416 may include various well-known components for establishing and maintaining communication between peripheral devices and processing system 404. Audio circuitry 410 may be coupled to audio speaker 450 and microphone 452 and may include well-known circuitry for processing voice signals received from interface 416 to enable a user to communicate with other users in real time. In some examples, audio circuitry 410 may include a headphone jack (not shown).
[0047] The sensor circuit 411 can be coupled to various sensors, including but not limited to one or more light emitting diodes (LEDs) or other light emitters that can sense remaining battery life, power consumption, processor speed, CPU load, etc., one or more photodiodes or other light sensors, one or more photothermal sensors, a magnetometer, an accelerometer, a gyroscope, a barometer, a compass, a proximity sensor, a camera, an ambient light sensor, a thermometer, a GPS sensor, an electro-oculogram (EOG) sensor, and various system sensors. In examples such as those involving head-mounted devices, one or more sensors can be employed in conjunction with functionality related to the user's eyes, such as tracking the user's eye movements or identifying the user based on images of his or her eyes.
[0048] Peripheral device interface 416 can couple the system's input and output peripherals to processor 418 and computer-readable medium 401. One or more processors 418 can communicate with one or more computer-readable media 401 via controller 44. Computer-readable medium 401 can be any device or medium (excluding signals) that can store code and / or data for use by one or more processors 418. In some examples, medium 401 can be a non-transitory computer-readable storage medium. Medium 401 can include a memory hierarchy, including but not limited to cache, main memory, and secondary memory. This memory hierarchy can be implemented using any combination of RAM (e.g., SRAM, DRAM, DDRAM), ROM, FLASH, magnetic and / or optical storage devices (e.g., disk drives, tapes, CDs (compact disks), and DVDs (digital video disks)). Medium 401 can also include a transmission medium for carrying information-bearing signals indicating computer instructions or data (but not including signals and not including the carrier wave upon which the signals are modulated). For example, transmission media may include communication networks including but not limited to the Internet (also known as the World Wide Web), intranets, local area networks (LANs), wide area networks (WLANs), storage area networks (SANs), metropolitan area networks (MANs), etc.
[0049] One or more processors 418 can run various software components stored in the medium 401 to perform various functions for the system 400. In some examples, the software components may include an operating system 422, a communication module (or instruction set) 424, an I / O processing module (or instruction set) 426, a graphics module (or instruction set) 428, and one or more applications (or instruction sets) 430. Each of these modules and the above-mentioned applications can correspond to an instruction set for performing one or more of the above-mentioned functions and methods described in this application (e.g., the computer-implemented methods and other information processing methods described herein). These modules (i.e., instruction sets) need not be implemented as separate software programs, processes, or modules, and therefore, in various examples, various subsets of these modules can be combined or otherwise rearranged. In some examples, the medium 401 can store a subset of the modules and data structures identified above. In addition, the medium 401 can store additional modules and data structures not described above.
[0050] Operating system 422 may include various processes, instruction sets, software components, and / or drivers for controlling and managing general system tasks (eg, memory management, storage device control, power management, etc.) and facilitating communications between various hardware and software components.
[0051] Communications module 424 may facilitate communications with other devices through one or more external ports 436 or via RF circuitry 408 , and may include various software components for processing data received from RF circuitry 408 and / or external ports 436 .
[0052] Graphics module 428 may include various known software components for rendering, animating, and displaying graphical objects on one or more display surfaces. The display surface may include a 2D or 3D display. The display surface may be directly or indirectly coupled to one or more components of example system 400. In examples involving a touch-sensitive display (e.g., a touchscreen), graphics module 428 may include components for rendering, displaying, and animating objects on the touch-sensitive display. In some examples, graphics module 428 may include components for rendering to a remote display. In some examples (e.g., those that include a camera), graphics module 428 may include components for creating and / or displaying images formed by synthesizing camera data (e.g., captured from a head-mounted camera) or photographic data (e.g., satellite-captured imagery) with rendered graphical objects. In some examples, the graphics module may include components for rendering images to a head-mounted display. In some examples, the images may include views of elements of virtual content (e.g., objects in a three-dimensional virtual environment) and / or views of the physical world (e.g., camera input indicating a user's physical environment). In some examples, the display may present a composite of virtual content and views of the physical world. In some examples, the view of the physical world may be a rendered image; in some examples, the view of the physical world may be an image from a camera.
[0053] The one or more applications 430 may include any application installed on the system 400, including but not limited to a browser, an address book, a contact list, email, instant messaging, word processing, keyboard emulation, widgets, JAVA-enabled applications, encryption, digital rights management, voice recognition, voice replication, location determination functionality (such as that provided by a global positioning system (GPS)), music players, etc.
[0054] The I / O subsystem 406 can be coupled to an eye I / O device 412 and one or more other I / O devices 414 for controlling or performing various functions. For example, the eye I / O device 412 can communicate with the processing system 404 via an eye I / O device controller 432, which can include various components for processing eye input (e.g., a sensor for eye tracking) or user gesture input (e.g., an optical sensor). One or more other input controllers 434 can receive / send electrical signals from / to other I / O devices 414. Other I / O devices 414 can include physical buttons, dials, slider switches, joysticks, keyboards, touchpads, other display screens, or any combination thereof.
[0055] The I / O processing module 426 may include various software components for performing various tasks associated with the eye I / O device 412 and / or other I / O devices 414, including, but not limited to, receiving and processing input received from the eye I / O device 412 via the eye I / O device controller 432 or from the other I / O devices 414 via the I / O controller 434. In some examples, the I / O device 414 and / or the I / O processing module 426 may perform various tasks associated with gesture input that may be provided by tactile or non-tactile means. In some examples, gesture input may be provided by a camera or another sensor for detecting movement of, for example, the user's eye, arm, hand, and / or finger. In some examples, the I / O device 414 and / or the I / O processing module 426 may be configured to identify an object of the display that the user wishes to interact with, such as a GUI element that the user is pointing to. In some examples, eye I / O device 412 and / or I / O processing module 426 can be configured to perform eye tracking tasks (e.g., with the assistance of optical or EOG sensors), such as identifying an object or area on a display that a user is looking at. In some examples, a device (e.g., a hardware "beacon") can be worn or held by a user to assist touch I / O device 412 and / or I / O processing module 426 in performing gesture-related tasks, such as identifying the position of a user's hand relative to a 2D or 3D environment. In some examples, eye I / O device 412 and / or I / O processing module 426 can be configured to identify a user based on sensor input (e.g., data related to the user's eyes from a camera sensor).
[0056] In some examples, graphics module 428 can display visual output to the user in GUI.Visual output can include text, graphics, video and any combination thereof.Some or all of the visual output can correspond to user interface objects.In some examples, I / O device 412 and / or 414 and / or controller 432 and / or 434 (and any associated modules and / or instruction sets in medium 401) can detect and track gestures and / or eye movements, and detected gestures and / or eye movements can be converted into interactions with graphic objects (such as one or more user interface objects).In the example where eyes I / O device 412 and / or eyes I / O device controller 432 are configured to track the eye movements of the user, the user can directly interact with the graphic objects by viewing them.
[0057] Feedback may be provided, for example, by the eye I / O device 412 or another I / O device 414 based on the content being displayed and / or one or more states of the computing system. Feedback may be sent optically (e.g., a light signal or a displayed image), mechanically (e.g., haptic feedback, touch feedback, force feedback, etc.), electrically (e.g., electrical stimulation), olfactory, auditory (e.g., beeps, etc.), etc., or any combination thereof, and in a variable or constant manner.
[0058] System 400 may also include a power system 444 for powering the various hardware components, and may include a power management system, one or more power supplies, a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and any other components typically associated with the generation, management, and distribution of power in a portable device.
[0059] In some examples, peripherals interface 416, one or more processors 418, and memory controller 420 can be implemented on a single chip, such as processing system 404. In some other examples, they can be implemented on separate chips.
[0060] In some examples, a method is disclosed. The method may include: receiving first graph data comprising a plurality of first nodes from a first client application of a computer system; and receiving second graph data comprising a plurality of second nodes from a second client application of the computer system; and generating a scene graph, wherein the scene graph describes relationships (such as occlusion relationships) between the first nodes and the second nodes, and the scene graph is configured to render a scene comprising the nodes when traversed by a processor of the computer system. Additionally or alternatively to one or more of the above examples, the method may further include traversing the scene graph by the processor of the computer system. Additionally or alternatively to one or more of the above examples, the computer system may be configured to communicate with a display, and the method may further include presenting an output on the display comprising at least one node that is not occluded by another node in the rendered scene. In some embodiments, occlusion is the visual obstruction of one node by another node when viewing the rendered scene of the object from a given angle. Additionally or alternatively to one or more of the above examples, the computer system may be configured to communicate with a display, and the method may further include displaying the output on the display, for example, by displaying the rendered scene as present on the scene graph, or by displaying only those nodes in the first plurality of nodes or the second plurality of nodes that are not occluded and not displaying other nodes. For example, if the second plurality of nodes obscures a portion of the first plurality of nodes, the displayed output may only be the unobstructed nodes in the first plurality of nodes, without displaying any of the second plurality of nodes. Additionally or alternatively to one or more of the above examples, the method may further include applying optimizations to the output at the computer system. Additionally or alternatively to one or more of the above examples, applying the optimizations may include culling surfaces. Additionally or alternatively to one or more of the above examples, the method may further include applying visual effects to the output at the computer system. Additionally or alternatively to one or more of the above examples, applying the visual effects may include calculating light values. Additionally or alternatively to one or more of the above examples, applying the visual effects may include executing a shading program. Additionally or alternatively to one or more of the above examples, the method may further include applying physical effects to the output at the computer system. Additionally or alternatively to one or more of the above examples, applying the physical effects may include detecting collisions. Additionally or alternatively to one or more of the above examples, the first client application may be a first application executing on the computer system, the second client application may be a second application executing on the computer system, and the first client application may be sandboxed on the computer system relative to the second client application.Additionally or alternatively to one or more of the above examples, the first graphics data may correspond to a first client scene graph associated with a first client application, the second graphics data may correspond to a second client scene graph associated with a second client application, the first client scene graph may be sandboxed on the computer system relative to the second client scene graph, the first client scene graph may be sandboxed on the computer system relative to the scene graph, and the second client scene graph may be sandboxed on the computer system relative to the scene graph. Additionally or alternatively to one or more of the above examples, the scene graph may correspond to a version of a versioned scene graph. Additionally or alternatively to one or more of the above examples, the first graphics data may be transferred to the scene graph using a first processing thread of the computer system, and independent of the first processing thread, the second graphics data may be transferred to the scene graph using a second processing thread of the computer system.
[0061] In some examples, a method is disclosed. The method may include traversing a scene graph of a computer system having a display, wherein the scene graph includes first 3D data associated with a first application, wherein the first 3D data includes one or more nodes, the scene graph includes second 3D data associated with a second application, wherein the second 3D data includes one or more nodes, the first application is sandboxed on the computer system relative to the second application, and the scene graph includes relationships between the nodes of the first 3D data and the nodes of the second 3D data; and displaying an image corresponding to the scene graph on the display, wherein the image corresponds to an output of traversing the scene graph, the image reflects the relationships, and the image is a partial or complete display of the data. Additionally or alternatively to one or more of the above examples, the relationships may be spatial relationships. Additionally or alternatively to one or more of the above examples, the method may further include applying, at the computer system, optimizations to the output of the traversed scene graph. Additionally or alternatively to one or more of the above examples, applying the optimizations may include culling surfaces. Additionally or alternatively to one or more of the above examples, the method may further include applying, at the computer system, visual effects to the output of the traversed scene graph. Additionally or alternatively to one or more of the above examples, applying the visual effects may include calculating light values. Additionally or alternatively to one or more of the above examples, applying the visual effect may include executing a shader. Additionally or alternatively to one or more of the above examples, the method may further include applying, at the computer system, a physics effect to the output of the traversal of the scene graph. Additionally or alternatively to one or more of the above examples, applying the physics effect may include detecting collisions. Additionally or alternatively to one or more of the above examples, the scene graph may correspond to a version of a versioned scene graph. Additionally or alternatively to one or more of the above examples, graphics data corresponding to the first 3D data may be transmitted to the scene graph by a host application executing on the computer system. Additionally or alternatively to one or more of the above examples, graphics data corresponding to the first 3D data may be transmitted to the host application by a client of the host application. Additionally or alternatively to one or more of the above examples, first graphics data corresponding to the first 3D data may be transmitted to the scene graph by the host application using a first processing thread, and second graphics data corresponding to the second 3D data may be transmitted to the scene graph by the host application using a second processing thread, independent of the first processing thread.
[0062] In some examples, a computer system is disclosed. The system may include one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform one or more of the methods described above.
[0063] In some examples, a non-transitory computer-readable storage medium is disclosed. The non-transitory computer-readable storage medium may store instructions that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising: receiving first graph data comprising a plurality of first nodes from a first client application of a computer system; receiving second graph data comprising a plurality of second nodes from a second client application of the computer system; and generating a scene graph, wherein the scene graph describes relationships between the first nodes and the second nodes, and the scene graph is configured to render a scene based on occlusion relationships when traversed by the processor of the computer system, wherein one or more of the first or second nodes in the first or second plurality of nodes occludes the other. In some embodiments, occlusion is the visual obstruction of one node from another node when viewing the rendered scene of the object from a given angle. Additionally or alternatively to one or more of the above examples, the method may further include traversing the scene graph by the processor of the computer system. Additionally or alternatively to one or more of the above examples, the computer system may be configured to communicate with a display, and the method may further include displaying output on the display, for example, by displaying the rendered scene as present on the scene graph, or by displaying only those nodes in the first or second plurality of nodes that are not occluded and not displaying other nodes. For example, if the second plurality of nodes occludes a portion of the first plurality of nodes, the displayed output may only display the unoccluded nodes in the first plurality of nodes, without displaying any of the second plurality of nodes. Additionally or alternatively to one or more of the above examples, the method may further include applying optimizations to the output at the computer system. Additionally or alternatively to one or more of the above examples, applying the optimizations may include culling surfaces. Additionally or alternatively to one or more of the above examples, the method may further include applying visual effects to the output at the computer system. Additionally or alternatively to one or more of the above examples, applying the visual effects may include calculating light values. Additionally or alternatively to one or more of the above examples, applying the visual effects may include executing a shading program. Additionally or alternatively to one or more of the above examples, the method may further include applying physical effects to the output at the computer system. Additionally or alternatively to one or more of the above examples, applying the physical effects may include detecting collisions. Additionally or alternatively to one or more of the above examples, the first client application may be a first application executing on the computer system, the second client application may be a second application executing on the computer system, and the first client application may be sandboxed relative to the second client application on the computer system.Additionally or alternatively to one or more of the above examples, the first graphics data may correspond to a first client scene graph associated with a first client application, the second graphics data may correspond to a second client scene graph associated with a second client application, the first client scene graph may be sandboxed on the computer system relative to the second client scene graph, the first client scene graph may be sandboxed on the computer system relative to the scene graph, and the second client scene graph may be sandboxed on the computer system relative to the scene graph. Additionally or alternatively to one or more of the above examples, the scene graph may correspond to a version of a versioned scene graph. Additionally or alternatively to one or more of the above examples, the first graphics data may be transferred to the scene graph using a first processing thread of the computer system, and independent of the first processing thread, the second graphics data may be transferred to the scene graph using a second processing thread of the computer system.
[0064] In some examples, a non-transitory computer-readable storage medium is disclosed. The non-transitory computer-readable storage medium may store instructions that, when executed by one or more processors, cause the one or more processors to perform a method, comprising: traversing a scene graph of a computer system having a display, wherein the scene graph includes first 3D data associated with a first application, wherein the first 3D data includes one or more nodes; the scene graph includes second 3D data associated with a second application, wherein the second 3D data includes one or more nodes; the first application is sandboxed on the computer system relative to the second application; and the scene graph includes relationships between the nodes of the first 3D data and the nodes of the second 3D data; and displaying an image corresponding to the scene graph on the display, wherein the image corresponds to an output of traversing the scene graph, and the image reflects the relationships. Additionally or alternatively to one or more of the above examples, the relationships may be spatial relationships. Additionally or alternatively to one or more of the above examples, the method may further include applying, at the computer system, optimizations to the output of the traversed scene graph. Additionally or alternatively to one or more of the above examples, applying the optimizations may include culling surfaces. Additionally or alternatively to one or more of the above examples, the method may further include applying, at the computer system, visual effects to the output of the traversed scene graph. Additionally or alternatively to one or more of the above examples, applying the visual effect may include calculating light values. Additionally or alternatively to one or more of the above examples, applying the visual effect may include executing a shading program. Additionally or alternatively to one or more of the above examples, the method may further include applying, at the computer system, a physics effect to the output of traversing the scene graph. Additionally or alternatively to one or more of the above examples, applying the physics effect may include detecting collisions. Additionally or alternatively to one or more of the above examples, the scene graph may correspond to a version of a versioned scene graph. Additionally or alternatively to one or more of the above examples, graphics data corresponding to the first 3D data may be transmitted to the scene graph by a host application executing on the computer system. Additionally or alternatively to one or more of the above examples, graphics data corresponding to the first 3D data may be transmitted to the host application by a client of the host application. Additionally or alternatively to one or more of the above examples, first graphics data corresponding to the first 3D data may be transmitted to the scene graph by the host application using a first processing thread, and second graphics data corresponding to the second 3D data may be transmitted to the scene graph by the host application using a second processing thread, independent of the first processing thread.
[0065] In some examples, a computer system is disclosed. The system may include: one or more processors; a memory configured to receive, at the computer system, first scene data from a first client application and second scene data from a second client application; and the memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform a method comprising: generating a graphics data structure based on the first scene data and the second scene data, the graphics data structure configured to, when provided as input to a rendering operation performed by the one or more processors, result in output corresponding to an image on a display. Additionally or alternatively to one or more of the above examples, the graphics data structure may be at least one of a display list and a display tree. Additionally or alternatively to one or more of the above examples, the method may further include performing the rendering operation using the graphics data structure as input. Additionally or alternatively to one or more of the above examples, the computer system may further include a display, and the method may further include displaying the image on the display. Additionally or alternatively to one or more of the above examples, the first client application may be a first application executed by one or more processors of a first device, and the second client application may be a second application executed by one or more processors of the first device. Additionally or alternatively to one or more of the above examples, the first client application may be a first application executed by one or more processors of the first device, and the second client application may be a second application executed by one or more processors of the second device. Additionally or alternatively to one or more of the above examples, the memory may be further configured to receive third scene data from a third client application. Additionally or alternatively to one or more of the above examples, the method may further include deleting the first scene data from the memory. Additionally or alternatively to one or more of the above examples, the graphics data structure may include first data and second data, and the method may further include: determining whether the first data corresponds to an occluded view or a non-occluded view; in response to determining that the first data corresponds to a non-occluded view, rendering an image including the non-occluded view based on the first data; and in response to determining that the first data corresponds to an occluded view, rendering an image not including the occluded view. Additionally or alternatively to one or more of the above examples, the memory may be further configured to, in response to receiving the first scene data, store the first scene data as a first version in a version control system. Additionally or alternatively to one or more of the above examples, the memory may be further configured to: receive third scene data from the first client application; and store the third scene data as a second version in the version control system. Additionally or alternatively to one or more of the above examples, the method may further include, in response to generating the graph data structure, deleting the first version from the memory.In addition to or in lieu of one or more of the above examples, the method may be performed in parallel with the memory receiving the third scene data. In addition to or in lieu of one or more of the above examples, the memory may be configured to receive the first scene data in parallel with receiving the second scene data. In addition to or in lieu of one or more of the above examples, the memory may be configured to receive the third scene data at a first interval corresponding to a first data rate, and the method may further include adjusting the length of the first interval to correspond to the second data rate. In addition to or in lieu of one or more of the above examples, the first scene data may include at least one of new data, deleted data, and a change in the relationship between the data.
[0066] In some examples, a computer system is disclosed. The computer system may include a server, server data, a first client application, and a second client application, and may be configured to: receive first unprocessed scene data from the first client application at the server; receive second unprocessed scene data from the second client application at the server; merge the first unprocessed scene data from the first client application, the second unprocessed scene data from the second client application, and the server data into a centralized scene data structure at the server; execute at least a portion of the data contained in the centralized scene data structure at the server; and create a graphics data structure based on the data executed within the centralized scene data structure. Additionally or alternatively to one or more of the above examples, the graphics data structure may be a display list or a display tree. Additionally or alternatively to one or more of the above examples, the computer system may further include a rendering engine configured to render the graphics data structure into a processed image. Additionally or alternatively to one or more of the above examples, the computer system may further include a display configured to display the processed image. Additionally or alternatively to one or more of the above examples, the display may be capable of displaying virtual content while maintaining at least a partial view of the physical world. Additionally or alternatively to one or more of the above examples, the first client application and the second client application may be two different applications running on a single physical device. Additionally or alternatively to one or more of the above examples, the first client application and the second client application may be two different applications, each running on a separate physical device. Additionally or alternatively to one or more of the above examples, the server may be configured to receive third unprocessed scene data from a third client application. Additionally or alternatively to one or more of the above examples, the server may be configured to delete the unprocessed scene data from the first client application after executing the unprocessed scene data from the first client application. Additionally or alternatively to one or more of the above examples, the rendering engine may further include an occlusion module configured to separate data within the graphics data structure into a first occlusion category and a second occlusion category and display the second occlusion category. Additionally or alternatively to one or more of the above examples, the server may be configured to store the first unprocessed scene data from the first client application as a first version. Additionally or alternatively to one or more of the above examples, the server may be configured to store the third unprocessed scene data from the first client application as a second version. In addition to or in lieu of one or more of the above examples, the computer system can be configured to store the first version of the first unprocessed scenario data from the first client application starting from the time the server receives the first version of the first unprocessed scenario data from the first client application until the time the first unprocessed scenario data from the first client application is read and executed.In addition to or in lieu of one or more of the above examples, the server may be configured to receive first unprocessed scene data from the first client while receiving second unprocessed scene data from the second client. In addition to or in lieu of one or more of the above examples, the server may be configured to slow down the rate at which the first client application sends unprocessed scene data to the server. In addition to or in lieu of one or more of the above examples, the data received from the first client application and the second client application may be at least one selected from the group consisting of: new data, deleted data, a change in a relationship between previously transmitted data, and modified data.
[0067] Although the disclosed examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. For example, one or more elements of the implementations may be combined, deleted, modified, or supplemented to form further implementations. Such changes and modifications are understood to be included within the scope of the disclosed examples as defined by the appended claims.
Claims
1. A method comprising: transmitting first graph data including a plurality of first nodes from a first client application to a computer system, the computer system being in communication with a second client application; A scene graph is received from the computer system, wherein: The scene graph describes a first occlusion relationship between at least one first node of the plurality of first nodes and at least one second node of the plurality of second nodes of the second graphics data of the second client application, and The scene graph further describes a second occlusion relationship between at least one third node among the plurality of first nodes and at least one fourth node among the plurality of second nodes; and A rendering scene is created based on the first occlusion relationship and further based on the second occlusion relationship, wherein at least one second node occludes at least one first node so that a first portion of the second graphic data occludes a first portion of the first graphic data, and wherein at least one third node occludes at least one fourth node so that a second portion of the first graphic data occludes a second portion of the second graphic data.
2. The method according to claim 1, wherein: Creating the rendered scene includes traversing the scene graph.
3. The method according to claim 1, further comprising: The rendering scene is displayed, wherein displaying the rendering scene includes: displaying at least one first node among the plurality of first nodes and displaying at least one second node among the plurality of second nodes.
4. The method according to claim 1, wherein: The first client application comprises a first application executing on the computer system, and the second client application comprises a second application executing on the computer system.
5. The method according to claim 4, wherein: The first client application is sandboxed on the computer system relative to the second client application.
6. The method according to claim 1, wherein: The first client application comprises an application executing on a second computer system, the second computer system is in communication with the computer system via a network, and receiving the scene graph from the computer system comprises receiving the scene graph from the computer system via the network at the second computer system.
7. The method according to claim 1, wherein: The scene graph corresponds to a version of a versioned scene graph.
8. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising: transmitting first graph data including a plurality of first nodes from a first client application to a computer system, the computer system being in communication with a second client application; A scene graph is received from the computer system, wherein: The scene graph describes a first occlusion relationship between at least one first node of the plurality of first nodes and at least one second node of the plurality of second nodes of the second graphics data of the second client application, and The scene graph further describes a second occlusion relationship between at least one third node among the plurality of first nodes and at least one fourth node among the plurality of second nodes; and A rendering scene is created based on the first occlusion relationship and further based on the second occlusion relationship, wherein at least one second node occludes at least one first node so that a first portion of the second graphic data occludes a first portion of the first graphic data, and wherein at least one third node occludes at least one fourth node so that a second portion of the first graphic data occludes a second portion of the second graphic data.
9. The non-transitory computer readable medium of claim 8, wherein: Creating the rendered scene includes traversing the scene graph.
10. The non-transitory computer readable medium of claim 8, the method further comprising: The rendering scene is displayed, wherein displaying the rendering scene includes: displaying at least one first node among the plurality of first nodes and displaying at least one second node among the plurality of second nodes.
