Application sharing

Abstract

To provide suitable application sharing.SOLUTION: A host device having a first processor executes an application via the first processor. The host device determines a state of the application. A scene graph is generated corresponding to the state of the application, and the scene graph is presented at a remote device having a display and a second processor. The remote device is configured to, in response to receiving the scene graph, render a view corresponding to the scene graph on the display without executing the application via the second processor.SELECTED DRAWING: Figure 4A

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Provisional Application No. 62 / 741,761, filed October 5, 2018, U.S. Provisional Application No. 62 / 702,844, filed July 24, 2018, U.S. Provisional Application No. 62 / 726,863, filed September 4, 2018, and U.S. Provisional Application No. 62 / 742,259, filed October 5, 2018, the contents of which are incorporated herein by reference in their entireties.

[0002] The present disclosure relates generally to systems and methods for presenting data of a computer application, and more particularly to systems and methods for presenting data of a computer application running on a local device on a remote device. [Background technology]

[0003] The proliferation of increasingly smaller, lighter, and mobile personal devices, and the accompanying proliferation of content-on-demand services, are changing user expectations for software applications. Software is often expected to be immediately accessible, have a small resource footprint, and be easily shareable with others, e.g., like streamed video content. Traditional software, which may need to be purchased, downloaded, and installed before it can be used, may not meet the needs of some users. Furthermore, traditional software may be excessively resource-intensive for increasingly mobile computing devices, including wearable devices, which contend with limitations on physical size, shape, and weight that limit the storage, processing power, and battery capacity of those devices, and may need to fit limited-bandwidth cellular data plans.

[0004] One response to the above is for software applications to feature "sharing" functionality, which allows users to remotely access (e.g., view or interact with) applications running on a host device. Because application sharing eliminates the need for remote users to install or run applications on their own computing devices, the barrier to entry is lowered, encouraging the use and proliferation of "sharable" applications.

[0005] Application sharing is not without potential problems. Some examples of application sharing involve streaming pre-rendered video data representing the visual output of the application; however, such video data can be bandwidth-intensive, limiting practical use of these applications to high-bandwidth environments and preventing their use on many mobile data plans. It is desirable for a host computing device to share a software application with one or more remote computing devices so that users of all such devices can simultaneously view and / or interact with the software application without the need for the remote devices to install or run the software application locally. It is further desirable to minimize the amount of data that must be transferred between the host device and the remote devices to facilitate use in low-bandwidth environments. Summary of the Invention [Means for solving the problem]

[0006] Systems and methods for sharing software applications for computing devices are disclosed. According to some embodiments, a host device having a first processor executes the application via the first processor. The host device determines a state of the application. A scene graph is generated corresponding to the state of the application, and the scene graph is presented to a remote device having a display and a second processor. In response to receiving the scene graph, the remote device is configured to render a view corresponding to the scene graph on the display without executing the application via the second processor. For example, the present application provides the following: (Item 1) 1. A method comprising: executing a first application via a first processor in a host device having the first processor; determining a first state of the first application; generating a first scene graph corresponding to a first state of the first application; presenting the first scene graph on a first remote device having a first display and a second processor; Including, the first remote device is configured, in response to receiving the first scene graph, to render on the first display a first view corresponding to the first scene graph; The method, wherein rendering the first view on the first display does not include executing the first application via the second processor. (Item 2) receiving, at the host device, an input from the first remote device; determining a second state of the first application based on the input, the second state being different from the first state; and generating a second scene graph corresponding to the second state, the second scene graph being different from the first scene graph; presenting the second scene graph to the remote device; further comprising the first remote device is configured, in response to receiving the second scene graph, to render on the first display a second view corresponding to the second scene graph without executing the first application; 2. The method of claim 1, wherein rendering the second view on the first display does not include executing the first application via the second processor. (Item 3) The host device has a second display, and the method further comprises: Item 10. The method of item 1, further comprising: rendering, at the host device, on the second display, a second view corresponding to the first scene graph. (Item 4) Item 4. The method of item 3, wherein the second view corresponding to the first scene graph is different from the first view corresponding to the first scene graph. (Item 5) Item 10. The method of item 1, wherein the first application is installed on the host device and the first application is not installed on the first remote device. (Item 6) Item 10. The method of item 1, wherein the first application is installed on the host device and the first application is installed on the first remote device. (Item 7) Item 10. The method of item 1, wherein the first remote device is a wearable computing device and the first display is configured to be head-mounted on a user of the first remote device. (Item 8) presenting the first scene graph on a second remote device having a second display and a third processor; the second remote device is configured, in response to receiving the first scene graph, to render on the second display a second view corresponding to the first scene graph; Rendering the second view on the second display does not include executing the first application via the third processor. The method according to item 1. (Item 9) determining a first state of the second application; the first scene graph further corresponds to a first state of the second application; Rendering the first view on the first display does not include executing the second application via the second processor. The method according to item 1. (Item 10) receiving a request from the first remote device to present the first scene graph; the request to present the first scene graph is generated in response to an interaction between the first remote device and a trigger; presenting the first scene graph on the first remote device is performed in response to receiving the request. The method according to item 1. (Item 11) Item 11. The method of item 10, wherein the trigger comprises a QR code. (Item 12) Item 11. The method of item 10, wherein the trigger comprises location information. (Item 13) Item 11. The method of item 10, wherein the request comprises an identification of the first application. (Item 14) A non-transitory machine-readable medium having instructions stored thereon that, when executed by a first device having one or more processors, including a first processor, cause the first device to: executing a first application via the first processor; determining a first state of the first application; generating a first scene graph corresponding to a first state of the first application; presenting the first scene graph on a second device having a first display and a second processor; carrying out a method comprising: the second device is remote from the first device; the second device is configured, in response to receiving the first scene graph, to render on the first display a first view corresponding to the first scene graph; Rendering the first view on the first display does not include executing the first application via the second processor. Non-transitory machine-readable media. (Item 15) The method further comprises: receiving, at the first device, an input from the second device; determining a second state of the first application based on the input, the second state being different from the first state; and generating a second scene graph corresponding to the second state, the second scene graph being different from the first scene graph; presenting the second scene graph to the second device; Including, the second device is configured, in response to receiving the second scene graph, to render, on the first display, a second view corresponding to the second scene graph without executing the first application; Rendering the second view on the first display does not include executing the first application via the second processor. Item 15. The non-transitory machine-readable medium of item 14. (Item 16) The first device has a second display, and the method further comprises: Rendering, at the first device, a second view corresponding to the first scene graph on the second display. Item 15. The non-transitory machine-readable medium of item 14, comprising: (Item 17) Item 17. The non-transitory machine-readable medium of item 16, wherein the second view is different from the first view. (Item 18) Item 15. The non-transitory machine-readable medium of item 14, wherein the first application is installed on the first device and the first application is not installed on the second device. (Item 19) Item 15. The non-transitory machine-readable medium of item 14, wherein the first application is installed on the first device and the second application is installed on the second device. (Item 20) Item 15. The non-transitory machine-readable medium of item 14, wherein the second device is a wearable computing device and the first display is configured to be head-mounted on a user of the second device. (Item 21) The method further includes presenting the first scene graph on a second remote device having a second display and a third processor; the second remote device is configured, in response to receiving the first scene graph, to render on the second display a second view corresponding to the first scene graph; Rendering the second view on the second display does not include executing the first application via the third processor. Item 15. The non-transitory machine-readable medium of item 14. (Item 22) The method further includes determining a first state of the second application; the first scene graph further corresponds to a first state of the second application; Rendering the first view on the first display does not include executing the second application via the second processor. Item 15. The non-transitory machine-readable medium of item 14. (Item 23) The method further includes receiving a request from the first remote device to present the first scene graph; the request to present the first scene graph is generated in response to an interaction between the first remote device and a trigger; presenting the first scene graph on the first remote device is performed in response to receiving the request. Item 1. The non-transitory machine-readable medium of item 1. (Item 24) Item 24. The non-transitory machine-readable medium of item 23, wherein the trigger comprises a QR code. (Item 25) Item 24. The non-transitory machine-readable medium of item 23, wherein the trigger comprises location information. (Item 26) Item 24. The non-transitory machine-readable medium of item 23, wherein the request comprises an identification of the first application. (Item 27) A first device having one or more processors including a first processor, the first device comprising: executing a first application via the first processor; determining a first state of the first application; generating a first scene graph corresponding to a first state of the first application; presenting the first scene graph on a second device having a first display and a second processor; It is possible to carry out a method comprising: the second device is remote from the first device; the second device is configured, in response to receiving the first scene graph, to render on the first display a first view corresponding to the first scene graph; Rendering the first view on the first display does not include executing the first application via the second processor. First device. (Item 28) The method further comprises: receiving, at the first device, an input from the second device; determining a second state of the first application based on the input, the second state being different from the first state; and generating a second scene graph corresponding to the second state, the second scene graph being different from the first scene graph; presenting the second scene graph to the second device; Including, the second device is configured, in response to receiving the second scene graph, to render, on the first display, a second view corresponding to the second scene graph without executing the first application; Rendering the second view on the first display does not include executing the first application via the second processor. Item 28. The device according to item 27. (Item 29) The first device has a second display, and the method further comprises: Rendering, at the first device, a second view corresponding to the first scene graph on the second display. 28. The device according to item 27, comprising: (Item 30) Item 30. The device of item 29, wherein the second view is different from the first view. (Item 31) 28. The device of claim 27, wherein the first application is installed on the first device and the first application is not installed on the second device. (Item 32) 28. The device of claim 27, wherein the first application is installed on the first device and the second application is installed on the second device. (Item 33) Item 28. The device of item 27, wherein the second device is a wearable computing device and the first display is configured to be head-mounted on a user of the second device. (Item 34) The method further includes presenting the first scene graph on a second remote device having a second display and a third processor; the second remote device is configured, in response to receiving the first scene graph, to render on the second display a second view corresponding to the first scene graph; Rendering the second view on the second display does not include executing the first application via the third processor. Item 28. The device according to item 27. (Item 35) The method further includes determining a first state of the second application; the first scene graph further corresponds to a first state of the second application; Rendering the first view on the first display does not include executing the second application via the second processor. Item 28. The device according to item 27. (Item 36) The method further includes receiving a request from the first remote device to present the first scene graph; the request to present the first scene graph is generated in response to an interaction between the first remote device and a trigger; presenting the first scene graph on the first remote device is performed in response to receiving the request. Item 28. The device according to item 27. (Item 37) Item 37. The device of item 36, wherein the trigger comprises a QR code. (Item 38) Item 37. The device of item 36, wherein the trigger comprises location information. (Item 39) Item 37. The device of item 36, wherein the request comprises an identification of the first application. [Brief explanation of the drawings]

[0007] [Figure 1] 1A-1E illustrate example computer systems that may participate in application sharing according to embodiments of the present disclosure.

[0008] [Figure 2A] FIG. 2A illustrates an exemplary flow of data in an exemplary computer system, according to an embodiment of the present disclosure.

[0009] [Figure 2B] FIG. 2B illustrates an exemplary flow of data in an exemplary computer system including multiple independent applications, according to an embodiment of the present disclosure.

[0010] [Figure 3A] FIG. 3A illustrates components of an exemplary computer system that may render 3D data from multiple independent applications to a display using a unified scene graph, according to an embodiment of the present disclosure.

[0011] [Figure 3B] FIG. 3B illustrates aspects of an example client application for an example computer system including multiple independent client applications, according to an embodiment of the present disclosure.

[0012] [Figure 3C] FIG. 3C illustrates aspects of an example renderer for an example computer system including multiple independent applications, according to an embodiment of the present disclosure.

[0013] [Figure 4A] 4A-4B illustrate an exemplary diagram of a system for application sharing according to an embodiment of the present disclosure. [Figure 4B] 4A-4B illustrate an exemplary diagram of a system for application sharing according to an embodiment of the present disclosure.

[0014] [Figure 5] FIG. 5 illustrates an exemplary diagram of a system for application sharing, according to an embodiment of the present disclosure.

[0015] [Figure 6] FIG. 6 illustrates an exemplary flowchart of application sharing according to an embodiment of the present disclosure.

[0016] [Figure 7]FIG. 7 illustrates an example system architecture that may be embodied in a portable or non-portable device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0017] In the following description of the embodiments, reference is made to the accompanying drawings which form a part hereof, and in which is shown, by way of illustration, specific embodiments which may be practiced. It is to be understood that other embodiments may be used and structural changes may be made without departing from the scope of the disclosed embodiments.

[0018] 1A-1E illustrate various exemplary computer systems with displays. FIG. 1A shows an exemplary desktop computer connected to an external monitor. FIG. 1B shows an exemplary laptop including a display. FIG. 1C shows an exemplary mobile device including an integrated display. FIG. 1D shows an exemplary television including a display. FIG. 1E shows an exemplary computer system including a head-mounted display. The present disclosure is not limited to any particular type of computer system, any particular type of display, or any particular means of connecting a computer system to a display. The present disclosure is further not limited to two-dimensional displays; three-dimensional displays, such as stereoscopic displays, are specifically contemplated.

[0019] This disclosure contemplates the use of centralized rendering techniques for application sharing. Such techniques are described, for example, in U.S. Patent Application Nos. 15 / 940,892 and 16 / 011,413 (both of which are incorporated herein by reference in their entireties). As described below and in the aforementioned applications, using centralized rendering techniques allows local and remote users sharing an application to independently render graphical data that accurately presents a compelling view of the shared application. For example, by independently rendering graphical data rather than receiving pre-rendered graphical data from a host device, the bandwidth requirements of the application sharing systems and methods described herein can be reduced.

[0020] Centralized Rendering

[0021] In some exemplary computer systems, data to be presented graphically (as a “rendered scene”) on a display includes data (“3D data”) representing objects (e.g., 2D or 3D geometric primitives, including polygons) in three-dimensional space, and presenting the 3D data on a display includes presenting an image (“displayed scene”) corresponding to the objects in three-dimensional space as viewed from a viewing origin oriented along a viewing axis. For example, in a software application (e.g., a video game using a 3D engine) running on a computer system, the 3D data may include data describing the spatial coordinates, orientation, and / or visual properties of objects within the three-dimensional game world, and the viewing origin and viewing axis within the game world. The 3D data may also include data related to textures associated with objects to be rendered, shader parameters associated with the objects, and other information that affects how the objects may be displayed. A game may, for example, during a “rendering” or “drawing” phase, instruct a software and / or hardware “pipeline” to create a rendered scene for presentation on a display as a displayed scene. The presentation may include a view of the content within the scene. Such content can include digital content, alone or in conjunction with "real-world" content (e.g., digital content overlaid on real-world content viewed through a see-through display). A view containing virtual content can represent what a user observing the content would expect to see. For example, a first view presented while the user is in a first orientation can depict the content as it would be seen from the first orientation. When the user rotates to a second orientation, the second view can present the same content as it would be seen from the second orientation, i.e., from a perspective that reflects the user's rotation relative to the first orientation. In general, it is desirable for the resulting image to reflect the user's expectations of the visual world.In particular, it is generally desirable for a first opaque object closer to the viewing origin to occlude a second object behind the first object. An incorrectly occluded object may confuse a user and not clearly indicate where the object is located in three-dimensional space. In some exemplary computer systems, occlusion is achieved through sorting, in which objects closer to the viewing origin are sorted or drawn above objects farther from the viewing origin.

[0022] Sorting multiple objects for presentation on a display so that one object realistically occludes another requires information about the relationships between the objects, e.g., the spatial relationships between the objects in three-dimensional space. Some exemplary computer systems utilize scene graphs to represent relationships (e.g., hierarchical relationships) between one or more objects, such as objects to be rendered as a scene. As used herein, a scene graph is any data structure that represents such relationships. For example, in a scene graph, rendered objects to be presented may be represented as nodes in the graph, and the relationships between the nodes represent logical or spatial relationships between the objects. A renderer can then traverse the scene graph according to techniques known in the art and render or prepare at least one of the objects for display in a manner that will achieve proper occlusion. For example, a renderer may create a scene of objects with nodes, but the corresponding presentation on the display may be only a subset of the rendered objects, such that an object occluded by another object in the renderer will be only partially presented in the resulting displayed scene (e.g., the non-occluded portion of the object is displayed, while the occluded portion is not displayed). Such selective display can be beneficial, for example, it can be efficient to occlude a first object initiated from a first application when only a second object initiated from a second application needs to be visible within a given period of time. In some embodiments, a scene graph is an intermediate data structure that sits between an application containing 3D data and a renderer for rendering that 3D data for presentation to a screen; in some embodiments, an application writes scene information to a scene graph, which may later be used by the renderer to render a scene or output a displayed scene.

[0023] 2A shows an example flow of data in an example computer system 200. In system 200, a single application 210 can write data to a scene graph 240 that a renderer 250 can use to render an object 220 for presentation on a display 290. For example, object 220 may include several objects (e.g., polygons) that together comprise 3D representations of two human hands, i.e., hand 232 and hand 236, and an application may, for example, during the rendering or drawing phase, instruct object 220 to be presented on the display from the perspective of a viewing origin oriented along a viewing axis. In an embodiment, hands 232 and 236 are interlocked in a handshake, and because of the relative positioning of the hands, a viewer would expect some portions of hand 232 to occlude portions of hand 236 and some polygons making up hand 236 to occlude portions of hand 232, relative to the viewing origin and the viewing axis. Application 210 can write information to scene graph 240 describing relationships between objects 220, such as spatial relationships between polygons that make up object 220, that can be used to identify polygons that should occlude other polygons, i.e., polygons that should be sorted to appear on top of one another. For example, scene graph 240 can reflect that polygon 234 (belonging to hand 232) is located between the viewing origin and the polygons that make up hand 236, and therefore should occlude those polygons in hand 236, and that polygon 238 (belonging to hand 236) is located between the viewing origin and the polygons that make up hand 232, and therefore should occlude those polygons in hand 232. The renderer 250 may then output the object 220 or a subset of the object 220 (e.g., only the hand 232, only the unoccluded portion of the hand 232, or only the hand 236) via a display 290 consistent with the desired occlusion for presentation.

[0024] Figure 2B shows an example flow of data in example computer system 201 using two independent applications. When the embodiment of Figure 2A is extended to render a scene modified by multiple independent applications, as shown in Figure 2B, rendering problems can arise. For example, in example computer system 201, application 212 and application 214 both write data to scene graph 240 and render their individual 3D data to a single display 290. In Figure 2B, application 212 attempts to render and present object 222 (including the objects that make up hand 232), and application 214 attempts to render and present object 224 (including the objects that make up hand 236). The embodiment shown in FIG. 2B may have difficulty achieving realistic occlusion of objects to be rendered: if application 212 and application 214 are independent (“sandboxed”) applications, application 212 cannot access data related to application 214's object 224 (including hand 236 and its constituent objects), and similarly, application 214 cannot access data related to application 212's object 222 (including hand 232 and its constituent objects). That is, in some embodiments, neither application 212 nor application 214 can fully identify the relationship between object 222 and object 224. Thus, neither application 212 nor application 214 can write information to scene graph 240 that may be necessary to identify objects that occlude other objects or the order in which objects should be sorted on the display. Additionally, various subsystems, such as those for lighting, shadowing, animation, particles, and collision detection, may not behave as expected. As a result, unrealistic, unwieldy, and confusing graphical data may be rendered.Furthermore, such rendering systems may not be able to take advantage of rendering optimizations such as culling, making these systems computationally inefficient.

[0025] Centralized rendering, such as that described in U.S. Patent Application Nos. 15 / 940,892 and 16 / 011,413, provides a solution to such problems that may arise in systems that render 3D data from multiple independent applications. A unified scene graph can be used in place of a traditional scene graph, such as scene graph 240 of FIG. 2B, in a system in which multiple independent applications provide 3D data to be rendered (such as exemplary computer system 201 of FIG. 2B). As described herein, in some embodiments, a unified scene graph can include a system that receives 3D data from multiple individual input sources, writes information corresponding to the 3D data to a central location, and maintains that information for access by a renderer that creates a rendered scene comprising objects based on the 3D data. The rendered scene may be used to generate realistic object occlusion, computational efficiency, visual effects (such as lighting and shadowing), or output (such as graphical output) that reflects physical effects (such as collision detection) or partial displays of occluded objects that would otherwise be difficult or impossible to achieve in a system that does not utilize a unified scene graph.

[0026] In some embodiments, an exemplary computer system includes multiple applications, each including 3D data representing one or more objects within a common 3D environment. Each of the multiple applications may exist in a “sandboxed” environment such that it remains independent of other applications; e.g., each individual application's data may be independent of each other application's data, each application may not have access to each other application's data, and each application's respective 3D data may correspond to the same 3D environment, while each application maintains its own instance of the 3D environment. For example, each application may represent a player in an online multiplayer video game, where each player exists in the same game world instance or 3D environment but lacks direct access to other players' data. In such embodiments, it may be desirable for all players to be rendered simultaneously in a single instance of the game world, but it may be undesirable (or computationally prohibitive) for each player to maintain the information necessary to render each other client participant's 3D data. Furthermore, for security purposes, it may be desirable to limit player information that is available to other players.

[0027] In some embodiments, multiple sandboxed applications can each independently write information corresponding to their 3D data to a local scene graph, which is later written to a common unified scene graph. The unified scene graph can then be traversed by a renderer to render a scene for presentation on a display as an image based on the collective 3D data provided by each application. By communicating 3D data from each of the multiple sandboxed applications to a single unified scene graph, the renderer can apply beneficial techniques, such as occlusion, lighting effects, and rendering optimizations (such as surface culling), that require or benefit from concurrent knowledge of all applications' 3D data. These benefits are achieved while limiting the required computational overhead of each sandboxed application; from a single application's perspective, all the application needs to do is update a single scene graph to reflect its 3D data; other operations are performed by other components of the system. Furthermore, security benefits can be obtained by maintaining isolation between sandboxed applications.

[0028] FIG. 3A illustrates components of an exemplary computer system 300 that can render 3D data from multiple independent applications to a display using a unified scene graph. The illustrated embodiment utilizes a client-server topology; however, this disclosure is not limited to client-server embodiments. In the exemplary computer system 300, a first client application 310 and a second client application 320 each communicate 3D data (in some embodiments, over a network) to a client-server interface 330. In some embodiments, the client applications 310 and 320 are “sandboxed” applications that operate independently of each other and independently communicate their 3D data to the client-server interface 330. The client-server interface 330 can receive updated 3D data from the client applications 310 and 320 and communicate that 3D data (in some embodiments, over a network) to a server-side host application 340. In some embodiments, the client-server interface 330 uses multithreading techniques and multiple processor threads to receive, process, and / or communicate 3D data to the host application 340. In some embodiments, the client-server interface includes logic for controlling (such as by throttling) the rate at which 3D data is communicated to the host application 340. The host application 340 can use the 3D data received from the client-server interface to update the unified scene graph 350 so that it reflects the 3D data received from the client applications 310 and 320. In some embodiments, the unified scene graph 350 comprises multiple versions of the scene graph, and known versioning techniques are used to allow updates to the unified scene graph 350 to occur in parallel.Renderer 360 can then traverse unified scene graph 350, apply optimizations and effects as appropriate, and generate output (e.g., graphical output comprising data for at least one of client applications 310 and 320 and, in some embodiments, only the occluded portions of one client application without the occluding application data) for display on display 370, such as a computer monitor.

[0029] 3B illustrates aspects of an exemplary client application 310 for the exemplary computer system 300 shown in FIG. 3A. In the illustrated embodiment, 3D data 312 represents graphical objects (geometric primitives, e.g., polygons, etc.) within a 3D environment to be presented on a display 370. The 3D data 312 may be updated 314 by the client application 310. For example, if the client application 310 is an application with a rendering loop that iterates 60 times per second, the client application 310 may update the 3D data 312 60 times per second to reflect changes in that data during the course of the application's operation that should be reflected in the rendered output. In some embodiments, the 3D data 312 is represented as a local scene graph 316, which may be local to each client application 310. In some embodiments, the local scene graph 316 may include data (e.g., nodes) that correspond to data in the unified scene graph 350. When the 3D data 312 is updated (314), the client application 310 can update the local scene graph 316 to reflect the most recent version of the 3D data 312. When the local scene graph 316 is updated, it can be used by the client application 310 to generate (317) client data 318. In some embodiments, the client data 318 may represent the local scene graph 316 in its entirety. In some embodiments, the client data 318 may represent changes made to the local scene graph 316 since the previous client data 318 was sent to the client-server interface 330. For example, the client data 318 may include nodes added to or removed from the local scene graph 316, changes to relationships between nodes in the local scene graph 316, or changes to the properties of nodes in the local scene graph 316.In some embodiments, the client data 318 may use identifiers, such as identification numbers, corresponding to scene graph nodes to identify relationships between data from the local scene graph 316 and corresponding data on the unified scene graph 350. The client data 318 may then be communicated to the client-server interface 330 for eventual communication to the host application 340. In some embodiments, communication of the client data 318 to the client-server interface 330 may occur over a network. In some embodiments, a client helper application may be used in conjunction with the client application 310 to generate the client data 318 from the local scene graph 316 or from the 3D data 312.

[0030] Aspects described with respect to client application 310 may similarly describe client application 320 or other client applications that (together with client application 310) comprise exemplary computer system 300. Those skilled in the art will appreciate that the systems and methods described herein may be extended to include any number of client applications and client data, and the disclosure is not limited to any such number. Furthermore, some benefits (e.g., improved computational efficiency) may become more apparent with an increasing number of client applications. As described above, client applications 310 and 320 may be sandboxed applications that do not share data or functionality. For example, in exemplary computer system 300, client application 320 may have its own 3D data and local scene graph that are separate from 3D data 312 and local scene graph 316 belonging to client application 310. However, in some embodiments, including exemplary computer system 300, a single client-server interface 300 is shared by multiple client applications, such as client applications 310 and 320.

[0031] Although the above examples are described with respect to 3D data, the present disclosure is not limited to three-dimensional data or data within a 3D environment. The above examples can be generalized to graphical data of any number of dimensions, including 2D graphical data.

[0032] 3C illustrates aspects of an example renderer 360 with respect to the example computer system 300 shown in FIGS. 3A-3B. In some embodiments, the renderer 360 comprises part of the host application 340. In some embodiments, the renderer 360 may be part of another component of the example computer system 300, or may be a separate component or application. In some embodiments, the renderer 360 may be implemented in different physical hardware than one or more components of the example computer system 300 and may communicate with one or more of those components over a network.

[0033] 3C , renderer 360 operates on version 352 of unified scene graph 350. In this embodiment, the renderer's role is to create a rendered scene comprising data, such as output or graphical output, for presentation on display 370 based on version 352 of unified scene graph 350. As part of this process, renderer 360 may traverse 362 version 352 using known scene graph traversal techniques. During or after traversal 362, renderer 360 may update 364 unified scene graph 350 as appropriate to reflect the results of the traversal. For example, as part of traversal 362, renderer 360 may identify abandoned nodes that should be removed from unified scene graph 350. Following traversal 362 and / or update 364, renderer 360 may apply various optimizations 366 to the scene. For example, renderer 360 may cull obscured or invisible surfaces to avoid consuming unnecessary computational resources. Following traversal 362 and / or update 364, renderer 360 may apply one or more visual effects 367 to the scene. For example, in some embodiments, renderer 360 may apply lighting or shadow effects, apply one or more shaders, apply particle effects, and / or apply physics effects. Finally, renderer 360 may output data to a graphical output pipeline, which may display the output on display 370.

[0034] In some cases, optimizations may be applied to the systems and methods described above to reduce the size of scene graph data, e.g., the amount of scene graph data that needs to be transmitted over a computer network (e.g., via client-server interface 330). For example, if a scene graph includes nodes (or referenced assets) that may not be visible in the rendered graphical output, those nodes may be excluded from the scene graph data to be transmitted. Similarly, assets may be unloaded when the node that references them is removed from the scene graph. Furthermore, to improve performance, nodes may be inserted only when they need to be rendered, avoiding the overhead of including nodes in the scene graph data when they may not have any apparent effect on the rendered scene (e.g., when the absence of the node may be imperceptible to the user).

[0035] In some cases, different levels of detail (LODs) can be applied to assets to reduce overall data size. Different LODs can help ensure that assets are less data-intensive than is warranted in a particular viewing context. For example, a lower-resolution image can be substituted for a higher-resolution image when the expected display size does not justify the data size of the higher-resolution image. This can improve system performance, for example, by reducing the amount of network bandwidth required to transmit a larger asset (e.g., a higher-resolution image) than necessary.

[0036] Application sharing

[0037] Embodiments of the systems and methods disclosed herein can be used to enable multiple clients to share (e.g., view and interact with) the content of a common software application running on a single host client. This may be desirable to avoid the need to install the software application on each remote client that communicates with the host client, and further, it may be desirable to synchronize input and control information across multiple users of the application. This may be advantageous in embodiments where clients have limited computing resources (e.g., disk storage) or users may lack the time, aptitude, or inclination to install a software application. Similarly, software providers may desire to encourage users to use a software application by reducing the number of steps required to interact with that application (e.g., by allowing a user to share another user's version of the application). Furthermore, the systems and methods disclosed herein can minimize network bandwidth usage by rendering content locally at each remote device rather than by forwarding video output from the host device to the remote device, e.g., by leveraging the centralized rendering systems and methods described above. In some examples, a computing device for hosting, sharing, viewing, or interacting with a software application may correspond to a head-mounted display device such as that corresponding to the exemplary device illustrated in FIG. 1E described above, and the shared content may be displayed on the display of the head-mounted device, and the user may interact with the shared software application via sensors or input devices of the head-mounted device.

[0038] 4A illustrates an exemplary system 400 for implementing the application sharing example described above. In the figure, four users (Praveen, Lisa, Leonid, and Naruto) of four separate computing devices (410, 420, 430, and 440) wish to share three separate software applications (chess, calculator, and Scrabble), with each application running on only one of the client computing devices. System 400 includes server 450, which in some embodiments may correspond to a cloud server. In exemplary system 400, computing devices 410, 420, 430, and 440 each communicate with server 450 but do not necessarily communicate directly with other computing devices.

[0039] In exemplary system 400, three software applications (chess, calculator, and Scrabble) each execute on one of client computing devices 410, 420, 430, or 440, but can be viewed and / or interacted with by each of the client computing devices on which it is not executing. For example, as shown in the figure, the chess application executes on device 410, and devices 420, 430, and 440 can each view the chess application by rendering a chess scene graph local to the respective device. (The scene graph can preferably be rendered on a display using a centralized rendering system and method such as those described above.) Similarly, in an embodiment, the calculator application is shown executing on device 410 in parallel with the chess application, and devices 420, 430, and 440 can each render a view of the calculator application via a calculator scene graph local to the respective device. Additionally, in the example embodiment, the Scrabble application is shown running on device 420, and devices 410, 430, and 440 are each capable of rendering a view of the Scrabble application via a Scrabble scene graph local to the respective device. Thus, each of the four devices 410, 420, 430, and 440 is capable of viewing each of the three applications either by running the application locally (as in device 410 running the chess and calculator applications) or by rendering a view from a local scene graph (as in device 430, which stores a local scene graph for each of the chess, calculator, and Scrabble applications). Server 450 operates to exchange data between devices 410, 420, 430, and 440, as described below.In some embodiments, such as those described below, server 450 can maintain its own local copy of the scene graph corresponding to each application (e.g., chess, calculator, Scrabble), which can provide the base scene graph for devices joining a pre-existing application sharing session.

[0040] In exemplary system 400, a host system runs an application (e.g., device 410 runs a chess application) and shares it with one or more remote devices (e.g., devices 420, 430, and 440). It may be advantageous for the remote devices to view and interact with the shared application without installing the application on the device itself. For example, in the illustrated embodiment, remote devices 420 and 440 do not have the chess application installed, and when initiating a sharing session, device 410, which hosts the chess application, may provide necessary data, such as requested assets, to remote devices 420 and 440. However, in some embodiments, the overhead of providing such data may be reduced or avoided by installing such data on the remote device. For example, in the illustrated embodiment, remote device 430 has the chess application installed on it before joining the sharing session, which may eliminate the need for device 410 to provide assets and other data related to the chess application to remote device 430. However, in an embodiment, the chess application is not executed on the remote device 430 (i.e., by a processor of the remote device 430), but rather it is executed on the device 410, and the remote device 430 relies on its own local installation of the chess application for required application data, for example, during initialization.

[0041] FIG. 5 illustrates an example of a host device (e.g., device 410) running an application 510 (e.g., chess) and sharing the application with a remote device (e.g., device 420) so that a user of the remote device can view and interact with the application, such as that described above with respect to exemplary system 400. As noted above, the remote device need not install the application; for example, at the start of an application sharing session, the host device may provide the remote device with required initialization data, such as required assets (e.g., textures, model data, audio files, particle effects, animation data), without the remote device having to perform a separate installation procedure. While the application is being shared, the host device may provide the remote device with application state data and / or one or more scene graphs, such as those described below. In some examples, the remote device may provide the host device with input data (e.g., representing button inputs, gesture inputs, touch inputs), user data (e.g., the user's identification details, activity history, and / or social media presence information), and / or sensor data (e.g., GPS coordinates, camera data, microphone data). In FIG. 5, application 510 represents an application executing on host device 410. In some embodiments, multiple applications, such as the chess and calculator applications shown executing on device 410 in FIG. 4A, may run concurrently on device 410; while the illustrated embodiment describes a single shared application, this can be extended to multiple shared applications. In some such embodiments, the host device may maintain and share two or more independent scene graphs (e.g., three scene graphs, each corresponding to one of the three shared applications). As noted above, in some embodiments, a single scene graph may correspond to two or more applications (e.g., a single scene graph corresponding to chess and Scrabble).Further, as described herein, while remote device 420 is remote with respect to host device 410, it need not be physically or electronically separate from host device 410. In some examples, remote device 420 may be physically or electronically connected to host device 410 or may be a component or module of a single physical device. In this manner, application sharing can be implemented at the platform level. For example, applications can be shared between components or areas of a single platform, such as a mobile device operating system (e.g., iOS, Android) or an operating system for a head-mounted device. In some examples, applications can be shared in this manner across platforms, which can, for example, enable behavior in which an application is shared between a user of a first platform (e.g., a mobile device running iOS) and a user of a second, different platform (e.g., a head-mounted virtual reality system running a custom operating system).

[0042] The copresence service 540 is a helper application that enables a primary application to be shared with a remote device. In an embodiment, the copresence service 540 runs on the host device 410 alongside the application 510, sandboxed to the application 510, allowing the application 510 to be shared with the remote device 420. In some embodiments, as described below, each device with which the application 510 is shared (e.g., devices 420, 430, 440 of FIG. 4A ) can run a separate copresence service application. For example, in the embodiment shown in FIG. 5 , the remote device 420 runs its own remote copresence service 570, which can include some or all of the features of the host copresence service 540. For example, the remote copresence service 570 can include a main logic loop 574 and a remote rendering engine client 576.

[0043] In the illustrated embodiment, host coexistence service 540 incorporates transport functionality for sending and receiving data (e.g., graphical data, input data) to and from remote devices (e.g., 420) via cloud server 450. For example, in an embodiment, local endpoint 542 can act as a terminal for exchanging data with application 510, and endpoint 550 can act as a terminal for exchanging data with cloud server 450. In an embodiment, host coexistence service 540 further comprises main application logic loop 544, which can perform various functions associated with host coexistence service 540. These functions can include mediating the exchange of data between host device 410 and remote devices, adding and removing remote devices to and from shared application 510, user interface functions (which may include “avatar” functionality for representing remote users), chat functionality, file transfer (e.g., exchanging data including assets associated with application 510 between host device 410 and remote devices such as remote device 420), and / or generating graphical output. The graphical output of the copresence service 540 (which may include a scene graph) can be presented to a rendering engine client 546, which in some embodiments may correspond to the client-server interface 330 described above with respect to Figures 3A and 3B. The rendering engine client 546 can then present the graphical data to a rendering engine server 560 (which in some embodiments may correspond to the server 340 described above with respect to Figures 3A and 3B) for rendering on the host device 410 (e.g., as described above with respect to process 360 shown in Figure 3C). However, in some embodiments, the copresence service 540 may include only limited features and may be a background process that runs without any user input or display output.

[0044] 5 , application 510 may include an input receiver 512 that may receive and handle host input 506 from device 410 (e.g., via an input service associated with device 410). Host input 506 may include input received via a traditional input device such as a keyboard or mouse, via a sensor (e.g., a camera, eye-tracking sensor, microphone) of a head-mounted device, via a handheld input peripheral, or by any other suitable device. In some examples, input 506 may be provided by one or more sensors or input devices associated with the head-mounted augmented reality device. In some examples, input receiver 512 may receive input provided by a remote device, such as those described below, instead of or in addition to host input from host device 410.

[0045] The application 510 may further include a main application logic stage 514. Input received by 512 may be provided as input to the application logic stage 514. The application logic stage 514 may include a main application loop that receives input, executes application logic (e.g., game logic for a chess application), maintains some application state, and provides some output (e.g., display output). The application logic stage 514 may include steps for determining an application scene graph for the host device 410 and for each remote device (e.g., remote device 420) with which the application is shared; in some embodiments, one or more of the local and remote devices may share the same scene graph. An exemplary graphical process implemented by the application logic stage 514 is described above with respect to the client application 310 shown in FIG. 3B . However, the present disclosure is not limited to any particular application or any particular application logic 514.

[0046] The application 510 may further comprise a rendering engine client 516, which in some embodiments may correspond to the client-server interface 330 described above with respect to FIGS. 3A and 3B. In some embodiments, the rendering engine client 516 may be identical to or share common components with the rendering engine client 546 described above. From the application logic stage 514, display data for the application logic may be provided to the rendering engine client 516, such as those described above with respect to FIGS. 3A and 3B. The rendering engine client 516 may be a client-side utility local to the host device 410 that generates one or more scene graphs, each scene graph corresponding to the application 510, based on the output of the application logic stage 514. In some embodiments, the application 510 may generate a scene graph for each device (e.g., 420) with which the application 510 is to be shared. Such scene graphs may be the same or unique to one or more of the devices. In some embodiments, the scene graph may be optimized for each remote device, such as by including only the data needed by each remote device. For example, the remote device 420 may provide the host device 410 with data indicating the field of view and camera origin of the user of the remote device 420, and the host device 410 may accordingly calculate elements of the scene graph that may be visible to the user of the remote device 420 (i.e., elements that may be visible in a view corresponding to the camera origin and field of view) and prune non-visible elements from the scene graph for the remote device 420. Similar optimizations may also be performed based on information regarding elements of the scene graph that may be visible on the remote device (e.g., level of detail settings or layering information on the remote device).

[0047] To present a view of application 510 to a local user of host device 410 (e.g., representing the user's view of the application), rendering engine client 516 may generate a scene graph corresponding to the local user's view and then present the scene graph and associated content to rendering engine server 560, which may display the corresponding view of application 510 on the display of device 410. This process may correspond to that described above with respect to FIGS. 3A and 3B. Rendering engine client 516 may, but need not, present the scene graph and associated content directly to rendering engine server 560. In some embodiments, rendering engine client 516 may present the scene graph and content to an endpoint, a socket pipe, or any other suitable inter-process communication mechanism for presentation to rendering engine client 560.

[0048] In some embodiments, the rendering engine server 560 may be local to the host device 410, and in some embodiments, the rendering engine server may be on a remote device, such as a dedicated server, or on another device connected to the device 410, such as in a peer-to-peer network. Neither the rendering engine client 516 nor the rendering engine server 560 need be a dedicated hardware module; in some embodiments, the rendering engine client 516 and / or the rendering engine server 560 may be pure software modules, such as separate applications (e.g., separate applications running in parallel on a single platform) or separate threads of a multi-threaded application. However, in some embodiments, one or more rendering engine functions (e.g., those performed by the renderer 360 shown in FIG. 3C ) may be implemented on dedicated hardware, such as a dedicated server and / or dedicated graphics processing hardware (e.g., a GPU). By implementing computationally intensive functions on dedicated hardware, computing resources can be freed up on the user device (e.g., the device 410). This can be particularly beneficial when user devices include mobile hardware (eg, mobile phones, headsets) that have limited battery life, computing resources, and physical space.

[0049] The rendering engine server 560 may include a unified scene graph and renderer, such as those described above. In some embodiments, multiple client applications (e.g., a chess application and a calculator application running in parallel on the device 410) may simultaneously provide content and / or scene graph data to the rendering engine server 560. Furthermore, in some embodiments, the rendering engine server 560 may not include any client application code and may be agnostic to the particular source of such data presented to it by the rendering engine clients 516. This may enable its use with a variety of applications, such as the chess and calculator applications described above, without the need for the rendering engine server to include custom logic for each such application. Similarly, this may enable applications, such as the chess and calculator applications, to collaborate with coexistence services, enabling application sharing functionality without any custom programming, and facilitating the production, adoption, and use of such applications.

[0050] 5, rendering engine client 516 can provide graphical data to host coexistence service 540. In some embodiments, this graphical data can be provided to an endpoint (e.g., endpoint 520) or to a socket or another suitable inter-process communication mechanism. The graphical data can include remote graphical data intended for presentation to remote device 420. For example, the remote graphical data can include a scene graph or partial scene graph corresponding to application 510 that is to be rendered and displayed on remote device 420. With respect to such remote graphical data, endpoint 520 can provide the remote graphical data to another endpoint 550 corresponding to the remote device, from which the data can be communicated to remote coexistence service 570 running on remote device 420 (e.g., via cloud server 450 and / or remote endpoint 580 corresponding to the remote coexistence service).

[0051] In some embodiments, once the remote graphical data (e.g., including a scene graph) is received at the remote copresence service 570, it can be provided to the remote copresence service's main logic loop 574, which can use the remote graphical data to determine a view to be rendered on the remote device 420. For example, the remote copresence service 570 can generate and update a local scene graph, such as that described above. In some embodiments, the remote graphical data can be provided directly to the rendering engine client 576. The rendering engine client 576 can then provide the data to the rendering engine server 590, which can be rendered on a display, such as the display of the device 420.

[0052] In some embodiments, the remote device 420 cannot interact with the application 510, but instead acts as a spectator that simply renders a view of the application. However, in other embodiments, the remote device 420 can interact with the application 510, such as through input provided to the remote device 420. In some cases, this input can affect the remote device's view of the application 510 (e.g., by adjusting display parameters) without communication back to the host device 410. However, in some embodiments, the remote device 420 can provide input back to the host device 410, which can be passed as input to the application 510 running on the host device 410. For example, if the application 510 is a chess application, the remote device 420 may provide user input to the host device 410 indicating the user's desired chess move. In the illustrated embodiment, the remote coexistence service 570 comprises a remote input receiver 572, which receives input from the remote device 420 (e.g., via the input service 566). The remote input receiver 572 and input services 566 may correspond to the input receiver 512 and input services 506, respectively, described above with respect to the host device 410. For example, the remote input receiver 572 may receive input provided via a conventional input device (e.g., a keyboard or mouse), via a sensor in a head-mounted device, via a handheld input peripheral, or by any other suitable device. This input data may be provided as input to the main loop 574 of the remote coexistence service 570.

[0053] In embodiments in which input data affects a remote device's view of an application without affecting the execution of the application itself, main loop 574 can determine how the input data affects the application's view and how corresponding graphical data should be presented to rendering engine client 576 (which, in turn, provides the graphical data to rendering engine server 590). In embodiments in which input data is communicated back to host device 410, main loop 574 can present the data to endpoint 580, which can communicate the data to endpoint 550 of host coexistence service 540. From there, the data can be communicated back to application 510 (e.g., via endpoints 542 and 520 and input receiver 512), where it can be presented as remote input to main application logic loop 514. Application logic loop 514 can then incorporate the remote input into an updated scene graph of application 510; for example, in a chess application, application logic loop 514 can update the game state according to user input received from the remote device, move chess pieces accordingly, and generate a new scene graph reflecting the updated positions of the chess pieces. The updated scene graph can then be presented to the remote device 420, such as via the mechanisms described above. In this manner, the remote device 420 can provide input to the application 510 and receive an updated scene graph that takes into account the remote device's input.

[0054] The graphical data provided by the rendering engine client 516 may also include host graphical data intended to be rendered and presented on the host device 410, instead of or in addition to the remote graphical data described above. This host graphical data may be presented to the host coexistence service 540 via the endpoint 520, for example, as input to the main loop 544. This graphical data may then be presented to the rendering engine client 546 of the host coexistence service 540, from where it may be presented to the host rendering engine server 560. The rendering engine server 560 may render a single view that incorporates graphical data from two sources, such as those described above, e.g., graphical data corresponding to the local application 510 and also graphical data corresponding to the host coexistence service 540.

[0055] While the example shown in FIG. 5 illustrates a single application 510 running on a host device and shared with a remote device, the example can be extended to multiple applications running on a host device and shared with a remote device. As noted above, graphical data for such multiple applications may be represented by a single scene graph (e.g., one scene graph corresponding to two applications) or by two or more scene graphs (e.g., two scene graphs, each corresponding to one application). Furthermore, the remote device 420 illustrated in the example shown in FIG. 5 may also run its own additional applications, which it may share with other devices (including device 410). For example, FIG. 4A shows device 420 running a Scrabble application that is shared with device 410. In such an example, remote device 420 may be considered a host device for the applications it runs. Other configurations of devices and shared applications are apparent, and the present disclosure is not limited to any particular such configuration. For example, applications may be shared between applications running on a single platform (e.g., iOS), between applications running on different platforms (e.g., iOS and Android), or some combination.

[0056] 6 shows a flowchart illustrating an example flow of data between a host device (e.g., device 410 of FIGS. 4A and 5) running an application (e.g., application 510 of FIG. 5, which may be a chess application) and a remote device (e.g., device 420 of FIGS. 4A and 5) sharing that application in accordance with the present disclosure. (As used with respect to this example, "host" refers to processes and data local to host device 410, the device running application 510 in a sharing session, and "remote" refers to processes and data remote from device 410, such as processes and data on device 420. Device 420 may have application 510 downloaded or installed but is not running it in the sharing session.) In the example process illustrated in FIG. 6, a user of a remote device can view and interact with the application without the application having to be installed on the remote device. In example step 610, a user of host device 410 initiates sharing of application 510. Application sharing can be initiated using any suitable technique, such as those described further below, for example, via interaction with a “Share” option presented via a context menu or system dialog on the host device 410. (In some embodiments, a user of a remote device, rather than the host device, may initiate application sharing.) In response, in stage 612, a host copresence service (e.g., host copresence service 540) running on device 410 registers application 610 for sharing and performs any necessary initialization. The host copresence service communicates (e.g., via server 450) to a remote copresence service (e.g., remote copresence service 570 running on device 420) that application 510 is being shared. (In some embodiments, such as when the remote device does not have application 510 downloaded or installed, this stage may include sending requested assets or other application data to the remote device.)) In response, in stage 614, the remote coexistence service may create a remote scene graph (or, in an embodiment, a sub-scene graph) local to the remote device 420 that corresponds to the application 510. As described above, a rendering engine server running on (or in communication with) the remote device 420 may render this remote scene graph, such as that described above with respect to FIGS. 3A-3C , and provide a display output (e.g., on a display of the device 420) that corresponds to a view of the application 510 belonging to the user of the device 420.

[0057] At stage 620, the host device 410 may determine a scene graph for the application 510 to be presented on the device 420 (e.g., a 3D camera view of the application 510 corresponding to the camera position of the user of the device 420), such as that described above, and sends data reflecting the scene graph to the host copresence service 540. At stage 622, in response to receiving the data reflecting the scene graph, the host copresence service 540 forwards the data to the remote copresence service 570 via the server 450, e.g., using a remote endpoint, a socket, or other suitable mechanism for communicating data. At stage 624, in response to receiving the data reflecting the scene graph, the remote copresence service 570 updates or replaces the remote scene graph created in stage 614 as described above (e.g., via the rendering engine client 576). This remote scene graph can then be rendered at the device 420 via a rendering engine server 590, such as that described above. By only communicating this updated data from the host device to the remote device, bandwidth usage is limited compared to transmitting raw image data (e.g., video frames) or other storage-intensive data.

[0058] The device 410 can repeat the process in stage 620 for additional devices, including the host device 410, such as remote devices 430 and 440. For each individual device, the rendering engine client 516 can determine a scene graph for the application 510 to be presented on that device, such as that described above. For a scene graph to be displayed on the host device 410, the rendering engine client 516 can generate or update a scene graph, such as that described above with respect to Figures 3A-3C, and present it to the rendering engine server 560 for rendering.

[0059] In embodiments in which the remote device 420 interacts with the application 510 through providing input rather than simply passively displaying a view of the application 510, the remote copresence service 570 in stage 630 can receive input from the remote device 420, for example, via a remote input receiver 572 (whose behavior may correspond to the host input receiver 512 and input service 506 as described above) that communicates with the input service 566. In stage 632, the remote copresence service 570 can forward the input event data to the host copresence service 540, which receives the input event data in stage 634. In stage 636, the host copresence service 540 can forward the input event data to the local application 510, which can be received in stage 638 (e.g., by the input receiver 512) and provided as input to the application logic 514.

[0060] In stage 640, application logic 514 may update the host scene graph according to host input received from host device 410, such as those described above; in embodiments in which a remote device (e.g., remote device 420) provides input, application logic 514 may additionally incorporate the remote input when updating the scene graph. For example, if application 510 is a chess application and the remote input received by application 510 in stage 630 corresponds to a user of remote device 420 moving a chess piece, application 510 may update the game state accordingly (which may affect data presented to all devices sharing the application as well as host device 410) and generate a corresponding updated scene graph reflecting the new game state (e.g., with the chess piece in its updated position on the chess board).

[0061] In step 650, the host application 510 sends updates to the remote device 420 based on the results of step 640. These updates are provided to the host coexistence service 540 in step 652 and to the remote coexistence service 570 in step 654. Steps 650, 652, and 654 may correspond to steps 620, 622, and 624, respectively, described above, and the steps described above may be repeated indefinitely as the host device 410 hosts the application 510 and the remote device 420 views and / or interacts with its shared version of the application. In some embodiments, bandwidth usage of the above process can be controlled by modulating the rate at which updates are provided by the host device. For example, increasing the time that elapses between updates (e.g., between step 620 and step 650) can reduce the required bandwidth. Exemplary systems and methods for adapting to changes in data throughput and for synchronizing devices that may have different rates of data throughput are described in U.S. Patent Application Nos. 15 / 940,892 and 16 / 011,413. Additionally, in some embodiments, the data size of updates between the host application 510 and the remote device 420 can be optimized by limiting the contents of the scene graph sent in the update. For example, as described above, the host application can determine (e.g., in stage 650) elements of the scene graph that may be invisible at the remote device 420 (e.g., because they reside outside the remote device 420's expected camera view) and cull those elements from the scene graph before sending the scene graph to the remote device 420.

[0062] In some embodiments, a cloud server or other suitable storage device may maintain base scene graphs for shared applications to facilitate new remote devices joining a shared session already in progress. For example, with reference to FIG. 4A , cloud server 450 may maintain scene graphs corresponding to a chess application, a calculator application, and a Scrabble application, respectively. In such embodiments, when an application becomes shared with an additional remote device (e.g., remote device 430), the device may initially be presented with the base scene graph residing on the cloud server, allowing the new remote device to join the ongoing application without the overhead of creating a new scene graph corresponding to the application. In such embodiments, it may be necessary for host device 410 to present incremental updates (e.g., changes since the previous update) to the remote device; because the base state scene graph is constantly available from cloud server 450, the remote device need only retrieve the base state scene graph and the corresponding changes since the base state to reconstruct a scene graph corresponding to the current state of application 510.

[0063] In the embodiments described above, a host device (e.g., device 410) runs an application and shares the application with a remote device (e.g., device 420). The host device generates a scene graph and sends it to the remote device, from which the remote device renders a view for display. In the embodiments described above, the host device also generates a scene graph for itself (which may or may not be the same scene graph as the one sent to the remote device), and the host device uses the scene graph to render a view for display on the host device. This may be desirable in a situation where the host device and remote device belong to two users who want to participate in the same shared application; for example, a user of the host device may want to play against a user of a remote device in a shared chess application. However, in some embodiments, the host device may simply send a scene graph to each remote device without rendering a view for itself.

[0064] FIG. 4B illustrates one such “headless” embodiment. In the embodiment 400A shown in FIG. 4B, the host device (e.g., 450A in FIG. 4B) may be a dedicated hosting entity, such as a data center, a cloud server, or one or more networked servers, configured for the primary purpose of sharing applications with remote devices (e.g., remote devices 410A, 420A, 430A, and 440A) such as those described above. This configuration may be advantageous, for example, when the host device may have access to more computing resources than the remote devices with which it communicates. In the illustrated embodiment, host device 450A runs one or more applications (e.g., chess, calculator, Scrabble). Host device 450A may generate and maintain scene graphs for each of those applications, such as those shown in FIG. 4B, and transmit those scene graphs to each remote device, such as those described above. Each remote device may then determine a local view and display that view to its individual user.

[0065] Logic executing on the host device can update the state of applications, such as those described above, to the remote device. For example, the host device logic can receive an updated location of the remote device, e.g., when a user of the device moves from the living room to the kitchen. When the user enters the kitchen, the new location can be provided from the remote device to the host device, and an updated scene graph and / or associated assets can then be provided from the host device to the remote device, which can then view the new content.

[0066] In some embodiments, a host device may be configured to host multiple applications, and in some embodiments, remote devices connecting to the host device never need to download or install applications, but instead rely on the host device to provide the remote device with all necessary data about the application when a sharing session is initiated. Applications can be launched as server-side programs on the host device using any suitable hosting framework. In some cases, a single instance of an application on the host device can be shared with different client participants. Seamless cross-platform application sharing can be enabled in this way.

[0067] In some examples, a request to run or share an application can be initiated by either a host device (e.g., host device 450A of FIG. 4B) or a remote device (e.g., remote devices 410A, 420A, 430A, or 440A of FIG. 4B), such as those described above. The share request can be approved or denied by the other party to the request (e.g., the remote device in a request by the host device or the host device in a request by the remote device).

[0068] Any suitable method can be used to initiate application execution or sharing. In some examples, a user can explicitly initiate application execution or sharing, such as by interacting with a traditional user interface (e.g., a menu listing application selections). In some examples, a user of a computing device (e.g., device 410A with reference to FIG. 4B ) can initiate application execution or sharing by interacting with a trigger, such as by scanning (e.g., via a camera in a head-mounted device) a visual target containing a QR code, barcode, URL, or other data that may include addressing information for the host device (e.g., host device 450A with reference to FIG. 4B ). In some cases, the trigger may include entering or exiting a particular location, such as that detected by a GPS unit, and the addressing information may be associated with the location (e.g., via a lookup table). In some examples, a single trigger can be associated with two or more applications.

[0069] In response to the interaction, the user's device can use the addressing information to communicate a request to the host device. In response to receiving the request, the host device can initiate sharing of an application with the user's device (e.g., as described above with respect to step 610 of FIG. 6 ), serve a file (e.g., an executable file to be launched on the user's device) to the user's device, or perform some other action. In examples in which the user's device interacts with the virtual environment, such triggers can be placed in the virtual environment (in some cases by other users of the shared virtual environment using spatial information such as persistent coordinate frame (PCF) data), and the user's interaction with those triggers in the virtual environment can result in the execution or sharing of an application as described above. Further, in some cases, triggers can be associated with sensor data, such as images received via a camera on a head-mounted device. For example, a user may encounter a chessboard at a particular location (e.g., a real-world location or a location in the virtual environment). By viewing, approaching, touching, or otherwise interacting with the chessboard, a user can initiate a request to run or share a chess application (e.g., as described above with respect to FIG. 6). In this manner, the application can be run and / or shared not only at the user's own initiative (e.g., via a traditional user interface), but also through organic interaction with objects and places within a real, virtual, or hybrid (e.g., augmented reality) environment.

[0070] In some cases, data related to the user's environment (e.g., sensor data such as GPS data or camera data from a head-mounted device) can be used to provide one or more parameters to an application. For example, when a chess application requests to be shared, the user's current location can be used to identify the current weather and time of day, which can be used to determine visual environmental effects (e.g., time-sensitive lighting) related to the chess application. Similarly, in some examples, a request to run or share an application may be based on contextual information; for example, a user may be able to send a share request to other devices in their vicinity. In some examples, it may be desirable for a “headless” host device (e.g., 450A) to present a remote user (e.g., a user of 410A, 420A, 430A, 440A) with an application share request based on contextual information; for example, the host device may be configured to present a request to share a theme park application to the user when the user's location is detected to be in a theme park. In some embodiments, permission may not be required and content may be automatically shared with the user. In some embodiments, data may be used to launch an application but may not be provided to the application itself. For example, personal data reflecting a user's current location may be used to trigger an application based on that location, but the application data may be prevented from accessing that personal data, such as to promote user privacy and confidentiality. Other methods of initiating an application sharing request will be apparent and are within the scope of this disclosure.

[0071] The above exemplary processes of a computer system may be provided by any suitable logic circuitry. Suitable logic circuitry may include one or more computer processors (e.g., CPUs, GPUs, etc.) that perform the processes upon execution of instructions implemented in a software program. In addition, such processes may also be provided via corresponding logic designs implemented in hardware logic circuitry, such as programmable logic (e.g., PLDs, FPGAs, etc.) or customized logic (e.g., ASICs, etc.), that implement the logic design providing the processes. Furthermore, such processes may be provided via implementations that combine one or more processors running both software and hardware logic circuitry.

[0072] 7 illustrates an example system 700 that can be used to implement any or all of the above-described embodiments. The above-described embodiments (in whole or in part) may be embodied within any portable (including wearable) or non-portable device, such as a communications device (e.g., a mobile phone, a smartphone), a multimedia device (e.g., an MP3 player, a TV, a radio), a portable or handheld computer (e.g., a tablet, a netbook, a laptop), a desktop computer, an all-in-one desktop, a peripheral device, a head-mounted device (e.g., which may include an integrated display), or any other system or device compatible for inclusion in the example system architecture 700. The above-described embodiments may be embodied in two or more physically separate devices, such as two or more computers that communicate via a wireless network. The above-described embodiments may be embodied in two or more physically distinct devices, such as a beltpack that communicates data to and / or from a head-mounted display. 7 is a block diagram of one embodiment of a system 700 that generally includes one or more computer-readable media 701, a processing system 704, an I / O subsystem 706, radio frequency (RF) circuitry 708, audio circuitry 710, and sensor circuitry 711. These components may be coupled by one or more communication buses or signal lines 703.

[0073] It should be apparent that the architecture shown in Figure 7 is only one example architecture for system 700, and that system 700 may have more or fewer components or a different configuration of components than those shown. The various components shown in Figure 7 may be implemented in hardware, software, firmware, or any combination thereof, including one or more signal processing and / or application specific integrated circuits.

[0074] Referring to the exemplary system architecture 700 of FIG. 7, RF circuitry 708 can be used to send and receive information via a wireless link or network to one or more other devices and includes well-known circuitry for performing this function. RF circuitry 708 and audio circuitry 710 can be coupled to processing system 704 via a peripherals interface 716. Interface 716 can include various well-known components for establishing and maintaining communications between peripherals and processing system 704. Audio circuitry 710 can be coupled to audio speaker 750 and microphone 752 and can include well-known circuitry for processing audio signals received from interface 716 and enabling a user to communicate with other users in real time. In some embodiments, audio circuitry 710 can include a headphone jack (not shown).

[0075] The sensor circuitry 711 may be coupled to various sensors, including, but not limited to, one or more light-emitting diodes (LEDs) or other light emitters, one or more photodiodes or other light sensors, one or more photothermal sensors, magnetometers, accelerometers, gyroscopes, barometers, compasses, proximity sensors, cameras, ambient light sensors, thermometers, GPS sensors, electro-oculography (EOG) sensors, and various system sensors that may sense remaining battery life, power consumption, processor speed, CPU load, and the like. In embodiments involving head-mounted devices, one or more sensors may be employed in connection with functionality related to the user's eyes, such as tracking the user's eye movements or identifying the user based on an image of their eye.

[0076] The peripherals interface 716 can couple input and output peripherals of the system to the processor 718 and the computer-readable medium 701. One or more processors 718 may communicate with one or more computer-readable media 701 via the controller 74. The computer-readable medium 701 can be any device or medium (excluding signals) that can store code and / or data for use by the one or more processors 718. In some embodiments, the medium 701 can be a non-transitory computer-readable storage medium. The medium 701 can include a memory hierarchy including, but not limited to, cache, primary memory, and secondary memory. The memory hierarchy can be implemented using any combination of magnetic and / or optical storage devices, such as RAM (e.g., SRAM, DRAM, DDRAM), FLASH®, disk drives, magnetic tape, CDs (compact discs), and DVDs (digital video discs). The medium 701 can also include a transmission medium for carrying information-bearing signals indicative of computer instructions or data (although excluding signals and excluding carrier waves on which the signals are modulated). For example, the transmission medium may include communication networks, including, but not limited to, the Internet (also known as the World Wide Web), an intranet, a local area network (LAN), a wide local area network (WLAN), a storage area network (SAN), a metropolitan area network (MAN), and the like.

[0077] The one or more processors 718 can execute various software components stored within the medium 701 to perform various functions for the system 700. In some embodiments, the software components may include an operating system 722, a communications module (or set of instructions) 724, an I / O processing module (or set of instructions) 726, a graphics module (or set of instructions) 728, and one or more applications (or sets of instructions) 730. These modules and applications may each correspond to sets of instructions for performing one or more functions described above and methods described herein (e.g., computer-implemented methods and other information processing methods described herein). These modules (i.e., sets of instructions) need not be implemented as separate software programs, procedures, or modules; thus, various subsets of these modules may be combined or otherwise rearranged in various embodiments. In some embodiments, the medium 701 may store a subset of the modules and data structures identified above. Additionally, the medium 701 may store additional modules and data structures not described above.

[0078] Operating system 722 may include various procedures, instruction sets, software components, and / or drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware and software components.

[0079] The communications module 724 can facilitate communications with other devices via one or more external ports 736 or through the RF circuitry 708 and can include various software components for handling data received from the RF circuitry 708 and / or the external port 736.

[0080] Graphics module 728 may include various known software components for rendering, animating, and displaying graphical objects on one or more display surfaces. The display surfaces may include 2D or 3D displays. The display surfaces may be directly or indirectly coupled to one or more components of exemplary system 700. In examples involving a touch-sensitive display (e.g., a touch screen), graphics module 728 may include components for rendering, displaying, and animating objects on the touch-sensitive display. In some examples, graphics module 728 may include components for rendering to a remote display. In some examples, such as those incorporating a camera, graphics module 728 may include components for creating and / or displaying images formed by combining rendered graphical objects with camera data (e.g., captured from a head-mounted camera) or photographic data (e.g., imagery captured by a satellite). In some examples, the graphics module may include components for rendering images to a head-mounted display device. In some examples, the images may include views of elements of virtual content (e.g., objects in a three-dimensional virtual environment) and / or views of the physical world (e.g., camera feed showing the user's physical surroundings). In some examples, the display may present a composite image of the virtual content and a view of the physical world, in some examples, the view of the physical world may be a rendered image, and in some examples, the view of the physical world may be an image from a camera.

[0081] The one or more applications 730 may include any application installed on the system 700, including, but not limited to, a browser, an address book, a contact list, email, instant messaging, word processing, keyboard emulation, widgets, JAVA-enabled applications, encryption, digital rights management, voice recognition, voice duplication, location determination capabilities (such as those provided by a Global Positioning System (GPS)), a music player, etc.

[0082] The I / O subsystem 706 can be coupled to an ocular I / O device 712 and one or more other I / O devices 714 for controlling or implementing various functions. For example, the ocular I / O device 712 can communicate with the processing system 704 via an ocular I / O device controller 732, which may include various components for processing ocular input (e.g., a sensor for eye tracking) or user gesture input (e.g., an optical sensor). The one or more other input controllers 734 can receive and send electrical signals from and to the other I / O device 714. The other I / O device 714 may include physical buttons, dials, slider switches, sticks, keyboards, touchpads, additional display screens, or any combination thereof.

[0083] I / O processing module 726 may include various software components for performing various tasks associated with ocular I / O device 712 and / or other I / O devices 714, including, without limitation, receiving and processing input received from ocular I / O device 712 via ocular I / O device controller 732 or from other I / O devices 714 via I / O controller 734. In some examples, I / O device 714 and / or I / O processing module 726 may perform various tasks associated with gesture input, which may be provided by tactile or non-tactile means. In some examples, gesture input may be provided by, for example, a camera or another sensor for detecting movement of a user's eyes, arms, hands, and / or fingers. In some examples, I / O device 714 and / or I / O processing module 726 may be configured to identify an object on the display with which the user wishes to interact, e.g., a GUI element at which the user is pointing. In some examples, ocular I / O device 712 and / or I / O processing module 726 may be configured (such as with the assistance of optical or EOG sensors) to perform eye-tracking tasks, such as identifying an object or area on a display that a user is looking at. In some examples, a device (such as a hardware “beacon”) may be worn or held by a user to assist touch I / O device 712 and / or I / O processing module 726 with gesture-related tasks, such as identifying the location of the user's hands relative to a 2D or 3D environment. In some examples, ocular I / O device 712 and / or I / O processing module 726 may be configured to identify a user based on sensor input, such as data from a camera sensor associated with the user's eyes.

[0084] In some examples, graphics module 728 can display visual output to a user within a GUI. The visual output may include text, graphics, video, and any combination thereof. Some or all of the visual output may correspond to user interface objects. In some examples, I / O devices 712 and / or 714 and / or controllers 732 and / or 734 (along with any associated modules and / or sets of instructions in medium 701) can detect and track gestures and / or eye movements and translate the detected gestures and / or eye movements into interactions with graphical objects, such as one or more user interface objects. In examples in which eye I / O device 712 and / or eye I / O device controller 732 are configured to track the user's eye movements, the user can interact directly with the graphical objects by looking at them.

[0085] Feedback may be provided based on the content being displayed and / or the state or states of the computing system, such as by the ocular I / O device 712 or another I / O device 714. Feedback may be transmitted optically (e.g., light signals or displayed images), mechanically (e.g., haptic feedback, touch feedback, force feedback, or the like), electrically (e.g., electrical stimulation), olfactory, acoustically (e.g., beeps or the like), or the like, or any combination thereof, and in a variable or non-variable manner.

[0086] System 700 may also include a power system 744 for powering the various hardware components and may include a power management system, one or more power sources, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and any other components typically associated with the generation, management, and distribution of power in a portable device.

[0087] In some embodiments, peripherals interface 716, one or more processors 718, and memory controller 720 may be implemented on a single chip, such as processing system 704. In some other embodiments, they may be implemented on separate chips.

[0088] Although the disclosed embodiments have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will be apparent to those skilled in the art. For example, elements of one or more implementations may be combined, deleted, modified, or supplemented to form further implementations. Such changes and modifications are to be understood as being included within the scope of the disclosed embodiments as defined by the appended claims.

Claims

[Claim 1] Application sharing, etc.

Images