Application sharing

Abstract

A host device having a first processor executes an application via the first processor. A host device determines a state of an application. A scene graph corresponding to the application state is generated and presented to a remote device having a display and a second processor. The remote device is configured to, in response to receiving the scene graph, render a view corresponding to the scene graph to the display without executing the application via the second processor.

Description

[0001] This application is a divisional application of Chinese patent application No. 201980055804.2, the original application was filed on July 22, 2019, the priority date was July 24, 2018, the international application number was PCT / US2019 / 0412881, the date of entry into the Chinese national phase was February 25, 2021, and the invention title was "Application Sharing".

[0002] Cross-reference to related applications

[0003] This application claims priority to U.S. Provisional Application Serial No. 62 / 741,761, filed October 5, 2018; U.S. Provisional Application Serial No. 62 / 702,844, filed July 24, 2018; U.S. Provisional Application Serial No. 62 / 726,863, filed September 4, 2018; and U.S. Provisional Application Serial No. 62 / 742,259, filed October 5, 2018, the entire contents of which are incorporated herein by reference. Technical Field

[0004] This disclosure generally relates to systems and methods for presenting data of computer applications, and particularly to systems and methods for presenting data of computer applications executed on a local device to a remote device. Background Technology

[0005] With the rise of smaller, lighter, and more portable personal devices, and the proliferation of content-on-demand services, user expectations for software applications have evolved. For example, software is often expected to be as readily accessible as streaming video content; to have lower resource consumption; and to be easily shared. Traditional software (which may require purchase, download, and installation before use) may not meet the requirements of some users. Furthermore, for the increasing number of mobile computing devices (including wearables), traditional software may be too resource-intensive, as these devices must contend with limitations in physical size, shape, and weight (which restrict their storage, processing power, and battery capacity), and may need to be compatible with cellular data schemes with limited bandwidth.

[0006] One response to the above is to enable "sharing" functionality in software applications, allowing users to remotely access (e.g., observe or interact with) the application running on the host device. Since application sharing eliminates the need for remote users to install or run the application on their computing devices, it lowers their barrier to entry, thereby encouraging the use and increase of "shared" applications.

[0007] Application sharing is not without its potential problems. Some examples of application sharing involve streaming pre-rendered video data that represents the application's visual output; however, because such video data can be bandwidth-intensive, the practical use of these applications may be limited to high-bandwidth environments, thus excluding their use on many mobile data scenarios. There is a desire for a host computing device to share software applications with one or more remote computing devices, allowing users on all such devices to simultaneously view and / or interact with the software application without requiring the remote device to have the application installed or executed locally. Furthermore, there is a desire to minimize the amount of data that must be transferred between the host and remote devices for use in low-bandwidth environments. Summary of the Invention

[0008] Systems and methods for software applications using shared computing devices are disclosed. According to some examples, a host device having a first processor executes the application via the first processor. The host device determines the state of the application. A scene graph corresponding to the application state is generated and presented to a remote device having a display and a second processor. The remote device is configured to, in response to receiving the scene graph, render a view corresponding to the scene graph to the display without executing the application via the second processor. Attached Figure Description

[0009] Figure 1A-1E An example computer system that can participate in application sharing is shown according to an example of this disclosure.

[0010] Figure 2A An example data stream is shown in an example computer system according to an example of this disclosure.

[0011] Figure 2B An example data stream is shown in an example computer system comprising multiple independent applications according to the present disclosure.

[0012] Figure 3A The components of an example computer system according to the present disclosure are shown, which can use a centralized scene graph to render 3D data from multiple independent applications to a display.

[0013] Figure 3B An aspect of an example client application for an example computer system comprising multiple independent client applications is shown according to examples of this disclosure.

[0014] Figure 3C An example renderer for an example computer system comprising multiple independent applications is shown according to examples of this disclosure.

[0015] Figures 4A-4B An example diagram of a system for application sharing, according to an example of this disclosure, is shown.

[0016] Figure 5 An example diagram of a system for application sharing, according to an example of this disclosure, is shown.

[0017] Figure 6 An example flowchart of an application sharing based on the examples of this disclosure is shown.

[0018] Figure 7 Examples of system architectures that can be embodied in portable or non-portable devices according to this disclosure are shown. Detailed Implementation

[0019] In the following illustrative description, reference is made to the accompanying drawings, which form part of the description, and specific examples that can be practiced are shown illustratively in the drawings. It should be understood that other examples may be used and structural changes may be made without departing from the scope of the disclosed examples.

[0020] Figures 1A to 1E Various example computer systems with displays are shown. Figure 1A An example desktop computer connected to an external monitor is shown. Figure 1B An example laptop computer including a display is shown. Figure 1C An example mobile device including an integrated display is shown. Figure 1D An example television including a display is shown. Figure 1E An example computer system including a head-mounted display is shown. This disclosure is not limited to any particular type of computer system, any particular type of display, or any particular component connecting the computer system to the display. This disclosure is further not limited to two-dimensional displays; in particular, three-dimensional displays such as stereoscopic displays are conceivable.

[0021] This disclosure envisions the use of centralized rendering technology for application sharing. This technology is described, for example, in U.S. Patent Application Serial Nos. 15 / 940,892 and 16 / 011,413, the entire contents of which are incorporated herein by reference. By using centralized rendering technology, as described below and in the aforementioned applications, local and remote users of the shared application can independently render graphics data that accurately presents a compelling view of the shared application. By rendering graphics data independently, rather than receiving pre-rendered graphics data from a host device, for example, the bandwidth requirements of the application sharing systems and methods described herein can be reduced.

[0022] Centralized rendering

[0023] In some example computer systems, the data to be graphically rendered on a display (as a “rendered scene”) includes data representing objects in three-dimensional space (“3D data”), such as 2D or 3D geometric primitives including polygons, and rendering 3D data on a display includes rendering images corresponding to objects in three-dimensional space as viewed from the origin of the view along the view axis (“display scene”). For example, in software applications running on a computer system (such as video games using a 3D engine), 3D data may include spatial coordinates, orientation, and / or visual characteristics of objects in a three-dimensional game world, as well as data describing the view origin and view axis in the game world. 3D data may also include data related to textures associated with objects to be rendered, shader parameters related to objects, and other information that affects how objects can be displayed. For example, during the “rendering” or “painting” phase, a game may instruct software and / or hardware “pipelines” to create a rendered scene to be presented on a display as a display scene. This presentation may include a view of the content within the scene. Such content can include digital content (e.g., digital content overlaid on real-world content viewed through a transmission display) alone or in combination with "real-world" content. A view including virtual content can represent what a user observing that content expects to see. For example, a first view presented when the user is in a first orientation can depict what will be seen from that first orientation. If the user rotates to a second orientation, the second view can present the same content as what will be seen from the second orientation (i.e., from a perspective reflecting the user's rotation about the first orientation). Typically, it is desirable that the resulting image reflects the user's expectations of the visual world. In particular, it is often desirable that a first opaque object closer to the view's origin occludes a second object behind it. Incorrectly occluded objects can confuse the user and may not clearly represent the object's position in three-dimensional space. In some example computer systems, occlusion is achieved through sorting, where objects closer to the view's origin are sorted or drawn on top of objects farther from the view's origin.

[0024] Arranging multiple objects for display on a monitor, such that one object actually occludes another, requires information about the relationships between the objects, such as spatial relationships between objects in three-dimensional space. Some example computer systems utilize scene graphs to represent relationships (e.g., hierarchical relationships) between one or more objects, such as objects to be rendered as a scene. As used herein, a scene graph is any data structure that represents such relationships. For example, in a scene graph, the objects to be rendered can be represented as nodes in the graph, where the relationships between nodes represent logical or spatial relationships between objects. The renderer can then traverse the scene graph according to techniques known in the art to render or prepare at least one of the objects for display in a manner that achieves appropriate occlusion. For example, the renderer can create a scene of objects with nodes; however, the corresponding presentation on the monitor can only be a subset of the rendered objects, such that an object occluded by another object in the renderer will only be partially presented in the resulting display scene (e.g., displaying the unoccluded portion of the object while not displaying the occluded portion). This selective display can be beneficial: for example, if it is only necessary to be able to view a second object running from a second application within a given time period, a first object running from a first application can be effectively occluded. In some examples, the scene graph is an intermediate data structure between the application that includes 3D data and the renderer that renders that 3D data to be presented on the screen: in some examples, the application writes scene information into the scene graph, and the scene graph can later be used by the renderer to render the scene or output the scene for display.

[0025] Figure 2AAn example data flow in an example computer system 200 is shown. In system 200, a single application 210 can write data to a scene graph 240, which a renderer 250 can use to render an object 220 for presentation on a display 290. For example, object 220 may include multiple objects (e.g., polygons) that together comprise a 3D representation of two human hands (hands 232 and 236); and the application may, for example, guide the object 220 to be presented on the display from an angle along the view origin oriented along the view axis during the rendering or drawing phase. In this example, hands 232 and 236 are interlocked in a handshake; due to the relative positioning of the hands, the viewer expects portions of hand 232 that occlude the portion of hand 236 with respect to the view origin and view axis, as well as some polygons of hand 236 that include the portion of hand 232 that occludes hand 236. Application 210 can write information describing the relationships between objects 220 (such as the spatial relationships between polygons that include object 220) into scene graph 240. This information can be used to identify which polygons should occlude other polygons, that is, which polygons should be ordered to appear on top of other polygons. For example, scene graph 240 may reflect that polygon 234 (which belongs to hand 232) is located between the view origin and the polygons that include hand 236, and therefore should occlude those polygons in hand 236; and polygon 238 (which belongs to hand 236) is located between the view origin and the polygons that include hand 232, and therefore should occlude those polygons in hand 232. Renderer 250 can then output object 220 or a subset of object 220 (e.g., only hand 232; only the unoccluded portion of hand 232; or only hand 236) for rendering via display 290 consistent with the desired occlusion.

[0026] Figure 2B This illustrates a sample data flow in a sample computer system 201 using two independent applications. When... Figure 2A When the example is expanded to render a scene modified by multiple independent applications, such as Figure 2B As shown, rendering problems may occur. For example, in example computer system 201, applications 212 and 214 both write data to scene graph 240 to render their respective 3D data to a single display 290. Figure 2B In the process, application 212 attempts to render and present object 222 (which includes an object containing hand 232); and application 214 refuses to render and present object 224 (which includes an object containing hand 236). Figure 2BThe examples shown may present difficulties in achieving actual occlusion of the objects to be rendered: if applications 212 and 214 are independent (“sandboxed”) applications, application 212 cannot access data relating to object 224 (including hand 236 and its constituent objects) of application 214, and similarly, application 214 cannot access data relating to object 222 (including hand 232 and its constituent objects) of application 212. That is, in some examples, neither application 212 nor application 214 can fully recognize the relationship between objects 222 and 224. Therefore, neither application 212 nor application 214 can write information to scene graph 240 that might be necessary to identify which objects occlude other objects or in what order objects should be ordered on the display. Furthermore, various subsystems (such as lighting, shadows, animation, particles, and collision detection) may not perform as expected. The result may be unrealistic, clunky, and confusing rendered graphics data. Moreover, such rendering systems may be computationally inefficient because they may not be able to take advantage of rendering optimizations (such as culling).

[0027] Centralized rendering, as described in U.S. Patent Application Serial Nos. 15 / 940,892 and 16 / 011,413, provides a solution to this problem that may occur in systems that render 3D data from multiple independent applications. Systems that provide 3D data to be rendered from multiple independent applications (such as...) Figure 2B In the example computer system 201, a centralized scene graph can be used instead of a traditional scene graph, such as Figure 2B Scene graph 240. As described herein, in some examples, a centralized scene graph may include a system that receives 3D data from multiple separate input sources; writes information corresponding to the 3D data to a central location; and maintains this information for access by a renderer that creates a rendering scene including objects based on the 3D data. This rendering scene can be used to generate outputs that reflect real-world object occlusion (such as graphics output); computational efficiency; visual effects (such as lighting and shadow casting); or physical effects (such as collision detection); or otherwise, partial display of occluded objects that would be difficult or impossible to achieve in a system that does not utilize a centralized scene graph.

[0028] In some examples, the sample computer system includes multiple applications, each containing 3D data representing one or more objects in a shared 3D environment. Each of these applications can exist in a “sandbox” environment, making it agnostic to other applications: for example, the data of each corresponding application may be independent of the data of each other application; each application may not be able to access the data of each other application; and although the 3D data of each application may correspond to the same 3D environment, each application maintains its own instance of the 3D environment. For example, each application may represent a player in an online multiplayer video game, where each player exists in an instance of the same game world or 3D environment but cannot directly access the data of other players. In such examples, it may be necessary to render all players simultaneously in a single instance of the game world, but it may not be desirable (or computationally prohibited) to maintain the information required to render the 3D data of each other client participant for each player. Furthermore, for security purposes, it may be desirable to restrict the player information available to other players.

[0029] In some examples, each sandbox application among multiple sandbox applications can independently write information corresponding to its 3D data to a local scene graph, which is then written to a shared, centralized scene graph. The renderer can then traverse the centralized scene graph to render the scene as an image on the display based on the set of 3D data provided by each application. By passing 3D data from each sandbox application across multiple applications to a single centralized scene graph, the renderer can apply beneficial techniques, such as occlusion, lighting effects, and rendering optimizations (such as surface culling), which require or benefit from knowing the 3D data of all applications simultaneously. These benefits are achieved while limiting the computational overhead required by each sandbox application: from the perspective of a single application, all that all applications need to do is update a single scene graph to reflect their 3D data, while other operations are performed by another component of the system. Furthermore, security benefits can be gained by maintaining the separation between sandbox applications.

[0030] Figure 3AComponents of an example computer system 300 are shown, which can render 3D data from multiple independent applications to a display using a centralized scene graph. The illustrated example utilizes a client-server topology; however, this disclosure is not limited to the client-server example. In the example computer system 300, a first client application 310 and a second client application 320 each (in some examples, via a network) transmit 3D data to a client-server interface 330. In some examples, client applications 310 and 320 are “sandboxed” applications that operate independently of each other and transmit their 3D data independently to the client-server interface 330. The client-server interface 330 can receive updated 3D data from client applications 310 and 320 and (in some examples, via a network) transmit that 3D data to a server-side host application 340. In some examples, the client-server interface 330 uses multithreading techniques to use multiple processor threads to receive, process, and / or transmit 3D data to the host application 340. In some examples, the client-server interface includes logic for controlling (e.g., by throttling) the rate at which 3D data is transmitted to the host application 340. The host application 340 can use the 3D data received from the client-server interface to update a centralized scene graph 350, such that the centralized scene graph 350 reflects the 3D data received from client applications 310 and 320. In some examples, the centralized scene graph 350 includes multiple versions of the scene graph, and known version control techniques are used to allow updates to the centralized scene graph 350 to occur in parallel. The renderer 360 can then traverse the centralized scene graph 350, apply optimizations and effects appropriately, and generate output to be displayed on a monitor 370, such as a computer monitor (e.g., the graphics output includes data from at least one of the client applications 310 and 320, and in some embodiments, only occluded portions of one client application are included without occluding application data).

[0031] Figure 3B Showing about Figure 3AThe example computer system 300 shown represents an aspect of the example client application 310. In the example shown, 3D data 312 represents graphical objects (such as geometric primitives, e.g., polygons) in a 3D environment to be rendered on display 370. 3D data 312 can be updated by the client application 310 (314). For example, if the client application 310 is an application with a rendering loop that iterates sixty times per second, the client application 310 can update the 3D data 312 sixty times per second to reflect changes in that data that should be reflected in the rendering output during the application's operation. In some examples, the 3D data 312 is represented as a local scene graph 316, which can be local to each client application 310. In some examples, the local scene graph 316 may include data (such as nodes) corresponding to data in a centralized scene graph 350. As the 3D data 312 is updated (314), the client application 310 can update the local scene graph 316 to reflect the latest version of the 3D data 312. As the local scene graph 316 is updated, the client application 310 can use it to generate (317) client data 318. In some examples, client data 318 may completely represent the local scene graph 316. In some examples, client data 318 may represent changes made to the local scene graph 316 since the previous client data 318 was sent to the client-server interface 330. For example, client data 318 may include nodes added to or removed from the local scene graph 316; changes in relationships between nodes in the local scene graph 316; or changes in the characteristics of nodes in the local scene graph 316. In some examples, client data 318 may use identifiers (such as identifiers corresponding to scene graph nodes) to identify the relationship between data from the local scene graph 316 and corresponding data on the centralized scene graph 350. Client data 318 may then be transmitted to the client-server interface 330 for final delivery to the host application 340. In some examples, the transmission of client data 318 to the client-server interface 330 may occur over a network. In some examples, the client assistant application can be used in conjunction with the client application 310 to generate client data 318 from the local scene graph 316 or 3D data 312.

[0032] The aspects described with respect to client application 310 can be similarly described for client application 320, or other client applications (together with client application 310) including the example computer system 300. Those skilled in the art will recognize that the systems and methods described herein can be extended to include any number of client applications and client data, and this disclosure is not limited to any such number; furthermore, as the number of client applications increases, some benefits (e.g., improvements in computational efficiency) may become more apparent. As mentioned above, client applications 310 and 320 can be sandboxed applications that do not share data or functionality. For example, in the example computer system 300, client application 320 may have its own 3D data and local scene graph belonging to client application 310 and independent of 3D data 312 and local scene graph 316. However, in some examples including the example computer system 300, a single client-server interface 300 is shared by multiple client applications (such as client applications 310 and 320).

[0033] Although the above examples describe 3D data, this disclosure is not limited to three-dimensional data or data in a 3D environment. The above examples can be generalized to graphical data of any number of dimensions, including 2D graphical data.

[0034] Figure 3C Showing about Figure 3A and Figure 3B The example computer system 300 shown represents an aspect of the example renderer 360. In some examples, renderer 360 is part of host application 340. In some examples, renderer 360 may be part of another component of the example computer system 300, or it may be a separate component or application. In some examples, renderer 360 may be implemented in physical hardware different from one or more components of the example computer system 300, and may communicate with one or more of those components over a network.

[0035] exist Figure 3CIn the example shown, renderer 360 operates on version 352 of centralized scene graph 350. In this example, the renderer's role is to create a rendered scene based on version 352 of centralized scene graph 350, including data such as output or graphical output for presentation on display 370. As part of this process, renderer 360 may traverse (362) version 352 using known scene graph traversal techniques. During or after traversal 362, renderer 360 may appropriately update (364) centralized scene graph 350 to reflect the results of the traversal. For example, as part of traversal 362, renderer 360 may identify orphaned nodes that should be removed from centralized scene graph 350. After traversal 362 and / or update 364, renderer 360 may apply various optimizations 366 to the scene. For example, renderer 360 may cull occluded or invisible surfaces to avoid consuming unnecessary computational resources. After traversing 362 and / or updating 364, renderer 360 can apply one or more visual effects 367 to the scene. For example, in some examples, renderer 360 can apply lighting or shadow effects; apply one or more shaders; apply particle effects; and / or apply physical effects. Finally, renderer 360 can output data to the graphics output pipeline, and the results can be displayed on monitor 370.

[0036] In some cases, optimizations can be applied to the systems and methods described above to reduce the size of scene graph data, such as the amount of scene graph data to be sent over a computer network (e.g., via client-server interface 330). For example, if the scene graph includes nodes (or reference assets) that may not be visible to the rendered graphics output, these nodes can be excluded from the scene graph data to be sent. Similarly, assets can be unloaded when a node referencing an asset is removed from the scene graph. Furthermore, to improve performance, nodes should only be inserted when rendering is required, avoiding the overhead of including nodes in the scene graph data when nodes may not have a significant impact on the rendered scene (e.g., the user may not perceive the node's absence).

[0037] In some cases, different levels of detail (LOD) can be applied to assets to reduce the overall data size. Different LODs can help ensure that assets are not more data-intensive than guaranteed in a particular viewing context. For example, when the expected display size cannot guarantee the data size of a higher-resolution image, a lower-resolution image can be used instead. This can improve system performance, for example, by reducing the amount of network bandwidth required to transmit assets that are larger than necessary (e.g., higher-resolution images).

[0038] Application sharing

[0039] Examples of the systems and methods disclosed herein can be used to allow multiple clients to share (e.g., view and interact with) the content of a common software application running on a single host client. It may be desirable to avoid the need to install the software application on each remote client communicating with the host client; furthermore, it may be desirable for multiple users across the application to synchronize input and control information. This can be advantageous in examples where clients have limited computing resources (e.g., disk storage) or where users may lack the time, complexity, or willingness to install the software application. Similarly, software providers may wish to encourage user adoption of the software application by reducing the number of steps required to interact with it (e.g., by allowing users to share another user version of the application). Furthermore, the systems and methods disclosed herein can minimize network bandwidth usage by rendering content locally at each remote device (e.g., by utilizing the centralized rendering systems and methods described above, rather than by transmitting video output from the host device to the remote device). In some examples, the computing device used for hosting, sharing, viewing, or interacting with the software application can correspond to a head-mounted display device, such as those described above. Figure 1E The example device shown; shared content can be displayed on the head-mounted device's display; and users can interact with the shared software application via the head-mounted device's sensors or input devices.

[0040] Figure 4A Example system 400 is shown for implementing the aforementioned application sharing. In this figure, four users (Praveen, Lisa, Leonid, and Naruto) on four corresponding computing devices (410, 420, 430, and 440) wish to share three separate software applications (chess, calculator, and spelling bee), where each application runs only on one of the client computing devices. System 400 includes server 450, which in some examples may correspond to a cloud server. In example system 400, each of computing devices 410, 420, 430, and 440 communicates with server 450, but does not necessarily communicate directly with the other computing devices.

[0041] In example system 400, each of the three software applications (chess, calculator, and word game) executes on one of client computing devices 410, 420, 430, or 440, but can be viewed and / or interacted with by each of the client computing devices not running on it. For example, as shown, the chess application is executing on device 410; and each of devices 420, 430, and 440 can view the chess application by rendering a chess scene graph local to the respective device (as described above, a centralized rendering system and method can preferably be used to render the scene graph on the display). Similarly, in this example, the calculator application is shown running concurrently with the chess application on device 410, and each of devices 420, 430, and 440 can render a view of the calculator application via a calculator scene graph local to the respective device. Additionally, in this example, a word game application is shown running on device 420, and each of devices 410, 430, and 440 is able to render a view of the word game application via a local word game scene graph. Thus, each of the four devices 410, 420, 430, and 440 can view each of the three applications: either by running the application locally (e.g., running the chess and calculator applications on device 410), or by rendering a view from a local scene graph (e.g., storing a local scene graph for each of the chess, calculator, and word game applications on device 430). Server 450 operates to exchange data between devices 410, 420, 430, and 440, as described below. In some examples, such as described below, server 450 may maintain its own local copy of the scene graph corresponding to each application (e.g., chess, calculator, word game); this can provide a base scene graph for devices joining a pre-existing application sharing session.

[0042] In example system 400, the host system executes an application (e.g., device 410 executes a chess application) and shares it with one or more remote devices (e.g., devices 420, 430, and 440). It can be advantageous for remote devices to view and interact with the shared application without installing the application on the device itself. For example, in the example shown, remote devices 420 and 440 do not have the chess application installed; when a sharing session is initiated, device 410, which hosts the chess application, can provide remote devices 420 and 440 with necessary data, such as required assets. However, in some examples, the overhead of providing such data can be reduced or avoided by installing it on the remote device. For example, in the example shown, remote device 430 has the chess application installed on its device before joining the sharing session, which eliminates the need for device 410 to provide remote device 430 with assets and other data for the chess application. However, in this example, the chess application is not executed on remote device 430 (i.e., not by the processor of remote device 430), but on device 410. For example, during initialization, remote device 430 relies on its own local installation of the chess application for the required application data.

[0043] Figure 5 This illustrates an example where a host device (e.g., device 410) executes application 510 (e.g., chess) and shares the application with a remote device (e.g., device 420) so that a user on the remote device can view and interact with an application such as that described above with respect to example system 400. As mentioned above, the remote device does not need to install the application: for example, at the start of the application sharing session, the host device can provide the remote device with the necessary initialization data, such as required assets (e.g., textures, model data, audio files, particle effects, animation data), without requiring the remote device to perform a separate installation process. As described below, when sharing an application, the host device can provide the remote device with application state data and / or one or more scene graphs. In some examples, the remote device can provide the host device with input data (e.g., presenting button input, gesture input, touch input); user data (e.g., user identity details, activity history, and / or social media presence information); and / or sensor data (e.g., GPS coordinates, camera data, microphone data). Figure 5 In this context, application 510 refers to the application running on host device 410. In some examples, multiple applications can run simultaneously on device 410, such as... Figure 4AThe chess and calculator applications running on device 410 are illustrated; although the example described depicts a single shared application, it can be extended to multiple shared applications. In some such examples, the host device may maintain and share two or more independent scene graphs (e.g., three scene graphs, each corresponding to one of the three shared applications). As mentioned above, in some examples, a single scene graph may correspond to two or more applications (e.g., a single scene graph corresponding to chess and Scrabble). Furthermore, as stated herein, although remote device 420 is remote with respect to host device 410, it does not necessarily have to be physically or electronically separated from host device 410. In some examples, remote device 420 may be physically or electronically connected to host device 410, or it may be a component or module of a single physical device. In this way, application sharing can be implemented at the platform level. For example, applications can be shared between components or regions of a single platform (such as a mobile device operating system (e.g., iOS, Android) or a head-mounted device operating system). In some examples, applications can be shared across platforms in this way; for example, this can enable the following behavior: sharing applications between users on a first platform (e.g., mobile devices running iOS) and users on a second different platform (e.g., head-mounted virtual reality systems running a custom operating system).

[0044] Coexistence service 540 is an assistant application that enables the main application to share with remote devices. In this example, coexistence service 540 executes concurrently with application 510 on host device 410 and sandboxes application 510, enabling application 510 to share with remote device 420. In some examples, as described below, application 510 shares with each device (e.g., Figure 4A Devices 420, 430, and 440 can execute corresponding coexistence service applications. For example, in Figure 5 In the example shown, remote device 420 executes its own remote coexistence service 570, which may include some or all of the features of host coexistence service 540. For example, remote coexistence service 570 may include main logic loop 574 and remote rendering engine client 576.

[0045] In the example shown, host coexistence service 540 incorporates transmission functions for sending data (e.g., graphical data, input data) to and receiving data from remote devices (e.g., 420) via cloud server 450. For example, in this example, local endpoint 542 can act as a terminal for exchanging data with application 510, and endpoint 550 can act as a terminal for exchanging data with cloud server 450. In this example, host coexistence service 540 further includes a main application logic loop 544 that can perform various functions associated with host coexistence service 540. These functions may include mediating data exchange between host device 410 and remote devices; adding and removing remote devices from shared application 510; user interface functions (which may include "avatars" representing remote users); chat functions; file transfer (e.g., exchanging data between host device 410 and remote devices such as remote device 420, including assets associated with application 510); and / or generating graphical output. The graphics output of the coexistence service 540 (which may include a scene graph) can be presented to the rendering engine client 546, which in some examples may correspond to the above regarding 3A and Figure 3B The client-server interface 330 is described above. The rendering engine client 546 can then communicate with the rendering engine server 560 (which in some examples may correspond to the above regarding...). Figure 3A and Figure 3B The described server 340) presents graphical data for rendering on the host device 410 (e.g., as described above regarding...). Figure 3C (As shown in process 360). However, in some examples, the coexistence service 540 may include only a limited set of features and may be a background process that is performed without any user input or displayed output.

[0046] refer to Figure 5 As illustrated in the example, application 510 may include input receiver 512 that can receive and process host input 506 from device 410 (e.g., via an input service associated with device 410). Host input 506 may include input received via conventional input devices such as a keyboard or mouse; via sensors of the head-mounted device (e.g., a camera, eye-tracking sensor, microphone); via handheld input peripherals; or via any other suitable device. In some examples, input 506 may be provided by one or more sensors or input devices associated with the head-mounted augmented reality device. In some examples, as described below, instead of host input from host device 410 or in addition to host input, input receiver 512 may receive input provided by a remote device.

[0047] Application 510 may further include a main application logic level 514. Inputs received by 512 may be provided as inputs to application logic level 514. Application logic level 514 may include a main application loop that accepts inputs, executes application logic (e.g., game logic for a chess application), maintains some application state, and provides some outputs (e.g., display output). Application logic level 514 may include determining a scene graph for the host device 410 and for each remote device (e.g., remote device 420) sharing the application; in some examples, one or more of the local and remote devices may share the same scene graph. The above is about... Figure 3B The client application 310 shown illustrates example graphics processing performed by application logic level 514. However, this disclosure is not limited to any particular application or any particular application logic 514.

[0048] Application 510 may further include a rendering engine client 516, which in some examples may correspond to the above regarding Figure 3A and Figure 3B The client-server interface 330 is described above. In some examples, the rendering engine client 516 may be the same as the rendering engine client 546 described above, or may share common components with the rendering engine client 546 described above. From the application logic level 514, the display data of the application logic can be provided to the rendering engine client 516, for example, as described above regarding... Figure 3A and Figure 3B The rendering engine client 516 may be a client utility native to the host device 410, which generates one or more scene graphs based on the output of the application logic level 514, each scene graph corresponding to the application 510. In some examples, the application 510 may generate scene graphs for each device (e.g., 420) with which it will share the scene graph. Such scene graphs may be identical or unique for one or more devices. In some examples, the scene graph may be optimized on a per-remote-device basis, such as by including only the data required for each remote device. For example, the remote device 420 may provide the host device 410 with data indicating the view and camera origin of the user of the remote device 420; the host device 410 may accordingly calculate which elements of the scene graph may be visible to the user of the remote device 420 (i.e., which elements may be visible in the view corresponding to the camera origin and view); and remove invisible elements from the scene graph for the remote device 420. Similar optimizations may also be performed based on information about which elements in the scene graph are visible on the remote devices (e.g., detail setting levels or hierarchical information on the remote devices).

[0049] To present the view of application 510 (e.g., representing the user's application view) to a local user on host device 410, rendering engine client 516 can generate a scene graph corresponding to the local user's view, and then present the scene graph and associated content to rendering engine server 560, which can then display the corresponding view of application 510 on the display of device 410. This process corresponds to the above regarding... Figure 3A and Figure 3B The process described above. Rendering engine client 516 may, but does not necessarily, directly present scene graphs and associated content to rendering engine server 560. In some examples, rendering engine client 516 may present scene graphs and content to endpoints, sockets, pipes, or any other suitable inter-process communication mechanism to render engine server 560.

[0050] In some examples, the rendering engine server 560 may be local to the host device 410; in others, it may be on a remote device (such as a dedicated server) or on another device connected to device 410, such as in a peer-to-peer network. Neither the rendering engine client 516 nor the rendering engine server 560 needs to be a dedicated hardware module; in some examples, the rendering engine client 516 and / or the rendering engine server 560 may be purely software modules, such as a standalone application (e.g., a standalone application running concurrently on a single platform) or a separate thread of a multi-threaded application. However, in some examples, one or more rendering engine functions (e.g., functions generated by a dedicated server and / or dedicated graphics processing hardware (e.g., a GPU) may be executed on dedicated hardware. Figure 3C (Those executed by renderer 360 shown). By performing computationally intensive functions on dedicated hardware, computing resources can be freed up on user devices (e.g., device 410). This can be particularly valuable when user devices include mobile hardware where battery life, computing resources, and physical space are very precious (e.g., mobile phones, headsets).

[0051] Rendering engine server 560 may include a centralized scene graph and renderer as described above. In some examples, multiple client applications (e.g., a chess application and a calculator application running simultaneously on device 410) may simultaneously provide content and / or scene graph data to rendering engine server 560. Furthermore, in some examples, rendering engine server 560 includes client application code and may be agnostic to the specific source of this data presented to it by rendering engine client 516. This allows it to be used with a variety of applications (such as the chess and calculator applications mentioned above) without requiring the rendering engine server to include custom logic for each such application. Similarly, this allows applications such as chess and calculator applications to cooperate with coexistence services to enable application sharing without any custom programming, encouraging the production, adoption, and use of such applications.

[0052] exist Figure 5 In the example shown, the rendering engine client 516 can provide graphics data to the host coexistence service 540. In some examples, the graphics data can be provided to an endpoint (e.g., endpoint 520), or to a socket or another suitable inter-process communication mechanism. The graphics data may include remote graphics data intended to be rendered to a remote device 420. For example, remote graphics data may include a scene graph or a portion of a scene graph corresponding to application 510 that will be rendered and displayed on the remote device 420. For such remote graphics data, endpoint 520 can provide the remote graphics data to another endpoint 550 corresponding to the remote device; from endpoint 550, the data can be transmitted to the remote coexistence service 570 executing on the remote device 420 (e.g., via cloud server 450 and / or a remote endpoint 580 corresponding to the remote coexistence service).

[0053] In some examples, once remote graphics data (e.g., including a scene graph) is received at the remote coexistence service 570, it can be provided to the main logic loop 574 of the remote coexistence service, which can use the remote graphics data to determine the view to be rendered on the remote device 420. For example, as described above, the remote coexistence service 570 can generate and update a local scene graph. In some examples, the remote graphics data can be provided directly to the rendering engine client 576. The rendering engine client 576 can then provide the data to the rendering engine server 590, where it can be rendered to a display, such as the display of device 420.

[0054] In some examples, remote device 420 cannot interact with application 510, but instead acts as a spectator simply rendering a view of the application. However, in other examples, remote device 420 can interact with application 510, for instance, via input provided to remote device 420. In some cases, this input can affect the remote device's view of application 510 (e.g., by adjusting display parameters) without requiring communication back to host device 410. However, in some examples, remote device 420 can provide input back to host device 410, which can then be passed as input to application 510 executing on host device 410. For example, if application 510 is a chess application, remote device 420 can provide user input to host device 410 instructing the user on desired moves. In the illustrated example, remote coexistence service 570 includes a remote input receiver 572 that receives input from remote device 420 (e.g., via input service 566). Remote input receiver 572 and input service 566 may correspond to input receiver 512 and input service 506 described above with respect to host device 410, respectively. For example, the remote input receiver 572 can receive input via conventional input devices (e.g., keyboard or mouse); via sensors of a head-mounted device; via handheld input peripherals; or via any other suitable device. This input data can be provided as input to the main loop 574 of the remote coexistence service 570.

[0055] In the example where input data affects the application view on a remote device without affecting the application's own execution, the main loop 574 can determine how the input data should affect the application's view and present the corresponding graphics data to the rendering engine client 576 (which subsequently provides the graphics data to the rendering engine server 590). In the example where the input data is sent back to the host device 410, the main loop 574 can present the data to endpoint 580, which can then send the data back to endpoint 550 of the host coexistence service 540. From here, the data can be sent back to the application 510 (e.g., via endpoints 542 and 520 and input receiver 512), where it can be presented as remote input to the main application logic loop 514. The application logic loop 514 can then incorporate the remote input into the updated scene graph of the application 510; for example, in a chess application, the application logic loop 514 can update the game state based on user input received from the remote device to move pieces accordingly; and generate a new scene graph reflecting the updated positions of the pieces. The updated scene graph can then be presented to the remote device 420, as described above. In this way, the remote device 420 can provide input to the application 510 and receive an updated scene diagram that takes into account the input from the remote device.

[0056] In addition to the aforementioned remote graphics data, or other graphics data provided by the rendering engine client 516, the graphics data may also include host graphics data intended for rendering and presentation on the host device 410. This host graphics data can be presented to the host coexistence service 540 via endpoint 520, for example, as input to the main loop 544. The graphics data can then be presented to the rendering engine client 546 of the host coexistence service 540; from there, it can be presented to the host rendering engine server 560. The rendering engine server 560 can render a single view that combines graphics data from both sources, such as those described above: for example, graphics data corresponding to the local application 510, and graphics data corresponding to the host coexistence service 540.

[0057] Although Figure 5 The example shown illustrates a single application 510 executing on a host device and shared with remote devices; however, this example can be extended to multiple applications executing on a host device and shared with remote devices. As mentioned above, the graphical data for such multiple applications can be represented by a single scene graph (e.g., one scene graph corresponding to two applications) or by two or more scene graphs (e.g., two scene graphs, each corresponding to one application). Furthermore, in Figure 5 The remote device 420 described in the example shown can also execute its own additional applications that can be shared with other devices, including device 410. For example, Figure 4A A device 420 is shown executing a word game application shared with device 410. In such examples, remote device 420 can be considered a host device regarding those applications executed by remote device 420. Other configurations of the devices and shared applications will be apparent, and this disclosure is not limited to any particular such configuration. For example, applications can be shared between applications running on a single platform (e.g., iOS), between applications running on different platforms (e.g., iOS and Android), or between combinations thereof.

[0058] Figure 6 A flowchart is shown illustrating the execution of an application (e.g., according to this disclosure) Figure 5 Application 510, which may be a chess application, is a host device (e.g., Figure 4A and Figure 5 Device 410 in the middle) and remote devices sharing that application (e.g., Figure 4A and Figure 5Example data flow between device 420 (as used in this example), where "host" refers to processes and data local to host device 410, which is the device executing application 510 in the shared session, and "remote" refers to processes and data remote from device 410, such as processes and data at device 420. Device 420 may download or install application 510, but does not execute it in the shared session. Figure 6 In the example process described, a user on a remote device can view and interact with the application without installing it on the remote device. At stage 610 of the example, a user on host device 410 initiates the sharing of application 510. Application sharing can be initiated using any suitable technology, such as those further described below, for example, by interacting with a “Share” option presented via a context menu or system dialog box on host device 410 (in some examples, the user on the remote device, rather than the host device, can initiate application sharing). In response, at stage 612, a host coexistence service (e.g., host coexistence service 540) executing on device 410 registers application 610 for sharing and performs any necessary initialization. The host coexistence service (e.g., via server 450) communicates with the remote coexistence service (e.g., remote coexistence service 570 executing on device 420) that is sharing application 510 (in some examples, such as when application 510 has not been downloaded or installed on the remote device, this stage may include sending required assets or other application data to the remote device). In response, at stage 614, the remote coexistence service can create a remote scene graph (or, in this example, a sub-scene graph) local to the remote device 420 corresponding to application 510. As described above, the rendering engine server executing on (or communicating with) the remote device 420 can render scenes such as those described above. Figures 3A-3C The remote scene diagram is provided, and a display output corresponding to the view of the application 510 belonging to the user of device 420 is provided (e.g., on the display of device 420).

[0059] At stage 620, host device 410 can determine a scene graph (e.g., a 3D camera view of application 510 corresponding to the camera position of the user of device 420) that should be presented to application 510 such as device 420 as described above, and send data reflecting the scene graph to host coexistence service 540. At stage 622, upon receiving the data reflecting the scene graph, host coexistence service 540 forwards the data to remote coexistence service 570, for example via server 450 using a remote endpoint, socket, or other suitable mechanism for transmitting data. At stage 624, upon receiving the data reflecting the scene graph, remote coexistence service 570 updates or replaces (e.g., via rendering engine client 576) the remote scene graph created at stage 614 as described above. As described above, the remote scene graph can then be rendered at device 420 via rendering engine server 590. By only transmitting the updated data from the host device to the remote device, bandwidth usage is limited compared to sending raw image data (e.g., video frames) or other storage-intensive data.

[0060] Device 410 can repeat this process at stage 620 for additional devices (such as remote devices 430 and 440, and including host device 410). For each corresponding device, as described above, rendering engine client 516 can determine the scene graph that should be presented to application 510 on that device. For the scene graph to be displayed on host device 410, rendering engine client 516 can generate or update the scene graph and present it to rendering engine server 560 for rendering, as described above. Figures 3A-3C As stated above.

[0061] In an example where the remote device 420 interacts with the application 510 by providing input rather than simply passively displaying a view of the application 510, at stage 630, the remote coexistence service 570 can, for example, accept input from the remote device 420 via a remote input receiver 572 communicating with the input service 566 (its behavior can correspond to that of the host input receiver 512 and input service 506 as described above). At stage 632, the remote coexistence service 570 can forward the input event data to the host coexistence service 540, which receives the input event data at stage 634. At stage 636, the host coexistence service 540 can forward the input event data to the local application 510, where it can be received at stage 638 (e.g., by the input receiver 512) and provided as input to the application logic 514.

[0062] At stage 640, application logic 514 can update the host scene graph based on host inputs received from host device 410 as described above, and in an example where input is provided by a remote device (e.g., remote device 420), application logic 514 can further incorporate that remote input when updating the scene graph. For example, if application 510 is a chess application, and the remote input received by application 510 at stage 630 corresponds to a user of remote device 420 moving a piece, application 510 can update the game state accordingly (which may affect data presented to all devices sharing the application as well as host device 410); and generate a corresponding updated scene graph reflecting the new game state (e.g., pieces in their updated positions on the chessboard).

[0063] At stage 650, host application 510 sends an update to remote device 420 based on the result of stage 640. These updates are provided to host coexistence service 540 at stage 652 and to remote coexistence service 570 at stage 654. Stages 652, 652, and 654 may correspond to stages 620, 622, and 624 described above, respectively, and these steps may be repeated indefinitely as host device 410 hosts application 510 and remote device 420 views and / or interacts with a shared version of its application. In some examples, the bandwidth usage of the above process can be controlled by adjusting the rate at which updates are provided by the host device. For example, increasing the time elapsed between updates (e.g., between stage 620 and stage 650) can reduce the required bandwidth. Example systems and methods for adapting to changes in data throughput and for synchronizing devices that may have different data throughput rates are described in U.S. Patent Application Serial Nos. 15 / 940,892 and 16 / 011,413. Additionally, in some examples, the data size of updates between the host application 510 and the remote device 420 can be optimized by limiting the content of the scene graph sent in the update. For example, as described above, the host application can determine (e.g., at stage 650) elements of the scene graph that may not be visible at the remote device 420 (e.g., because they exist outside the predicted camera view of the remote device 420), and can remove those elements from the scene graph before sending the scene graph to the remote device 420.

[0064] In some examples, cloud servers or other suitable storage devices can maintain a basic scenario graph of the shared application, facilitating the joining of new remote devices into an existing shared session. For example, regarding... Figure 4AThe cloud server 450 can maintain scene graphs for each of the chess, calculator, and word games applications, respectively. In such an example, when an application begins to share with an attached remote device (e.g., remote device 430), the device can initially use the basic scene graph present on the cloud server, allowing new remote devices to join the ongoing application without the overhead of creating a new scene graph corresponding to the application. In such an example, the host device 410 may need to present incremental updates (e.g., changes since a previous update) to the remote device; since the basic state scene graph is always available from the cloud server 450, the remote device only needs to obtain the basic state scene graph and the changes since the corresponding basic state in order to reconstruct a scene graph corresponding to the current state of application 510.

[0065] In the example above, the host device (e.g., device 410) executes the application and shares it with a remote device (e.g., device 420). The host device generates a scene graph and sends it to the remote device, which renders a view from the scene graph for display. In the example above, the host device also generates a scene graph for itself (which may or may not be the same scene graph sent to the remote device), and uses the scene graph to render a view for display on the host device. This might be desirable when the host device and the remote device belong to two users who wish to participate in the same shared application; for example, a user of the host device might want to act as a user of the remote device in a shared chess application. However, in some examples, the host device could simply send a scene graph to each remote device without rendering a view for itself.

[0066] Figure 4B Describe an example of this "headless" feature. Figure 4B In Example 400A shown, the host device (e.g., Figure 4B The 450A in the example can be a dedicated host entity, such as a data center, cloud server, or one or more networked servers, configured to share the primary purpose of an application with remote devices such as those described above (e.g., remote devices 410A, 420A, 430A, and 440A). This configuration may be advantageous, for example, where the host device has access to greater computing resources than the remote device it communicates with. In the example shown, host device 450A executes one or more applications (e.g., chess, calculator, word game). Such as Figure 4B As shown, host device 450A can generate and maintain scene graphs for each of those applications, and send those scene graphs to each remote device as described above. Each remote device can then determine its local view and display that view to its corresponding user.

[0067] The logic executing on the host device can update the state of applications such as those described above regarding remote devices. For example, the host device logic could receive updated locations from remote devices, such as when a user moves from the living room to the kitchen. When a user enters the kitchen, the new location can be provided from the remote device to the host device; then, the host device can provide the remote device with an updated scene map and / or associated assets, which the remote device can subsequently view.

[0068] In some examples, the host device can be configured to host a large number of applications, and in others, remote devices connecting to the host device never need to download or install the application; instead, they rely on the host device to provide all the necessary data for the application when initiating a sharing session. Applications can run as server-side programs on the host device using any suitable hosting framework. In some cases, a single instance of an application on the host device can be shared with different client participants. This approach enables seamless cross-platform application sharing.

[0069] In some examples, requests to execute or share an application can be made by a host device such as those described above (e.g., Figure 4B The host device 450A) or remote device (e.g., Figure 4B The request is initiated by a remote device (410A, 420A, 430A, or 440A). The sharing request can be accepted or rejected by the requesting party (e.g., a host device requesting a remote device, or a remote device requesting a host device).

[0070] Application execution or sharing can be initiated using any suitable method. In some examples, users can explicitly initiate application execution or sharing, such as by interacting with a traditional user interface (e.g., a menu listing selections for applications). In some examples, computing devices (e.g., regarding...) Figure 4B Users of device 410A can interact with triggers, such as by scanning (e.g., via the camera of the head-mounted device) a visual target containing a QR code, barcode, URL, or may include a host device (e.g., regarding...). Figure 4B Other data, such as addressing information from the host device 450A, can be used to initiate the execution or sharing of applications. In some cases, triggers may include entry or exit to a specific location, such as that detected by a GPS unit, and addressing information may be associated with that location (e.g., via a lookup table). In some examples, a single trigger may be associated with two or more applications.

[0071] In response to the interaction, the user device can use addressing information to send a request to the host device. Upon receiving this request, the host device can initiate application sharing with the user device (e.g., as mentioned above regarding...). Figure 6(as described in stage 610); providing files to the user device (e.g., an executable file to be run on the user device); or taking other actions. In examples of user device interaction with a virtual environment, such triggers can be placed in the virtual environment (in some cases, spatial information such as persistent coordinate system (PCF) data is used by other users of the shared virtual environment), and user interaction with those triggers in the virtual environment can result in the execution or sharing of the application as described above. Furthermore, in some cases, triggers can be associated with sensor data (such as images received via a camera on a head-mounted device). For example, a user may encounter a chessboard at a specific location (e.g., a real-world location or a location in a virtual environment). By viewing, approaching, touching, or otherwise interacting with the chessboard, the user can initiate a request to execute or share a chess application (e.g., as described above regarding...). Figure 6 In this way, the application can not only be actively executed and / or shared by the user (e.g., via a traditional user interface); but also through organic interaction with objects and locations in real, virtual, or hybrid (e.g., augmented reality) environments.

[0072] In some cases, data related to the user's environment (e.g., sensor data such as GPS data or camera data from a head-mounted device) can be used to provide one or more parameters to the application. For example, when requesting to share a chess application, the user's current location can be used to identify the current weather and time of day, which can be used to determine the visual environmental impact of the chess application (e.g., time-sensitive lighting). Similarly, in some examples, requests to execute or share an application can be predicted based on contextual information; for example, a user may be able to send a sharing request to other devices nearby. In some examples, it may be desirable for a "headless" host device (e.g., 450A) to provide application sharing requests to remote users (e.g., users of 410A, 420A, 430A, 440A) based on contextual information; for example, the host device can be configured to present a request to share a theme park application to the user when the user's location is detected at a theme park. In some embodiments, permission may not be required, and content may be automatically shared with the user. In some embodiments, data may be used to initiate the application but may not be provided to the application itself. For example, personal data reflecting a user's current location can be used to trigger applications based on that location, but applications may be prevented from accessing that personal data, such as to enhance user privacy and confidentiality. Other ways to initiate application sharing requests will be obvious and are within the scope of this disclosure.

[0073] The above example process of a computer system can be provided by any suitable logic circuit. Suitable logic circuitry may include one or more computer processors (e.g., CPU, GPU, etc.) that perform the process when executing instructions implemented in a software program. Alternatively, the process can be provided via a corresponding logic design implemented in hardware logic circuitry, such as programmable logic (e.g., PLD, FPGA, etc.) or custom logic (e.g., ASIC, etc.) that implements the logic design providing the process. Furthermore, the process can be provided via an implementation that combines one or more processors running software and hardware logic circuitry.

[0074] Figure 7 An example system 700 is shown that can be used to implement any or all of the above examples. All or part of the above examples can be embodied in any portable device (including wearable devices) or non-portable device, such as communication devices (e.g., mobile phones, smartphones), multimedia devices (e.g., MP3 players, televisions, radios), portable or handheld computers (e.g., tablet computers, netbooks, laptop computers), desktop computers, all-in-one computers, peripherals, head-mounted devices (which may include, for example, an integrated display), or any other system or device suitable for including the example system architecture 700, including combinations of two or more of these types of devices. The above examples can be embodied in two or more physically separate devices, such as two or more computers communicating via a wireless network. The above examples can be embodied in two or more physically distinct devices, such as a belt bag for transmitting data to and / or from a head-mounted display. Figure 7 This is a block diagram of an example system 700, which typically includes one or more computer-readable media 701, a processing system 704, an I / O subsystem 706, radio frequency (RF) circuitry 708, audio circuitry 710, and sensor circuitry 711. These components may be coupled via one or more communication buses or signal lines 703.

[0075] It is obvious that Figure 7 The architecture shown is merely an example architecture of system 700, and system 700 may have more or fewer components than shown, or different configurations of components. Figure 7 The various components shown can be implemented in hardware, software, firmware, or any combination thereof, including one or more signal processing and / or application-specific integrated circuits.

[0076] refer to Figure 7In the example system architecture 700, RF circuitry 708 can be used to send and receive information to one or more other devices via a wireless link or network, and includes well-known circuitry for performing this function. RF circuitry 708 and audio circuitry 710 can be coupled to processing system 704 via peripheral device interface 716. Interface 716 can include various known components for establishing and maintaining communication between peripheral devices and processing system 704. Audio circuitry 710 can be coupled to audio speaker 750 and microphone 752, and can include known circuitry for processing voice signals received from interface 716 to enable a user to communicate with other users in real time. In some examples, audio circuitry 710 may include a headphone jack (not shown).

[0077] Sensor circuit 711 can be coupled to a variety of sensors, including but not limited to one or more light-emitting diodes (LEDs) or other light emitters, one or more photodiodes or other optical sensors, one or more photothermal sensors, magnetometers, accelerometers, gyroscopes, barometers, compasses, proximity sensors, cameras, ambient light sensors, thermometers, GPS sensors, electrooculography (EOG) sensors, and various system sensors capable of sensing remaining battery life, power consumption, processor speed, CPU load, etc. In examples such as those involving head-mounted devices, one or more sensors can be incorporated in conjunction with functions related to the user's eyes, such as tracking the user's eye movements or identifying the user based on images of his or her eyes.

[0078] Peripheral interface 716 couples the system's input and output peripherals to processor 718 and computer-readable medium 701. One or more processors 718 can communicate with one or more computer-readable media 701 via controller 74. Computer-readable medium 701 can be any device or medium (excluding signals) capable of storing code and / or data for use by one or more processors 718. In some examples, medium 701 can be a non-transitory computer-readable storage medium. Medium 701 can include a memory hierarchy, including but not limited to cache, main memory, and secondary memory. The memory hierarchy can be implemented using any combination of RAM (e.g., SRAM, DRAM, DDRAM), ROM, FLASH, magnetic and / or optical storage devices (such as disk drives, magnetic tape, CDs (optical discs), and DVDs (digital video discs)). Medium 701 can also include a transmission medium for carrying information-bearing signals (excluding signals and carrier waves when modulated) that indicate computer instructions or data. For example, the transmission medium can include communication networks, including but not limited to the Internet (also known as the World Wide Web), intranets, local area networks (LANs), wide area networks (WLANs), storage area networks (SANs), metropolitan area networks (MANs), etc.

[0079] One or more processors 718 can run various software components stored in medium 701 to perform various functions of system 700. In some examples, the software components may include an operating system 722, a communication module (or instruction set) 724, an I / O processing module (or instruction set) 726, a graphics module (or instruction set) 728, and one or more applications (or instruction sets) 730. Each of these modules and the applications mentioned above may correspond to a set of instructions (e.g., computer-implemented methods and other information processing methods described herein) for performing one or more of the functions described above and the methods described herein. These modules (i.e., instruction sets) do not need to be implemented as separate software programs, processes, or modules, and therefore, in various examples, various subsets of these modules may be combined or otherwise rearranged. In some examples, medium 701 may store a subset of the modules and data structures identified above. In addition, medium 701 may store additional modules and data structures not described above.

[0080] Operating system 722 may include various procedures, instruction sets, software components and / or drivers for controlling and managing common system tasks (e.g., memory management, storage device control, power management, etc.) and facilitating communication between various hardware and software components.

[0081] The communication module 724 can facilitate communication with other devices via one or more external ports 736 or via RF circuitry 708, and may include various software components for processing data received from RF circuitry 708 and / or external ports 736.

[0082] The graphics module 728 may include various known software components for rendering, animate, and displaying graphical objects on one or more display surfaces. The display surface may include a 2D or 3D display. The display surface may be directly or indirectly coupled to one or more components of the example system 700. In examples involving touch-sensing displays (e.g., touchscreens), the graphics module 728 may include components for rendering, displaying, and animateting objects on the touch-sensing display. In some examples, the graphics module 728 may include components for rendering to a remote display. In some examples, such as those including cameras, the graphics module 728 may include components for creating and / or displaying images formed by compositing camera data (such as that captured from a head-mounted camera) or photographic data (such as images captured by satellite) with rendered graphical objects. In some examples, the graphics module may include components for rendering images to a head-mounted display. In some examples, the image may include a view of elements of virtual content (e.g., objects in a 3D virtual environment) and / or a view of the physical world (e.g., camera input indicating the user's physical environment). In some examples, the display may present a composite of virtual content and views of the physical world. In some examples, the view of the physical world can be a rendered image; in other examples, the view of the physical world can be an image from a camera.

[0083] One or more applications 730 may include any application installed on system 700, including but not limited to browsers, address books, contact lists, email, instant messaging, word processing, keyboard emulation, widgets, Java-enabled applications, encryption, digital rights management, speech recognition, voice copying, location determination functions (such as those provided by the Global Positioning System (GPS), music players, etc.

[0084] I / O subsystem 706 may be coupled to eye I / O device 712 and one or more other I / O devices 714 for controlling or performing various functions. For example, eye I / O device 712 may communicate with processing system 704 via eye I / O device controller 732, which may include various components for processing eye input (e.g., sensors for eye tracking) or user gesture input (e.g., optical sensors). One or more other input controllers 734 may receive / send electrical signals to other I / O devices 714. Other I / O devices 714 may include physical buttons, dials, slider switches, joysticks, keyboards, touchpads, other displays, or any combination thereof.

[0085] I / O processing module 726 may include various software components for performing various tasks associated with eye I / O device 712 and / or other I / O devices 714, including but not limited to receiving and processing input received from eye I / O device 712 via eye I / O device controller 732 or from other I / O devices 714 via I / O controller 734. In some examples, I / O device 714 and / or I / O processing module 726 may perform various tasks associated with gesture input, which may be provided by tactile or non-tactile means. In some examples, gesture input may be provided, for example, by a camera or another sensor for detecting the movement of a user's eyes, arm, hand, and / or fingers. In some examples, I / O device 714 and / or I / O processing module 726 may be configured to recognize objects on a display that the user wishes to interact with, such as GUI elements that the user is pointing at. In some examples, the eye I / O device 712 and / or I / O processing module 726 can be configured (e.g., by means of optical or EOG sensors) to perform eye-tracking tasks, such as identifying an object or area on a display that the user is looking at. In some examples, devices (such as hardware “beacons”) can be worn or held by the user to assist the touch I / O device 712 and / or I / O processing module 726 in performing gesture-related tasks, such as identifying the position of the user’s hand relative to a 2D or 3D environment. In some examples, the eye I / O device 712 and / or I / O processing module 726 can be configured to identify the user based on sensor input related to the user’s eyes, such as data from a camera sensor.

[0086] In some examples, the graphics module 728 can display visual output to the user in a GUI. The visual output can include text, graphics, video, and any combination thereof. Some or all of the visual output can correspond to user interface objects. In some examples, I / O devices 712 and / or 714 and / or controllers 732 and / or 734 (and any associated modules and / or instruction sets in media 701) can detect and track gestures and / or eye movements, and can translate detected gestures and / or eye movements into interactions with graphical objects, such as one or more user interface objects. In examples where eye I / O devices 712 and / or eye I / O device controller 732 are configured to track the user's eye movements, the user can interact directly with the graphical objects by looking at them.

[0087] Feedback can be provided, for example, by eye I / O device 712 or another I / O device 714, based on the content being displayed and / or one or more states of the computing system. The feedback can be optical (e.g., light signals or displayed images), mechanical (e.g., haptic feedback, touch feedback, force feedback, etc.), electrical (e.g., electrical stimulation), olfactory, auditory (e.g., beeping sounds, etc.), or any combination thereof, and can be displayed in a variable or constant manner.

[0088] System 700 may also include a power system 744 for powering various hardware components, and may include a power management system, one or more power supplies, a charging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and any other components typically associated with the generation, management, and distribution of power in portable devices.

[0089] In some examples, the peripheral interface 716, one or more processors 718, and memory controller 720 can be implemented on a single chip, such as the processing system 704. In other examples, they can be implemented on separate chips.

[0090] Although the disclosed examples have been fully described with reference to the accompanying drawings, it should be noted that various changes and modifications will become apparent to those skilled in the art. For example, elements of one or more implementations may be combined, deleted, modified, or supplemented to form further implementations. Such changes and modifications should be understood to be included within the scope of the disclosed examples as defined by the appended claims.

Claims

1. A method for application sharing, comprising: At a host device having a first display and a first processor, a first application is executed via the first processor; Determine the first state of the first application; Generate a first scene graph corresponding to the first state of the first application; The first scene image is presented to a first remote device having a second display and a second processor; At the first remote device, the second application is executed via the second processor; Generate a second scene diagram corresponding to the first state of the second application; as well as The second scene diagram is presented to the host device; in: The first remote device is configured to render a first view corresponding to the first scene image to the second display in response to receiving the first scene image. Rendering the first view to the second display does not include executing the first application via the second processor. The host device is configured to render a second view corresponding to the second scene map to the first display in response to receiving the second scene map. Rendering the second view to the first display does not include executing the second application via the first processor. The first view includes virtual content in the environment of the first remote device, and the second display includes a transmissive display for simultaneously viewing the virtual content and the environment of the first remote device.

2. The method according to claim 1, further comprising: At the host device, input is received from the first remote device; A second state of the first application is determined based on the input, and the second state is different from the first state. Generate a third scene diagram corresponding to the second state, wherein the third scene diagram is different from the first scene diagram; Present the third scene image to the first remote device. in: The first remote device is configured to, in response to receiving the third scene graph, render a third view corresponding to the third scene graph to the second display without executing the first application, and Rendering the third view to the second display does not include executing the first application via the second processor. The input includes outputs from one or more sensors of the first remote device, the first view includes virtual content, and the outputs from the one or more sensors indicate interaction with the virtual content and / or the updated position of the first remote device.

3. The method according to claim 1, wherein, The first application is installed on the host device, and the first application is not installed on the first remote device.

4. The method according to claim 1, wherein, The first application is installed on the host device, and the first application is also installed on the first remote device.

5. The method according to claim 1, wherein, The first remote device is a wearable computing device, and the first display is configured to be mounted on the first remote device by a user.

6. The method of claim 1, further comprising: The first scene image is presented to a second remote device having a third display and a third processor. in: The second remote device is configured to render a second view corresponding to the first scene map to the third display in response to receiving the first scene map; and Rendering the second view to the third display does not include executing the first application via the third processor.

7. The method of claim 1, further comprising determining a first state of the third application, wherein: The first scene diagram further corresponds to the first state of the third application, and Rendering the first view to the third display does not include executing the third application via the second processor.

8. The method of claim 1, further comprising receiving a request from the first remote device to present the first scene graph, wherein: The request to present the first scene graph is generated in response to the interaction between the first remote device and the trigger, and In response to receiving the request, the system performs the action of presenting the first scene map to the first remote device.

9. The method according to claim 8, wherein, The trigger includes a QR code.

10. The method according to claim 8, wherein, The trigger includes location information.

Images