Multi-user cross-device synchronization for meta-universe instantiation

Abstract

A method includes receiving a request to establish a multi-player session for a user to allow participation in a meta universe. The method includes determining whether an application generating the meta universe is installed on a local device of the user selected by the user to participate in the meta universe. The method includes launching a corresponding local instance of the application when the application is installed on a corresponding local device. The method includes launching a corresponding cloud instance of the application on a cloud-based streaming server when the application is not installed on the corresponding local device. The method includes determining that each instance of the application for the user has been launched, wherein each instance of the user is a local instance or a cloud instance. The method includes allowing the start of the multiplayer session when the instances of the application program for the user have been launched.

Description

Technical Field

[0001] The present disclosure relates to gaming, and more particularly to monitoring computing devices available to users and applications installed on those devices to facilitate loading instances of applications locally or at a cloud-based streaming server for use by groups of users in a multiplayer session. In addition, an artificial intelligence model may be built for the users so that recommendations for applications may be provided to groups of users participating in a multiplayer session at a particular point in time. Background Art

[0002] Video games and / or game applications and their related industries (e.g., playing video games) are very popular and account for a large percentage of the global entertainment market. Video games can be played anytime, anywhere using various types of platforms (including game consoles, desktop or laptop computers, mobile phones, etc.).

[0003] Users expect an immersive social experience that can include one or more metaverses and seamlessly transfer between virtual environments. For example, a user group that is playing together in a virtual environment of an application may want to transfer their comprehensive social experience to another virtual environment of another application (i.e., start playing together in a new virtual environment). The application should be executable across different platforms of the devices currently used by the user group. However, problems may arise when the current devices of some users in the group cannot install the application on the local device or do not install the application on the local device to provide the best user experience. The affected users have to wait for the application to be installed on their devices and then contact their friends. Since this installation process may last for a long time (e.g., more than 10 to 20 minutes), the movement of the user group between virtual environments will be incomplete and interrupted. That is, the immersive social experience will be broken by the suddenness of reality because some users will be excluded from the new virtual environment and the interaction performed in the new virtual environment.

[0004] It is against this background that embodiments of the present disclosure are presented. Summary of the invention

[0005] Embodiments of the present disclosure relate to a backend server supervisor configured to monitor and supervise users' computing devices to determine which applications are locally installed on which devices of the users. In this way, when a group of players wishes to participate together in a virtual world or metaverse (such as a virtual world or metaverse created by an application) in a multiplayer session, the supervisor can facilitate such participation by launching local instances of the locally installed application on the corresponding devices and / or launching cloud instances of the application for users whose devices do not have the application locally installed. The supervisor can then coordinate the initiation and management of the multiplayer session to allow the user group to participate in the virtual world or metaverse of the application.

[0006] In one embodiment, a method is disclosed. The method includes generating game state data for a player's game play during execution of a video game. The method includes receiving a request to establish a multiplayer session for multiple users over a network to allow participation in a metaverse. The method includes determining whether an application configured to generate the metaverse is installed on multiple local devices of the multiple users selected by the multiple users to participate in the metaverse. The method includes launching a corresponding local instance of the application on the corresponding local device when the application is installed on the corresponding local device. The method includes launching a corresponding cloud instance of the application on a cloud-based streaming media server when the application is not installed on the corresponding local device. The method includes determining that each of the multiple instances of the application for the multiple users has been launched, wherein each of the multiple instances is a local instance or a cloud instance. The method includes allowing the multiplayer session to start when each of the multiple instances of the application for the multiple users has been launched.

[0007] In another embodiment, a non-transitory computer-readable medium storing a computer program for implementing a method is disclosed. The computer-readable medium includes program instructions for receiving a request to establish a multiplayer session for a plurality of users over a network to allow participation in a metaverse. The computer-readable medium includes program instructions for determining whether an application configured to generate the metaverse is installed on a plurality of local devices of the plurality of users selected by the plurality of users to participate in the metaverse. The computer-readable medium includes program instructions for starting a corresponding local instance of the application on the corresponding local device when the application is installed on the corresponding local device. The computer-readable medium includes program instructions for starting a corresponding cloud instance of the application on a cloud-based streaming media server when the application is not installed on the corresponding local device. The computer-readable medium includes program instructions for determining that each of a plurality of instances of the application for the plurality of users has been started, wherein each of the plurality of instances is a local instance or a cloud instance. The computer-readable medium includes program instructions for allowing the multiplayer session to start when each of the plurality of instances of the application for the plurality of users has been started.

[0008] In yet another embodiment, a computer system is disclosed, wherein the computer system includes a processor and a memory, the memory coupled to the processor and having instructions stored therein, the instructions, if executed by the computer system, causing the computer system to perform a method. The method includes receiving a request to establish a multiplayer session for a plurality of users over a network to allow participation in a metaverse. The method includes determining whether an application configured to generate the metaverse is installed on a plurality of local devices of the plurality of users selected by the plurality of users to participate in the metaverse. The method includes launching a corresponding local instance of the application on the corresponding local device when the application is installed on the corresponding local device. The method includes launching a corresponding cloud instance of the application on a cloud-based streaming media server when the application is not installed on the corresponding local device. The method includes determining that each of the plurality of instances of the application for the plurality of users has been launched, wherein each of the plurality of instances is a local instance or a cloud instance. The method includes allowing the multiplayer session to start when each of the plurality of instances of the application for the plurality of users has been launched.

[0009] Other aspects of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present disclosure may be best understood by referring to the following description taken in conjunction with the accompanying drawings, in which:

[0011] Figure 1 A system including a supervisor configured to monitor devices and applications installed on user devices to facilitate loading an instance of an application providing a metaverse or virtual world locally or at a cloud-based streaming server to support initiating a multi-player session of the application conducted by multiple users in a group is illustrated according to one embodiment of the present disclosure.

[0012] Figure 2 is a flow chart illustrating a method for launching an instance of a metaverse or application to support a multi-player session by a group of users according to one embodiment of the present disclosure.

[0013] Figure 3 An example of information captured and monitored by a supervisor configured to monitor devices and applications installed on user devices according to one embodiment of the present disclosure, wherein the supervisor is configured to launch instances of applications configured to generate a metaverse or virtual world on a local device or at a cloud-based streaming server, and / or make recommendations for applications for a group of users to participate in a multi-player session.

[0014] Figure 4 is a diagram illustrating a supervisor that facilitates transfers between devices of users of applications as the users move through a physical environment, according to one embodiment of the present disclosure.

[0015] Figure 5 Components of an example device that may be used to perform aspects of various embodiments of the present disclosure are illustrated. DETAILED DESCRIPTION

[0016] Although the following detailed description contains many specific details for illustrative purposes, those skilled in the art will appreciate that many variations and modifications of the following details are within the scope of the present disclosure. Therefore, the aspects of the present disclosure described below are set forth without causing any loss of generality to the appended claims of this specification and without imposing limitations on these claims.

[0017] In general, various embodiments of the present disclosure describe systems and methods for implementing supervisor functionality at a backend server that is configured to monitor a user's computing devices and the applications installed on those computing devices. In this way, the supervisor knows which applications are installed on and / or are being executed on the device currently used by the user. Knowledge of applications and devices allows the supervisor to facilitate and / or manage the initiation of multiplayer sessions for groups of users or multiple users who wish to participate together in a virtual world or metaverse generated by a corresponding application. In the case of a local installation on the corresponding user's selected device, a local instance of the application will be launched on the local selected device because the local execution of the corresponding application that generates the metaverse can give the best user experience. On the other hand, if no local installation exists on the corresponding user's selected device, a cloud instance of the application is launched on a cloud-based streaming server that is configured to stream the game play of the corresponding user within the metaverse to the selected device. In this way, a multiplayer session can be established using a local or cloud-based instance of an application that generates a metaverse for a user. In addition, artificial intelligence (AI) techniques can be implemented to build an AI model of the user that predicts the pattern of game participation. In this way, multiple users in a group may be provided with recommendations made by a moderator that recommend applications and / or corresponding metaverses for a multi-player session at any time or at a recommended time.

[0018] Advantages of methods and systems configured to implement a supervisor include: when the supervisor launches a corresponding instance of the application for each of the users to support a multi-player session, the users seamlessly transition to the metaverse generated by the application. The supervisor is configured to facilitate instances of the application that support multi-player sessions on a local device or a cloud-based streaming server for multiple users (i.e., in a group). Thus, the supervisor can manage the launch of instances of the application for users even if the application is not installed locally on the corresponding device (i.e., without having to coordinate and / or wait for local installation of the application). Because the supervisor is able to launch local instances and cloud-based instances, each of the multiple users in the group can immediately participate in the multi-player session without having to wait for the application to be downloaded and installed on the corresponding device. Other advantages include: the supervisor is configured to facilitate application installation during or after a streaming session on the corresponding device, wherein the supervisor can query the user about the application installation and facilitate the application installation after receiving authorization from the user. Still other advantages include: seamless transition of multiple users in a group from one metaverse to another, as the supervisor can facilitate the generation of instances of applications of the new metaverse, wherein the instances of the applications (local and / or cloud-based) support multi-player sessions for multiple users. Still other advantages include: seamless transition from one device executing the application to another device to support game sessions for users traveling from one location to another, wherein the transfer of game sessions between devices is tailored to provide the best gaming experience to the users. Still other advantages include enabling the supervisor to build an AI model for each of the users that predicts the user's game play patterns. In this way, the supervisor can provide recommendations of applications for multiple users in a group who want to participate in a multi-player session together at a certain point in time, within a general time period or a recommended time period.

[0019] Throughout this specification, references to "games" or "video games" or "gaming applications" are intended to refer to any type of interactive application that is directed by executing input commands. For purposes of illustration only, interactive applications include applications for gaming, word processing, video processing, video game processing, and the like. Additionally, the terms "virtual world" or "virtual environment" or "metaverse" are intended to refer to any type of environment generated by one or more corresponding applications for interaction between multiple users in a multi-player session or multi-player gaming session. Further, the terms introduced above are interchangeable.

[0020] With the above general understanding of the various embodiments, example details of the embodiments will now be described with reference to the various figures.

[0021] Figure 1A system 100 including a supervisor 120 according to one embodiment of the present disclosure is illustrated, the supervisor being configured to monitor devices and applications installed on user devices to facilitate loading instances of applications providing a metaverse or virtual world locally or at a cloud-based streaming server to support initiating a multiplayer session of the application by multiple users in a group. The system 100 can be implemented to monitor application usage and application installation status on various devices of multiple users for various purposes. For example, the supervisor 120 of the system 100 can be configured to facilitate the initiation of a multiplayer session of the application by automating the launch of a local instance of the application or a cloud-based instance of the application for selected devices of users in a user group participating in the multiplayer session. In another example, the supervisor is capable of building an artificial intelligence (AI) model that can predict the gaming behavior of users, such that the AI ​​model is implemented to recommend applications and / or devices for a multiplayer session for a user group at any time or at a specific time.

[0022] As shown, the system 100 can provide games to one or more client devices 110 and between the one or more client devices via a network 150. Specifically, according to one embodiment of the present disclosure, the system 100 can be configured to provide games to users participating in a multi-player game session (e.g., participating in a metaverse generated by an application) via a cloud gaming network 190, wherein the game can be executed locally (e.g., on a local client device of the corresponding user) or can be executed remotely from the corresponding client device 110 of the corresponding user who is playing a video game (e.g., acting as a thin client). In at least one capability, the cloud gaming network 190 supports multi-player game sessions for user groups, including delivering and receiving game data of players for coordinating and / or aligning objects and players' actions within a scene of a game world or metaverse, managing communications between users, etc., so that users participating in a multi-player game session in distributed locations can interact with each other in the game world or metaverse in real time.

[0023] Specifically, the system 100 may provide game control to one or more users playing one or more applications (e.g., video games) in a multiplayer session through a local instance operated on a client device or through a cloud-based instance operated in a cloud gaming network 190 via a network 150. The network 150 may include one or more communication technologies, including a fifth generation (5G) network technology with an advanced wireless communication system (e.g., cellular network technology). In some embodiments, the cloud gaming network 190 may include a plurality of virtual machines (VMs) running on a host's hypervisor, wherein one or more virtual machines are configured to utilize hardware resources available to the host's hypervisor to execute a game processor module. It should be noted that access services delivered over a wide geographic area (such as providing access to games of the current embodiment) typically use cloud computing. Cloud computing is a computing method in which dynamically scalable and typically virtualized resources are provided as a service over the Internet.

[0024] In a multiplayer session that allows a group of users to participate in an interaction in a game world or metaverse generated by an application (which may be a video game), some users may execute an instance of the application locally on a client device to participate in the multiplayer session. Other users who do not have the application installed on the selected device or when the computing power of the selected device is insufficient to execute the application may participate in the multiplayer session via a cloud-based instance of the application executed at the cloud gaming network 190.

[0025] As shown, the cloud gaming network 190 includes a game server 160 that provides access to multiple video games. Most applications played in a corresponding multi-player session are played through a network 150 connected to the game server 160. For example, in a multi-player session involving multiple instances of an application (e.g., generating a virtual environment, a game world, a metaverse, etc.), a dedicated server application (session manager) collects data from a user and distributes it to other users so that all instances are updated with respect to objects, characters, etc. to allow real-time interaction within the virtual environment of the multi-player session, where the user can execute a local instance or a cloud-based instance of the corresponding application. The game server 160 can be any type of server computing device available in the cloud, and can be configured as one or more virtual machines executed on one or more hosts. For example, the game server 160 can manage a virtual machine for a game processor that supports a cloud-based instance of an instantiated application for a user. Thus, multiple game processors of the game server 160 associated with multiple virtual machines are configured to execute multiple instances of one or more applications associated with the game play of multiple users. In this manner, the backend server supports streaming of media (e.g., video, audio, etc.) providing gaming entertainment for multiple applications (e.g., video games, gaming applications, etc.) to multiple corresponding users. That is, the game server 160 is configured to stream data (e.g., rendered images and / or frames of the corresponding gaming entertainment) back to the corresponding client devices 110 via the network 150. In this manner, computationally complex gaming applications can be executed at the backend servers in response to controller inputs received and forwarded by the client devices 110. Each server is capable of rendering images and / or frames, which are then encoded (e.g., compressed) and streamed to the corresponding client devices for display.

[0026] In a multi-player session, an instance of the application may be executed locally on the client device 110 or at the cloud gaming network 190. In either case, the application as game logic 115 is executed by a game engine 111 (e.g., a game title processing engine). For the purpose of clarity and brevity, the implementation of game logic 115 and game engine 111 is described in the context of a cloud gaming network 190. Specifically, the application may be executed by a distributed game title processing engine (referred to herein as a "game engine"). Specifically, the game server 160 and / or the game title processing engine 111 include basic processor-based functions for executing applications and services associated with the application. For example, processor-based functions include 2D or 3D rendering, physics, physics simulation, scripting, audio, animation, graphics processing, lighting, shading, rasterization, ray tracing, shadows, culling, transformation, artificial intelligence, etc. In this way, the game engine implements game logic, performs game calculations, physics, geometric transformations, rendering, lighting, shading, audio, and additional in-game or game-related services. In addition, the application's services include memory management, multi-threading management, Quality of Service (QoS), bandwidth testing, social networking, social friend management, communication with friends on social networks, social utilities, communication channels, audio communication, texting, messaging, instant messaging, chat support, game entertainment replay function, help function, etc.

[0027] The user accesses the remote service using a client device 110, which includes at least a CPU, a display, and input / output (I / O). For example, the user can access the cloud gaming network 190 via the communication network 150 using a corresponding client device 110 configured to update a session controller (e.g., deliver and / or receive user game state data) and receive streaming media. The client device 110 can be a personal computer (PC), a mobile phone, a netbook, a personal digital assistant (PAD), a handheld device, etc.

[0028] In one embodiment, the client device 110 may be configured with a game title processing engine and game logic for at least some local processing of the application, and may be further utilized to receive streaming content generated by the application executed at the back-end server, or other content provided by the back-end server support. In yet other embodiments, for independent local processing, the game title processing engine 111 includes basic processor-based functions for executing the application and the services associated with the application, as previously described. For local processing, the game logic 115 is stored on the local client device 110 and used to execute the application. For example, an instance of the application is executed by the game title processing engine 111 of the corresponding client device 110. The game logic 115 (e.g., executable code) that implements the application is stored on the corresponding client device 110 and used to execute the application. For the purpose of illustration, the game logic 115 can be delivered to the corresponding client device 110 via a portable medium (e.g., optical media) or via a network (e.g., downloaded from a game provider via the Internet).

[0029] In one embodiment, the client device 110 may be configured as a thin client that provides an interface with a backend server (e.g., a game server 160 of the cloud gaming network 190) configured to provide computing functionality (e.g., including a game title processing engine 111). Specifically, the client device 110 of a corresponding user (not shown) is configured to request access to an application through a communication network 150 such as the Internet, and to render images generated by a video game executed by the game server 160 for display, wherein the encoded images are delivered (i.e., streamed) to the client device 110 for display in association with the corresponding user. For example, a user may interact with an instance of an application associated with the game play of the corresponding user executed on a game processor of the game server 160 through the client device 110, such as through input commands for driving game play. The client device 110 may receive input from various types of input devices (such as a game controller, a tablet computer, a keyboard, gestures captured by a camera, a mouse, a touchpad, an audio input terminal, etc.). More specifically, an instance of an application is executed by the game title processing engine 111 and is configured to generate a rendered image that is delivered over the network 150 for display on a corresponding display associated with the client device 110. That is, the client device 110 is configured to receive an encoded image (e.g., encoded from a game rendered image generated by execution of a video game), and to display the image rendered for display. The game title processing engine 111 is capable of supporting multiple applications using multiple game logics, each of which is selectable by a user.

[0030] In addition, the system 100 includes a backend supervisor 120, the functionality of which is configured to monitor the computing devices of users, including, in part, monitoring the applications installed on those devices, when the applications are used, when the devices are used, and when the applications are used on which devices during which time periods. The supervisor 120 may be located at the cloud gaming network 190, or may be remote from the cloud gaming network 190. For example, the supervisor 120 is able to utilize application installation and device usage to facilitate the initiation of a multiplayer session for multiple users (e.g., a group) who wish to participate and interact with each other in a virtual environment or metaverse. Specifically, the supervisor 120 may manage the launching of local and / or cloud-based instances of applications configured to generate a virtual environment or metaverse based on whether the applications are installed on the corresponding devices when instantiating a multiplayer session.

[0031] In one embodiment, the supervisor 120 acts in cooperation with the client-based application 112 installed on the client device 110. For example, a corresponding client-based application 112 exists on each of the multiple client devices associated with the user, wherein each of the client-based applications communicates with the supervisor 120 to understand which applications are installed and / or run on which devices of the user. Specifically, the supervisor 120 communicates with the corresponding client-based application 112 to access and / or collect information about which applications are installed and / or run on the corresponding client devices. In addition, the supervisor is able to control the corresponding client device 110 via the corresponding client-based application 112, including causing the client device to start a local instance of the application on the client device. In one embodiment, the client-based application 112 can be a very small program, such as a daemon process that has no user interface and runs in the background. In another embodiment, the client-based application 112 can be part of a mature client application, such as an application loaded onto a gaming platform or a computer or an equivalent device, etc.

[0032] In some embodiments, the supervisor 120 includes artificial intelligence (AI), including a deep / machine learning engine 130, which is configured to build or train and implement an AI model 135 for each of the multiple users of the device monitored by the supervisor 120. In one embodiment, the AI ​​learning model 135 is a machine learning model that is configured to apply machine learning to learn the user's game usage and / or play behavior and make recommendations based on the behavior. In another embodiment, the AI ​​learning model is a deep learning model that is configured to apply deep learning to learn the user's game usage behavior and make recommendations based on the behavior, where machine learning is a subclass of artificial intelligence and deep learning is a subclass of machine learning. Thus, artificial intelligence is used to learn the user's game usage behavior and make recommendations based on the behavior. Specifically, the corresponding AI model 135 is configured to learn the usage patterns of the corresponding user, including learning which applications the user likes to play during which time periods, and learning which devices the user prefers to play these applications on. In this way, the AI ​​model 135 can predict which applications the corresponding user prefers during which time periods and on which devices. For a group of users participating in a virtual world or metaverse (e.g., a game world of a video game) in a multi-player session, the supervisor 120 is configured, via the corresponding AI model of the user group, to recommend to the user group specific applications for the requested multi-player session, and more specifically for any time period, or for a specific time period.

[0033] For illustration purposes only, according to one embodiment of the present disclosure, the deep / machine learning engine 190 may be configured as a neural network for training and / or implementing the AI ​​model 135. In general, a neural network represents a network of interconnected nodes that respond to inputs (e.g., extracted features) and generate outputs (e.g., learning a user's game usage and / or play behavior and providing recommendations). In one implementation, the AI ​​neural network includes a hierarchical structure of nodes. For example, there may be an input layer of nodes, an output layer of nodes, and an intermediate or hidden layer of nodes. The input nodes are interconnected to the hidden nodes in the hidden layer, and the hidden nodes are interconnected to the output nodes. The interconnections between the nodes may have digital weights, which can be used to link multiple nodes together between inputs and outputs, such as when defining the rules of the AI ​​model 135.

[0034] Through Figure 1 A detailed description of the system 100, Figure 2Flowchart 200 of the present disclosure discloses a method for monitoring application installations on a device and device usage by a user in order to facilitate the initiation of a multiplayer session for multiple users (e.g., a group) according to one embodiment of the present disclosure. The operations performed in the flowchart may be performed by one or more of the previously described components and also Figure 1 Specifically, Figure 2 is a flow chart illustrating a method for initiating an instance of a metaverse or application to support a multiplayer session by a group of users according to one embodiment of the present disclosure. A multiplayer session may be initiated by launching a corresponding application (e.g., for generating a desired virtual world and / or metaverse, etc.) across different platforms of different devices (e.g., mobile devices, mobile phones, tablets, thin clients, game consoles, computer towers, servers, etc.). For example, the different platforms may include different operating systems for the respective devices, such as a first local device executing a first operating system and a second local device executing a second operating system.

[0035] At 205, the method includes monitoring a plurality of applications installed on a plurality of devices available to a plurality of users. The plurality of users includes all users using the supervisor service. For example, the plurality of users may include one or more user groups, each of which desires to participate together in a corresponding virtual world or metaverse in a corresponding multiplayer session. That is, a user group may utilize the supervisor and the method implemented in flowchart 300 to facilitate a multiplayer session conducted within a corresponding virtual world or metaverse. Specifically, for a corresponding user associated with a plurality of devices and one or more selected devices currently being used by the user, each device of the user is monitored to track which applications are installed on the device and, further, which applications may be installed on the device based on the computing power, memory capacity, etc. of the device.

[0036] At 210, the method includes receiving, over a network, a request to establish a multiplayer session for a plurality of users (e.g., a group) to allow participation in a virtual world and / or metaverse generated by an application. Specifically, the users may be friends or a social group that enjoys playing in any of a plurality of virtual worlds and may wish to play together in a multiplayer session at a specific time (e.g., a current time, a future time, or any suggested time, etc.). For example, in one implementation, the application is a video game, and in a multiplayer session (e.g., a multiplayer game session), the users are participating in a virtual world or metaverse generated by the video game.

[0037] Generally, a multiplayer session is initiated and / or established by instantiating multiple instances of an application that generates a virtual world and / or metaverse for multiple users (e.g., a group of users). In some cases, a user may have to transfer (e.g., terminate) from one application to another application that generates a virtual world and / or metaverse (e.g., to move between virtual experiences), which is coordinated by a supervisor implementing flowchart 300. The method of flowchart 300 determines whether an application configured to generate a virtual world and / or metaverse is installed on multiple local devices of multiple users selected by the multiple users to participate in the metaverse. Depending on whether the application is installed locally, a local instance or a cloud-based instance of the application is launched to facilitate the initiation and / or establishment of a multiplayer session, thereby allowing multiple users (e.g., a group) to participate in the virtual world and / or metaverse. The following operations are repeated and / or looped until each device of the user has a corresponding instance of the application executed for participating in the multiplayer session.

[0038] At 220, the method includes selecting a local device of a plurality of local devices that is selected for a plurality of users to participate in a virtual world and / or a metaverse generated by a corresponding application. The local device is associated with the corresponding user, wherein the local device may be a current device used by the corresponding user or selected by the corresponding user to participate in a multi-player session, or may be automatically selected when implementing the method of flowchart 300 (e.g., by a supervisor based on a corresponding AI model of the corresponding user).

[0039] At decision step 230 , the method includes determining whether the local device of the corresponding user has an application installed, wherein the application is configured to generate a virtual world and / or metaverse within which the users wish to interact with each other in a multiplayer session.

[0040] At 240, if the application is installed on the local device of the corresponding user, the method includes launching (e.g., instantiating, starting, etc.) a local instance of the application on the local device. For example, this may include launching the locally installed application on the corresponding local device (i.e., as an instance). Specifically, for the first user, the method may include determining that the application is installed on a first local device of the first user. As previously described, the first local device may be selected by the first user to participate in the virtual world and or metaverse generated by the application in the multiplayer session. Thus, a first local instance of the application is launched on the first local device for the first user to participate in the metaverse for the first time.

[0041] At 245, the method includes sending feedback back to an entity (e.g., a manager or supervisor that facilitated the initiation of the multiplayer session) that the local instance of the application was launched. In this way, one entity is configured to track the launch of instances of the application for all users in a plurality of users (e.g., a group) participating in the multiplayer session. The method then proceeds to decision step 260.

[0042] In one embodiment, for legacy games that do not report back to any entity and provide no feedback, instructions may be sent to the local device to generate and / or display a pop-up user interface (UI) that asks the user to confirm when the application is available for use (e.g., installed on the local device). For example, when the application is launched, the UI may display a message asking the user to click a button, such as a message including "Once the application is running, click 'Ready' to continue," where "Ready" is placed in a window that allows the user to select. In this way, a message indicating that the local instance has been installed on the local device is returned to the entity managing the initiation of the multiplayer session (e.g., a supervisor) without the application sending information back as feedback. In another embodiment, the local device may be instructed and / or requested to send a screenshot of the display of the local device back to the entity managing the initiation of the multiplayer session (e.g., a supervisor) every few seconds when the application is launched. Further, the backend entity managing the initiation of the multiplayer session (e.g., a supervisor) may use computer vision and / or AI to perform image recognition to recognize when the application is running (e.g., recognize the name screen of the application).

[0043] At 250, if the application is not installed on the local device of the corresponding user, the method includes launching (e.g., instantiating, starting, etc.) a cloud-based instance of the application on a cloud-based streaming server (e.g., at the cloud gaming network 190). The cloud-based instance is used to execute the application so that the corresponding user can participate in a multi-player session in a virtual world and / or metaverse generated by the application. Continuing with the example provided above, after determining that the application is not installed on the first local device of the first user, the first cloud instance of the application is launched on the cloud-based streaming server for the first user to participate in the virtual world and / or metaverse for the first time. In this way, even if the local instance of the application cannot be launched to play the game in the multi-player session, the corresponding user does not need to wait to download and install the application on the local device before participating in the multi-player session, because the cloud-based instance of the application allows the corresponding user to participate in the virtual world and / or metaverse of the application with other users in the multi-player session.

[0044] At 255, the method includes sending feedback back to an entity (e.g., a manager or supervisor that facilitated the initiation of the multi-player session) that the cloud-based instance of the application has been launched. In this way, the entity is configured to track the launch of instances (local instances and / or cloud-based instances) of the application for all users in a plurality of users (e.g., a group) participating in the multi-player session. The method then proceeds to decision step 260.

[0045] At decision step 260, the method includes determining whether all instances of the application associated with the plurality of devices have been launched. That is, the method determines whether all local and / or cloud-based instances of the application for the plurality of users (e.g., in a group) have been launched, thereby allowing all users to participate in the multiplayer session. If there is at least one remaining instance that has not been launched, the method proceeds back to operation 220 to launch a local or cloud-based instance of the application for the next corresponding local device. On the other hand, if all instances of the application (e.g., local instances and / or cloud-based instances) have been launched, the method proceeds to 270.

[0046] At 270, the method includes allowing a multiplayer session to start when each of the multiple instances of the application for the multiple users has been launched. That is, the user has joined the multiplayer session. As previously described, the instance can be a local instance executed on a local device of the corresponding user, or a cloud-based instance executed on a gaming cloud system. For example, the method can include sending a notification to a session manager of a cloud-based streaming server or a cloud gaming network that each of the multiple instances of the application for the multiple users has been launched, wherein the session manager supports the multiplayer session, as previously described.

[0047] In another embodiment, when the method determines that the application is not installed on the local device of the corresponding user, in addition to launching the cloud-based instance of the application, a request is sent to the corresponding user via the local device with a query asking whether the user would like to install the application on the local device. The delivery of the query may occur during or shortly after the launch of the cloud-based instance, during game play of the multi-player session, or after the multi-player session ends. After receiving authorization to perform the installation, the method may include instructing the local device to download the application. The user may be presented with a time for downloading the application (e.g., immediately, after the end of the multi-player session, within a specified time period, etc.).

[0048] As previously described, a query or prompt to the user may occur at game launch (e.g., launch of a cloud-based instance), where the query may ask the user whether to install the application on the local device. For example, the message may include the following: "This game can be downloaded from your device. Would you like to install it?" In response, several options for the user to select may include: "Install now | Install later | Remind me later | No, thanks." For example, the user may choose to install the application or get a reminder later so that the download does not interfere (i.e., introduce connection instability) and consume bandwidth during the streaming of the current game play in the current multi-player session. More specifically, the "install later" option may tell the supervisor to send another message (e.g., a download message) when the game play of the multi-player session is completed, or when the device is idle, or during the middle of the night, or at a time period specified by the user, or at a predefined time period, etc. Further, the query or prompt may occur at the end of the multi-player session. For example, the inquiry and / or prompt may include: "For best results the next time you play, please install the corresponding application on your device." Several options for user selection in response may include: "Install now | Install later | Remind me later | No thanks."

[0049] In one embodiment, a query to install the application may be provided for the benefit of the user. When the cloud stream is interrupted or the connection is dropped, a query or prompt may be generated. For example, the query and / or prompt may include: "Your connection seems unstable. For best results, please install the game on your device." In response, several options for the user to select may include: "Install now | Install later | Remind me later | No, thanks." As an example, during a multi-player session, one user may be participating using a cloud-based instance of the application and may further experience connection instability (e.g., periodic interference in the connection). This may occur in the case where the user is using a device that can install but has not yet installed the game (e.g., using a mobile device). In the case where the user participates in a multi-player application with a group of users (e.g., a first-person shooter game in a war setting), even if the application is installable on the mobile device, the user has not yet installed the application on the mobile device. As previously described, the method determines that the user should stream the game play back to the user (using a cloud-based instance of the application). Since the stream may be fragile and may be interrupted / dropped for various reasons, the supervisor implementing the method may know through tracking that the mobile device has an installable version. In this case, the method includes sending a message to the mobile device with instructions to query the user whether the user is willing to download the application to obtain a better user experience (e.g., better stability in game play through the local instance of the application) in the current multi-player session or future multi-player sessions.

[0050] In another embodiment, there may be a connection problem with the locally installed instance of the application. For example, the connection through the network back to the session manager used to coordinate the game play of one of the users in the multiplayer session may be interrupted or disconnected. In this case, the game play of the user who suffered the connection problem or has been disconnected cannot be transmitted to the multiplayer server (e.g., the session manager). As a solution, a method (e.g., implemented by a supervisor) can periodically check (ping) each of the devices used by the users in the multiplayer session to check whether the connection is still there.

[0051] If the corresponding device cannot be reached or the corresponding device fails to send a verification request, the method may include sending a message to the devices of other users in the multiplayer session that the disconnected user appears to be experiencing connection problems. In this way, other users are notified that there may be problems with communication with the disconnected user, or that the character of the disconnected user may be experiencing problems. For example, the supervisor may be configured to periodically verify each of a plurality of local devices used by a plurality of users in a multiplayer session. The method may include determining that a connection to a first local device of a first user is terminated during the multiplayer session. The first local device is used by the first user to participate in a multiplayer session for the first time. The method may include providing a notification to each of a plurality of local devices connected via a network (e.g., to other users in the multiplayer session) that the connection to the first user is experiencing connection problems. In this way, other users are notified that the disconnected user is experiencing problems participating in the multiplayer session.

[0052] As previously described, an AI model can be built for each user, and the supervisor monitors each user's device. For example, the AI ​​model predicts the corresponding user's gaming behavior, including learning the following: when the corresponding user plays video games, which video games they play, on which devices they play video games, and at what time the user plays these video games. Thus, the AI ​​model can be used to provide recommendations. For example, an AI model for a group of users can be analyzed to find various ways to optimize the group's gaming experience with minimal interference.

[0053] In one embodiment, the supervisor analyzing the AI ​​models of the users in the group may proactively suggest playtime based on the times when each person is typically in a stable environment. For example, the suggested times may occur when each user is able to interface with a game console or personal computer, is connected via a local area network (LAN) rather than Wi-Fi, is not communicating (e.g., a stable connection) when using a mobile device so that the global positioning satellite (GPS) location is static, etc.

[0054] In one embodiment, a supervisor analyzing the AI ​​models of users in a group may determine that everyone who wants to play together in a multiplayer session has a particular application (e.g., a video game, etc.) installed on their current device. In this case, the supervisor may recommend a particular game as the "best experience" because each player can launch a local instance of the application and no player must stream their game play (i.e., there is no cloud instance of the application), where streaming the game play may introduce additional latency, causing the user to experience a delay in the multiplayer session compared to other users. For example, a method implemented by a supervisor may include receiving an initial request for coordinated interaction between multiple users. The method includes determining a plurality of applications installed on a plurality of local devices of a plurality of users (e.g., in a group). The method includes determining that an application is installed on each of the plurality of local devices. The method includes providing a response to the initial request to each of the plurality of local devices, the response recommending an application configured to generate a metaverse for coordinated interaction. For example, Figure 3 The data collected by the moderator is illustrated to make recommendations based on which applications are installed on which devices of a group of users who wish to participate in a multi-player session.

[0055] Specifically, Figure 3 Table 300 including information captured and monitored by a supervisor configured to monitor devices and applications installed on user devices according to one embodiment of the present disclosure is illustrated. According to an embodiment of the present disclosure, the supervisor is configured to launch an instance of an application configured to generate a metaverse or virtual world on a local device or at a cloud-based streaming server, and / or make recommendations for applications for a group of users to participate in a multiplayer session. Specifically, the information in table 300 can be used to recommend applications to a group of users who wish to participate in a multiplayer session.

[0056] As shown, table 300 includes a row 310 listing a plurality of users. In the current example, the plurality of users are user groups that wish to participate in a multiplayer session. In other words, table 300 is specific to user groups. Of course, in other embodiments, the supervisor can monitor a large number of users, including one or more groups. For example, the user group in row 310 may include user 1, user 2, and user 3.

[0057] Table 300 includes row 320 that lists the devices of each of the users. For each user, at least one device is selected to participate in the multiplayer session, as previously described (e.g., the current device used by the corresponding user, selected by the corresponding user, selected by the AI ​​model, etc.). For example, user 1 has device 1, device 2, and device 3, where device 1 is the currently selected device. Furthermore, user 2 has device 1 and device 2, where device 1 is the currently selected device. Further, user 3 has device 1 and device 2, where device 1 is the currently selected device.

[0058] Table 300 includes a list of available applications in rows 330A-N. This may include all applications or a selected library of applications. For example, the applications may include application AN (also referred to as app AN), which is available for local installation on one or more of the devices utilized by the users in row 310. As shown, each of the users may not have all applications installed, but may choose to install a few applications on each device. In fact, for a user with multiple devices, each device may have a different set of applications installed on those devices. For example, the applications installed for user 1 include: app 1 and app 4 installed on device 1; app 1-2 and app 4-6 installed on device 2; and app 1 and app 3 installed on device 3. In addition, the applications installed for user 2 include: app 4 on device 1; and app 1 and app 6 on device 2. Further, the applications installed for the user include: app 3-4 on device 1, and app 2-4 and app 6 on device 2.

[0059] As shown in table 300, connection 340 (solid and bold lines) shows the relationship of applications to users, which can be determined by the supervisor's monitoring of devices. Specifically, connection 340 shows that app 1 is installed on the device of each of users 1, 2, and 3. That is, each of the users has app 1 installed on their corresponding device 1 (e.g., the selected or current device). As previously described, because users in the group who want to play in a multi-player session immediately install a specific application (e.g., a video game, etc.) on their current devices, the supervisor can recommend app 1 to the user group. That is, the supervisor can determine that app 1 can give the user group the "best experience" because each player can launch a local instance of app 1 (i.e., no player has to stream their game play from a cloud instance).

[0060] As shown in table 300, connection 350 (e.g., a dotted and bold line) can be used to show recommended applications for future time points. Specifically, the supervisor can use the corresponding AI models of the users in the group to predict that app 6 is installed on the device of each user in the user. For example, user 1 may have app 6 installed on device 2, and user 2 may have app 6 installed on device 2, and user 3 may also have app 6 installed on device 2. These devices may not be selected as current devices by any user. However, the corresponding AI model may indicate that users 1, 2, and 3 each use their corresponding device 2 during a common time period. For example, device 2 may be a game console that is active between 6 and 9 p.m. for each of users 1-3, or a game console that the user can access (e.g., at home). Since app 2 is installed on the corresponding device 2 of each user in users 1-3, the supervisor can recommend app 6 to the group including users 1-3 for entertainment in a multi-player session between 6 and 9 p.m.

[0061] Figure 4 1 is a diagram illustrating a supervisor 120 that facilitates transfers between devices of a user of an application as the user moves through a physical environment according to one embodiment of the present disclosure. As shown, the user is traveling along a geographic location route 140 to return to a home environment 160. For example, the user is traveling on bus No. 130.

[0062] If the Metaverse is considered an "always connected" experience, there may be times when a user wants to switch between devices during an experience (e.g., gaming of an application in single-player or multi-player mode). For example, perhaps a user is attending a virtual concert with other friends, each of whom is also virtually attending the virtual concert in a multi-player session. For example, the virtual concert is a two-hour experience. For the first hour, the user is attending the virtual concert while riding bus No. 130. That is, the user is watching and / or participating in the virtual concert via a mobile device (e.g., client device 110A). At some point in time, the user arrives at residence (arrives at residence environment 160) and wants to attend the second hour of the virtual concert from a game console or stand-alone computer (e.g., client device 110B).

[0063] As shown, throughout the user's journey along the geographic location route 140, the supervisor is connected to each device in the devices 110A and 110B through the network 150. When the user is within a certain proximity, or at a pre-programmed time, the supervisor 120 can "preheat" the console (client device 110B). For example, the supervisor 120 can determine that the user is close to the residential environment 160 (for example, the client device 110A currently used by the user is detected in the geographic area 145 adjacent to the residential environment 160), and take further action. The geographic area 145 may include a bus stop where the user usually gets off the bus on the way to the residential environment 160. When the user arrives and enters the geographic area 145 (for example, when the bus is at the stop), the supervisor can send a message to the game console (for example, the client device 110B) telling it to wake up, start the virtual concert and prepare for the user to join the session. Thus, when the user returns to the residential environment 160, the user can interact with the console (for example, the client device 110B) and see the interactive button 180. The user simply clicks a "Join Here" button on the console (e.g., associated with client device 110B), and the multiplayer session is seamlessly transferred to the user's new device (e.g., client device 110B). In this way, the user is able to seamlessly participate in the virtual concert because the multiplayer session of the virtual concert is transferred to client device 110B in the home environment 160 in a timely manner before the user returns to the home environment 160.

[0064] In another embodiment, the extension of the supervisor functionality 120 includes transferring the Bluetooth connection for the corresponding audio device. For example, when the user is traveling on a bus 130 and is using a client device 110A, the earbuds can be paired with a mobile phone (e.g., client device 110A). The supervisor 120 can coordinate the seamless transfer of the pairing of the same earbuds to the client device 110B in the residential environment 160, where the earbuds can be connected as auxiliary devices 170 via the client device 110B (e.g., console). In one embodiment, the "join from here" button 180 can be configured to also trigger the client device 110B (e.g., console) to connect to the earbuds. In this case, the supervisor interacts with the client device 110A (e.g., mobile phone) to determine which earbuds the user uses in advance (e.g., before Bluetooth pairing). The supervisor 120 delivers information about the earbuds to the console (e.g., client device 110B), so the "join from here" button 180 also triggers the plus action that connects the client device 110B to the correct earbuds.

[0065] In yet another embodiment, if the earbuds have not been previously paired with the client device 110B, the supervisor may coordinate a pre-pairing process. For example, the supervisor 120 may communicate pairing information from the client device 110A (e.g., mobile phone) to the client device 110B (e.g., console) and instruct the client device 110B to pre-pair the earbuds before the user arrives at the residential environment 160. For example, all the pairing information may be loaded into the client device 110B (e.g., console) so that when the user arrives at the residential environment 160, all the client device 110B needs to do is connect to the earbuds via Bluetooth or any other wireless connection format.

[0066] In another embodiment, the transfer between devices may be performed in the reverse direction. For example, the supervisor 120 may know that the user is leaving the residential environment 160, as predicted by the AI ​​model. For example, the user may leave for work at a specific time during the day and use the bus 130 to go to work. Thus, when the user is interacting with the application in the residential environment 160 but may be leaving the residential environment 160, the supervisor 120 may pre-launch or start the application on the client device 110A (e.g., a mobile phone). In another case, when the client device 110A does not have the application installed, the supervisor may pre-launch or start a cloud-based instance of the application so that the cloud-based instance is ready to stream the execution of the application to the client device 110A at the appropriate time (e.g., when the user instructs to transfer to the stream, or when entering an area 145 of the geographic route 145).

[0067] Figure 5Components of an example device 500 that can be used to perform aspects of various embodiments of the present disclosure are illustrated. This block diagram illustrates a device 500 that can be combined with or can be a personal computer, video game console, personal digital assistant, server, or other digital device suitable for practicing embodiments of the present disclosure. The device 500 includes a central processing unit (CPU) 502 for running software applications and optionally an operating system. The CPU 502 can be composed of one or more homogeneous or heterogeneous processing cores. For example, the CPU 502 is one or more general-purpose microprocessors with one or more processing cores. Further embodiments can be implemented using one or more CPUs with a microprocessor architecture that is particularly suitable for highly parallel and computationally intensive applications such as processing operations for interpreting queries, which identify context-related resources and immediately implement and render context-related resources in a video game. The device 500 can be localized to a player playing a game segment (e.g., a game console), or remote from the player (e.g., a back-end server processor), or is one of many servers in a game cloud system that uses virtualization to remotely stream game play to a client or to implement additional services such as supervisor functionality.

[0068] Specifically, the CPU 502 may be configured to implement a supervisor 120 having functionality configured to monitor computing devices of users and applications installed on those computing devices in order to facilitate and / or manage the initiation of a multiplayer session for multiple users (e.g., in a group) who wish to participate together in a virtual world and / or metaverse generated by a corresponding application (e.g., to participate in a multiplayer gaming session). For example, the supervisor 120 may launch local instances and / or cloud-based instances of an application for the users participating in the multiplayer session. In this manner, the initiation and establishment of a multiplayer session may occur automatically, without requiring user input other than a request and confirmation to play in the multiplayer session, and without waiting for any application to be installed to any user's local device.

[0069] Memory 504 stores applications and data for use by CPU 502. Storage 506 provides non-volatile storage and other computer-readable media for applications and data, and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD or other optical storage devices, as well as signal transmission and storage media. User input device 508 transmits user input from one or more users to device 500, examples of which may include keyboards, mice, joysticks, touch pads, touch screens, static or video recorders / cameras, tracking devices for gesture recognition, and / or microphones. Network interface 514 allows device 500 to communicate with other computer systems via an electronic communication network, and may include wired or wireless communications through local area networks and wide area networks (such as the Internet). Audio processor 512 is adapted to generate analog or digital audio output using instructions and / or data provided by CPU 502, memory 504, and / or storage device 506. The components of device 500 , including CPU 502 , memory 504 , data storage 506 , user input device 508 , network interface 510 , and audio processor 512 , are connected via one or more data buses 522 .

[0070] The graphics subsystem 520 is further connected to the data bus 522 and the components of the device 500. The graphics subsystem 520 includes a graphics processing unit (GPU) 516 and a graphics memory 518. The graphics memory 518 includes a display memory (e.g., a frame buffer) for storing pixel data for each pixel of an output image. The graphics memory 518 may be integrated with the GPU 516 in the same device, connected to the GPU 516 as a separate device, and / or implemented in the memory 504. The pixel data may be provided directly from the CPU 502 to the graphics memory 518. Alternatively, the CPU 502 may provide the GPU 516 with data and / or instructions defining the desired output image, and the GPU 516 generates pixel data for one or more output images according to the data and / or instructions. The data and / or instructions defining the desired output image may be stored in the memory 504 and / or the graphics memory 518. In one embodiment, the GPU 516 includes a 3D rendering capability to generate pixel data for an output image from instructions and data defining the geometry, lighting, shading, texture, motion, and / or camera parameters of a scene. The GPU 516 may further include one or more programmable execution units capable of executing shader programs. In one embodiment, the GPU 516 may be implemented within an AI engine (e.g., machine learning engine 190) to provide additional processing capabilities, such as for AI, machine learning functionality, or deep learning functionality.

[0071] Graphics subsystem 520 periodically outputs pixel data for an image to be displayed on display device 510 from graphics memory 518. Display device 510 may be any device capable of displaying visual information in response to a signal from device 500, including CRT, LCD, plasma, and OLED displays. For example, device 500 may provide analog or digital signals to display device 510.

[0072] In other embodiments, the graphics subsystem 520 includes multiple GPU devices that are combined to perform graphics processing for a single application executed on a corresponding CPU. For example, multiple GPUs may perform alternating forms of frame rendering, where GPU 1 renders the first frame, and GPU 2 renders the second frame, and so on, in sequential frame cycles, until the last GPU is reached, whereupon the initial GPU renders the next video frame (e.g., if there are only two GPUs, GPU 1 renders the third frame). That is, the GPUs rotate when rendering frames. Rendering operations may overlap, where GPU 2 may start rendering the second frame before GPU 1 finishes rendering the first frame. In another implementation, different shader operations in rendering and / or graphics pipelines may be assigned to multiple GPU devices. The master GPU performs master rendering and compositing. For example, in a group including three GPUs, master GPU 1 may perform a primary rendering (e.g., a first shader operation) and synthesize outputs from slave GPU 2 and slave GPU 3, where slave GPU 2 may perform a second shader (e.g., a fluid effect such as a river) operation, and slave GPU 3 may perform a third shader (e.g., a particle smoke) operation, where master GPU 1 synthesizes the results from each of GPU 1, GPU 2, and GPU 3. In this way, different GPUs may be assigned to perform different shader operations (e.g., flag waving, wind, smoke generation, fire, etc.) to render a video frame. In yet another embodiment, each of the three GPUs may be assigned to a different object and / or scene portion corresponding to a video frame. In the above embodiments and implementations, these operations may be performed in the same frame cycle (simultaneous parallelism) or in different frame cycles (sequential parallelism).

[0073] Thus, in various embodiments, the present disclosure describes systems and methods that are configured to monitor and / or oversee computing devices available to users and the applications installed on those devices to facilitate loading instances of applications locally or at a cloud-based streaming server for use by groups of users in multi-player sessions, and to build AI models of users that predict their gaming usage and behavior to recommend applications to groups of users at any time or at a specific point in time.

[0074] It should be noted that access services delivered over a wide geographic area (such as providing access to games of the current embodiment) typically use cloud computing. Cloud computing is a computing method in which dynamically scalable and usually virtualized resources are provided as services over the Internet. Users do not need to be experts in the technical infrastructure in the "cloud" that supports them. Cloud computing can be divided into different services, such as infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). Cloud computing services typically provide common applications (such as video games) accessed from a web browser online, while software and data are stored on servers in the cloud. Based on the way the Internet is depicted in computer network diagrams, the term cloud is used as a metaphor for the Internet and is an abstraction of the complex infrastructure that it hides.

[0075] In some embodiments, a game server can be used to perform the operation of a duration information platform for video game players. Most video games played over the Internet are operated by connection to a game server. Typically, games use a dedicated server application that collects data from players and distributes it to other players. In other embodiments, video games can be performed by a distributed game engine. In these embodiments, a distributed game engine can be performed on multiple processing entities (PEs) so that each PE performs a functional fragment of a given game engine on which a video game runs. The game engine simply regards each processing entity as a computing node. The game engine typically performs a series of operations with diverse functions to perform video game applications and additional services for user experience. For example, a game engine implements game logic, performs game calculations, physical effects, geometric transformations, rendering, lighting, shading, audio, and additional in-game or game-related services. Additional services may include, for example, messaging, social utilities, audio communications, game entertainment replay functions, help functions, etc. Although a game engine can sometimes be performed on an operating system virtualized by a hypervisor of a specific server, in other embodiments, the game engine itself is distributed between multiple processing entities, and each of the multiple processing entities can reside on different server units in a data center.

[0076] According to this embodiment, the corresponding processing entity used to perform the operation can be a server unit, a virtual machine, or a container, depending on the needs of each game engine fragment. For example, if a game engine fragment is responsible for camera transformations, a virtual machine associated with a graphics processing unit (GPU) can be provided to this particular game engine fragment because it will perform a large number of relatively simple mathematical operations (e.g., matrix transformations). Other game engine fragments that require fewer but more complex operations can be provided with processing entities associated with one or more higher-powered central processing units (CPUs).

[0077] By distributing the game engine, the game engine is provided with elastic computing properties that are not constrained by the capabilities of the physical server units. Instead, more or fewer computing nodes are provided to the game engine as needed to meet the needs of the video game. From the perspective of the video game and the video game player, a game engine distributed across multiple computing nodes is no different than a non-distributed game engine executed on a single processing entity, because the game engine manager or supervisor distributes the workload and seamlessly integrates the results to provide the video game output components to the end user.

[0078] The user uses a client device to access the remote service, and the client device includes at least a CPU, a display, and an I / O. The client device can be a PC, a mobile phone, a notebook computer, a PDA, etc. In one embodiment, the network identification client used on the game server performs and adjusts the communication method adopted. In other cases, the client device uses a standard communication method such as html to access the application on the game server through the Internet. It should be understood that a given video game or game application can be developed for a specific platform and a specific associated controller device. However, when such a game is made available via a game cloud system as presented herein, the user may access the video game with different controller devices. For example, a game may have been developed for a game console and its associated controller, and the user may access the cloud-based version of the game from a personal computer using a keyboard and mouse. In such scenarios, the input parameter configuration can define the mapping of the input generated from the available controller device (in this case, keyboard and mouse) of the user to the execution of the video game that is acceptable.

[0079] In another example, a user may access the cloud gaming system via a tablet computing device, a touch screen smart phone, or other touch screen driven device. In this case, the client device and the controller device are integrated together in the same device, where input is provided by means of a detected touch screen input / gesture. For such a device, an input parameter configuration may define a specific touch screen input corresponding to a game input of a video game. For example, during the running of a video game, a button, a directional key, or other type of input element may be displayed or overlaid to indicate a position on the touch screen where the user may touch to generate a game input. Gestures (such as swipes or specific touch movements in a particular direction) may also be detected as game inputs. In one embodiment, guidance may be provided to the user, indicating, for example, how to provide input for game play via a touch screen before starting the game play of a video game, so as to adapt the user to the operation of the controls on the touch screen.

[0080] In some embodiments, the client device is used as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection to transmit input from the controller device to the client device. The client device can then process these inputs and then transmit the input data to the cloud game server via a network (e.g., accessed via a local networking device such as a router). However, in other embodiments, the controller itself can be a networked device with the ability to directly communicate input to the cloud game server via a network without first communicating such input through a client device. For example, the controller can be connected to a local networking device (such as the above-mentioned router) to send data to and receive data from the cloud game server. Therefore, although the client device may still be required to receive video output from a cloud-based video game and render it on a local display, input latency can be reduced by allowing the controller to send input directly to the game cloud server over the network, thereby bypassing the client device.

[0081] In one embodiment, the networked controller and client device may be configured to send certain types of input directly from the controller to the cloud gaming server, and to send other types of input via the client device. For example, inputs whose detection does not depend on any additional hardware or processing other than the controller itself may be sent directly from the controller to the gaming cloud server via the network, thereby bypassing the client device. Such inputs may include button inputs, joystick inputs, embedded motion detection inputs (e.g., accelerometers, magnetometers, gyroscopes), etc. However, inputs that utilize additional hardware or need to be processed by the client device may be sent by the client device to the cloud gaming server. These may include video or audio captured from the gaming environment, which may be processed by the client device before being sent to the cloud gaming server. In addition, inputs from the motion detection hardware of the controller may be processed by the client device in conjunction with the captured video to detect the position and motion of the controller, which is then communicated to the cloud gaming server by the client device. It should be understood that the controller device according to various embodiments may also receive data (e.g., feedback data) from the client device or directly from the cloud gaming server.

[0082] The access of the client device to the cloud gaming network can be achieved through a communication network that implements one or more communication technologies. In some embodiments, the network may include a 5th generation (5G) network technology with an advanced wireless communication system. 5G is the fifth generation of cellular network technology. The 5G network is a digital cellular network in which the service area covered by the provider is divided into small geographical areas called cells. The analog signals representing sound and images are digitized in the phone, converted by an analog-to-digital converter and transmitted as a bit stream. All 5G wireless devices in the cell communicate with the local antenna array and low-power automatic transceiver (transmitter and receiver) in the cell through radio waves on the frequency channel allocated by the transceiver from the frequency pool reused in other cells. The local antenna is connected to the telephone network and the Internet through a high-bandwidth optical fiber or wireless backhaul connection. As in other cell networks, a mobile device that crosses from one cell to another automatically transfers to the new cell. It should be understood that the 5G network is only an exemplary type of communication network, and the embodiments of the present disclosure can utilize earlier generations of wireless or wired communications, as well as later generations of wired or wireless technologies that appear after 5G.

[0083] In one embodiment, various technical examples can be implemented using a virtual environment through a head-mounted display (HMD). The HMD may also be referred to as a virtual reality (VR) head-mounted device. As used herein, the term "virtual reality" (VR) generally refers to the interaction of a user with a virtual space / environment, which involves viewing the virtual space in a real-time response to the movement of the HMD (controlled by the user) through the HMD (or VR head-mounted device) to provide the user with a feeling in a virtual space or metaverse. For example, a user can see a three-dimensional (3D) view of the virtual space when facing a given direction, and when the user turns to one side and thus turns the HMD in the same manner, the view of the side in the virtual space is rendered on the HMD. The HMD can be worn in a manner similar to glasses, goggles, or a helmet, and is configured to display video games or other metaverse content to the user. The HMD can provide a very immersive experience to the user by means of its display mechanism being close to the user's eyes. Therefore, the HMD can provide a display area that occupies most of the user's or even the entire field of view to each eye of the user, and can also provide viewing with three-dimensional depth and perspective.

[0084] In one embodiment, the HMD may include a gaze tracking camera configured to capture images of the user's eyes as the user interacts with the VR scene. The gaze information captured by the gaze tracking camera may include information related to the user's gaze direction and specific virtual objects and content items in the VR scene that the user is paying attention to or interested in interacting with. Thus, based on the user's gaze direction, the system can detect specific virtual objects and content items that may be of potential attention to the user (where the user is interested in interacting with and engaging with them), such as game characters, game objects, game props, etc.

[0085] In some embodiments, the HMD may include an externally facing camera configured to capture images of the user's real-world space, such as the user's body movements and any real-world objects that may be located in the real-world space. In some embodiments, the images captured by the external camera may be analyzed to determine the position / orientation of the real-world objects relative to the HMD. Using the known position / orientation of the HMD, real-world objects, and inertial sensor data from the user's posture and movement, the user's posture and movement during the user's interaction with the VR scene can be continuously monitored and tracked. For example, when interacting with a scene in a game, a user may make various gestures, such as pointing and walking toward a specific content item in the scene. In one embodiment, the gestures may be tracked and processed by the system to generate predictions of interactions with specific content items in the game scene. In some embodiments, machine learning may be used to facilitate or assist predictions.

[0086] During the use of the HMD, various single-hand and two-hand controllers can be used. In some implementations, the controller itself can be tracked by tracking the lights included in the controller or tracking the shapes, sensors and inertial data associated with the controller. Using these different types of controllers or even just using gestures made and captured by one or more cameras, the virtual reality environment or metaverse rendered on the HMD can be interfaced, controlled, manipulated, interacted with and participated in the virtual reality environment or metaverse. In some cases, the HMD can be wirelessly connected to the cloud computing and gaming system through a network. In one embodiment, the cloud computing and gaming system maintain and execute the video game played by the user. In some embodiments, the cloud computing and gaming system are configured to receive input from the HMD and interface objects through the network. The cloud computing and gaming system are configured to process input to affect the game state of the video game being executed. Output (such as video data, audio data and tactile feedback data) from the video game being executed is transmitted to the HMD and the interface object. In other implementations, the HMD can communicate wirelessly with the cloud computing and gaming system through an alternative mechanism or channel such as a cellular network.

[0087] In addition, although the implementations in the present disclosure may be described with reference to a head-mounted display, it should be understood that in other implementations, a non-head-mounted display may be substituted, including but not limited to a portable device screen (e.g., a tablet, a smart phone, a laptop, etc.) or any other type of display that can be configured to render video and / or provide a display of an interactive scene or virtual environment according to the implementation of the present invention. It should be understood that the various embodiments defined herein may be combined or assembled into a specific implementation using the various features disclosed herein. Therefore, the examples provided are only some possible examples and are not limited to various implementations that can define more implementations by combining various elements. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0088] Embodiments of the present disclosure may be practiced with various computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through wired or wireless networks.

[0089] Although the method operations are described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be adjusted so that they occur at slightly different times, or the operations may be distributed in a system that allows processing operations to occur at various intervals associated with the processing as long as the processing of telemetry and game state data used to generate the modified game state is performed in the desired manner.

[0090] In view of the above embodiments, it should be understood that the embodiments of the present disclosure may employ various computer-implemented operations involving data stored in a computer system. These operations are those operations that require physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments of the present disclosure are useful machine operations. The embodiments of the present disclosure also relate to devices or equipment for performing these operations. The device may be specially constructed for the desired purpose, or the device may be a general-purpose computer selectively activated or configured by a computer program stored in a computer. Specifically, various general-purpose machines may be used together with computer programs written according to the teachings of this article, or it may be more convenient to construct a more specialized device to perform the desired operations.

[0091] One or more embodiments may also be manufactured as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data that can then be read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read-only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical and non-optical data storage devices. A computer readable medium may include a computer readable tangible medium distributed on a network coupled computer system so that the computer readable code is stored and executed in a distributed manner.

[0092] In one embodiment, the video game is executed locally on a game console, a personal computer, or on a server. In some cases, the video game is executed by one or more servers in a data center. When executing the video game, some instances of the video game may be simulations of the video game. For example, the video game may be executed by an environment or server that generates a simulation of the video game. In some embodiments, the simulation is an instance of the video game. In other embodiments, the simulation may be produced by an emulator. In either case, if the video game is represented as a simulation, the simulation may be executed to render interactive content that may be interactively streamed, executed, and / or controlled by a user input.

[0093] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, it is apparent that certain changes and modifications may be practiced within the scope of the appended claims. Therefore, the present embodiments are to be considered illustrative rather than restrictive, and the embodiments are not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.

Claims

1. A method comprising: receiving, over a network, a request to establish a multiplayer session for a plurality of users to allow participation in the metaverse; determining whether an application configured to generate the metaverse is installed on a plurality of local devices of the plurality of users selected by the plurality of users to participate in the metaverse; launching a corresponding local instance of the application on the corresponding local device when the application is installed on the corresponding local device; launching a corresponding cloud instance of the application on a cloud-based streaming media server when the application is not installed on the corresponding local device; determining that each of a plurality of instances of the application for the plurality of users has been launched, wherein each of the plurality of instances is a local instance or a cloud instance; as well as The multi-player session is allowed to begin when each of the plurality of instances of the application for the plurality of users has been launched.

2. The method of claim 1 , wherein allowing the multi-player session to begin comprises: The cloud-based streaming server is notified that each of the multiple instances of the application for the multiple users has been launched, wherein the cloud-based streaming server supports the multi-player session.

3. The method of claim 1 , wherein launching the corresponding local instance of the application comprises: determining that the application is installed on a first local device of a first user, wherein the first local device is selected by the first user to participate in the metaverse; as well as A first local instance of the application is launched on the first local device for the first user to participate in the metaverse for the first time.

4. The method of claim 1 , wherein launching the corresponding cloud instance of the application comprises: determining that the application is not installed on a first local device of the first user; as well as A first cloud instance of the application is launched on the cloud-based streaming server for the first user to participate in the metaverse for the first time.

5. The method of claim 4, further comprising: Sending, via the first local device, a request to the first user to install the application on the first local device; obtain authorization to perform said installation; as well as Instruct the first local device to download the application.

6. The method of claim 1, wherein the application is a video game.

7. The method of claim 1, further comprising: A plurality of applications installed on a plurality of devices available to the plurality of users are monitored.

8. The method of claim 1, further comprising: periodically pinging each local device of the plurality of local devices; determining that a connection to a first local device of a first user is terminated during the multi-player session, wherein the first local device is used by the first user to participate in the multi-player session for the first time; as well as A notification is provided to each of the plurality of local devices connected via the network that the connection to the first user is experiencing a connection problem.

9. The method of claim 1, further comprising: receiving an initial request for a coordinated interaction between the plurality of users; determining a plurality of applications installed on the plurality of local devices of the plurality of users; determining that the application is installed on each local device of the plurality of local devices; as well as A response to the initial request is provided to each of the plurality of local devices, the response recommending an application configured to generate the metaverse for the coordinated interaction.

10. The method according to claim 1, wherein the first local device executes a first operating system, and The second local device executes a second operating system.

11. A non-transitory computer-readable medium storing a computer program for executing a method, the computer-readable medium comprising: program instructions for receiving, over a network, a request to establish a multiplayer session for a plurality of users to allow participation in the Metaverse; program instructions for determining whether an application configured to generate the metaverse is installed on a plurality of local devices of the plurality of users selected by the plurality of users to participate in the metaverse; program instructions for launching a corresponding local instance of the application on the corresponding local device when the application is installed on the corresponding local device; program instructions for launching a corresponding cloud instance of the application on a cloud-based streaming media server when the application is not installed on the corresponding local device; program instructions for determining that each of a plurality of instances of the application for the plurality of users has been launched, wherein each of the plurality of instances is a local instance or a cloud instance; as well as Program instructions for allowing the multi-player session to begin when each of the plurality of instances of the application for the plurality of users has been launched.

12. The non-transitory computer-readable medium of claim 11, wherein the program instructions that allow the multi-player session to begin include: Program instructions for notifying the cloud-based streaming server that each of the plurality of instances of the application for the plurality of users has been launched, wherein the cloud-based streaming server supports the multi-player session.

13. The non-transitory computer-readable medium of claim 11, wherein the program instructions for launching the corresponding local instance of the application program include: program instructions for determining that the application is installed on a first local device of a first user, wherein the first local device is selected by the first user to participate in the Metaverse; as well as Program instructions for launching a first local instance of the application on the first local device for the first user to participate in the metaverse for the first time.

14. The non-transitory computer-readable medium of claim 11, wherein the program instructions for launching the corresponding cloud instance of the application program include: program instructions for determining that the application is not installed on a first local device of a first user; as well as Program instructions for launching a first cloud instance of the application on the cloud-based streaming server for the first user to participate in the metaverse for the first time.

15. The non-transitory computer-readable medium of claim 11, wherein the application is a video game.

16. The non-transitory computer readable medium of claim 11, further comprising: Program instructions for monitoring a plurality of applications installed on a plurality of devices available to the plurality of users.

17. A computer system comprising: processor; a memory coupled to the processor and having instructions stored therein that, if executed by the computer system, cause the computer system to perform a method comprising: receiving, over a network, a request to establish a multiplayer session for a plurality of users to allow participation in the metaverse; determining whether an application configured to generate the metaverse is installed on a plurality of local devices of the plurality of users selected by the plurality of users to participate in the metaverse; launching a corresponding local instance of the application on the corresponding local device when the application is installed on the corresponding local device; launching a corresponding cloud instance of the application on a cloud-based streaming media server when the application is not installed on the corresponding local device; Determining that each of a plurality of instances of the application for the plurality of users has been launched, wherein each of the plurality of instances is a local instance or a cloud instance; and The multi-player session is allowed to begin when each of the plurality of instances of the application for the plurality of users has been launched.

18. The computer system of claim 17, wherein in the method, allowing the multi-player session to begin comprises: The cloud-based streaming server is notified that each of the multiple instances of the application for the multiple users has been launched, wherein the cloud-based streaming server supports the multi-player session.

19. The computer system of claim 17, wherein in the method, launching the corresponding local instance of the application comprises: determining that the application is installed on a first local device of a first user, wherein the first local device is selected by the first user to participate in the metaverse; as well as A first local instance of the application is launched on the first local device for the first user to participate in the metaverse for the first time.

20. The computer system of claim 17, wherein in the method, launching the corresponding cloud instance of the application comprises: determining that the application is not installed on a first local device of the first user; as well as A first cloud instance of the application is launched on the cloud-based streaming server for the first user to participate in the metaverse for the first time.

Images