In-game location-based game play companion application

Abstract

To provide a method and a system of a location based companion application for supporting a user's game play.SOLUTION: A method includes reception of location-based information of game play of a user who plays a game application displayed on a first computing device, and the location-based information is created by referring to a location within the user's game play in a game world correlated with the game application. The method includes integration of location-based information of multiple pieces of game play of multiple users who play a game application. The method includes generation of context-related information of a character's location on the basis of location-based information of multiple pieces of game play. The method includes generation of a companion interface including context-related information.SELECTED DRAWING: Figure 5E

Description

Technical Field

[0001] The present disclosure relates to video games. In particular, the present disclosure describes a method and system for a location-based companion application that assists a user's gameplay in playing a game application.

Background Art

[0002] A cloud-based system that uses computing resources (hardware and software) delivers services via a network (e.g., the Internet). In games, this service enables the streaming of content to remote clients, with most processing being performed on the server and being distributable. In some cases, video games are executed within a cloud-based system, and the input provided by a remote client similarly drives the execution of the video game without requiring dedicated game hardware at the client's location. Thus, it has become easier for users to access more video game titles without the limitations of complex hardware, and it has become easier for game suppliers to manage game code from a centralized location, so the popularity of cloud-based processing and games is increasing. Also, due to the higher processing power of cloud servers, video games are becoming more complex and expansive.

[0003] However, as video games become more complex, it can become increasingly difficult for a user to progress and / or complete a video game. For example, a video game may become more extensive and include options that are available to millions, or even trillions, of users. Thus, it may be impossible for a user to even try each of these available options or even understand which of these options are available. In another scenario, a user may reach a point within the video game that seems impossible to progress or solve. This happens quite frequently to users, and the users gradually become more dissatisfied with the game due to its difficulty or inability to progress, and ultimately quit the video game. For example, in the gameplay of a video game, a user may attempt to reach another part of the game world through an obvious passageway in order to progress the gameplay. Illustratively, the user may need to pass through a waterfall that functions as a passageway and seems to have exhausted all the options available to attempt to pass through. However, a solution is available and would be simple if known (e.g., lift a stabilizing rock to pass through the waterfall), but for some reason, the user is unable to find the correct passageway or sequence of actions to enter. After spending useless time, the user may ultimately quit the game because they are unable to find a way to move forward.

[0004] In such situations, embodiments of the present disclosure arise. SUMMARY OF THE INVENTION

[0005] Embodiments of the present disclosure relate to a location-based companion interface that assists in the gameplay of a user playing a game application. Some inventive embodiments of the present disclosure are described below.

[0006] In one embodiment, a game method is disclosed. The method includes receiving location-based information of a user's game play of a game application displayed on a first computing device, the location-based information being created with reference to the position of a character within the user's game play in a game world associated with the game application. The method includes receiving and aggregating location-based information of a plurality of game plays of a plurality of users playing the game application. The method includes generating context-related information of the character's position based on the location-based information of the plurality of game plays. The method includes generating a companion interface including the context-related information. The method includes transmitting the companion interface to a second computing device associated with the user for display simultaneously with the user's game play.

[0007] In another embodiment, a game method is disclosed. The method includes instantiating a first instance of a game application associated with a user's game play. The method includes delivering data representing the user's game play via a first communication channel to a first computing device for interaction by the user. The method includes identifying location-based information of a character within the user's game play, the location information being created with reference to a game world associated with the game application. The method includes aggregating location-based information of a plurality of game plays of a plurality of users playing the game application. The method includes generating context-related information of the character's position based on the location-based information of the plurality of game plays. The method includes generating a companion interface including the context-related information. The method includes transmitting the companion interface via a second communication channel to a second computing device associated with the user for display simultaneously with the user's game play.

[0008] In yet another embodiment, another gaming method is disclosed. The method includes instantiating a first instance of a game application associated with a user's game play. The method includes delivering data representative of the user's game play to a first computing device via a first communication channel for interaction by the user. The method includes identifying position-based information of a character within the user's game play, the position information being created with reference to a game world associated with the game application. The method includes generating context-related information of the character's position based on the position-based information, the context-related information including an offer to complete a task within the user's game play. The method includes generating a companion interface including the context-related information. The method includes transmitting the companion interface to a second computing device associated with the user via a second communication channel for display concurrently with the user's game play. The method includes receiving acceptance of the offer. The method includes instantiating a second instance of the game application in a first jump game play based on a first snapshot generated during execution of the first instance. The first snapshot includes a first game state of the first instance and the position-based information, and the first jump game play starts from a location within the game application corresponding to the first snapshot. The method includes generating a second snapshot including a second game state of the second instance during execution of the second instance of the game application. The second instance is controlled by an expert and the task is completed in the first jump game play. The method includes instantiating a third instance of the game application in a second jump game play based on the second snapshot, the third instance being controlled by the user and the task being completed in the second jump game play.

[0009] In another embodiment, a non-transitory computer-readable medium storing a computer program for a game is disclosed. The computer-readable medium includes program instructions for receiving location-based information of a user's game play for playing a game application displayed on a first computing device, where the location-based information is created with reference to the position of a character within the user's game play in a game world associated with the game application. The computer-readable medium includes program instructions for aggregating location-based information of a plurality of game plays of a plurality of users playing the game application. The computer-readable medium includes program instructions for generating context-related information of the character's position based on the location-based information of the plurality of game plays. The computer-readable medium includes program instructions for generating a companion interface including the context-related information. The computer-readable medium includes program instructions for transmitting the companion interface to a second computing device associated with the user for display simultaneously with the user's game play.

[0010] Other aspects of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate embodiments of the principles of the present disclosure.

Brief Description of the Drawings

[0011] The present disclosure may be best understood by reference to the following description, taken in conjunction with the accompanying drawings.

[0012]

Figure 1A

[0013]

Figure 1B

[0014]

Figure 1C

[0015]

Figure 2

[0016]

Figure 3

[0017]

Figure 4A

[0018]

Figure 4B

[0019]

Figure 4C

[0020]

Figure 5A

[0021]

Figure 5B

[0022]

Figure 5C

[0023]

Figure 5D

[0024]

Figure 5E

[0025]

Figure 6

[0026]

Figure 7

[0027]

Figure 8

DETAILED DESCRIPTION OF THE INVENTION

[0028] The following detailed description includes a number of specific details for illustrative purposes, but any person skilled in the art will understand that many variations and modifications of the following details are within the scope of the present disclosure. Accordingly, the aspects of the present disclosure described below are set forth without losing the generality of the claims that follow this description and without imposing limitations on the claims.

[0029] Generally, various embodiments of the present disclosure describe systems and methods that implement a location-based companion interface configured to assist a user's gameplay. Embodiments of the present disclosure provide additional uses of a game application through the location-based companion interface. The companion interface includes context-related information (e.g., message communication, auxiliary information, etc.) generated based on the position of a character within the user's gameplay. The location-based information includes defined parameters generated for snapshots that are periodically collected during the user's gameplay. Specifically, the snapshot includes metadata and / or information regarding the user's gameplay, and at the jump location within the game application corresponding to the snapshot, another instance of the corresponding game application can be configured to be made available. The context-related information also includes information collected during the gameplay of other users playing the same game application. In this way, the user can receive context-related information based on the user's current progress (e.g., position in the game world, etc.). For example, the context-related information can provide assistance in the user's gameplay, and the information can be based on the gameplay position, past gameplay, and expected gameplay. Further, messages can be created using the companion interface. For example, the message can be a request for help, and the message can be delivered to the target user or broadcast to a defined group (e.g., the user's friends) and can be displayed within the corresponding companion interface. The companion interface can include messages to the user created by other users playing the game application. In other examples, the gameplay can be improved by creating and delivering messages to one or more target users. For example, when an event such as reaching a geographical location within the game world or defeating a boss occurs, the message can be triggered to be displayed on the companion interface of the target user.Messages may also be set personally, enabling person-to-person games or user-to-user communication. That is, the game application may be used in a multiplayer environment via the corresponding companion interface, allowing real-time communication between two or more users playing the game application, and each companion interface provides information related to the game play of other users. The context-related information may include an offer of assistance by an expert to guide the user through the user's game play, or an offer of assistance by an expert to accomplish tasks within the user's game play.

[0030] With the above general understanding of the various embodiments, exemplary details of the embodiments are now described with reference to the various drawings.

[0031] Throughout this specification, references to "game application" are intended to represent any type of interactive application directed by the execution of input commands. For illustrative purposes only, interactive applications include applications such as games, word processing, video processing, video game processing, etc. Further, the terms video game and game application are interchangeable.

[0032] FIG. 1A illustrates a system 10 used to implement a location-based companion interface configured to assist the game play of a user playing a game application, where the game application can be executed on a local computing device or via a cloud game network. The companion interface can be used to create content (such as auxiliary information, messages, etc.) for interaction by other users playing the game application, and the interaction can be via the corresponding companion interface. The content is created based on location-based information captured during the game play of the user playing the game application, such as snapshot information. The companion interface can also be configured to assist the user's game play by providing context-related information to the user based on the user's past game play, the aggregated game play of multiple users playing the same game application, the location information of the user's game play, and the current progress of the user's game play (including property and skill accumulation, task completion, level completion, etc.).

[0033] As shown in FIG. 1A, the game application can be executed locally on the client device 100 of user 5 or by a backend game execution engine 211 operating on a backend game server 205 of a cloud game network or a game cloud system. The game execution engine 211 can operate within one of the multiple game processors 201 of the game server 205. In either case, the cloud game network is configured to provide a location-based companion interface to assist the game play of one or more users playing the game application. Further, the game application can be executed in single-player mode or multiplayer mode, and embodiments of the present invention provide multiplayer enhancements (such as assistance, communication, etc.) for both operating modes.

[0034] In some embodiments, the cloud game network may include a plurality of virtual machines (VMs) operating on a hypervisor of a host machine, and one or more of the virtual machines are configured to execute a game processor module 201 that utilizes the hardware resources available to the host hypervisor when assisting single-player or multiplayer video games. In other embodiments, the cloud game network is configured to support a plurality of local computing devices that support a plurality of users, and each local computing device may execute an instance of a video game, such as within a single-player or multiplayer video game. For example, in multiplayer mode, while the video game is being executed locally, the cloud game network can simultaneously receive information (e.g., game state data) from each local computing device and appropriately distribute that information across one or more of the local computing devices, whereby each user can interact with other users in the game environment of the multiplayer video game (e.g., via the corresponding characters within the video game). In this way, the cloud game network coordinates and combines the gameplay of each user within the multiplayer game environment.

[0035] As shown, system 10 includes a game server 205 that executes a game processor module 201 that provides access to a plurality of interactive game applications. The game server 205 can be any type of server computing device available within the cloud and can be configured as one or more virtual machines executed on one or more hosts as described above. For example, the game server 205 can manage a virtual machine that supports the game processor 201. The game server 205 is also configured to provide additional services and / or content to user 5. For example, the game server can be configured to provide a companion interface viewable by user 5 to generate and / or receive context-related information, as will be described further below.

[0036] The client device 100 is configured to request access to a game application via a network 150 such as the Internet, and to render an instance of a video game or game application that is executed by the game server 205 and delivered to the display device 12 associated with the user 5. For example, the user 5 can interact with an instance of the game application executed on the game processor 201 through the client device 100. The client device 100 may also include a game execution engine 111 configured for local execution of the game application as described above. The client device 100 can receive inputs from various types of input devices such as the game controller 6, the tablet computer 11, the keyboard, and gestures captured by a video camera, a mouse, a touchpad, etc. The client device 100 can be any type of computing device having at least a memory and a processor module capable of connecting to the game server 205 via the network 150. Some examples of the client device 100 include a personal computer (PC), a game console, a home theater device, a general-purpose computer, a mobile computing device, a tablet, a phone, and any other type of computing device capable of interacting with the game server 205 to execute an instance of a video game.

[0037] The client device 100 is configured to receive the rendered image and display the rendered image on the display 12. For example, through a cloud-based service, the rendered image can be delivered by an instance of a game application executed on the game execution engine 211 of the game server 205 associated with the user 5. In another embodiment, through local game processing, the rendered image can be delivered by the local game execution engine 111. In either case, the client device 100 is configured to interact with the execution engine 211 or 111 associated with the game play of the user 5, such as via input commands used to operate the game play.

[0038] Furthermore, the client device 100 is configured to interact with the game server 205 to capture and store a snapshot of the game play of user 5 when playing a game application, and each snapshot includes information related to the game play (such as the game state, etc.). For example, the snapshot may include position-based information corresponding to the positions of the characters in the game world of user 5's game play. Further, the snapshot enables the corresponding user to jump into the saved game play at the jump location in the game application corresponding to the capture of the snapshot. Thus, user 5 can jump into their own game play saved at the jump location corresponding to the selected snapshot, another user may jump into user 5's game play, or user 5 may jump into the game play of another user saved at the jump location corresponding to the selected snapshot. Furthermore, the client device 100 is configured to interact with the game server 205 to display a location-based companion interface from the companion interface generator 213, and the companion interface is configured to receive and / or generate context-related content such as auxiliary information, message communication, interactive quests, and tasks. Specifically, the information included in the snapshots captured during user 5's game play, such as position-based information related to the game play, as well as the information captured during the game play of other users, is used to generate context-related content.

[0039] More specifically, the game processor 201 of the game server 205 is configured to generate and / or receive a snapshot of the game play of the user 5 when playing a game application. For example, the snapshot may be generated and output by the local game execution engine 111 on the client device 100 and distributed to the game processor 201 via the network 150. Further, the snapshot may be generated by the game execution engine 211 within the game processor 201, such as by an instance of the game application running on the engine 211. Further, other game processors of the game server 205 associated with other virtual machines are configured to execute instances of game applications associated with the game play of other users, capture snapshots between these game plays, and use this additional information to create context-related information.

[0040] The snapshot generator 212 is configured to capture a plurality of snapshots generated from the gameplay of user 5. Each snapshot provides information that enables the start of an instance of the video game from a location within the video game associated with the corresponding snapshot. Snapshots are automatically generated during the gameplay of the game application by user 5. In embodiments, portions of each snapshot are stored in a related database configured independently or in a related database configured under the data store 140. In another embodiment, snapshots can be manually generated via an instruction from user 5. In this way, any user can jump into the gameplay of user 5 at a location within the game application associated with the corresponding snapshot through the selection of the corresponding snapshot. Additionally, snapshots of the gameplay of other users who play multiple game applications can also be captured. Thus, the game processor 201 is configured to access the information in the database 140 to enable jumping to the saved gameplay of any user based on the corresponding snapshot. That is, the requesting user can start playing the video game from the jump location corresponding to the selected snapshot using the game character of the original user who generated and saved the snapshot.

[0041] A detailed discussion regarding the creation and specifications of snapshots is provided in U.S. Patent Application No. 15 / 411,421, entitled "Method And System For Saving A Snapshot of Game Play And Used To Begin Later Execution Of The Game Play By Any User As Executed On A Game Cloud System", which is hereby incorporated by reference in its entirety. The following is a brief description of the creation and embodiments of snapshots.

[0042] Specifically, each snapshot includes metadata and / or information that enables the execution of the start of an instance of the game application from a location within the game application corresponding to the snapshot. For example, in the game play of user 5, at a specific location during the progress of the game application, such as in the middle of a level, a snapshot can be generated. The related snapshot information is stored in one or more databases of database 140. Pointers can be used to associate information with each database corresponding to a specific snapshot. In this way, another user who wants to experience the game play of user 5, or the same user 5 who wants to experience their previous game play again, may select a snapshot corresponding to a location within the game application of interest.

[0043] The metadata and information within each snapshot can provide and / or be analyzed to provide additional information related to the user's game play. For example, the snapshot can help identify where the user (e.g., the user's character) was within the game application, where the user is within the game application, what the user did, the property and skills the user accumulated, and where the user is trying to go within the game application. This additional information can be used to generate context-related content (e.g., auxiliary information) to be displayed to the user at the companion application interface, or this additional information can be used to generate content (e.g., quests, tasks, messages, etc.) configured for interaction with other users. Further, the context-related content can be generated based on other information (e.g., location-based information) collected from the other game plays of other users playing the game application.

[0044] A snapshot includes a snapshot image of the scenery rendered at that location. The snapshot image is stored in the snapshot image database 146. The snapshot images presented in a time-series thumbnail format provide a view of the user's gameplay at the corresponding location where the user is progressing through the video game.

[0045] More specifically, the snapshot also includes game state data that defines the state of the game at that location. For example, the game state data may include game characters, game objects, game object attributes, game attributes, game object states, graphic overlays, and the like. In this way, the game state data enables the generation of the game environment that existed at the corresponding location within the video game. The game state data may also include the states of the CPU, GPU, and memory, register values, program counter values, programmable DMA states, DMA-buffered data, audio chip states, CD-ROM states, and the like, i.e., the states of all devices used to render the gameplay. The game state data may also identify which portions of the executable code need to be loaded in order to execute the video game from that location. It is not necessary to capture and store all of the game state data; only the data sufficient for the executable code to start the game from the location corresponding to the snapshot needs to be captured and stored. The game state data is stored in the game state database 145.

[0046] The snapshot also includes user - saved data. Generally, user - saved data includes information for personally configuring the user's video game. Since this includes information associated with the user's character, the video game is rendered with a character (e.g., shape, appearance, clothing, weapons, etc.) that can be unique to that user. In this way, the user - saved data enables the generation of a character for the corresponding user's gameplay, and the character has a state corresponding to the location within the video game associated with the snapshot. For example, the user - saved data may include the game difficulty level, game level, character attributes, character position, remaining lives, total possible number of available lives, protective gear, trophies, time counter value, and other property information selected by user 5 when playing the game. The user - saved data may also include, for example, user profile data for identifying user 5. The user - saved data is stored in database 141.

[0047] Furthermore, the snapshot also includes random seed data generated by the artificial intelligence (AI) module 215. The random seed data may not be part of the original game code, but by adding it as an overlay, the game environment can become more realistic and / or attractive to the user. That is, the random seed data provides additional features to the game environment existing at the corresponding location within the user's gameplay. For example, AI characters can be randomly generated and provided as an overlay. The AI characters are not associated with any particular user playing the game, but are placed within the game environment to improve the user experience. By way of illustration, these AI characters can randomly walk along the streets within the cityscape. Additionally, other objects can also be generated and presented as an overlay. For example, background clouds and birds flying in the sky can be generated and presented as an overlay. The random seed data is stored in the random seed database 143.

[0048] In this way, another user who wants to experience the gameplay of user 5 can select a snapshot corresponding to a location within the video game of interest. For example, by the user selecting a snapshot image presented in chronological order or a node within a node graph, the jump execution engine 216 of the game processor 201 can access the corresponding snapshot, instantiate another instance of the video game based on the snapshot, and start the execution of the video game from the location within the video game corresponding to the snapshot. In this way, the snapshot enables the requesting user to jump into the gameplay of user 5 at the location corresponding to the snapshot. Further, user 5 may access the gameplay of other users, or even use the corresponding snapshot to access their previous gameplay within the same or another game application. Specifically, by user 5 selecting a snapshot (e.g., from chronological order or via a message), the execution engine 216 can collect the snapshot (e.g., metadata and / or information) from various databases (e.g., from database 140) in order to start executing the corresponding game application from the location where the corresponding snapshot was captured within the game application.

[0049] The game processor 201 includes a location-based companion application generator 213 configured to generate a companion interface that assists the game play of user 5 when playing a game application. Using the generator 213, context-related information (such as auxiliary information, messages, etc.) to be delivered to user 5 or received from user 5 based on the game play of user 5 can be created, and the context-related information is created using location-based information (such as a snapshot). The context-related information can also be based on information collected from the game play of other users playing the game application. Specifically, the generator 213 identifies the progress of the game play of user 5 of a specific game application with respect to a specific context of the game play (such as the current position of the character, game state information, etc.) (for example, based on a snapshot), and is configurable to identify context-related information that may be delivered to a companion interface that can be displayed on a device 11 separate from the device that displays the game play of user 5. For example, the context-related information may provide information that provides assistance in progressing the game application. The context-related information takes into account information provided by a prediction engine 214 configured to predict where the game play of user 5 will go, and may include areas visited by the character, tasks required to advance the game play, properties required to advance the game play (such as properties required to complete the required tasks), etc. Using the companion interface, context-related information by user 5 may be created for interactions by other users. For example, the location-based information (such as radar mapping, waypoints, etc.) displayed on the companion interface may facilitate the creation of interactive content (such as quests, tasks, messages, etc.). That is, user 5 may create interactive context-related content using location-based information (such as a snapshot).In another embodiment, the companion interface can be used to create a message from the user, and the message can be directed to asking a user's friends for help or facilitating interactive communication (such as games with multiple players, teasing, etc.). The companion interface can be used to display messages from other users. The companion interface can be used to display offers of assistance.

[0050] For example, in an embodiment, the location-based information can be obtained based on the current and / or past game play of a plurality of users playing the same game application within a crowdsourcing environment, and thus the information can be determined through the observation and / or analysis of a plurality of game plays. In this way, crowdsourced content may be discovered during game play, and the content may be useful to other players playing the same game application or may provide an improved user experience to these other players. In another embodiment, the information provided within the companion interface may be related to the game play of users playing the same game application simultaneously (e.g., the information may be related to the game play of a friend of a user playing the game application simultaneously, and the information provides real-time interaction between friends), and the information advances the user's game play or provides an improved user experience. That is, each companion interface provides real-time interaction between users. In yet another embodiment, the user plays the game application in isolation (e.g., plays alone) and receives, via the companion interface, information that helps advance the game play of the first user or provides an improved user experience. In the case of single-player play, the information (e.g., help, coaching, etc.) can be delivered in the form of recorded video, images, and / or text content. This information is not generated in real time and can be generated by a person associated with a game developer (e.g., a game developer employed by a game software maker, a game console maker, etc.) that plays a role in generating information that benefits the user and is transmitted via each companion interface. Further, a third party (for-profit, non-profit, etc.) can undertake the role of generating information that benefits the public playing the game. Further, the information may be generated by a friend of the first user for the benefit of only that player or for the benefit of only the friends within the group. Also, the information may be generated by other related and / or unrelated users playing the game interface (e.g., a cloud-sourced game interface).Furthermore, the components used in embodiments of the companion interface may be included within the user's local game processor (e.g., disposed on device 11 or client device 100), or may be included within a backend server (e.g., game server 2015).

[0051] As shown, the companion interface is delivered to device 11 (e.g., a tablet) for display and interaction, and device 11 may be separate from client device 100 configured to execute and / or assist in executing a game application for user 5's interaction. For example, a first communication channel may be established between game server 205 and client device 100, and a separate second communication channel may be established between game server 205 and device 11.

[0052] FIG. 1B illustrates a system 106B that provides game control to one or more users playing one or more game applications executed locally by the corresponding user, according to an embodiment of the present disclosure, where backend server support (e.g., accessible via game server 205) may implement a location-based companion interface that assists the corresponding user's game play. In one embodiment, system 106B operates in cooperation with system 10 of FIG. 1A and system 200 of FIG. 2 to implement a location-based companion interface that assists the corresponding user's game play. Referring now to the drawings, like reference numerals indicate identical or corresponding parts.

[0053] As shown in FIG. 1B, a plurality of users 115 (e.g., user 5A, user 5B, ···, user 5N) are playing a plurality of game applications, and each of the game applications is executed locally on a corresponding client device 100 (e.g., a game console) of the corresponding user. Further, each of the plurality of users 115 can access a device 11 configured to receive and / or generate a companion interface for display on the device 11 as introduced above, and the companion interface provides context-related information to the corresponding user playing the corresponding game application as described above. Each of the client devices 100 can be similarly configured in that the local execution of the corresponding game application is performed. For example, user 5A may be playing a first game application on the corresponding client device 100, and an instance of the first game application is executed by the corresponding game title execution engine 111. Game logic 126A (e.g., executable code) for implementing the first game application is stored on the corresponding client device 100 and is used to execute the first game application. By way of example, the game logic can be distributed to the corresponding client device 100 via a portable medium (e.g., a flash drive, a compact disk, etc.) or via a network (e.g., downloaded from a game provider via the Internet 150). For example, user 5B is playing a second game application on the corresponding client device 100, and an instance of the second game application is executed by the corresponding game title execution engine 111. The second game application can be the same as or different from the first game application being executed for user 5A. Game logic 126B (e.g., executable code) for implementing the second game application is stored on the corresponding client device 100 as described above and is used to execute the second game application.Furthermore, user 115N is playing the Nth game application on the corresponding client device 100, and an instance of the Nth game application is executed by the corresponding game title execution engine 111. The Nth game application may be the same as the first or second game application, or may be a completely different game application. The game logic 126N (e.g., executable code) for implementing the third game application is stored on the corresponding client device 100 as described above and is used to execute the Nth game application.

[0054] As described above, the client device 100 can receive inputs from various types of input devices such as game controllers, tablet computers, keyboards, and gestures captured by video cameras, mice, touch pads, etc. The client device 100 can be any type of computing device having at least a memory and a processor module capable of connecting to the game server 205 via the network 150. Also, the corresponding client device 100 of the user is configured to generate a rendering image executed by the game title execution engine 111 operating locally or remotely and display the rendering image on the display. For example, the rendering image can be associated with an instance of the first game application executed on the client device 100 of user 5A. For example, the corresponding client device 100 is configured to interact with an instance of the corresponding game application executed locally or remotely to implement the corresponding user's game play, such as via input commands used to drive the game play.

[0055] In one embodiment, the client device 100 operates in single-player mode for the corresponding user who is playing the game application. As will be described later, according to one embodiment of the present disclosure, a location-based companion interface service that supports the game play of the corresponding user can be provided by backend server support via the game server 205.

[0056] In another embodiment, a plurality of client devices 100 operate in multiplayer mode for the corresponding users who are each playing a specific game application. In this case, the backend server support via the game server can provide a multiplayer function, such as through the multiplayer processing engine 119. Specifically, the multiplayer processing engine 119 is configured to control a multiplayer game session of a specific game application. For example, the multiplayer processing engine 119 communicates with a multiplayer session controller 116 that is configured to establish and maintain a communication session with each of the users and / or players participating in the multiplayer game session. In this way, by being controlled by the multiplayer session controller 116, the users within the session can communicate with each other.

[0057] Furthermore, the multi-player processing engine 119 communicates with the multi-player logic 118 to enable interactions between users within the corresponding game environment of each user. In particular, the state sharing module 117 is configured to manage the state of each user within the multi-player game session. For example, the state data may include game state data that defines the game play state of the corresponding user at a specific location (of the game application). For example, the game state data may include game characters, game objects, game object attributes, game attributes, game object states, graphic overlays, and the like. In this way, the game state data enables the generation of the game environment existing at the corresponding location within the game application. The game state data may also include the states of the CPU, GPU, and memory, register values, program counter values, programmable DMA states, DMA-buffered data, audio chip states, CD-ROM states, and the like, that is, the states of all devices used to render the game play. The game state data may also identify which portions of the executable code need to be loaded in order to execute the video game from that location. The game state data may be stored in the databases 140 of FIGS. 1A and 2 and is accessible by the state sharing module 117.

[0058] Furthermore, the status data may include user save data that contains information for personally configuring the video game of the corresponding player. Since this includes information associated with the character the user plays, the video game is rendered with a character (e.g., position, shape, appearance, clothing, weapons, etc.) that may be unique to that user. In this way, the user save data makes it possible to generate a character for the corresponding user's game play, and the character has a status corresponding to the location within the game application that the corresponding user is currently experiencing. For example, the user save data may include the game difficulty, game level, character attributes, character position, remaining lives, total number of available lives, protective gear, trophies, time counter value, etc. that the corresponding user 115A selected when playing the game. The user save data may also include, for example, user profile data that identifies the corresponding user 115A. The user save data may be stored in the database 140.

[0059] In this way, the multiplayer processing engine 119 that uses the status sharing data 117 and the multiplayer logic 118 can overlay / insert objects and characters into each of the game environments of the users participating in the multiplayer game session. For example, the character of the first user is overlaid / inserted into the game environment of the second user. This enables interaction between the users within the multiplayer game session via their respective game environments (e.g., displayed on the screen).

[0060] Furthermore, backend server support via the game server 205 can provide location-based companion application services provided through the companion interface generated by the companion application generator 213. As previously introduced, the generator 213 is configured to create context-related information (such as auxiliary information, messages, etc.) to be delivered to or received from the user 5. The information is generated based on the user 5's game play of a particular application (e.g., based on the information provided within a snapshot). In this way, the generator 213 can identify the context of the user 5's game play and provide context-related information that can be delivered to a companion interface that can be displayed on the device 11 (e.g., a device separate from the device that displays the user 5's game play).

[0061] FIG. 1C illustrates a system 106C that provides game control to a plurality of users 115 (e.g., users 5L, 5M, ···, 5Z) who play a game application executed via a cloud game network, according to one embodiment of the present disclosure. In some embodiments, the cloud game network can be a game cloud system 210 that includes a plurality of virtual machines (VMs) operating on a hypervisor of a host machine, and one or more virtual machines are configured to execute a game processor module that utilizes the hardware resources available to the host hypervisor. In one embodiment, the system 106C operates in cooperation with the system 10 of FIG. 1A and / or the system 200 of FIG. 2 to implement a location-based companion interface that assists the corresponding user's game play. Referring now to the drawings, like reference numerals indicate identical or corresponding parts.

[0062] As shown, game cloud system 210 includes a game server 205 that provides access to a plurality of interactive video games or game applications. Game server 205 can be any type of server computing device available within the cloud and can be configured as one or more virtual machines executed on one or more hosts. For example, game server 205 can manage a virtual machine that assists a game processor in instantiating an instance of a user's game application. Thus, a plurality of game processors of game server 205 associated with a plurality of virtual machines are configured to execute a plurality of instances of game applications associated with the game play of a plurality of users 115. In this way, backend server support provides streaming of media (e.g., video, audio, etc.) for the game play of a plurality of game applications to a plurality of corresponding users.

[0063] A plurality of users 115 access the game cloud system 210 via the network 150, and users (e.g., users 5L, 5M, ···, 5Z) access the network 150 via the corresponding client device 100'. The client device 100' can be configured similarly to the client device 100 in FIGS. 1A - 1B (e.g., including a game execution engine 111, etc.), or can be configured as a sink client that provides an interface to a backend server that provides computing capabilities (e.g., including a game execution engine 211). Further, each of the plurality of users 115 can access a device 11 configured to receive and / or generate a companion interface for display on the device 11 as previously introduced. The companion interface provides context - relevant information to the corresponding user playing the corresponding game application as described above. Specifically, the client device 100' of the corresponding user 5L is configured to request access to a game application via a network 150 such as the Internet, and to render an instance of a game application (e.g., a video game) executed by the game server 205 and delivered to a display device associated with the corresponding user 5L. For example, user 5L can interact with an instance of a game application running on the game processor of the game server 205 through the client device 100'. More specifically, the instance of the game application is executed by the game title execution engine 211. The game logic (e.g., executable code) for implementing the game application is stored in and accessible from the data store 140 as described above and is used to execute the game application. The game title processing engine 211 can assist a plurality of game applications using a plurality of game logics 177 as shown.

[0064] As described above, the client device 100' can receive inputs from various types of input devices such as game controllers, tablet computers, keyboards, and gestures captured by video cameras, mice, touch pads, etc. The client device 100' can be any type of computing device having at least a memory and a processor module capable of connecting to the game server 205 via the network 150. Also, the client device 100' of the corresponding user is configured to generate a rendering image executed by the game title execution engine 211 operating locally or remotely and display the rendering image on the display. For example, the rendering image can be associated with an instance of a first game application executed on the client device 100' of user 5L. For example, the corresponding client device 100' is configured to interact with an instance of the corresponding game application executed locally or remotely to implement the game play of the corresponding user, such as via input commands used to drive the game play.

[0065] In another embodiment, the multiplayer processing engine 119 provides control of the multiplayer game session of the game application as described above. Specifically, when the multiplayer processing engine 119 manages a multiplayer game session, the multiplayer session controller 116 is configured to establish and maintain communication sessions with each of the users and / or players within the multiplayer session. Thus, by being controlled by the multiplayer session controller 116, the users within the session can communicate with each other.

[0066] Furthermore, the multiplayer processing engine 119 communicates with the multiplayer logic 118 to enable interactions between users within the corresponding game environment of each user. In particular, the state sharing module 117 is configured to manage the state of each user within a multiplayer game session. For example, the state data may include, as described above, game state data that defines the gameplay state of the corresponding user 115A at a specific location (of the game application). Furthermore, the state data may include, as described above, user save data that includes information for personally configuring the corresponding player's video game. For example, since the state data includes information associated with the user's character, the video game may be rendered with a character (e.g., shape, appearance, clothing, weapons, etc.) that is unique to that user. In this way, the multiplayer processing engine 119 that uses the state sharing data 117 and the multiplayer logic 118 can overlay / insert objects and characters into each of the game environments of the users participating in the multiplayer game session. This enables interactions between users within the multiplayer game session via their respective game environments (e.g., displayed on the screen).

[0067] Furthermore, backend server support via the game server 205 may provide location-based companion application services provided through a companion interface generated by the companion application generator 213. As previously introduced, the generator 213 is configured to create context-related information (such as auxiliary information, messages, etc.) to be distributed to or received from a corresponding user (e.g., user 5L). The information is generated based on the gameplay of the user of a particular application (e.g., based on the information provided in a snapshot). In this way, the generator 213 can identify the context of the corresponding user's gameplay and provide context-related information that can be distributed to a companion interface that can be displayed on a device 11 (e.g., a device separate from the device displaying the gameplay of user 5L).

[0068] Figure 2 illustrates a system diagram 200 that enables access to and play of a game application stored in a game cloud system (GCS) 210 according to an embodiment of the present disclosure. Generally, the game cloud system GCS 210 can be a cloud computing system that operates via a network 220 to support multiple users. Further, GCS 210 is configured to store snapshots generated during the game play of game applications of multiple users, and by using the snapshots, an instance of the start of a requesting user's game application can be started from a location within the game application corresponding to the snapshot. For example, the snapshot generator 212 is configured to generate and / or capture snapshots of the game play of one or more players playing a game application. The snapshot generator 212 can be executed outside or inside the game server 205. Further, GCS 210 enables a user to move within the game application and preview past and future scenes of the game application through the use of snapshots. Further, the snapshot allows a requesting user to jump to a selected location within the video game through the corresponding snapshot and experience the game play of another user. Specifically, the system 200 includes GCS 210, one or more social media providers 240, and user devices 230, all of which are connected via a network 220 (e.g., the Internet). One or more user devices are connected to the network 220 and can access the services provided by GCS 210 and the social media provider 240.

[0069] In one embodiment, the game cloud system 210 includes a game server 205, a video recorder 271, a tag processor 273, and an account manager 274 including a user profile manager, a game selection engine 275, a game session manager 285, user access logic 280, a network interface 290, and a social media manager 295. GCS 210 may further include a plurality of game storage systems such as a game state store, a random seed store, a user save data store, a snapshot store, etc., which may generally be stored in the data store 140. Other game storage systems may include a game code store 261, a recorded game store 262, a tag data store 263, a video game data store 264, and a game network user store 265. In one embodiment, GCS 210 is a system that may provide an interconnection between game applications, services, game-related digital content, and systems, applications, users, and social networks. GCS 210 may communicate with the user device 230 and the social media provider 240 via the network interface 290 through the social media manager 295. The social media manager 295 may be configured to be associated with one or more friends. In one embodiment, each social media provider 240 includes at least one social graph 245 indicating a user social network connection.

[0070] User U0 can access the services provided by GCS210 via game session manager 285, and user U0 may represent user 5 in FIG. 1. For example, account manager 274 enables authentication and access of user U0 to GCS210. Account manager 274 stores information regarding member users. For example, the user profile of each member user may be managed by account manager 274. In this way, account manager 274 can use member information for authentication purposes. For example, account manager 274 may be used to update and manage user information regarding member users. Further, the game titles owned by member users may be managed by account manager 274. In this manner, the game applications stored in data store 264 are available to all member users who own these game applications.

[0071] In one embodiment, a user, such as user U0, can access the services provided by GCS210 and social media provider 240 through a connection via network 220 by user device 230. User device 230 may include any type of wired or wireless, portable or non-portable device having a processor and a memory. In one embodiment, user device 230 may be in the form of a smartphone, a tablet computer, or a hybrid form factor that provides touch screen capabilities in a portable form. One exemplary device may include a cellular phone device that operates an operating system and provides access to various applications (apps) that can be obtained via network 220 and executed on a local portable device (such as a smartphone, tablet, laptop, desktop, etc.).

[0072] The user device 230 includes a display 232 that functions as an interface for the user U0 to send an input command 236 and display data and / or information 235 received from the GCS 210 and the social media provider 240. The display 232 can be configured as a display typically provided by a touch screen, or a flat panel display, a cathode ray tube (CRT), or other devices capable of rendering a display. Alternatively, the user device 230 can have its own display 232 separate from the device, similar to a desktop computer or a laptop computer. The user U0 may be able to utilize an additional device 231 (e.g., device 11 in FIG. 1A) to implement the location-based companion interface.

[0073] In one embodiment, the user device 13 is configured to communicate with the GCS 210 to enable user U0 to play game applications. In some embodiments, the GCS 210 may include a plurality of virtual machines (VMs) operating on a hypervisor of a host machine, and one or more of the virtual machines are configured to execute a game processor module that utilizes the hardware resources available to the host hypervisor. For example, user U0 may select an available game application in the video game data store 264 via the game selection engine 275 (e.g., by game title, etc.). The game application can be played within a single-player game environment or within a multiplayer game environment. In this way, the selected game application becomes available and is loaded to be executed by the game server 205 on the GCS 210. In one embodiment, since the game play is mainly executed on the GCS 210, the user device 230 receives a stream of game video frames 235 from the GCS 210, and user input commands 236 that drive the game play are sent back to the GCS 210. The video frames 235 received from the streaming of the game play are displayed on the display 232 of the user device 230. In other embodiments, the GCS 210 is configured to support a plurality of local computing devices for a plurality of users, and each local computing device can execute an instance of a game application, such as within a single-player game application or a multiplayer game application. For example, in a multiplayer game environment, while the game application is being executed locally, the cloud game network simultaneously receives information (e.g., game state data) from each local computing device and distributes the information appropriately across one or more of the local computing devices, whereby each user can interact with other users in the game environment of the multiplayer game application (e.g., via the corresponding characters in the video game).In this way, the cloud gaming network adjusts and combines the game play of each user within the multiplayer game environment.

[0074] In one embodiment, after user U0 selects an available game title to play, the game session of the selected game title can be started by user U0 through game session manager 285. So that user U0 can resume game play from the previous game play stop point, game session manager 285 first accesses the game state store in data store 140 and, if there is a saved game state of the last session that user U0 played (regarding the selected game), obtains this. Once the resume point or start point is identified, game session manager 285 can notify the game execution engine in game processor 201 to execute the game code of the game title selected from game code store 261. After the game session is started, game session manager 285 can deliver game video frames 235 (i.e., streaming of video data) to a user device, such as user device 230, via network interface 290.

[0075] During game play, game session manager 285 can communicate with game processor 201, recording engine 271, and tag processor 273 to generate or save a record (e.g., video) of the game play or game play session. In one embodiment, the video record of the game play can include tag content input or provided during the game play and other game-related metadata. The tag content can also be saved via a snapshot. The video record of the game play can be saved in recording game store 262 together with any game metrics corresponding to that game play. Any tag content can be saved in tag data store 263.

[0076] During gameplay, the game session manager 285 can communicate with the game processor 201 of the game server 205 to distribute and obtain user input commands 236 that are used to affect the results of the corresponding gameplay of the game application. The input commands 236 input by user U0 can be sent from the user device 230 to the game session manager 285 of the GCS 210. The input commands 236 including the input commands used to drive the gameplay may include user interactive inputs such as tag contents (e.g., text, images, video recording clips, etc.). The game input commands, as well as any user play metrics (such as the length of time the user plays the game), can be stored in the game network user store. Multiple functions that may be available to the user can be enabled using the selection information related to the gameplay of the game application.

[0077] Since the gameplay is executed on the GCS 210 by multiple users, the information generated and stored from these games enables any requesting user to experience the gameplay of other users, especially when the gameplay is executed via the GCS 210. Specifically, the snapshot generator 212 of the GCS 210 is configured to save snapshots generated by the gameplay of the users playing the game application through the GCS 210. In the case of user U0, the user device provides an interface that enables user U0 to interact with the game application during gameplay. A snapshot of the gameplay by user U0 is generated and saved on the GCS 210. The snapshot generator 212 can be executed outside the game server 205 as shown in FIG. 2, or can be executed inside the game server 205 as shown in FIG. 1A.

[0078] Furthermore, using the information collected from these gameplays, context-related information provided to user U0 in the corresponding companion application can be generated. For example, as previously introduced, the companion application generator 213 is configured to implement a location-based companion interface configured to assist user U0's gameplay, and the companion interface includes context-related information (such as message communication, auxiliary information, auxiliary offers, etc.) generated based on the location of the character within user U0's gameplay. The companion application generator 213 can be executed outside the game server 205 as shown in FIG. 2, or can be executed inside the game server 205 as shown in FIG. 1A. In these embodiments, for the display of the companion application interface including context-related information, the context-related information can be delivered to the user device 231 via the network 220. In another embodiment, the companion application generator 213 can exist locally to the user (e.g., implemented within the user device 231) and be configured to generate and display context-related information. In this embodiment, the user device 231 may communicate directly with the user device 230 via a local network (or through the external network 220) to implement the companion application interface, the user device 231 may deliver location-based information to the user device 230, and the device 230 is configured to generate and display a companion application interface including context-related information.

[0079] Furthermore, the user device 230 is configured to provide an interface that enables jumping to a selected location within the game application using a snapshot generated during the gameplay of user U0 or another user. For example, the jump game execution engine 216 is configured to access the corresponding snapshot, instantiate an instance of the game application based on the snapshot, and execute the start of the game application from the location within the game application corresponding to the snapshot. In this way, the snapshot enables the requesting user to jump into the corresponding user's gameplay at the location corresponding to the snapshot. For example, user U0 can experience the gameplay of any other user, or review and / or replay their own gameplay. That is, through the snapshot of the corresponding gameplay, the requesting user plays the game application using the character used within that gameplay and corresponding to that gameplay. The jump game execution engine 216 can be executed outside the game server 205 as shown in FIG. 2, or can be executed inside the game server 205 as shown in FIG. 1A.

[0080] FIGS. 3-8 are described in the context of a user playing a game application. Generally, the game application can be any interactive game that responds to user input. Specifically, FIGS. 3-8 illustrate a location-based companion interface configured to assist the user's gameplay, and the companion interface includes context-related information (such as message communication, auxiliary information, etc.) generated based on the position of the character within the user's gameplay.

[0081] A method of implementing a location-based companion interface that assists a corresponding user's game play according to an embodiment of the present disclosure is described herein in connection with the flowchart 300 of FIG. 3, along with detailed descriptions of various modules of a game server and a client device that communicate via a network. The flowchart 300 illustrates the process and data flow of operations involving the game server side for the purpose of generating location-based information included within a companion interface that is transmitted via the network for display on a user's client device. The client device may be separate from another device that displays the game play of a user playing a game application. Specifically, the method of flowchart 300 may be at least partially executed by the companion application generator 213 of FIGS. 1 and 2.

[0082] The embodiment of the present invention disclosed in FIG. 3 is described from the perspective of the game server side. However, other embodiments of the present invention are configured to assist a user's game play to execute a game application and include a location-based companion interface within a local user system having a game processor configured to generate location-based information of the game play, and a companion application generator of another device configured to receive the location-based information via a local network and display context-related information. For example, the companion interface is implemented within a local and isolated system, and information from the game play of other users is not necessarily used to generate context-related information. In another embodiment, information from the game play of other users is received from a backend game server via another network and may be used to generate context-related information.

[0083] Flowchart 300 includes operations 305, 310, and 315 that execute a game application and generate location-based information of a user's game play for playing the game application. Specifically, the method includes, in operation 305, instantiating a first instance of a game application associated with the user's game play. As described above, in one embodiment, an instance of the game application can be executed locally on the user's client device. In another embodiment, an instance of the game application may be executed on a backend game execution engine of a backend game server, and the server can be part of a cloud game network or a game cloud system. The method includes, in operation 310, delivering data representing the user's game play to a first computing device via a first communication channel for interaction by the user. The communication channel can be implemented through a network such as the Internet, for example. Thus, a rendering image displayed on the first computing device can be delivered, and the rendering image is generated by an instance of the game application in response to input commands associated with the user's game play.

[0084] The method includes, at operation 315, identifying location-based information of a character within a user's gameplay. Specifically, the location-based information is created with reference to the location of the character within the user's gameplay in a game world associated with the game application. The location-based information may be included within a snapshot generated, captured, and / or stored during the user's gameplay, as described above. For example, each snapshot may include metadata and / or information generated with reference to the character's location. In one embodiment, the metadata and / or information is configured to enable execution of an instance of the game application from a location within the game application corresponding to the snapshot (e.g., starting from a jump location corresponding to the state of the gameplay at the time the snapshot was captured, reflecting the location of the character within the gameplay). For example, the snapshot includes location-based information of the gameplay and game state data defining the state of the gameplay at the corresponding location (e.g., the game state data includes game characters, game objects, object attributes, graphic overlays, character possessions, character skill sets, character task completion history within the game application, the current geographical location of the character within the game world, the progress of the game application in the user's gameplay, the current state of the character's gameplay, etc.), such that the game state data enables generation of the game environment that existed at the corresponding location within the gameplay. The snapshot may include user save data used to personally configure the game application for the user, and the data may include information for personally configuring a character within the gameplay (e.g., shape, appearance, clothing, weapons, game difficulty, game level, character attributes, etc.). The snapshot may also include random seed data related to the game state, as described above.

[0085] The remaining operations of the flowchart 300 may be performed by the companion application generator, which, as described above, may be executed locally or on a backend server of a cloud gaming network. Specifically, the method includes, at operation 320, receiving location-based information of the game play of a user playing a game application. When the game application is executed locally on a first computing device, the location-based information generated during game play may be received from the local first computing device via a network. For example, the location-based information may be generated by the first computing device and stored locally. When the game application is executed by a game execution engine operating on a backend game server of a cloud gaming network or a game cloud system, the location-based information generated during game play may be received from the backend game server internally when the cloud gaming network is configured to execute the game application and generate a companion interface, or externally when the cloud gaming network is configured to execute the game application and another server generates the companion interface.

[0086] The method includes receiving and / or aggregating location-based information of a plurality of game plays of a plurality of users playing a game application at operation 325. For example, similar to taking a snapshot during a user's game play, a snapshot may be taken during another user's game play, and as described above, the snapshot includes location-based information related to another user's game play. The snapshot and / or the location-based information included therein may be generated locally on the user device and delivered to a backend server of a cloud game network or a game cloud system, or the snapshot may be generated at a backend server of a cloud game network that also executes an instance of the game application to assist other game plays. The location-based information may also be received from the backend server through a local network (the network when the server functions to execute an instance of the game application and generate a companion application), or through a remote network (the network when the backend server of the cloud game network functions independently of generating the companion application).

[0087] These snapshots, which contain metadata / information included within the snapshot or inferred from snapshots generated during other users' gameplay, can provide additional information related to a user's gameplay when combined with location-based information (generated during the user's gameplay). For example, using the location-based information generated during the user's gameplay, it is possible to identify where a character was during gameplay, tasks the character accomplished during gameplay, the character's current possessions and skills, the location where the character currently is (e.g., within the game world), etc. By combining and analyzing the location-based information generated during the user's gameplay with the metadata / information included within the snapshot or inferred from snapshots generated during other users' gameplay, actions / properties necessary to advance the user's gameplay, such as where the character within the gameplay is likely to go next, the next and / or necessary tasks to progress the gameplay, and the possessions or skills the character needs to accomplish these tasks, can be statistically identified (e.g., by a prediction engine).

[0088] The method includes, at operation 330, generating context-related information about the location of a character, the context-related information being generated in real-time based on location-based information of a plurality of gameplays including the user's gameplay. In some embodiments, the context-related information can provide assistance in advancing and / or progressing the game application in the embodiment based on statistical prediction as described above.

[0089] For example, the context-related information may be provided in the form of a message, and the message may be used to provide information useful for advancing the game play (e.g., steps to guide the user to achieve the required tasks, notifications of what comes next, etc.). In one embodiment, a series of related messages are generated according to the position and / or actions of the characters in the game world. For example, step-by-step instructions are generated that provide an indication of the chunk size, which is part of a larger set of instructions, and each chunk is delivered when the character reaches the corresponding position. In particular, the tasks required to advance the user's game play can be statistically predicted. One or more solutions for achieving the tasks can be identified. These solutions are identified from information (e.g., snapshots) collected from multiple game plays of other users and can provide insights into the user's future game play. For example, the collected information may indicate that the aforementioned tasks were typically addressed in other game plays by other users at the same location within the game play and that these users found one or more solutions for completing those tasks. These solutions may include the assets required to achieve these tasks. Further, the analyzed collected information may be further filtered by the user. For example, the user may be interested in solutions from all game plays (e.g., by default), the game plays of well-known experts, one or more selected game plays of the user's friends, etc.

[0090] The message may include an offer of assistance, such as advice from an expert, a friend or a third party, or a link to open two-way interactive communication that enables a friend or an expert to guide the user within the user's game play. As further described in relation to FIG. 6, in some embodiments, the message includes an offer for the user to play the game application and accomplish a task in a jump game. In some embodiments, the message is generated in response to a trigger such as recognizing that the user cannot advance to a particular location within the game application. In other examples, the context-related information may include an offer to receive or purchase downloadable content (DLC) that the user needs to accomplish the next task to be completed in order to advance the game play.

[0091] In yet another embodiment, the context-related information may be initially generated by the user. For example, the message may be for mass transmission to other users (e.g., a help flag) and include a request for assistance from other players of the game application. Further, the message including the request for assistance may be targeted at the user's friends or may be distributed to known experts or third parties that provide game assistance, and the friends, experts, and third parties may or may not be playing the game application simultaneously. In other embodiments, the message may also include an offer to open two-way communication with the user, which is generated by the user and may be targeted at the user's friends. In yet another embodiment, the message includes an offer to open two-way communication with another user, and the message is generated by another user.

[0092] For example, a broadcast message can be generated in response to receiving a help request from a user, who may be stuck in the progress of their gameplay. The message can be received by a backend server (such as a server running the companion application generator 213). For example, a beacon and / or flag associated with the help request can be broadcast across one or more companion interfaces of one or more friends of the user playing the game application. One or more beacons may be inserted into one or more radar mappings showing the game world in one or more user interfaces, and each beacon is placed at a location corresponding to the position of the user's character in the game world. The backend server may receive an acceptance of the request, and the acceptance can be generated by the user's friend. In one embodiment, data representing the user's gameplay is streamed to a third computing device associated with the friend. That is, the user's gameplay can be streamed to the friend via the corresponding companion interface. Further, a two-way communication session (such as message communication via the companion interface, voice, etc.) can be established so that the user and the friend can communicate. In this way, the friend may help provide a real-time guide when the user plays the game application.

[0093] In other embodiments, the context-related information may also provide other types of information provided to improve the user experience. The information can be related to the user's gameplay and can be generally useful, supportive, interesting information, etc. For example, the information can provide the positions of other users playing the game application (such as the positions where the characters of other users are in the radar mapping of the game world), particularly the positions of other users in the vicinity of the user's character. The context-related information can include navigation pointers, interesting game features and their locations, icons or links to other features (such as snapshots, jump game instantiations, etc.).

[0094] In yet other embodiments, the context-related information may include messages from other users (e.g., an egg drop containing information or descriptions from the user's friends, including taunts, words of encouragement, etc.). By way of example, a pre-defined event may be detected within the user's gameplay. The pre-defined event may include a character reaching a certain location within the game world. When detected, access may be had to a pre-generated message from the user's friend. This message may be delivered to the user as context-related information. For example, the message may include a taunt from a friend (e.g., "Aha, didn't think you'd get this far!"). In another example, the message may include words of encouragement from a friend (e.g., "Congratulations, you did it! It took me ages to get here. Get through the next section quickly and meet me behind (in the game)."). For example, FIGS. 4A - 4C illustrate messages generated by a friend and delivered to a companion interface of a user playing a game application, as described below.

[0095] The method includes, at 330, generating a companion interface that includes location-based information (e.g., snapshot information) from a plurality of gameplays and context-related information based on location-based information from the user's gameplay, the information being generated based on the position of a character within the user's gameplay (e.g., location-based context-related information). That is, generally the companion interface provides a function to assist the user's gameplay and enables the user or any other viewer to access in real time information generally useful to the user during play of the game application.

[0096] For example, in one embodiment, the companion interface includes a radar mapping that shows at least a portion of the game world of the game application, and the radar mapping includes at least objects / features placed within the game world, as well as the positions of the user's and other players' characters. Additional information such as message communication that provides general information to enhance the user experience, or solutions to achieve predictive tasks necessary to advance the user's gameplay, may also be included within or adjacent to the radar mapping. In the previously introduced examples, the solutions are statistically predicted and selected for presentation to the user in the companion interface.

[0097] At least one of the solutions may be selected for presentation to the user as context - relevant information, and the solution may include a series of steps necessary to achieve the task. The first step may be associated with a first position / first action combination, the second step may be associated with a second position / second action combination, and so on. The series of steps may further be associated with the user's position and / or actions within the game play. Each step of the solution may further be associated with a corresponding message, and each message may be presented to the companion application depending on the displayed position and / or actions of the character within the user's game play. For example, the first step of the series may be presented in a first message within the companion interface. Illustratively, the first step may include the properties required by the character to achieve the task and instructions for obtaining these properties. The method includes identifying that the first step has been completed within the user's game play, and the completion may further be associated with the character's first position and / or the first action taken by the character. At this point, the second step of the series may be presented in a second message within the companion interface.

[0098] The method includes, at 340, sending the companion interface to a second computing device associated with the user for display concurrently with the user's gameplay. For example, in one embodiment, there may be two communication channels for delivering information, such as a first communication channel established to deliver data representing the user's gameplay to a first computing device, and a second communication channel established to deliver data associated with the companion interface (e.g., deliver the interface and provide input commands to control the interface). In another embodiment, the companion interface may be delivered along with data representing the user's gameplay, such as via a split screen including a first screen for displaying the gameplay and a second screen for displaying the companion interface. More specifically, since the companion interface is generated in real time and delivered concurrently with the user's gameplay, the information provided through the interface assists the user's gameplay. In this way, the user's gameplay can be enhanced with the information provided by the companion interface.

[0099] Figures 4A - 4C illustrate messages generated by a friend and delivered to the companion interface of a user playing a game application, as previously introduced. The messages may be provided within a mapping 400 (e.g., a radar mapping) indicating the position of the user's character within the game world of the game application. The messages may be presented in response to an event trigger (e.g., the user's character reaching a specific location within the game world). In each of the radar mappings 400 shown in Figures 4A - 4C, a direction pointer 450 represents the position of the character within the game world and is positioned at the center of the radar mapping 400. Further, the direction pointer 450 also indicates the character's orientation (e.g., the character's viewpoint within the game world).

[0100] Specifically, FIG. 4A illustrates a mapping 400 of a user's gameplay while playing a game application, and the mapping 400 indicates that a character of a second user (e.g., the user's friend) is in the approximate vicinity of the position of the character within the user's gameplay. In FIG. 4A, the mapping 400 may be provided within a window 430. The second user is represented by an icon 420, which indicates the position of the second user within the game world. The mapping 400 also shows other characters 405 and 410 of other users in the approximate vicinity. Further, a message 450 related to the user's gameplay may be associated with the second user. The message may be presented in a selection window 425 together with the icon 420. The selection window may have a heading 441 (e.g., "Memo from John Doenavich") indicating that messages from another player of the game application are available. The selection window 425 may include a preview 442 of the message 450. In this way, the user is made aware that the message 450 from another player of the game application, namely John Doenavich, who may be the user's friend, is selectable.

[0101] In another embodiment, the characters of other users do not need to be near the user's character. In this case, the message 450 can be presented in the form of an egg drop. Specifically, FIG. 4B illustrates the mapping 400 of the user's gameplay introduced in FIG. 4A according to an embodiment of the present disclosure, and includes a location-based message 450 regarding the user's gameplay left by a second user, John Doenavich. As shown, the mapping 400 presented in the window 430 includes a message icon 421. The position of the message icon 421 within the mapping 400 can indicate the position of the character in the game world associated with John Doenavich's gameplay when the message 450 was created. The message, along with the icon 421, can be presented in a selection window 425' similar to the selection window 425 presented in FIG. 4A. The selection window can have the same heading 441 (e.g., "Note from John Doenavich") indicating that a message from another player of the game application is available. The selection window 425' can include the same preview 442 of the message 450. In this way, the user is made aware that a message 450 from another player of the game application, namely John Doenavich who can be the user's friend, is selectable.

[0102] Figure 4C illustrates the mapping 400 of the user's gameplay introduced in FIGS. 4A and 4B according to an embodiment of the present disclosure, and includes a window 435 that displays a location-based message 450 left by another player (e.g., John Doenavich). When there is no character of another player near the character played by the user and represented by the direction pointer 450, after the user activates the selection window 425', in the split screen, the mapping 400 including the icon 421 is included in the window 430'. When the character of another player is present near the character played by the user, in relation to FIG. 4A, the icon 421 can be replaced by the character icon 420 previously introduced in FIG. 4A. In particular, the window 435 displays a heading 441 and also includes the entire message 450. Also, with the icons 445 and 446, the user can save or discard the message 450.

[0103] FIGS. 5A-5E illustrate a companion interface 590 for a user playing a game application according to an embodiment of the present disclosure, and the companion interface facilitates two-way communication between two or more players of the game application. An opportunity may be provided to instantiate and continue two-way communication within a mapping 500 (e.g., a radar mapping) that indicates the position of the user's character within the game world of the game application. In each of the radar mappings 500 shown in FIGS. 5A-5E, a direction pointer 550 represents the position of the character within the game world and is located at the center of the radar mapping 500. Further, the direction pointer 550 also indicates the direction of the character (e.g., the perspective of the character within the game world).

[0104] Specifically, FIG. 5A illustrates a mapping 500 of a user's gameplay for playing a game application according to an embodiment of the present disclosure, and the mapping indicates that another player (e.g., John Doenavich first introduced in the examples of FIGS. 4A-4C) is currently playing the game application. In FIG. 5A, the mapping may be displayed within window 530. Outside the region defined by mapping 500, an arc 560 is disposed, and the arc 560 indicates the approximate direction of the characters of other players within the game world with respect to the position of the user's character represented by the central direction arrow 550 of mapping 500. The arc 560 is outside of mapping 500 and indicates that the characters of other players are not within the approximate vicinity of the user's character (shown by the inside of mapping 500), but are currently playing the game application. Other characters 505, 510, and 520 are shown to be within the approximate vicinity of the user's character (represented by the inside of mapping 500). An information window 570 is also provided within the companion interface 590 to inform the user that another player, John Doenavich, is playing the game application.

[0105] Figure 5B illustrates a mapping 500 of a user's gameplay introduced in Figure 5A according to an embodiment of the present disclosure. The mapping indicates that another player is approaching in the approximate vicinity of the position of the character within the user's gameplay. The approximate vicinity is defined by the interior of the mapping 500. The window 530 including the mapping 500 is updated to display that the character of another player is approaching the outer boundary line 501 of the mapping 500 but has not crossed the boundary line. Specifically, an icon 561 is displayed at a position within the mapping 500 adjacent to the boundary line 501, which indicates the approximate direction in which the character of another player is approaching in the approximate vicinity with respect to the position of the user's character indicated by a direction pointer 550 disposed at the center of the mapping 500. The icon 561 may represent an area of influence centered on the position of the character of another player within the game world (e.g., represented by a two-dimensional circle within the mapping 500). Since the character of another player is still outside the area defined by the mapping 500, the icon 561 is represented by a semi-circle. As the character approaches closer to the approximate vicinity defined by the mapping 500, the icon 561 becomes larger or darker within the mapping 500 and may begin to complete a full circle as displayed in the mapping 500.

[0106] Figure 5C illustrates a mapping 500 of a user's gameplay introduced in Figures 5A - 5B according to an embodiment of the present disclosure. The mapping indicates that another player has now arrived at the outer boundary line 501 of the mapping. The window 530 including the mapping 500 is updated to display that the character of the second player is now considered to be within the approximate vicinity of the user's character, i.e., within the mapping 500. Since the focus of the companion interface 590 illustrated in Figures 5A - 5E is to open communication between the user of the game application and another player, the icon 562 represents a person (e.g., John Doenavich of another player) and does not necessarily represent the character within the game application played by another player.

[0107] Figure 5D illustrates a mapping 500 of a user's gameplay introduced in FIGS. 5A-5C according to an embodiment of the present disclosure, and includes a window 535 that displays information about other players. The window 530' includes, in a split screen, the mapping 500 that includes an icon 562 located at the outer boundary line 501. Basically, the mapping 500 is the same in both FIGS. 5C and 5D. Further, the window 535 displays information about another player, John Doenavich, and includes a username 568 ("superCooluser88"), game consoles available to other users (e.g., SONY PS3 and SONY PS4), languages spoken by other players, and the like. Importantly, the window 535 includes an invitation to participate in a conversation with another player, John Doenavich, and the invitation is presented within a select icon 570, which, when activated, instantiates two-way communication between the user and the other player.

[0108] Figure 5E illustrates a mapping 500 of a user's gameplay introduced in FIGS. 5A-5D according to an embodiment of the present disclosure, and includes a window 535' that displays a two-way conversation between the user and another player, John Doenavich. The window 530' includes, in a split screen, the mapping 500 that includes an icon 562 located at the outer boundary line 501. The mapping 500 shown in FIG. 5E is similar to the mapping 500 shown in FIGS. 5C and 5D, but may be updated according to the duration of the conversation between the user and the other player shown in FIG. 5E. Specifically, the window 535' displays the text input during the conversation being conducted between the user and another player, John Doenavich. In the conversation, the user and the other player are attempting to meet their two characters within the game world.

[0109] A location-based companion interface according to an embodiment of the present disclosure that assists in the game play of a corresponding user, along with a detailed description of various modules of a game server and a client device that communicate via a network, the location-based companion interface including an offer that provides assistance in advancing the game play, is described herein in connection with the flowchart 600 of FIG. 6. Flowchart 600 illustrates the process and data flow of operations involving the game server side for the purpose of generating location-based information included within the companion interface that is transmitted via the network for display on the user's client device, and the client device may be separate from another device that displays the game play of the user playing the game application. Specifically, the method of flowchart 600 may be at least partially executed by the companion application generator 213 of FIGS. 1 and 2.

[0110] The method includes, at 610, generating context-related information about the position of a character within the game play of a user playing a game application. For example, an instance of the game application may be executed in association with the game play of the user, and data representing the game play may be delivered to the user's first computing device. The data is delivered via a first communication channel for interaction by the user. The location-based information of the character may be generated, captured, and included, for example, within a snapshot, and the location information is created with reference to the game world associated with the game application. As described above, each snapshot may also include information (e.g., metadata, game state data) that enables the generation of an environment corresponding to a location within the video game, so that based on the snapshot, the corresponding scene and environment can be generated in another instance of the game application. That is, the start of the jump game is executed from a location within the game application corresponding to the jump location corresponding to the state of the game play when the corresponding snapshot was captured.

[0111] Furthermore, location-based information of multiple game plays of other users playing the game application is also generated, received, and / or incorporated. That is, the snapshots captured during the game play of other users include location-based information related to these game plays, such as metadata / information, game state, etc. For example, multiple snapshots can be generated from multiple instances of the game application executed in association with multiple users. When each instance of the video game is running, one or more snapshots are captured, and as described above, the snapshot enables the start of the instance of the video game from the location within the video game corresponding to that snapshot. As described above, the location-based information collected from multiple game plays including the user's game play is analyzed, and additional context-related information related to the user's game play can be identified. For example, by combining and analyzing the location-based information generated during the user's game play with the metadata / information included in the snapshots of multiple game plays of other players, the prediction engine can statistically identify the actions / assets necessary to advance the user's game play. This context-related information is generated in real time, for example, to provide the user with real-time assistance in advancing and / or progressing their own game play when playing the game application.

[0112] In one embodiment, in the method, a task to be completed within the user's game play is identified. Furthermore, a solution configured to complete the task is also identified based on the metadata / information collected and analyzed in the game play of other players as described above, and also based on the position of the character within the user's game play. For example, the solution can include a series of steps that the character, i.e., the user, should execute as an input within the game play.

[0113] The method includes, at 615, generating a companion interface that includes context-related information. The companion interface provides functions to assist the user's gameplay and enables the user or any other viewer to access the information in real time. In one embodiment, the companion interface includes an offer to complete a task, as previously introduced, and the completion of the task may be required to advance the user's gameplay.

[0114] The method includes, at 620, transmitting the companion interface to a second computing device associated with the user for display concurrently with the user's gameplay. For example, there may be a first communication channel established to deliver data representing the user's gameplay to a first computing device, and a second communication channel established to deliver data associated with the companion interface (e.g., delivering the interface and providing input commands to control the interface). In this way, the user's gameplay can be enhanced with the information provided by the companion interface.

[0115] The method includes receiving acceptance of the offer at 625. For example, despite repeated efforts, the user may be dissatisfied with the progress of their game play, which may be due to being unable to progress beyond a certain point within the game application. Instead of taking the drastic measure of quitting the game, which is known to be a problem that can occur for any player regardless of the game application they are playing, the user may choose to receive the assistance of an expert provided within the offer, which may include having the expert complete the user's task. Specifically, the method includes instantiating a second instance of the game application in the jump game play based on a first snapshot generated during the execution of a first instance of the user's game play at 630. Specifically, the first snapshot includes the first game state of the first instance and position-based information of the character within the user's game play. In order for the game state data within the snapshot to enable the generation of an environment that includes the generation of the character and random seed data corresponding to a location within the game application (herein identified as the jump point), and for the start of the game application to be executed from a location within the game application corresponding to the snapshot, the first jump game play starts from a location within the game application corresponding to the first snapshot.

[0116] In this way, the expert can jump into the user's gameplay. For example, the expert can instruct the character used in the user's gameplay in a jump game to complete tasks in the jump game play. Further, the method includes, at 635, generating and / or capturing a second snapshot including a second game state of a second instance of the game application (during the execution of the jump game), the second instance being controlled by the expert, and the tasks in the jump game play captured in the second snapshot being completed. At this point, the second snapshot associated with the jump game play can be made available to the user. When the second snapshot is selected, the method includes, at 640, instantiating a third instance of the game application in a second jump game play based on the second snapshot. The third instance is controlled by the user, and the tasks in the second jump game play are completed. In this way, the user can continue his own gameplay executed by the third instance of the game application, and at this point the gameplay includes the completion of the tasks.

[0117] Specific embodiments have been provided to illustrate the generation and implementation of a location-based companion interface configured to assist the user's gameplay. The companion interface includes context-related information (such as message communication, auxiliary information, auxiliary offers, etc.) generated based on the position of the character within the user's gameplay. These embodiments are described by way of example and not limitation. Those skilled in the art who have read this disclosure will be able to implement additional embodiments that are within the spirit and scope of this disclosure.

[0118] Note that access services distributed over a wide area, such as providing access to the games of the present embodiment, often use cloud computing. Cloud computing is a computing paradigm in which dynamically scalable and often virtualized resources are provided as a service over the Internet. A user does not need to be an expert in the technical infrastructure of the "cloud" that supports the user. Cloud computing can be classified into different services such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services often provide common applications such as video games online for access from a web browser, but the software and data are stored on servers within the cloud. The term "cloud" is used as a metaphor for the Internet based on how the Internet is depicted in a computer network diagram and is an abstraction of the underlying complex infrastructure.

[0119] A Game Processing Server (GPS) (or simply "game server") is used for game clients to play single and multiplayer video games. Most video games played over the Internet operate via a connection to a game server. Typically, a game uses a dedicated server application that collects data from players and distributes the collected data to other players. This is more efficient and effective than a peer-to-peer structure but requires a separate server to host the server application. In another embodiment, the GPS establishes communication between the player and each game-playing device and exchanges information without relying on a centralized GPS.

[0120] A dedicated GPS is a server that operates independently of the client. Such a server typically runs on dedicated hardware located in a data center and provides more bandwidth and dedicated processing power. A dedicated server is a preferred way to host the game servers for most PC-based multiplayer games. Large-scale multiplayer online games operate on dedicated servers usually hosted by the software company that owns the game title, and the dedicated server can control and update the content.

[0121] Users access remote services through client devices that include at least a CPU, a display, and I / O. The client device can be a PC, a mobile phone, a netbook, a PDA, etc. In one embodiment, the network running on the game server recognizes the type of device used by the client and adjusts the communication method employed. In other cases, the client device uses a standard communication method such as HTML to access the application on the game server via the Internet.

[0122] Embodiments of the present disclosure can be implemented by various computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, and mainframe computers. The present disclosure can also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked through a wired or wireless network.

[0123] It should be understood that a given video game may be developed for a specific platform and a specific associated controller device. However, when such a game becomes available via a game cloud system as presented herein, a user may access the video game with a different controller device. For example, a game may have been developed for a certain gaming console and its associated controller, but a user may access a cloud-based version of the game from a personal computer using a keyboard and mouse. In such a scenario, input parameter settings can define a mapping from inputs that can be generated by the controller devices available to the user (in this case, the keyboard and mouse) to inputs acceptable for the execution of the video game.

[0124] In another example, a user may access the cloud game system via a tablet computing device, a touch screen smartphone, or other touch screen-driven device. In this case, the client device and the controller device are integrated together within the same device, and input is provided by detected touch screen input / gestures. For such devices, input parameter settings can define that certain touch screen inputs correspond to game inputs of the video game. For example, buttons, a direction pad, or other types of input elements may be displayed or overlaid during the operation of the video game to indicate positions on the touch screen where the user can touch to generate a game input. Gestures such as swipes in a particular direction or particular touch motions may also be detected as game inputs. In one embodiment, a tutorial showing how to input into the game play via the touch screen may be provided to the user, for example, to acclimate the user to control actions on the touch screen before beginning the game play of the video game.

[0125] In some embodiments, the client device functions as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection and sends inputs from the controller device to the client device. The client device then processes these inputs and may then send the input data via a network (e.g., via a local network device such as a router that is accessed by the cloud game server). However, in other embodiments, the controller itself may be a network device that has the ability to communicate inputs directly to the cloud game server via the network without first communicating such inputs through the client device. For example, the controller may connect to a local network device (such as the aforementioned router) to send and receive data with the cloud game server. Thus, while the client device still needs to receive video output from the cloud-based video game and render it on a local display, the input latency can be reduced by enabling the controller to bypass the client device and send inputs directly to the cloud game server via the network.

[0126] In one embodiment, a networked controller and client device can be configured such that certain types of input are sent directly from the controller to the cloud game server, and other types of input are sent via the client device. For example, input detected without relying on any additional hardware or processing separate from the controller itself can bypass the client device and be sent directly from the controller to the cloud game server over the network. Such input can include button input, joystick input, embedded motion detection input (e.g., accelerometer, magnetometer, gyroscope), etc. However, input that utilizes additional hardware or requires processing by the client device can be sent to the cloud game server by the client device. This can include video or audio captured from the game environment that can be processed by the client device before being sent to the cloud game server. Further, to detect the position and movement of the controller, input from the controller's motion detection hardware is processed by the client device in conjunction with the captured video, and the processed input is then communicated to the cloud game server by the client device. It should be understood that controller devices according to various embodiments can also receive data (e.g., feedback data) from the client device or directly from the cloud game server.

[0127] It should be understood that the embodiments described herein can be implemented on any type of client device. In some embodiments, the client device is a head-mounted display (HMD).

[0128] FIG. 7 is a diagram illustrating components of a head-mounted display 750 according to an embodiment of the present disclosure. The head-mounted display 750 includes a processor 700 that executes program instructions. The memory 702 is provided for storage purposes and may include both volatile and non-volatile memory. A display 704 is included that provides a visual interface visible to the user. A battery 706 is provided as a power source for the head-mounted display 750. The motion detection module 708 may include any of various types of motion sensing hardware such as a magnetometer 710, an accelerometer 712, and a gyroscope 714.

[0129] An accelerometer is a device that measures acceleration and gravity-induced reaction forces. Single and multi-axis models are available to detect the magnitude and direction of acceleration in various directions. Accelerometers are used to sense tilt, vibration, and shock. In one embodiment, three accelerometers 712 are used to provide the direction of gravity, which gives an absolute reference for two angles (world space pitch and world space roll).

[0130] A magnetometer measures the strength and direction of the magnetic field near the head-mounted display. In one embodiment, three magnetometers 710 are used within the head-mounted display to ensure an absolute reference for the world space yaw angle. In one embodiment, the magnetometer is designed to have a geomagnetic field range of ±80 microteslas. The magnetometer is affected by metal and provides a monotonic yaw measurement based on the actual yaw. The magnetic field may be distorted by metal in the environment, which causes distortion in the yaw measurement. If necessary, this distortion can be calibrated using information from other sensors such as a gyroscope or a camera. In one embodiment, the accelerometer 712 is used together with the magnetometer 710 to obtain the tilt and orientation of the head-mounted display 750.

[0131] A gyroscope is a device for measuring or maintaining orientation based on the principle of angular momentum. In one embodiment, three gyroscopes 714 provide information regarding movement over their respective axes (x, y, and z) based on inertial sensing. Gyroscopes are useful for detecting high-speed rotation. However, gyroscopes can vary over time without the presence of an absolute reference. This requires periodically resetting the gyroscope, which can be done using other available information such as visual tracking of objects, accelerometers, magnetometers, etc. to identify position / orientation.

[0132] To capture images and image streams of the real environment, a camera 716 is provided. A plurality of cameras, including a rear camera (oriented away from the user when the user is looking at the display of the head-mounted display 750) and a front camera (oriented towards the user when the user is looking at the display of the head-mounted display 750), can be included in the head-mounted display 750. Additionally, a depth camera 718 can be included in the head-mounted display 750 to sense depth information of objects within the real environment.

[0133] In one embodiment, warnings regarding safety can be provided using a camera integrated on the front of the HMD. For example, if the user is approaching a wall or an object, the user can be warned. In one embodiment, an external view of physical objects in the room can be provided to the user to warn the user of their presence. The outline can be overlaid, for example, within a virtual environment. In some embodiments, a view of reference markers overlaid on the floor, for example, can be provided to the HMD user. For example, the markers can provide the user with a reference to the center of the room in which the user is playing a game. This can provide the user with visual information about where to move so as not to collide with walls or other objects in the room, for example. When the user wears the HMD to play a game or manipulate content, tactile warnings and / or audio warnings can also be provided to the user to enhance safety.

[0134] The head-mounted display 750 includes a speaker 720 that provides audio output. Also, a microphone 722 may be included to capture audio from the real environment, including sounds from the surrounding environment, utterances by the user, and the like. The head-mounted display 750 includes a haptic feedback module 724 that provides haptic feedback to the user. In one embodiment, the haptic feedback module 724 is capable of causing movement and / or vibration of the head-mounted display 750 to provide haptic feedback to the user.

[0135] The LED 726 is provided as a visual indicator of the state of the head-mounted display 750. For example, the LED may indicate battery level, startup status, and the like. A card reader 728 is provided to enable the head-mounted display 750 to read information from and write information to a memory card. As an example of an interface that enables connection to a peripheral device or connection to other portable devices such as other portable devices and computers, a USB interface 730 is included. In various embodiments of the head-mounted display 750, any of various types of interfaces may be included to enable better connectivity of the head-mounted display 750.

[0136] A WiFi (registered trademark) module 732 is included to enable connection to the Internet via wireless network technology. Also, the head-mounted display 750 includes a Bluetooth (registered trademark) module 734 to enable wireless connection to other devices. A communication link 736 may also be included for connection to other devices. In one embodiment, the communication link 736 utilizes infrared communication for wireless communication. In other embodiments, the communication link 736 may utilize any of various wireless or wired transmission protocols to communicate with other devices.

[0137] To provide an input interface for the user, an input button / sensor 738 is included. Any of various types of input interfaces such as buttons, touch pads, joysticks, trackballs, etc. may be included. To facilitate communication with other devices via ultrasonic technology, an ultrasonic communication module 740 may be included in the head-mounted display 750.

[0138] To enable detection of physiological data from the user, a biosensor 742 is included. In one embodiment, the biosensor 742 includes one or more dry electrodes for detecting the user's bioelectrical signals through the user's skin.

[0139] The foregoing components of the head-mounted display 750 are described as merely representative components that may be included in the head-mounted display 750. In various embodiments of the present disclosure, the head-mounted display 750 may or may not include some of the various components described above. Other components known in the art but not described herein may further be included in embodiments of the head-mounted display 750 to simplify the aspects of the present disclosure described herein.

[0140] In various embodiments of the present disclosure, it will be understood by those skilled in the art that the foregoing handheld device may be used in conjunction with an interactive application displayed on a display to provide various interactive functions. The representative embodiments described herein are provided as examples only, not as limitations.

[0141] FIG. 8 is a block diagram of a game system 800 according to various embodiments of the present disclosure. The game system 800 is configured to provide a video stream to one or more clients 810 via a network 815. The game system 800 typically includes a video server system 820 and any game server 825. The video server system 820 is configured to provide a video stream with a minimum quality of service to one or more clients 810. For example, the video server system 820 may receive game commands that change the state of a video game or the viewpoint within a video game, and provide an updated video stream that reflects this state change to the client 810 with a minimum latency. The video server system 820 may be configured to provide the video stream in a variety of alternative video formats, including formats that are not yet defined. Further, the video stream may include video frames configured to be presented to the user at a variety of frame rates. Typical frame rates are 30 frames per second, 60 frames per second, and 820 frames per second. However, alternative embodiments of the present disclosure include higher or lower frame rates.

[0142] In this specification, clients 810, individually referred to as 810A, 810B, etc., may include head-mounted displays, terminals, personal computers, game consoles, tablet computers, telephones, set-top boxes, kiosks, wireless devices, digital pads, stand-alone devices, and / or handheld game play devices, etc. Usually, client 810 is configured to receive an encoded (i.e., compressed) video stream, decode the video stream, and present the resulting video to a user, such as a game player. The process of receiving the encoded video stream and / or decoding the video stream typically includes storing individual video frames in a client's reception buffer. The video stream may be presented to the user on a display integrated with client 810, or on a separate device such as a monitor or television. Client 810 is optionally configured to support multiple game players. For example, a game console may be configured to support two, three, four or more simultaneous players. Each of these players may receive a separate video stream, or a single video stream may include regions of frames specially generated for each player, for example based on each player's perspective. Client 810 is optionally geographically dispersed. The number of clients included in game system 800 can vary widely, from one or two to thousands, tens of thousands or more. As used in this specification, the term "game player" is used to refer to a person who plays a game, and the term "game play device" is used to refer to a device used to play a game. In some embodiments, a game play device may refer to a plurality of computing devices that cooperate to deliver a game experience to a user. For example, a game console and an HMD may cooperate with a video server system 820 to deliver a game viewable through the HMD. In one embodiment, the game console receives a video stream from the video server system 820, and the game console transfers or updates the video stream to the HMD that performs rendering.

[0143] Client 810 is configured to receive a video stream via network 815. Network 815 can be any type of communication network, including, for example, a telephone network, the Internet, a wireless network, a power line network, a local area network, a wide area network, and / or a private network. In an exemplary embodiment, the video stream is communicated via a standard protocol such as TCP / IP or UDP / IP. Alternatively, the video stream is communicated via a proprietary standard.

[0144] Exemplary embodiments of client 810 include a personal computer having a processor, non-volatile memory, a display, decoding logic, network communication capabilities, and an input device. The decoding logic can include hardware, firmware, and / or software stored on a computer-readable medium. Systems for decoding (and encoding) video streams are well known in the art and vary depending on the particular encoding scheme used.

[0145] Client 810 may further include a system configured, although not required, to modify the received video. For example, the client may be configured to perform additional rendering, such as overlaying one video over another and / or cropping the video. For example, client 810 may receive various types of video frames, such as I-frames, P-frames, and B-frames, and may be configured to process these frames into an image for display to the user. In some embodiments, a member of client 810 is configured to perform operations such as additional rendering, shading, or 3D transformation on the video stream. A member of client 810 is optionally configured to receive multiple audio or video streams. Input devices of client 810 may include, for example, a single-handed game controller, a two-handed game controller, a gesture recognition system, a gaze recognition system, a voice recognition system, a keyboard, a joystick, a pointing device, a force feedback device, a motion and / or position sensing device, a mouse, a touch screen, a neural interface, a camera, and / or a developing input device.

[0146] The video stream (and optionally the audio stream) received by client 810 is generated and provided by video server system 820. As further described elsewhere in this specification, this video stream includes video frames (and the audio stream includes audio frames). The video frames are configured to significantly contribute to the image displayed to the user (e.g., the video frames include pixel information of a suitable data structure). The term "video frame" as used herein is used to refer to a frame that mainly includes information configured to contribute, such as by acting, to the image displayed to the user. Most of the teachings of this specification regarding "video frames" can also be applied to "audio frames".

[0147] Client 810 is typically configured to receive input from a user. These inputs can include game commands configured to change the state of a video game or act on gameplay. Game commands can be received using an input device and / or automatically generated by computing instructions operating on client 810. The received game commands are communicated from client 810 via network 815 to video server system 820 and / or game server 825. For example, in some embodiments, game commands are communicated to game server 825 via video server system 820. In some embodiments, separate copies of game commands are communicated from client 810 to game server 825 and video server system 820. Communication of game commands optionally depends on identification of the commands. Game commands are optionally communicated from client 810A through different paths or communication channels used to provide an audio or video stream to client 810A.

[0148] The game server 825 is optionally operated by an entity different from the video server system 820. For example, the game server 825 can be operated by the publisher of a multiplayer game. In this embodiment, the video server system 820 is optionally regarded as a client by the game server 825 and is optionally configured to be a conventional client that executes a conventional game engine from the perspective of the game server 825. The communication between the video server system 820 and the game server 825 optionally occurs via the network 815. Thus, the game server 825 can be a conventional multiplayer game server that transmits game state information to a plurality of clients including the video server system 820. The video server system 820 can be configured to communicate with a plurality of instances of the game server 825 simultaneously. For example, the video server system 820 can be configured to provide a plurality of different video games to different users. Each of these different video games can be supported by a different game server 825 and / or issued by a different entity. In some embodiments, some geographically dispersed instances of the video server system 820 are configured to provide game videos to a plurality of different users. Each of these instances of the video server system 820 can communicate with the same instance of the game server 825. The communication between the video server system 820 and one or more game servers 825 optionally occurs via a dedicated communication channel. For example, the video server system 820 can be connected to the game server 825 via a high-bandwidth channel dedicated for communication between these two systems.

[0149] The video server system 820 includes at least a video source 830, an I / O device 845, a processor 850, and a non-transitory storage 855. The video server system 820 can include one computing device or can be distributed across a plurality of computing devices. These computing devices are optionally connected via a communication system such as a local area network.

[0150] The video source 830 is configured to provide a video stream, such as a streaming video or a series of video frames, that forms a video, for example. In some embodiments, the video source 830 includes a video game engine and rendering logic. The video game engine is configured to receive game commands from a player and maintain a replica of the state of the video game based on the received commands. This game state includes the positions of objects within the game environment, as well as typically the viewpoint. The game state may also include properties, images, colors, and / or the textures of the objects.

[0151] The game state is typically maintained based on game rules and game commands such as move, turn, attack, set focus, interact, and / or use. Optionally, a part of the game engine is located within the game server 825. The game server 825 may maintain a replica of the state of the game based on game commands received from a plurality of players using geographically distributed clients. In these cases, the game state is provided by the game server 825 to the video source 830, a replica of the game state is stored, and rendering is performed. The game server 825 may receive game commands directly from the client 810 via the network 815 and / or may receive game commands via the video server system 820.

[0152] The video source 830 typically includes rendering logic such as hardware, firmware, and / or software stored in a computer-readable medium such as storage 855. This rendering logic is configured to create video frames of a video stream based on the game state. All or part of the rendering logic is optionally disposed within a graphics processing unit (GPU). The rendering logic typically includes processing stages configured to identify three-dimensional spatial relationships between objects and / or apply suitable textures, etc., based on the game state and viewpoint. The rendering logic generates raw video that is typically encoded before communicating to the client 810. For example, the raw video can be encoded according to the Adobe Flash® standard,.wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith, MPG-x.Xvid.FFmpeg, x264, VP6-8, realvideo, or mp3, etc. The encoding process generates a video stream that is optionally packaged for delivery to a decoder on a remote device. The video stream is characterized by a frame size and a frame rate. Typical frame sizes include 800×600, 1280×720 (e.g., 720p), 1024×768, but any other frame size may also be used. The frame rate is the number of video frames per second. The video stream can include various types of video frames. For example, the H.264 standard includes "P" frames and "I" frames. An I-frame includes information to refresh all macroblocks / pixels on a display device, while a P-frame includes information to refresh a subset thereof. A P-frame typically has a smaller data size than an I-frame. As used herein, the term "frame size" is meant to refer to the number of pixels within a frame. The term "frame data size" is used to refer to the number of bytes required to store a frame.

[0153] In an alternative embodiment, the video source 830 includes a video recording device such as a camera. This camera can be used to generate delayed or live video that can be included in the video stream of a computer game. The resulting video stream can optionally include both rendered images and images recorded using a still or video camera. The video source 830 can also include a storage device configured to store previously recorded video for inclusion in the video stream. The video source 830 can also include a motion or position sensing device configured to detect the motion or position of an object such as a person, and logic configured to identify a game state or generate video based on the detected motion and / or position.

[0154] Optionally, the video source 830 is configured to provide overlays configured to be placed on top of other video. For example, these overlays can include a command interface, login instructions, messages to the game player, images of other game players, video feeds of other game players (e.g., webcam video). In embodiments where the client 810A includes a touch screen interface or a gaze detection interface, the overlay can include a virtual keyboard, joystick, and / or touch pad, etc. In one example of an overlay, the player's voice is overlaid on the audio stream. Optionally, the video source 830 further includes one or more audio sources.

[0155] In an embodiment where the video server system 820 is configured to maintain a game state based on inputs from multiple players, the viewpoints, including the position and orientation of the field of view, can be different for each player. Optionally, the video source 830 is configured to provide a separate video stream to each player based on the viewpoint of each player. Further, the video source 830 can be configured to provide different frame sizes, frame data sizes, and / or encodings to each of the clients 810. Optionally, the video source 830 is configured to provide three-dimensional video.

[0156] The I / O device 845 is configured to transmit and / or receive information such as video, commands, information requests, game states, line-of-sight information, device operations, device positions, user operations, client identifications, player identifications, game commands, security information, and / or audio for the video server system 820. The I / O device 845 typically includes communication hardware such as a network card or a modem. The I / O device 845 is configured to communicate with the game server 825, the network 815, and / or the client 810.

[0157] The processor 850 is configured to execute logic, such as software, included within various components of the video server system 820 discussed herein. For example, the processor 850 can be programmed with software instructions to perform the functions of the video source 830, the game server 825, and / or the client qualifier 860. The video server system 820 optionally includes multiple instances of the processor 850. The processor 850 can also be programmed with software instructions to execute commands received by the video server system 820 or to coordinate the operations of the various elements of the game system 800 discussed herein. The processor 850 can include one or more hardware devices. The processor 850 is an electronic processor.

[0158] Storage 855 includes a non-transitory analog and / or digital storage device. For example, storage 855 may include an analog storage device configured to store video frames. Storage 855 may include computer-readable digital storage such as, for example, a hard drive, an optical drive, or solid state storage. Storage 855 is configured (e.g., by a suitable data structure or file system) to store video frames, artificial frames, video streams including both video frames and artificial frames, audio frames, and / or audio streams, etc. Optionally, storage 855 is distributed across multiple devices. In some embodiments, storage 855 is configured to store software components of video source 830 discussed elsewhere in this specification. These components may be stored in a format that can be provided at any time when needed.

[0159] Video server system 820 optionally further includes a client qualifier 860. Client qualifier 860 is configured to remotely identify the performance of a client such as client 810A or 810B. These performances may include both the performance of client 810A itself and the performance of one or more communication channels between client 810A and video server system 820. For example, client qualifier 860 may be configured to attempt communication channels through network 815.

[0160] The client qualifier 860 can identify (e.g., discover) the performance of the client 810A, either manually or automatically. Manual identification includes communicating with the user of the client 810A and requesting the user to provide the performance. For example, in some embodiments, the client qualifier 860 is configured to display images and / or text, etc. within the browser of the client 810A. In one embodiment, the client 810A is an HMD including a browser. In another embodiment, the client 810A is a gaming machine having a browser that can be displayed on the HMD. The displayed object requests the user to input information such as the operating system, processor, type of video decoder, type of network connection, resolution of the display, etc. of the client 810A. The information input by the user is returned to the client qualifier 860.

[0161] Automatic identification can occur, for example, by executing an agent on the client 810A and / or by sending a test video to the client 810A. The agent can comprise computing instructions such as JavaScript embedded in a web page or installed as an add-on. The agent is optional and is provided by the client qualifier 860. In various embodiments, the agent can discover the processing power of the client 810A, the decoding and display capabilities of the client 810A, the latency, reliability, and bandwidth of the communication channel between the client 810A and the video server system 820, the type of display of the client 810A, the firewall present on the client 810A, the hardware of the client 810A, the software executed on the client 810A, and / or registry entries within the client 810A, etc.

[0162] Client Qualifier 860 includes hardware, firmware, and / or software stored on a computer-readable medium. Client Qualifier 860 is optionally disposed on a computing device separate from one or more other elements of video server system 820. For example, in some embodiments, Client Qualifier 860 is configured to identify the characteristics of a communication channel between Client 810 and multiple instances of video server system 820. In these embodiments, the information discovered by Client Qualifier can be used to identify which instance of video server system 820 is optimal for delivering streaming video to one of Clients 810.

[0163] It should be understood that the various embodiments defined herein can be combined or assembled into specific embodiments using the various features disclosed herein. Accordingly, the provided examples are only some possible examples and are not limited to the various embodiments that can define more embodiments by combining various elements. In some examples, some embodiments may include fewer elements without departing from the spirit of the disclosure or equivalent embodiments.

[0164] Embodiments of the present disclosure can be implemented by various computer system configurations including, for example, handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, and mainframe computers. Embodiments of the present disclosure can also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked through a wired or wireless network.

[0165] With the foregoing embodiments in mind, it should be understood that embodiments of the present disclosure can use various computer-implemented operations involving data stored in a computer system. These operations are operations that require physical manipulation of physical quantities. Any of the operations described herein that form part of the embodiments of the present disclosure are useful mechanical operations. Embodiments of the present invention also relate to devices or apparatuses for performing these operations. The apparatus can be specially constructed for the required purpose, or the apparatus can be a general-purpose computer selectively enabled or configured by a computer program stored in the computer. Specifically, various general-purpose machines can be used with a computer program written according to the teachings herein, or it may be more convenient to construct a more specialized apparatus for performing the required operations.

[0166] The present disclosure can also be embodied as computer-readable code on a computer-readable medium. A computer-readable medium is any data storage device that can store data that can later be read by a computer system. Examples of computer-readable media include hard drives, network attached storage (NAS), read-only memory, random access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer-readable medium can include computer-readable tangible media distributed across a computer system connected to a network so that the computer-readable code is stored and executed in a distributed fashion.

[0167] Although the method operations have been described in a particular order, other maintenance operations may be performed between the operations, or the operations may be adjusted to occur at slightly different times, or the operations may be distributed within the system to allow the processing operations to occur at various processing-related intervals, as long as the processing of the overlay operations is performed in the desired manner. It should be understood that this is possible.

[0168] Although the foregoing disclosure has been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Accordingly, the present embodiment should be considered as illustrative and not restrictive, and the embodiments of the present disclosure are not limited to the details provided herein and may be changed within the scope and equivalents of the appended claims.

Claims

1. Executing a first instance of a video game for a user's gameplay; Receiving position-based information associated with the game state during the user's gameplay, which is displayed on a first device associated with the user, wherein the position-based information is generated with reference to the virtual position of a character within the user's gameplay in a game world associated with the video game, and the first device is configured to execute the video game; Determining context-related information associated with the position-based information and the game state; Accessing a message from the user's friend based on the virtual position of the character, wherein the message is associated with the context-related information; Generating a companion interface including the message during the gameplay; Transmitting the companion interface to a second device associated with the user for displaying the message simultaneously with the user's gameplay, wherein the second device is not configured to execute the video game; A method comprising the above steps.

2. Determining that the virtual position of the character is a trigger point; Triggering access to the message based on the trigger point; The method according to claim 1, further comprising the above steps.

3. The gameplay is a first gameplay, The message is generated during a second gameplay of the friend of the video game; The method according to claim 1.

4. Further comprising detecting a predefined event within the user's gameplay that occurs at or near the virtual position, The message is accessed upon detection of the predefined event, The predefined event is detected in response to the character reaching a specific position within the game world during the gameplay; The method according to claim 1.

5. The position-based information for the user's gameplay includes The current property of the character, The current skill of the character, The task completion history within the video game, The current geographical location of the character within the game world, The progress of the video game in the user's game play, The current state of the user's game play, One or more actions of the character within the user's game play, The method according to claim 1, which is related to any of the above.

6. The companion interface is a first companion interface, The method according to claim 1, further comprising the step of establishing a communication session between the user's first companion interface and the friend's second companion interface.

7. Generating a snapshot during the user's game play executed in the first instance with respect to the virtual position of the character, wherein the snapshot includes the game state of the video game and the location-based information for the user's game play, and the snapshot enables the execution of another instance of the video game starting from a location within the video game corresponding to the place where the snapshot was captured. The method according to claim 1 further includes this step.

8. A computer system, A processor, A memory coupled to the processor for storing instructions, wherein if the instructions are executed by the computer system, the computer system will be caused to execute a method. The computer system includes the memory. The method includes, Executing a first instance of a video game for the user's game play, Receiving location-based information associated with the game state during the user's game play displayed on a first device associated with the user, wherein the location-based information is generated with reference to the virtual position of the character within the user's game play in a game world associated with the video game, and the first device is configured to execute the video game. This step, Determining context-related information associated with the location-based information and the game state, Accessing a message from the user's friend based on the virtual position of the character, wherein the message is associated with the context-related information. This step Generating a companion interface that includes the message during the gameplay; Transmitting the companion interface to a second device associated with the user for displaying the message concurrently with the user's gameplay, wherein the second device is not configured to execute the video game; A computer system comprising the above. **Claim 9** The method further includes: Determining that the virtual position of the character is a trigger point; Triggering access to the message based on the trigger point; The computer system according to claim 8, further comprising the above. **Claim 10** The gameplay is a first gameplay, In the method, the message is generated during a second gameplay of a friend of the video game. The computer system according to claim 8. **Claim 11** The method further includes: Detecting a predefined event within the user's gameplay that occurs at or near the virtual position; The message is accessed upon detection of the predefined event; The predefined event is detected in response to the character reaching a specific position within the game world during the gameplay. The computer system according to claim 8. **Claim 12** In the method, the location-based information regarding the user's gameplay is The current property of the character, The current skill of the character, The task completion history within the video game, The current geographical location of the character within the game world, The progress of the video game in the user's gameplay, The current state of the user's gameplay, One or more actions of the character within the user's gameplay. The computer system according to claim 8, related to any of the above. **Claim 13** The method further includes: Establishing a communication session between the user's companion interface and a second companion interface of a friend. The computer system according to claim 8. **Claim 14** The method further includes: A step of generating a snapshot during the user's gameplay executed in the first instance with respect to the virtual position of the character, wherein the snapshot includes the game state of the video game and the position-based information regarding the user's gameplay, and the snapshot enables execution of another instance of the video game starting from a location within the video game corresponding to the location where the snapshot was captured. The computer system according to claim 8, further comprising the step.

15. A non-transitory computer-readable medium storing a computer program for a method, comprising: Program instructions for executing a first instance of a video game for a user's gameplay; Program instructions for receiving position-based information associated with a game state during the user's gameplay displayed on a first device associated with the user, wherein the position-based information is generated with reference to a virtual position of a character within the user's gameplay in a game world associated with the video game, and the first device is configured to execute the video game; Program instructions for determining context-related information associated with the position-based information and the game state; Program instructions for accessing a message from the user's friend based on the virtual position of the character, wherein the message is associated with the context-related information; Program instructions for generating a companion interface including the message during the gameplay; Program instructions for transmitting the companion interface to a second device associated with the user for displaying the message simultaneously with the user's gameplay, wherein the second device is not configured to execute the video game; A non-transitory computer-readable medium comprising the above.

16. Program instructions for determining that the virtual position of the character is a trigger point; Program instructions for triggering access to the message based on the trigger point; The non-transitory computer-readable medium according to claim 15, comprising the above.

17. The game play is a first game play, In the method, the message is generated during a second game play of the friend of the video game, The non-transitory computer-readable medium according to claim 15.

18. Further comprising program instructions for detecting a predefined event within the game play of the user that occurs at or near the virtual location, The message is accessed upon detection of the predefined event, The predefined event is detected in response to the character reaching a specific location within the game world during the game play, The non-transitory computer-readable medium according to claim 15.

19. In the method, the location-based information regarding the user's game play is The current property of the character, The current skill of the character, The task achievement history within the video game, The current geographical location of the character within the game world, The progress of the video game in the user's game play, The current state of the user's game play, One or more actions of the character within the user's game play, The non-transitory computer-readable medium according to claim 15, which is related to any of the above.

20. The companion interface is a first companion interface, Further comprising program instructions for establishing a communication session between the user's first companion interface and the friend's second companion interface, The non-transitory computer-readable medium according to claim 15.

Images