Method, medium and system for providing game assistance

Abstract

The invention provides a method, a medium and a system for providing game assistance. The method for providing game assistance comprises the following steps: determining a current game scene of a game application program, wherein the current game scene at least comprises a game state related to the game application program and user information; sending a query including the current game scenario from the game application; and receiving, by the game application, a response to the query, the response generated based on a comparison of the current game scenario to a plurality of game playing processes of a plurality of users.

Description

[0001] This application is a divisional application of the Chinese invention patent application with application date of March 21, 2019 and application number 201980030234.1. Technical Field

[0002] The present disclosure relates to gaming applications. Among other things, the present disclosure describes methods and systems for providing gaming assistance in response to user queries related to gaming applications. Background of the Invention

[0004] As processing power increases, video games become more complex and extended. However, as complexity increases, users may find it increasingly difficult to navigate and / or complete video games. For example, video games may become more extended and include millions or even trillions of available options for users to use. In this way, users may not even be able to try each of these available options, or even understand which of these options are available. In other cases, users may reach a point in the video game that seems impossible to navigate or solve. This often happens to users, in which case the user becomes increasingly frustrated with the game due to the difficulty or inability to advance through the game, and eventually quits the video game. For example, during the game play of a video game, a user may try to reach another part of the game world through an obvious gate in order to advance the game play process. For example, a user may need to pass through a waterfall that acts as a gate, and seems to have exhausted all available options when trying to pass through. However, even if a known solution is available and simple (e.g., picking a stable rock and running through the waterfall), for whatever reason, the user cannot find the correct path or sequence of operations to enter. After hours of futility, the user may eventually quit the game because the user cannot find a way to advance.

[0005] It is against this background that embodiments of the present disclosure emerge. Summary of the invention

[0006] Embodiments of the present disclosure are directed to providing gaming assistance in response to a player query related to a gaming application, wherein a response is provided to a player device that may or may not be displaying a user's game play, wherein the response may provide assistance during the user's game play, and wherein the response may provide assistance independent of any game play of the user. Several inventive embodiments of the present disclosure are described below.

[0007] In one embodiment, a method for providing game assistance is disclosed. The method includes receiving information related to multiple game play processes of players of a game application over a network at a back-end server. The method includes receiving a query from a first player playing the game application, the query being related to a first game play process of the first player. The method includes determining a current game scenario of the first game play process from the information. The method includes generating a response to the query and the current game scenario based on the information related to the multiple game play processes. For example, the response can be generated by a deep learning engine that is configured to match queries with responses given the current game scenario. The method includes sending the response to a device of the first player.

[0008] In another embodiment, a non-transitory computer-readable medium storing a computer program for providing game assistance is disclosed. The computer-readable medium includes program instructions for receiving information related to multiple game play processes of players of a game application through a network at a back-end server. The computer-readable medium includes program instructions for receiving a query from a first player playing the game application, and the query is related to the first game play process of the first player. The computer-readable medium includes program instructions for determining the current game scenario of the first game play process from the information. The computer-readable medium includes program instructions for generating a response to the query and the current game scenario based on the information related to the multiple game play processes. For example, the response can be generated by a deep learning engine, which is configured to match the query with the response given the current game scenario. The computer-readable medium includes program instructions for sending the response to the device of the first player.

[0009] In yet another embodiment, a computer system is disclosed, the computer system having a processor and a memory coupled to the processor, the memory having instructions stored therein, the instructions causing the computer system to perform a method for providing game assistance when executed by the computer system. The method includes receiving information related to multiple game play processes of players of a game application through a network at a back-end server. The method includes receiving a query from a first player playing a game application, the query being related to a first game play process of the first player. The method includes determining the current game scenario of the first game play process from the information. The method includes generating a response to the query and the current game scenario based on the information related to the multiple game play processes. For example, the response can be generated by a deep learning engine, which is configured to match the query with the response given the current game scenario. The method includes sending the response to a device of the first player.

[0010] In another embodiment, a method for providing game assistance is disclosed, comprising: determining a current game scenario of a game application, the current game scenario including at least a game status and user information related to the game application; sending a query including the current game scenario from the game application; and receiving, by the game application, a response to the query, the response being generated based on a comparison of the current game scenario with multiple game play processes of multiple users.

[0011] In another embodiment, one or more non-transitory computer-readable storage media storing instructions are disclosed, which, when executed by one or more processors of the system, cause the system to perform operations including: determining a current game scenario of a game application, wherein the current game scenario includes at least a game state and user information related to the game application; sending a query from the game application including the current game scenario; and receiving a response to the query by the game application, wherein the response is generated based on comparing the current game scenario with multiple game play processes of multiple users.

[0012] In another embodiment, a system is disclosed, comprising: one or more storage media storing first instructions; and one or more processors configured to execute the instructions to cause the system to perform the operations of the above-mentioned method for providing game assistance.

[0013] Other aspects of the present disclosure will become apparent from the following detailed description taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The present disclosure may be best understood by referring to the following description in conjunction with the accompanying drawings, in which:

[0015] Figure 1A A system for providing gaming assistance during the game play of a player playing a game application, wherein the game application may be executed on a local computing device or through a cloud gaming network, is shown according to one embodiment of the present disclosure.

[0016] Figure 1B A system for providing gaming assistance in response to a verbal query received by a game controller during a game play session of a player playing a game application, wherein the game application may be executed on a local computing device or through a cloud gaming network, is shown according to one embodiment of the present disclosure.

[0017] Figure 1CAn example neural network for building a saccadic motion model for one or more users viewing a VR scene in an HMD is shown according to one embodiment of the present disclosure.

[0018] Figure 2A A system for providing gaming assistance during the game play of a player playing a game application is shown according to one embodiment of the present disclosure.

[0019] Figure 2B A system for providing gaming assistance during a game play session of a player playing a game application that is executed locally by the player is shown according to one embodiment of the present disclosure.

[0020] Figure 2C A system for providing gaming assistance during the game play of a player playing a game application executed over a cloud gaming network is shown according to one embodiment of the present disclosure.

[0021] Figure 3A A data flow diagram is shown according to one embodiment of the present disclosure, which shows the flow of data between a user and a back-end server that provides game assistance during the game play of a player playing a game application, where the game application can be executed on a local computing device or through a cloud gaming network.

[0022] Figure 3B The game play process of a player playing a game application and the periodic collection of information including game context data and global context data according to one embodiment of the present disclosure are shown.

[0023] Figure 3C is a flow chart illustrating steps in a method for providing game assistance during the game play of a player playing a game application, wherein the game application may be executed on a local computing device or through a cloud gaming network, according to one embodiment of the present disclosure.

[0024] Figure 3D is a flow chart illustrating steps in a method for connecting a player to an expert through a communication session to provide real-time assistance during a player's game play of a game application according to one embodiment of the present disclosure.

[0025] Figure 3E is a data flow diagram illustrating the flow of data in a system or method for connecting a player to an expert through a communication session to provide real-time assistance during a player's game play of a game application according to one embodiment of the present disclosure.

[0026] Figure 4AA home screen of a companion mobile device configured to provide gaming assistance to a player playing a gaming application executed on a local gaming console or a cloud gaming network is shown according to one embodiment of the present disclosure.

[0027] Figure 4B According to one embodiment of the present disclosure, Figure 4A Selection of a quick search button on a companion mobile device configured to provide gaming assistance and presentation of information provided in response to a query of the quick search button on a display separate from the mobile device.

[0028] Figure 4C According to one embodiment of the present disclosure, Figure 4A Interaction between a companion mobile device configured to provide gaming assistance and a display that presents the game play progress of a player playing a game application and directs the player to the mobile device to obtain requested information.

[0029] Figure 4D A companion mobile device configured to provide gaming assistance to a user who is not currently playing a gaming application but can present queries related to the gaming application to a backend server for support is shown according to one embodiment of the present disclosure.

[0030] Figure 4E A companion mobile device 11 is shown according to one embodiment of the present disclosure, the companion mobile device being configured to provide gaming assistance to a user to support a player's gaming process.

[0031] FIG. 5A to FIG. 5B The use of a companion mobile device for game assistance associated with game play of a gaming application presented on a separate display is shown in accordance with one embodiment of the present disclosure.

[0032] FIG. 6A to FIG. 6G The use of a companion mobile device for game assistance associated with game play of a gaming application presented on a separate display to navigate a series of connected layers of information provided in response to a query is shown in accordance with one embodiment of the present disclosure.

[0033] Figure 7 The use of a companion mobile device for game assistance associated with a player's game play of a gaming application presented on a separate display is shown according to one embodiment of the present disclosure, wherein an option is presented to connect the player to an expert via a communication session.

[0034] FIG. 8A to FIG. 8GThe use of a display to present a player of a gaming application playing the game and browsing a series of connected layers of information provided in response to a query to acquire an asset is shown according to one embodiment of the present disclosure.

[0035] 9A to 9C The use of a display to present a player of a gaming application playing the game and browsing a series of connected layers of information provided in response to a query to defeat an opponent is shown in accordance with one embodiment of the present disclosure.

[0036] FIG. 10A to FIG. 10D The use of a display to present a player of a gaming application playing the game and browsing a series of connected layers of information provided in response to a query to find an object or location is shown according to one embodiment of the present disclosure.

[0037] FIG. 11A to FIG. 11B A display is shown according to one embodiment of the present disclosure, showing a player playing a game application and browsing a series of connected layers of information provided in response to a query having primary intent and secondary intent (such as a requirement to complete a job, e.g., meet General Tullius).

[0038] FIG. 12A to FIG. 12C The use of a display to present a player of a gaming application playing the game and browsing a series of connected layers of information provided in response to a query to learn how to complete a job is shown in accordance with one embodiment of the present disclosure.

[0039] FIG. 13A to FIG. 13B A display is shown presenting a player of a gaming application playing the game and browsing a series of connected layers of information provided in response to a query to know something about the gaming application (e.g., information about an enemy fighter) according to one embodiment of the present disclosure.

[0040] Fig.14 Components of an example device that may be used to perform aspects of various embodiments of the present disclosure are shown.

[0041] Fig.15 is a diagram showing components of a head mounted display according to an embodiment of the present disclosure.

[0042] Fig.16 is a block diagram of a gaming system according to various embodiments of the present disclosure. DETAILED DESCRIPTION

[0043] Although the following detailed description contains many specific details for the purpose of illustration, it will be appreciated by those skilled in the art that many variations and modifications of the following details are within the scope of the present disclosure. Therefore, in setting forth the various aspects of the present disclosure described below, the generality of the appended claims of this description is not compromised and no limitations are imposed on these claims.

[0044] In general, various embodiments of the present disclosure describe systems and methods for providing game assistance during or independently of a game playing process of a player playing a game application. For example, when a player is stuck on a part of a game application, the player can request help by querying a back-end game assistance server. The query can be in any format, or a combination of formats, including audio, text, video, etc. The game assistance server is configured to match the query with a modeled response based on the current game scenario from which the query is raised. The matching is performed by a deep learning engine, for example, by a game assistance server or by a third party providing artificial intelligence (AI) to include deep learning. The response can be presented to the user in any format or combination of formats (including audio, text, video, etc.). In this way, the game experience of the player is improved. During the game playing process of the game application, the player can generate a query at any time for any reason. The player may wish to obtain more information about certain aspects of the gaming application, such as character information, strategy information, the overall popularity of the gaming application, scheduled play times with friends in a multi-player format via a cloud gaming service, obtain information about social network friends who are playing the same gaming application or another gaming application (e.g., discover what gaming application a friend is playing, obtain information about the gaming application, obtain purchase information about the gaming application, etc.), obtain information about how to overcome certain obstacles that prevent the player from progressing in the gaming application, etc. Thus, the player has a more immersive experience with the gaming application, so that the gaming application is not simply played in an out-of-the-box manner, that is, the player can enhance his or her gaming process with information (e.g., audio, text, video, etc.) that is presented concurrently with the gaming process, and / or through an ongoing conversation with an automated gaming assistant (e.g., a game manager / navigator) that is configured to provide psychological and informational support to the player during the gaming process. Furthermore, the role of the gaming console is expanded in providing a more complete and richer set of information from which information models can be constructed for responding to queries. That is, in order to determine the situational information about the game application and / or the player, low-level operating system (OS) information can be transmitted from the game console during the player's game playing process. That is, not only the information about the game playing process is used, but also the information related to the gameplay is used to fill in the situational information for responding to one or more queries. OS level information can include button sequence, button actuation speed, game speed, the time the player plays, the duration of the player's play, the game the player plays, etc. Moreover, the game console can distribute game assistance functions to one or more devices. For example, response modeling and the matching of queries and responses are performed by the back-end game assistance server instead of the game console.In addition, the game console can distribute the responses delivered by the game assistance server to one or more other devices for presentation to the player. For example, the responses can be provided on a display simultaneously with the current game play, or the responses can be provided to a companion device (e.g., a mobile phone) that works in conjunction with the display presenting the game play, or the responses can be provided through a set of separate speakers dedicated to providing the responses independent of the audio of the game play (e.g., separate speakers, speakers located on the game controller, etc.). In this way, there is minimal disruption to the game play of the game application because there is no audio mixing, no or minimal video mixing, etc.

[0045] With the above general understanding of the various embodiments in mind, example details of the embodiments will now be described with reference to the various figures.

[0046] Throughout the specification, references to "game application" are intended to mean any type of interactive application directed by the execution of input commands. For purposes of illustration only, interactive applications include applications for gaming, word processing, video processing, video game processing, etc. Furthermore, the terms video game and game application are interchangeable.

[0047] Figure 1A A system 100A is shown for providing game assistance during the game play of a player 5 playing a game application, which may be executed on a local computing device 100 or through a cloud gaming network (not shown for clarity), according to one embodiment of the present disclosure. Game assistance may be provided in conjunction with the game play, such as when the player 5 requests information about an object in the game application, or requests assistance in overcoming an obstacle in the game application. In other embodiments, game assistance may be provided independent of the game play, such as when the player is in an environment that is not involved in any game play (e.g., at work, in a taxi, on public transportation, etc.), but is interested in making inquiries about the game application or generally about the game environment.

[0048] In particular, if Figure 1A As shown, the game application can be executed locally at the client device 100 of the player 5, or can be executed at a backend game execution engine running at a backend game server (not shown) of a cloud gaming network or gaming cloud system.

[0049] In addition, the client device 100 acts as a front end to a game assistance server 140 at the back end of the system 100A. In particular, the client device 100 includes a client game assistance module 120 that is configured to receive queries from the player 5 and, with the cooperation of the back end game assistance server 140, deliver responses to those queries back to the player 5, as will be described below. The user interface 110 at the client device 100 supports the client game assistance module 120 by receiving queries (e.g., text) and presenting responses (e.g., audio, video, etc.).

[0050] Furthermore, the game assistance module 120 cooperates with the client-side automatic speech recognition (ASR) engine 125 to optionally convert the query from a first format to a second format. For example, the query may come in the form of audio from the player 5, and the ASR engine 125 is configured to convert the audio query to a text format more suitable for downstream device processing (e.g., analysis, transmission, etc.). The ASR engine 125 can also be configured to convert text to audio so that responses received in text are broadcast in audio.

[0051] The client device 100 is configured to request access to a game application through a network 150 such as the Internet, and to render an instance of a video game or game application executed by a game execution engine 111 or a backend game server 205 (see FIG. 2 ) and pass the rendered instance (e.g., audio and video) to a display device 12 and / or a head mounted display (HMD) 102 associated with a player 5. For example, the player 5 may be interacting with an instance of a game application executed on a cloud game processor 201 through the client device 100. As shown, the game execution engine 111 is configured for local execution of the game application, as described above. The client device 100 can receive input from various types of input devices (such as a game controller 6, a tablet computer 11, a keyboard) and gestures captured by a camera, a mouse, a touchpad, etc. The client device 100 can be any type of computing device having at least a memory and a processor module that can be connected to the game server 205 through the network 150. Some examples of client devices 100 include personal computers (PCs), game consoles, home theater devices, general purpose computers, mobile computing devices, tablet computers, phones, or any other type of computing device that can interact with game server 205 to execute an instance of a video game. In an embodiment, HMD 102 can be configured to perform the functions of client device 100.

[0052] The client device 100 may interact with a local query agent 115 that acts as an interface with the player 5. For example, the query agent 115 includes a microphone 131 configured to receive and / or record audio from the player 5. In this manner, a verbal query 181 presented by the player 5 may be input into the system 100A through the query agent 115. In one embodiment, the query agent 115 passes the query 182 to the client game assistance module 120 of the client device 100 in the same format without any conversion. In another embodiment, the client device 100 further includes an ASR engine 141C to convert the query from a first format to a second format (e.g., audio to text, text to audio, etc.), and then passes the query 182 to the client game assistance module 120 in the second format.

[0053] In other embodiments, the query 181 may be presented through another communication path. For example, the query may be in the form of text or video and received through the mobile device 11 or display 12 (e.g., through a keyboard, etc.) or through an interface of the HMD 102. The query 181 may be delivered to the client device directly (e.g., from the mobile device 11) or through the user interface 110. In another embodiment, the query 181' is delivered directly from the mobile device 11 (or another type of device) to the game assistance server 140 through the network 150. Similarly, the response 195' is delivered directly back to the mobile device or any other device.

[0054] A query 181 in a specific format (e.g., a first or second text or audio format) is passed from the client game assistance module 120 to the back-end game assistance server 100A via the network 150, which is configured to provide game assistance to the player 5 in response to the query. The game assistance server 140 includes an ASR engine 141B, which is optionally configured to convert the query 181 from one format to another. For example, the ASR engine 141B can convert the query from audio to text (e.g., using a deep learning engine to perform speech recognition), or vice versa. It is important to note that the conversion can occur at any point along the path taken by the query 181 of the player 5, including at the game assistance server 140, or downstream of the game assistance server 140. In some embodiments, because the ASR engine may require a large amount of resources (e.g., including a deep learning engine), the ASR engine can be located at the game assistance server 140 or at another third-party server that performs ASR.

[0055] In addition, the game assistance server 140 includes a preliminary scenario analyzer 145, which is configured to provide a game scenario in association with at least the query 181. Scenario data 184 including game scenario data (including low-level OS scenario data) and global scenario data (e.g., information related to a user profile) is passed from the client device 100 (e.g., generated in part by the game execution engine 111) to the game assistance server 140. The preliminary scenario analyzer 145 can determine the current scenario 187 (e.g., the current game scenario) associated with the query 181. For example, the preliminary scenario analyzer 145 can determine that the query 181 is presented in association with the first game application. Additional scenarios can be determined, such as which scene or level the player participated in during the game. In this way, some scenarios from which responses may be matched or generated are provided for the query. For example, the first game application and the second game application may generally have the same obstacle (e.g., climbing over a mountain wall), while a general query without additional scenarios (how do I climb over this wall) may return an incorrect response to the wrong game application. However, a query along with the appropriate current context 187 will return an appropriate response.

[0056] In one embodiment, the response 195 is generated by a third-party artificial intelligence (AI) processor 160 including an ASR engine 141A, a natural language processing (NLP) engine 142, and a machine or deep learning engine 190. In particular, the query 181 and the current scenario 187 are passed from the game assistance server 140 to the AI ​​processor 160. If the query 181 has not yet been converted to an appropriate format suitable for processing by the AI ​​processor 141A, the ASR engine 141A is enabled to perform appropriate conversions (e.g., audio / speech to text, etc.), such as using speech recognition technology. In another embodiment, the response 195 is generated by a game assistance server 140 that also includes an NLP engine and a deep learning engine.

[0057] Additionally, once the query 181 has been converted to an appropriate format (e.g., text), the NLP engine 142 is configured to interpret the nature of the query, or understand what the player 5 is requesting (e.g., the content of the query). For example, the NLP engine may analyze the query in text form using a deep learning engine to determine the meaning of the query. In this manner, the AI ​​processor 160 may match or generate an appropriate response to the interpreted query for a given current scenario 187, as will be described below.

[0058] In particular, based on contextual information collected from the player 5 and other players playing the game application, the player 5 may be presented with a pre-generated response 195 that directly responds to the query 181 given the current context 187. The pre-generated response 195 may be a modeled response to that particular query 181 or to similar queries, as learned by the deep learning engine 190. In this way, the deep learning engine 190 is configured to match the query 181 to an appropriate modeled response 195. In particular, the deep learning engine 190 is configured to match the interpreted query to a response model or query / response model in order to provide a response to the query. In one embodiment, when no response (e.g., text, audio, video, etc.) matches the query, a new response may be generated based on one or more closest matching models.

[0059] In this way, the deep learning engine 190 returns the response 195 to the game assistance server 140. The response 195 may have an appropriate format. For example, the response 195 may have a first format, such as text. At a certain point during the transmission back to the player 5, the response 195 may be converted into a format suitable for the player 5. In one implementation, the response 195 is converted from text to audio at a certain point (e.g., at the ASR engine 141A of the AI ​​processor 141A, at the ASR engine 141B of the game assistance server 140, at the ASR engine 125 of the client device 100, or at the ASR engine 141C of the local query agent 115). In this way, the speaker 132 at the local query agent is able to broadcast the query 195 to the player 5. In another implementation, the response 195 is transmitted back to the client device 100 in text form and is transmitted to an appropriate device docked with the player 5, such as a display 12, a mobile device 11, an HMD 102, etc. Other implementations are also supported, such as presenting the response as a video via a display or by another appropriate means.

[0060] Figure 1B A system 100B for providing gaming assistance in response to a verbal query received by a game controller during a game session of a player playing a game application, wherein the game application may be executed on a local computing device or through a cloud gaming network, is shown according to one embodiment of the present disclosure. The functionality of the system 100B is similar to Figure 1AThe client device 100 may perform the same functions as the system 100A, except that the controller 6 is configured as a local query agent. In particular, the client device 100 may interact with the controller 6 that acts as an interface with the player 5. For example, the controller 6 includes a microphone 131' configured to receive and / or record audio from the player 5 (e.g., to receive the query 181). The controller 5 may include an ASR engine 141C' configured to optionally convert the query from a first format to a second format (e.g., audio to text, text to audio, etc.). In addition, the controller 6 includes a speaker 132' configured to broadcast the response 195 for the benefit of the player 5.

[0061] Figure 1C An example neural network for building a response model and / or a query / response model based on the context information and corresponding queries of a game application according to an embodiment of the present disclosure is shown. Specifically, for example, the deep learning or machine learning engine 190 in the AI ​​processor 160 is configured as an input to receive information related to the context data, the query, and the basic response. The deep learning engine 190 uses artificial intelligence (including deep learning algorithms, reinforcement learning, or other algorithms based on artificial intelligence) to build a response model and / or a query / response model. That is, during the learning and / or modeling phase, the deep learning engine 190 uses input data to create a response model and / or a query / response model that can be used to respond to one or more queries. In some embodiments, the deep learning engine 190 is also configured for ASR and NLP processing.

[0062] In particular, neural network 190 represents an example of an automated analysis tool for analyzing a data set to determine the responses, actions, behaviors, needs, and / or requirements of a corresponding user. Different types of neural networks 190 are possible. In an example, neural network 190 supports deep learning. Correspondingly, deep neural networks, convolutional deep neural networks, and / or recurrent neural networks using supervised or unsupervised training can be implemented. In another example, neural network 190 includes a deep learning network that supports reinforcement learning. For example, neural network 190 is configured as a Markov decision process (MDP) that supports a reinforcement learning algorithm.

[0063] In general, neural network 190 represents a network of interconnected nodes, such as an artificial neural network. Each node learns some information from the data. Knowledge can be exchanged between nodes through the interconnection. Input to neural network 190 activates a set of nodes. This set of nodes in turn activates other nodes, thereby propagating knowledge about the input. This activation process is repeated across other nodes until an output is provided.

[0064] As shown, neural network 190 includes a hierarchy of nodes. At the lowest level of the hierarchy, there is an input layer 191. Input layer 191 includes a set of input nodes. Each of these input nodes is mapped to local data 115 actively collected by actuators or passively collected by sensors, for example, during monitoring of a test user / object performing a corresponding glance (e.g., eye orientation data).

[0065] At the highest hierarchical level, there is an output layer 193. The output layer 193 includes a set of output nodes. The output nodes represent decisions (e.g., predictions) related to information about a response (e.g., for a given scenario, etc.). In this way, the output nodes can match a query to a specific response given a corresponding scenario.

[0066] These results can be compared to predetermined and true results obtained from previous interactions and monitoring of the test subject in order to refine and / or modify the parameters used by the deep learning engine 190 to iteratively determine the appropriate response model and / or query / response model. That is, the nodes in the neural network 190 will learn the parameters of the model that can be used to make such decisions as the parameters are refined. In this way, a given query can be associated with an increasingly refined modeled response, and potentially a new modeled response.

[0067] In particular, there is a hidden layer 192 between the input layer 191 and the output layer 193. The hidden layer 192 includes "N" hidden layers, where "N" is an integer greater than or equal to 1. Each hidden layer includes a set of hidden nodes. The input nodes are interconnected to the hidden nodes. Similarly, the hidden nodes are interconnected to the output nodes, so that the input nodes are not directly interconnected to the output nodes. If there are multiple hidden layers, the input nodes are interconnected to the hidden nodes of the lowest hidden layer. These hidden nodes are interconnected to the hidden nodes of the next hidden layer, and so on. The hidden nodes of the next highest hidden layer are interconnected to the output nodes. The interconnection connects two nodes. The interconnection has a numerical weight that can be learned so that the neural network 190 adapts to the input and is able to learn.

[0068] Typically, the hidden layer 192 allows knowledge about the input nodes to be shared between all jobs corresponding to the output nodes. To this end, in one implementation, the transformation f is applied to the input nodes through the hidden layer 192. In one example, the transformation f is nonlinear. Different nonlinear transformations f can be obtained, including, for example, a linear rectification function f(x)=max(0,x).

[0069] The neural network 190 also uses a cost function c to find the optimal solution. The cost function measures the deviation between the prediction output by the neural network 190 (defined as f(x) for a given input x) and the ground truth or target value y (e.g., the expected result). The optimal solution represents a situation in which no solution has a cost lower than the cost of the optimal solution. For data for which such ground truth labels are available, an example of a cost function is the mean squared error between the prediction and the ground truth. During the learning process, the neural network 190 can use a back-propagation algorithm to employ different optimization methods to learn the model parameters (e.g., the weights of the interconnections between nodes in the hidden layer 192) that minimize the cost function. An example of such an optimization method is stochastic gradient descent.

[0070] In one example, the training data sets for neural network 190 can be from the same data domain. For example, neural network 190 is trained to learn patterns and / or characteristics of similar queries based on a given set of inputs or input data. For example, the data domain includes queries related to specific scenarios for a given game scenario in a game application. In another example, the training data sets are from different data domains to include input data other than the baseline. In this way, neural network 190 can use other data domains to identify queries, or can be configured to generate response models for given queries based on those data domains.

[0071] Figure 2A A system 10 for providing gaming assistance during a game play session of a player playing a game application according to one embodiment of the present disclosure is shown. For example, the assistance may be provided through a user interface configured to support the game play session (e.g., through a companion device—mobile phone—or speaker, controller speaker, etc.). According to one embodiment of the present disclosure, the game application may be executed on a local computing device or through a cloud gaming network.

[0072] like Figure 2A As shown, the game application may be executed locally at the client device 100 of the player 5, or may be executed at a backend game execution engine 211 operating at a backend game server 205 of the cloud gaming network or gaming cloud system. The game execution engine 211 may operate within one of many game processors 201 of the game server 205. In either case, the cloud gaming network is configured to provide game assistance to the player by providing responses to queries. For example, the game server 205 and / or the game processor 201 may include a game assistance server 140 configured to provide game assistance to the player 5, and may include an AI processor 160', as previously described. In some embodiments, the AI ​​is provided by a third party (e.g., Figure 1A to Figure 1B The AI ​​processor 160) is executed.

[0073] In one implementation, the user interface 110 at the client device 100 can support the provision of game assistance, enabling a player to pose queries through the user interface 110 and interact (e.g., see, hear, be presented, etc.) with responses provided in the user interface 110. In other embodiments, the user interface is provided in a different device, such as a mobile device (e.g., a phone) or a speaker, as previously described. In addition, the game application can be executed in a single player mode or a multi-player mode, wherein embodiments of the present invention provide multi-player enhancements (e.g., assistance, communication, etc.) for both modes of operation.

[0074] In some embodiments, the cloud gaming network may include multiple virtual machines (VMs) running on a host's hypervisor, wherein one or more virtual machines are configured to execute the game processor module 201 using hardware resources available to the hypervisor of the host supporting single-player or multi-player video games. In other embodiments, the cloud gaming network is configured to support multiple local computing devices supporting multiple users, wherein each local computing device can execute an instance of a video game such as in a single-player or multi-player video game. For example, in multi-player mode, when the video game is executed locally, the cloud gaming network simultaneously receives information (e.g., game state data) from each local computing device, and distributes the information to one or more of the local computing devices accordingly, so that each user can interact with other users in the gaming environment of the multi-player video game (e.g., through corresponding characters in the video game). In this way, the cloud gaming network coordinates and combines the gaming process of each user within the multi-player gaming environment.

[0075] As shown, system 10 includes a game server 205 executing a game processor module 201, which provides access to multiple interactive game applications. Game server 205 can be any type of server computing device available in the cloud, and can be configured to one or more virtual machines executed on one or more hosts, as previously described. For example, game server 205 can manage the virtual machines supporting game processor 201. Game server 205 is also configured to provide additional services and / or content to user 5. For example, game server 205 can be configured to provide game assistance to the player 5 who is playing the corresponding game application or is not currently playing but is interested in the information related to the game application. Assistance can be in the form of a response to an inquiry. In some embodiments, if there is no response to match a given inquiry, the game server can be configured to connect the player of the game application to an expert to provide real-time assistance through a communication session, wherein the game server is configured to receive a request for assistance, matches the player with an appropriate expert, and during the player's game process (such as by a help session controller 220) establishes a help session that connects the player to an expert in real time.

[0076] The client device 100 is configured to request access to a game application over a network 150 (such as the Internet) and to render an instance of a video game or game application that is executed by a game server 205 and delivered to a display device 12 and / or a head mounted display (HMD) 102 associated with a user 5. For example, the user 5 may be interacting with an instance of a game application executed on a game processor 201 through the client device 100. The client device 100 may also include a game execution engine 111 configured for local execution of the game application, as described above. The client device 100 may receive input from various types of input devices (such as a game controller 6, a tablet computer 11, a keyboard) and gestures captured by a camera, a mouse, a touchpad, etc. The client device 100 may be any type of computing device having at least a memory and a processor module that is capable of connecting to the game server 205 over the network 150. Some examples of client devices 100 include personal computers (PCs), game consoles, home theater devices, general purpose computers, mobile computing devices, tablet computers, phones, or any other type of computing device that can interact with game server 205 to execute an instance of a video game. In an embodiment, HMD 102 can be configured to perform the functions of client device 100.

[0077] The client device 100 is configured to receive the rendered image and to display the rendered image on the display 12 and / or the HMD 102. For example, the rendered image may be delivered by an instance of a game application executing on a game execution engine 211 of the game server 205 in association with the user 5, via the network 150. In another example, the rendered image may be delivered by a local game execution engine 111, via local game processing. In either case, the client device 100 is configured to interact with the execution engine 211 or 111 in association with the game play process of the user 5, such as through input commands for driving the game play process.

[0078] In addition, the client device 100 is configured to interact with the game server 205 to capture and store one or more game scenarios (e.g., low-level OS scenarios) and global scenario information of the game process of the user 5 when playing the game application. Each game scenario includes information related to the game process (e.g., game state, user information, etc.), and may include low-level OS information related to hardware operation (e.g., actuated buttons, actuation speed, time of the game process, etc.). More specifically, the game processor 201 of the game server 205 is configured to generate and / or receive game and / or OS level scenarios of the game process of the user 5 when playing the game application. In addition, global scenario data is also collected, and the global scenario data is generally related to the user profile data (e.g., the length of time the player played the game application, the last time the player played the game application, the frequency of the player requesting assistance, how the player's skills are compared with other players, etc.). In another implementation, the game scenario including the OS level scenario and the global scenario can be generated by the local game execution engine 111 on the client device 100, and output and transmitted to the game processor 201 through the network 150. In addition, the game scenario including the OS level scenario and the global scenario can be generated by the game execution engine 211 within the game processor 201 at the cloud network, such as by the game scenario generator 222. The game scenario including the OS level scenario and the global scenario can be stored locally on the client device 100 and / or stored at the scenario profile database 242 of the game server 205.

[0079] In particular, each game scenario includes metadata and / or information related to the game playing process. Game scenarios can be captured at various points in the process of playing the game application (such as in the middle of the level). For illustration, the game scenario can help determine the position of the player (e.g., the player's character) in the game application, the position of the player in the game application, what the player has done, which assets and skills the player or character has accumulated, which tasks or jobs are presented to the player, and the whereabouts of the player in the game application. In addition, the metadata and information in each game scenario can provide and / or be analyzed to provide support related to the player's game playing process (such as when matching the query with the response), wherein the game playing process has a specific scenario related to the query, and the matched response is most suitable for answering the query, as determined by deep learning. Specifically, based on the game scenario, the client device 100 is configured to interact with the game server 205 to display a user interface, which can display the response associated with the query to provide game assistance during the player's game playing process or independently of any game playing process. In some embodiments, the response is provided by a separate device (e.g., a speaker).

[0080] More specifically, the game scenario also includes game state data that defines the game state at this point. For example, the game state data may include game characters, game objects, game object attributes, game attributes, game object states, graphic overlays, the position of the character in the game world of the game process of the player 5, the scene or game environment of the game process, the level of the game application, the assets of the character (e.g., weapons, tools, bombs, etc.), the type or race of the character (e.g., wizards, soldiers, etc.), the current task and / or work presented to the player, equipment, the skill set of the character, the game level, the character attribute, the character position, the remaining number of lives, the total number of possible lives available, armor, trophies, time counter values ​​and other asset information, etc. In this way, the game state data allows the generation of the game environment at the corresponding point in the video game. The game state data may also include the state of each device for rendering the game process, such as the state of the CPU, the state of the GPU, the state of the memory, the register value, the program counter value, the programmable DMA state, the buffer data of the DMA, the audio chip state, the CD-ROM state, etc. The game state data may include low-level OS data, such as the button actuated, the actuation speed, which game application was played, and other hardware-related data. The game state data is stored in the game state database 245 .

[0081] In addition, the game scenario may include user and / or player information associated with the player. Typically, the data saved by the user / player includes information that personalizes the video game for the corresponding player. This includes information associated with the player's character so that the video game is rendered with a character that may be unique to the player (e.g., body type, race, appearance, clothing, weapons, etc.). In this way, the data saved by the user / player enables the generation of a character for the corresponding player's game play, wherein the character has a state corresponding to a point in the game application associated with the game scenario.

[0082] In one implementation, a game scenario is associated with snapshot information that provides information that enables an instance of a video game to be executed from a point in the video game associated with the corresponding snapshot. Access to a particular snapshot captured and stored during a player's game play session allows another instance of the game application to be executed using the information in the snapshot (such as the game state and user information that may be associated with the previously described game scenario). For example, another user can jump into a parallel version of the game play session associated with the snapshot. A complete discussion of the creation and use of snapshots is provided in U.S. application serial number 15 / 411,421, entitled "Method And System For Saving A Snapshot of Game Play And Used To Begin Later Execution Of The Game Play By Any User As Executed On A Game Cloud System," the entire contents of which are incorporated herein by reference.

[0083] In one embodiment, the snapshot includes a snapshot image of the scene rendered at that point. The snapshot images are stored in the snapshot image database 246. The snapshot images can be presented in the form of thumbnails relative to the timeline, wherein the snapshots provide various views of the user's game play at corresponding points in the user's progression through the video game as shown in the timeline. The snapshot generator 212 can be configured to generate snapshots during the game play process.

[0084] In yet other embodiments, the contextual data includes global contextual data. For example, the global contextual data may include the player's skills or abilities, the player's overall readiness to seek help, the player's recentness in playing a game application, the game difficulty selected by the user 5 when playing a game, etc. For example, the user / player saved data may also include user profile data identifying the player 5. The user / player saved data is stored in the database 241 or the player profile database 243.

[0085] In addition, a player profile including information related to the corresponding player may be generated and stored in the profile database 243. The profile information may include name, age, residence, account information, user-related information from the gaming scenario (e.g., user-saved data stored in the database 241), etc. The player / expert gaming profile generator 221 is configured to create and manage player profiles.

[0086] As previously described, the game processor 201 may include a game assistance server 140 configured to provide game assistance to players. In particular, the game assistance server 140 receives a query, optionally converts the query into an appropriate format, and sends the query to an AI engine (e.g., including an ASR engine, an NLP engine, and a deep learning engine) to match the query to an appropriate modeled response. For example, when a player requests help to overcome an obstacle (e.g., climbing over a wall), the game assistance server 140 processes the query within its system (e.g., locally through a local AI processor 160'), or sends the query to a third-party AI processor 160 for processing, as described above. Once the modeled response is matched, the response is returned to the client device 100 for further distribution to, for example, a display or speaker, etc. In some embodiments, the game assistance can be passed to a device 11 (e.g., a tablet computer) for display and interaction, wherein the device 11 can be separated from a client device 100 (companion device) configured to execute and / or support the execution of a game application to interact with a user 5. For example, a first communication channel may be established between the game server 205 and the client device 100, and a separate second communication channel may be established between the game server 205 and the device 11 to deliver game assistance. The modeled responses and / or modeled queries / responses may be stored in the database 247 as they are generated by deep learning.

[0087] Figure 2B A system 206B is shown that provides game assistance to a player during or outside of a game play session of a player playing a game application using a game assistance server 140 at a backend server that is configured to match a player's query (e.g., text, audio, video, etc.) to an appropriate response (e.g., text, audio, video, etc.). Figure 2B As shown, the game application is executed locally by the corresponding player, and the backend server support (e.g., accessible through the game server 205) can realize the establishment and management of the help session. In one embodiment, the system 206B and Figure 2A The system 10 works in conjunction with the game cloud system 210 to provide game assistance to the player by providing a response to the query at the game cloud system 210, as described above. Now referring to the drawings, the same reference numerals represent the same or corresponding parts.

[0088] like Figure 2BAs shown, multiple players 215 (e.g., player 5A, player 5B, ..., player 5N) are playing multiple game applications, each of which is locally executed on a corresponding client device 100 (e.g., a game console) of a corresponding user. System 206B supports the game play process of multiple players 215 at one or more moments (such as within a period of time). In addition, each of the multiple players 215 can access device 11 or other devices previously introduced (e.g., speakers), which are configured to receive information that provides game assistance, as described above. Each client device 100 can be similarly configured to perform local execution of the corresponding game application. For example, player 5A may be playing a first game application on a corresponding client device 100, where an instance of the first game application is executed by a corresponding game title execution engine 111. The game logic 126A (e.g., executable code) that implements the first game application is stored on the corresponding client device 100 and is used to execute the first game application. For the purpose of illustration, the game logic may be delivered to the corresponding client device 100 via a portable medium (e.g., a flash drive, a compact disc, etc.) or via a network (e.g., downloaded from a game provider via the Internet 150). In addition, player 5B is playing a second game application on the corresponding client device 100, where an instance of the second game application is executed by the corresponding game title execution engine 111. The second game application may be the same as the first game application executed for player 5A or a different game application. As previously described, the game logic 126B (e.g., executable code) that implements the second game application is stored on the corresponding client device 100 and is used to execute the second game application. In addition, player 115N is playing an Nth game application on the corresponding client device 100, where an instance of the Nth game application is executed by the corresponding game title execution engine 111. The Nth game application may be the same as the first or second game application, or may be a completely different game application. As previously described, the game logic 126N (e.g., executable code) that implements the third game application is stored on the corresponding client device 100 and is used to execute the Nth game application.

[0089] As previously described, each client device 100 can receive input from various types of input devices (such as game controllers, tablet computers, keyboards) and gestures captured by cameras, mice, touch pads, etc. The client device 100 can be any type of computing device having at least a memory and a processor module that can be connected to the game server 205 via the network 150. Moreover, the client device 100 of the corresponding player is configured to generate a rendered image executed by the game title execution engine 111 executed locally or remotely, and to display the rendered image on a display. For example, the rendered image can be associated with an instance of a first game application executed on the client device 100 of the player 5A. For example, the corresponding client device 100 is configured to interact with the instance of the corresponding game application executed locally or remotely to implement the game play process of the corresponding player, such as through input commands for driving the game play process.

[0090] In one embodiment, the client device 100 operates in a single player mode for a corresponding player who is playing the game application.

[0091] In another embodiment, multiple client devices 100 operate in a multi-player mode for each corresponding player playing a particular game application. In that case, multi-player functionality can be provided, such as through a multi-player processing engine 119, via the back-end server support of the game server. In particular, the multi-player processing engine 119 is configured to control a multi-player game session for a particular game application. For example, the multi-player processing engine 130 communicates with a multi-player session controller 116, which is configured to establish and maintain a communication session with each user and / or player participating in the multi-player game session. In this way, the players in the session can communicate with each other under the control of the multi-player session controller 116.

[0092] In addition, the multi-player processing engine 119 communicates with the multi-player logic 118 to enable interaction between users within each user's corresponding game environment. In particular, the state sharing module 117 is configured to manage the state of each user in the multi-player game session. For example, the state data may include game state data that defines the state of the game play process (of the game application) for the corresponding user at a specific point. For example, the game state data may include game characters, game objects, game object properties, game properties, game object states, graphical overlays, etc. In this way, the game state data allows the generation of a game environment that exists at a corresponding point in the game application. The game state data may also include the state of each device used to render the game play process, such as the state of the CPU, the state of the GPU, the state of the memory, register values, program counter values, programmable DMA state, DMA buffer data, audio chip state, CD-ROM state, etc. The game state data may also identify from this point which parts of the executable code need to be loaded in order to execute the video game. The game state data may be stored in Figure 2A in the database 240 and can be accessed by the state sharing module 117 .

[0093] In addition, the state data may include data saved by the user, which includes information that personalizes the video game for the corresponding player. This includes information associated with the role played by the user, so as to render the video game with a role (e.g., position, body shape, appearance, clothing, weapons, etc.) that may be unique to the user. In this way, the data saved by the user enables the generation of a role for the corresponding user's game play process, wherein the role has a state corresponding to a point in the game application currently experienced by the corresponding user. For example, the data saved by the user may include the game difficulty, game level, character attributes, character position, remaining life number, available possible life total number, armor, trophy, time counter value, etc. selected by the corresponding user when playing the game. For example, the data saved by the user may also include user profile data that identifies the corresponding player. The data saved by the user may be stored in the database 240.

[0094] In this manner, the multiplayer processing engine 119 using the state sharing data 117 and the multiplayer logic 118 is able to overlay / insert objects and characters into each of the game environments of the users participating in the multiplayer game session. For example, a first user's character is overlaid / inserted into a second user's game environment. This allows interaction between the users in the multiplayer game session via each of their respective game environments (e.g., as displayed on a screen).

[0095] According to one embodiment of the present disclosure, assistance in supporting the game play process of the corresponding player can be provided via the back-end server support of the game server 205, such as providing a response that matches the player's query, as will be described below. For example, the game assistance server 140 is configured to cooperate with the game server 205 to provide game assistance. In particular, as previously described, the game assistance server 140 works in conjunction with the AI ​​processor 160 to match queries with responses using a deep learning engine. For example, the AI ​​processor 160 includes: an ASR engine 141A, which is configured to optionally convert the query into a format suitable for use; and an NLP engine, which is configured to interpret the query. In addition, the deep learning engine 190 is configured to match the query with a modeled response previously learned by the deep learning engine.

[0096] Figure 2C A system 206C for providing game assistance during the game play of a player playing a game application executed on a cloud gaming network according to one embodiment of the present disclosure is shown. As shown, according to one embodiment of the present disclosure, the system 106C provides game control to multiple players 215 (e.g., players 5L, 5M, ..., 5Z) playing a game application executed on the cloud gaming network. In some embodiments, the cloud gaming network can be a gaming cloud system 210, which includes multiple virtual machines (VMs) running on a host's hypervisor, wherein one or more virtual machines are configured to utilize hardware resources available to the host's hypervisor to execute a game processor module. In one embodiment, the system 206C is connected to Figure 2A The system 10 works in cooperation to provide game assistance associated with the query. Referring now to the drawings, like reference numerals designate like or corresponding parts.

[0097] As shown, the game cloud system 210 includes a game server 205 that provides access to multiple interactive video games or game applications. The game server 205 can be any type of server computing device available in the cloud, and can be configured as one or more virtual machines executed on one or more hosts. For example, the game server 205 can manage a virtual machine that supports a game processor that instantiates an instance of a user's game application. In this way, the multiple game processors of the game server 205 associated with multiple virtual machines are configured to execute multiple instances of the game application associated with the game playing process of multiple users 215. In this way, the back-end server supports providing the media streams (e.g., video, audio, etc.) of the game playing process of multiple game applications to multiple corresponding users.

[0098] A plurality of players 215 access the gaming cloud system 210 via the network 150, wherein the players (eg, players 5L, 5M, ..., 5Z) access the network 150 via corresponding client devices 100', wherein the client devices 100' may be similarly configured as Figure 1A to Figure 1B and FIG. 2A to FIG. 2B The client device 100 of the corresponding player 5L (e.g., including the game execution engine 111, etc.), or a thin client (e.g., including the game execution engine 211) that can be configured to provide an interface with a back-end server that provides computing functions. In addition, each of the multiple players 215 can access the device 11 or any other device previously introduced (e.g., a speaker) that is configured to present game assistance (e.g., a response) to answer one or more queries, as previously described. In particular, the client device 100' of the corresponding player 5L is configured to request access to a game application through a network 150 (such as the Internet), and to render an instance of a game application (e.g., a video game) executed by the game server 205 and delivered to a display device associated with the corresponding player 5L. For example, the player 5L may be interacting with an instance of a game application executed on a game processor of the game server 205 through the client device 100'. More specifically, the instance of the game application is executed by the game title execution engine 211. As previously described, the game logic (e.g., executable code) that implements the game application is stored and accessible through the data storage area 140, and is used to execute the game application. As shown, the game title processing engine 211 is capable of using multiple game logics 177 to support multiple game applications.

[0099] As previously described, the client device 100' can receive input from various types of input devices (such as game controllers, tablet computers, keyboards) and gestures captured by cameras, mice, touchpads, etc. The client device 100' can be any type of computing device having at least a memory and a processor module that can be connected to the game server 205 via the network 150. Moreover, the client device 100' of the corresponding player is configured to generate a rendered image executed by the game title execution engine 211 executed locally or remotely, and to display the rendered image on a display. For example, the rendered image can be associated with an instance of a first game application executed on the client device 100' of the player 5L. For example, the corresponding client device 100' is configured to interact with an instance of a corresponding game application executed locally or remotely to implement the game play process of the corresponding player, such as through input commands for driving the game play process.

[0100] In another embodiment, the previously described multi-player processing engine 119 provides a multi-player gaming session for controlling a gaming application. In particular, when the multi-player processing engine 119 is managing a multi-player gaming session, the multi-player session controller 116 is configured to establish and maintain a communication session with each user and / or player in the multi-player session. In this manner, the players in the session can communicate with each other under the control of the multi-player session controller 116.

[0101] In addition, the multi-player processing engine 119 communicates with the multi-player logic 118 so as to enable interaction between players in the corresponding game environment of each player. In particular, the state sharing module 117 is configured to manage the state of each player in the multi-player game session. For example, the state data may include game state data, which defines the state of the game process (of the game application) of the corresponding player 115A at a specific point, as described above. In addition, the state data may include data saved by the user / player, which includes information that personalizes the video game for the corresponding player, as described above. For example, the state data includes information associated with the user's role, so that the video game is rendered with a role (e.g., body shape, appearance, clothing, weapons, etc.) that may be unique to the user. In this way, the multi-player processing engine 119 using the state sharing data 117 and the multi-player logic 118 can superimpose / insert objects and roles into each game environment of the user participating in the multi-player game session. This allows each of the users in the multi-player game session to interact via their respective game environments (e.g., as displayed on the screen).

[0102] In addition, according to one embodiment of the present disclosure, assistance in supporting the game play process of the corresponding player can be provided via the back-end server support of the game server 205, such as providing matching responses to the player's query, as will be described below. For example, the game assistance server 140 is configured to cooperate with the game server 205 to provide game assistance. In particular, as previously described, the game assistance server 140 works in conjunction with the AI ​​processor 160 to match queries with responses using a deep learning engine. For example, the AI ​​processor 160 includes: an ASR engine 141A, which is configured to optionally convert the query into a format suitable for use; and an NLP engine, which is configured to interpret the query. In addition, the deep learning engine 190 is configured to match the query with the modeled response previously learned by the deep learning engine.

[0103] Figure 3AA data flow diagram is shown according to one embodiment of the present disclosure, which shows the data flow between a user and a back-end server that provides game assistance during the game play of a player playing a game application, where the game application can be executed on a local computing device or on a cloud gaming network.

[0104] As shown, context information 305 collected from multiple players' game play can be used to provide a newly generated or matched response to a query. As previously described, the collected context information 305 can include game context data 315 and global context data 310. The context information 305 is transmitted to the game assistance server 140 along with the query 182.

[0105] Game scenario data 315 may include metadata and / or information related to the game play process, such as game state data defining the game state, user / player information related to the player, and system-level data (e.g., OS data), as described above. For example, game state data includes information that allows the generation of a game environment at a corresponding point in a video game or game application (e.g., game objects, object attributes, graphic overlays, game characters, scenes / game environments, game path information, ratio information (ratios of kills to deaths and other factors), character position information, accuracy, rate or speed of completing work, etc.). In addition, game scenario data may include user / player information, which includes information that personalizes the video game for the corresponding player (e.g., character assets customized for the user: body shape, race, appearance clothing, weapons, etc.). Game scenario data may include system-level data (e.g., OS data), such as controller input data, speed of controller input actuation, number of times the game input is actuated, user-induced controller movement, number of times the game console is used and purpose of use (game, Internet, streaming, etc.), other actions unrelated to the game, etc. The global contextual data 310 may include user profile data for the user, such as the player's overall gaming skills, game controller skills and handling characteristics, how often the user plays games, how often the user launches and uses the console to play games and / or other features provided by the game console, the dates and times when video games are played, the duration of corresponding game play sessions, and the like.

[0106] The scenario data 305 is collected in a database 340B, which is accessible to the AI ​​engine 160 for generating modeled responses and for matching queries to corresponding responses. As shown, database 340B includes user data and statistics for each user / player. For example, database 305B includes information related to user 1 of each game in game 1, ..., N. Database 205B also includes information related to other users. In addition, database 205A includes modeled responses for each game application in game application 1, ..., N.

[0107] Based on the history 317 of contextual information (e.g., game and global contextual data) collected from the player requesting assistance and other players playing the same game application, modeled responses can be generated, such as through a deep learning process, to answer one or more queries related to the game application or other general game problems. In particular, each game application has its own set of modeled responses. In this way, given the current contextual data 305A, a general query (how do I get over the mountain wall) can be matched with an appropriate response. For example, a query can now be matched to a response for the corresponding game application instead of matching it to a response for another game application that also has a wall in the mountain that acts as an obstacle.

[0108] Thereafter, when a player makes a query 182, a pre-generated or modeled response 319 may be presented to the requesting player given the current context data 305A. In one embodiment, the game play is paused while the query is being serviced, as indicated by the pause identification box 320. The data collector is configured to collect the current context data 305A and the query 182 (e.g., how do I get over the mountain wall) and pass it to the game assistance server 140.

[0109] In addition, a newly modeled response (text, audio, video, etc.) that directly answers the request for help can be presented to the player requesting help. For example, a request or query can be made by a user in a first format (e.g., text, audio, video, etc.) and translated into a format understood by the server providing help. In one implementation, the query is made in an audio format and translated into a text format that the game assistance server can understand (e.g., using speech recognition technology). The reformatted request / query is then interpreted to understand the nature of the query (e.g., the content of the help query).

[0110] The artificial intelligence (AI) applied by the AI ​​engine 160 can be used to match the interpreted query and the current scenario data 305A with pre-generated content (recorded help sessions, text answers, audio answers, video answers, etc.). The AI ​​engine 160 includes a machine learning or deep learning engine 190 for matching queries with response models and / or query / response models to provide responses to queries. In one embodiment, the modeled response is provided in a specific format such as text, audio, video, etc. In addition, in other embodiments, the pre-generated or modeled response can be a recorded help session (e.g., audio and video recording) between another user and an expert, wherein the expert provides assistance for the same or related help query provided in the request. The pre-generated response can be a pre-generated response (e.g., text, audio, video, etc.) previously generated to provide assistance to another user for the same or related help query. When there is no pre-generated response available, AI can be used to generate a new modeled response to the query (e.g., text answer, audio answer, video answer, etc.) based in part on one or more closest matching models.

[0111] Additionally, embodiments of the present disclosure provide real-time assistance (eg, a live help session with an expert) to users / players requesting help when no pre-generated responses or newly generated responses are available.

[0112] Figure 3B The game play process of a player playing a game application according to one embodiment of the present disclosure and the periodic collection of context data including game context data and global context data are shown. As previously mentioned, the context data is used to provide responses to one or more queries raised by the player during the game play process as implemented by the backend game assistance server.

[0113] In particular, the player's current session is shown in timeline 350, which indicates the progress of the game play process through the game application. Timeline 350 includes the start point 301 of the session and the end point 302 of the session. During the session, the player's character performs various actions and tasks.

[0114] In addition, as shown in timeline 350, situational data is collected throughout the session. In one embodiment, situational data is collected periodically and / or at important moments during the game play process. For example, global situational data 310A to 310N are collected at various points during timeline 350, wherein the global situational data may include user profile data of the user and other metric data associated with the user, as described above. In addition, game situational data 315A to 315N are collected at various points during timeline 350, wherein the game situational data may include metadata and / or information related to the game play process, such as game state data defining the game state, user / player information related to the player, and system level data (e.g., OS data), as described above.

[0115] like Figure 3B As shown, the player has experienced a loop 370 during the game process, wherein the player cannot advance through a specific point or obstacle (e.g., defeating a small boss or boss, etc.) in the game application. For example, in screenshot 360, the player cannot defeat the boss at the end of the 3rd level, wherein screenshot 360 may show the interaction between Kratos and the giant boss in the game application God of War (God of War) issued by SONY Interactive Entertainment. In the God of War game application, Kratos is a Spartan warrior of Greek mythology, and his mission is to kill the god of war Ares. After multiple attempts, the player can issue a query 374 of "how do I defeat the boss", which is then passed back to the back-end game assistance server, as described above. Together with query 374, applicable scenario data is also passed, so that at least through the application of a deep learning engine, at least the game scenario is provided to the game assistance server for matching query 374 to appropriate response 375.

[0116] For example, response 375 may indicate that the key to defeating the boss is to perform a "hammer sequence". In addition, to help the player understand the sequence, response 375 may include a sequence of controller inputs (e.g., right button, left button, A button, A button, O button, X button, etc.). The response may be provided in one or more formats such as audio, text, video, etc. In one implementation, the sequence of controller inputs may be provided via an instructional video that visually illustrates how to perform a hammer sequence. In addition, a ghost hand may be presented, illustrating how the ghost hand interacts with each button or actuator in the sequence.

[0117] After the player follows the instructions and / or prompts provided in response 375 and defeats the boss, the player is released from loop 370 and is able to advance the game play, as shown at the end 371 of the game play timeline 350. For example, after defeating the boss at the end of level 3, the player is able to advance to level 4.

[0118] By describing in detail the various modules of the game assistance server and the client device communicating through the network, according to one embodiment of the present disclosure, now in combination with Figure 3C Flowchart 300C of describes a method for providing game assistance to support a corresponding player's game play process, wherein the game application can be executed on a local computing device or on a cloud gaming network. Flowchart 300C shows the process and data flow of operations involved on the game assistance server side for matching queries with responses through a deep learning engine. The response can be transmitted to the player's device that can be separate from another device that displays the player's game play process of the game application, or can be transmitted to the device that displays the game play process. In particular, the method of flowchart 300C can be at least partially composed of Figure 1A to Figure 1B and FIG. 2A to FIG. 2C The game assists the server 140 to execute.

[0119] At 380, the method includes receiving information related to a plurality of game play sessions of a player of a game application at a backend server over a network. The information may include context data, such as global context data and game context data, as previously described.

[0120] For example, the global contextual data may include user profile data for the user and other metric data associated with the user. As an illustration, the global contextual data may include player profile data; the player's play style; the player's skill level; whether the player is currently playing a game application; whether the player is online; the title of the game application; co-players the player is playing with; the frequency of input controller actuation; the player's friends, etc.

[0121] In addition, the game scenario data may include metadata and / or information related to the game play process, such as game state data defining the game state, user / player information related to the player, and system level data (e.g., OS data). For example, the game state information may define the state of the game play process at the corresponding point to include character information (e.g., type, race, etc.), game application, the location of the character, the level being played, the character's assets, game objects, game object attributes, game attributes, game object states, graphic overlays, character assets (e.g., clothing, weapons, etc.), equipment, the character's skill set, the character's geographic location in the game environment / world, the current task and / or work presented to the first character during the game play process; the next task of the first character during the game play process, equipment, the character's skill set, the location of the game play process in the game environment, etc. In addition, the game scenario data may include OS level data, wherein the OS level data may include game input; controller input; timestamp of controller input, etc. The game state data may include user / player information that may include information that personalizes the video game for the corresponding player, such as the player's skills or abilities, the player's overall readiness to ask for help, the player's recency in playing the game application, the game difficulty selected by the user 5 while playing the game, the game level, character attributes, character position, number of lives remaining, total possible lives available, armor, trophies, time counter values, and other asset information, etc. In general, the game state data allows the generation of a game environment that existed at a corresponding point during game play.

[0122] At 382, ​​the method includes receiving a query from a player playing the game application, wherein the query is related to the player's game play process. For example, the query may request information notifying the player how to solve an obstacle, general information about the game environment encountered by the player, information about game characters, information about friends playing the game application, information about the game application, how the player is performing relative to other players, etc.

[0123] In some embodiments, the query is converted to a format suitable for processing. For example, the query is converted from a first format to a second format. In one embodiment, the query is presented in one of an audio, text, and video format, but other formats are also supported. In one implementation, the first format is audio, and the second format is text, so that the query can be presented by the player through a microphone system (e.g., an independent transceiver) transmitted in an audio format. In this way, the query in text format can be interpreted, such as using an NPL engine to determine the content of the query, as described above.

[0124] At 384, the method includes determining the current game scenario of the game playing process. Scenario data is continuously collected during the game playing process, and includes global scenario data and game scenario data. The current game scenario can be determined from the scenario data, and used to match the query to the appropriate response. In particular, at 386, the method includes generating a response to the query and the current game scenario based on information related to a plurality of game playing processes. For example, the current game scenario includes the title of the game application, the current scene or game environment encountered by the player, etc. In this way, a game scenario can be given to a general query (for example, how to kill the boss) to better understand the content of the query and how to best respond to the query. As an illustration, the query may be unanswerable, but in the case of the scenario and level with a specific game application, and the boss of the level, the appropriate response indicating the boss's weakness can be returned to the player.

[0125] In one embodiment, the interpreted query and the current game scenario of the game application are provided as input to a deep learning engine, which is configured to match the interpreted query and the current game scenario with a response. The deep learning engine has previously been trained in terms of queries and responses to generate multiple modeled responses and / or queries / responses, wherein the matched response is one of the modeled responses of the game application. In one embodiment, the deep learning engine is part of a third-party AI engine. In this way, the query and the current game scenario are passed to a third-party AI engine, which can be configured to optionally interpret the query and match the interpreted query and the current game scenario with a response.

[0126] At 388, the method includes sending the response to the player's device. In one embodiment, the device may include a companion mobile device separate from the display that presents the game play process. For example, the companion device can be a mobile phone, which can be used to receive queries and pass queries to the back-end game assistance server, and receive responses and present the responses to the player. Similarly, the companion device can be an independent speaker. For example, the companion device can include audio, text and / or video presentation. In another embodiment, the device can be a display that simultaneously displays the response and the game play process.

[0127] In one embodiment, when there is no response available for matching, the game assistance server can find an expert who can provide real-time (e.g., live) help to the player. In particular, the current game scenario can be matched with the game scenario of other players classified as experts. Experts who have recently played game applications in a similar manner (e.g., using the same weapon to kill the leader of the query) can be matched with the current game scenario of the query based on the scenario (e.g., global and / or game scenario). For example, the current game scenario of the player (requesting help) can be compared with multiple historical expert game scenarios to view the matching degree of the expert and the player. In one embodiment, matching can be completed by a deep learning engine. That is, the comparison determines the matching degree of each expert's game playing process and the first player's game playing process. Experts can be selected based on various factors, including the highest matching rating, availability, the agility of the expert to respond to the broadcast notification of the required help, etc. If the player agrees, the expert is connected to the player via the communication session between the player and the expert's device so that the expert provides assistance to the player.

[0128] Figure 3D The present invention is a flowchart showing steps in a method for connecting a player to an expert through a communication session to provide real-time assistance during a player's game play of a game application according to one embodiment of the present disclosure. The help session can be transmitted to the player's device, which can be separate from another device that displays the player's game play of the game application.

[0129] In some embodiments, before matching the user with an expert in response to a request for help from the user, the situational information collected from the game play process of multiple players and / or experts can be used to provide a newly generated or regenerated response. The collected situational data may include metadata and / or information related to the game play process, such as game state data defining the game state, user / player information related to the player, and system-level data (e.g., OS data), as described above. For example, the game state data includes information that allows the generation of a game environment at a corresponding point in a video game or game application (e.g., game objects, object attributes, graphic overlays, game characters, scenes / game environments, game path information, ratio information (ratio of kills to deaths and other factors), character position information, accuracy, rate or speed of completing work, etc.). User / player information includes information that personalizes the video game for the corresponding player (e.g., character assets for user customization-body shape, race, appearance clothing, weapons, etc.). User / player information may also include user profile data of the user, such as the player's overall game skills, game controller skills and processing characteristics, the frequency of the user playing games, the frequency of the user starting and using the console to play games and / or other features provided by the game console, etc. In addition, system-level data (e.g., OS data) includes controller input data, speed of controller input actuation, date and time of video game play, duration of corresponding game play session, number of times game inputs are actuated, user-induced controller movement, number of times the game console is used and for what purpose (games, Internet, streaming, etc.), other non-game related actions, etc.

[0130] Based on the situation information collected from the user / player requesting help and other users playing the video game, a pre-generated response that directly responds to the help request can be presented to the user / player. The pre-generated response can be a recorded help session (e.g., audio and video recording) between another user and an expert, wherein the expert provides assistance for the same or related help query provided in the request. The pre-generated response can be a pre-generated response (e.g., text, audio, video, etc.) previously generated to provide assistance to another user for the same or related help query provided in the request. In addition, a newly generated response (text, audio, video, etc.) that directly answers the help request can be presented to the user / player requesting help. For example, a request or query can be proposed by a user in a first format (e.g., text, audio, video, etc.), and translated into a format that the server providing help understands. In one implementation, the request is presented in an audio format and translated into a text format that the help server can understand (e.g., using speech recognition technology). The reformatted request / query is then interpreted to understand the essence of the query (e.g., the content of the help query). Artificial intelligence can be used to match the explained query with the pre-generated response.

[0131] If no response is generated, then the implementation Figure 3D Method. In particular, at 390, the method includes receiving information related to multiple game play processes of multiple players of a game application at a back-end server via a network. The player may currently be playing a game application, or has played a game application. In some embodiments, the information includes a game play process. In some embodiments, the information includes metadata and / or information generated related to the game play process, such as game state data. For example, the information may include game state information and information saved by the user / player, as described above. The information may include snapshot information, which may provide information that enables the execution of an instance of a video game from a point associated with a corresponding snapshot in the video game. For example, the game state information may define the state of the game play process at the corresponding point, to include character information (e.g., type, race, etc.), game application, the location of the character, the level being played, the character's assets, game objects, game object properties, game properties, game object states, graphic overlays, character assets, character's skill set, the character's geographical location in the game environment / world, the current task and / or work presented to the player, equipment, character's skill set, etc. The game state data allows the generation of a game environment that exists at a corresponding point in the game play process. In addition, user / player information related to the player may include information that personalizes the video game for the corresponding player, such as the player's skills or abilities, the player's overall readiness to seek help, the player's recentness in playing the game application, the game difficulty selected by the user 5 when playing the game, the game level, character attributes, character position, number of lives remaining, total possible lives available, armor, trophies, time counter values, and other asset information.

[0132] At 391, the method includes determining a current game scenario of a first game play of a first player from the information. The first game scenario is associated with the current state of the game play of the first player. Specifically, information associated with the current game play of the first player is received. In one case, the current game play is live, so that the first player is currently playing a game application. The game scenario defines a game environment at a specific point in the game play. The current game scenario defines a game environment at a current point in the corresponding game play. Game scenarios can be defined for one or more points in the corresponding game play. For example, a game scenario can define a player's role, various characteristics of the player, assets associated with the player, work presented to the player, etc. The game scenario can be based on previously received metadata and / or generated information associated with the game play or closely related to them.

[0133] At 392, the method includes determining from the information a plurality of historical expert game scenarios of a plurality of expert game play processes of an expert who has played the game application. In one implementation, the expert may also be currently playing the game application, and generates a new historical expert game scenario by corresponding game play processes. The expert game play process is generated from a player who is classified as an expert of the game application. As previously mentioned, game scenarios can usually be defined for one or more points in the corresponding game play process, such as defining game scenarios for one or more experts. The expert game play process is obtained from a plurality of game play processes, particularly from the game play process of the player who is classified as an expert. Players can be classified by self-registration, by qualification certification or by any other method. In one embodiment, when determining the game scenarios of a plurality of game play processes of all players, expert game scenarios have been determined simultaneously. Like this, once the player is classified as an expert, the game scenario information of the corresponding game player for the expert can be identified as one or more expert game scenarios. In addition, game scenario information can be determined for a plurality of points during the corresponding game play process. For example, the game scenario information of the first expert may include a first game scenario at a first point in the game play process, a second game scenario at a second point in the game play process, ..., and an Nth game scenario at an Nth point in the game play process. For example, the game play process of the corresponding expert may have multiple game scenarios, including game scenarios about facing a boss of level 1, facing a boss of level 2, progress within a given side quest, etc. When multiple players have been classified as experts, the game scenario information of each expert may be determined.

[0134] The classified experts for a particular game application form a collection of multiple players. As previously described, experts can be self-registered, such as without any qualification criteria. In another implementation, experts can have some qualifications, such as player skills, completed work, completed tasks, completed a portion of the game within a period of time, completed the game within a period of time, etc., as previously described. After reaching the qualifications, the expert can self-register, and / or can be automatically marked as an expert (e.g., authorized).

[0135] Different players and / or experts playing the same game application may have the same or similar game scenarios within their corresponding game play sessions. For example, by collecting game scenarios for multiple players who all play the same game application, the game play sessions of different players may be aligned to have similar characters with the same assets, similar play styles of different players, similar routes through the game world of the game application, etc. The game scenario information may be used to match a player with another player who is classified as an expert (e.g., self-registered, qualified, etc.), so that the expert can provide assistance with the game play session of the player who requested assistance, as will be described below.

[0136] At 393, the method includes receiving an assistance query related to the first game play session. That is, the first player is also requesting assistance, or sending a request notification, etc. For example, the query may be specifically about how to beat a particular point in the game (e.g., a level boss, a mission, a job, etc.), or may be about obtaining information about an object (e.g., the name of a boss, an object encountered during game play), or may be about the player's overall goal at that point in the game application.

[0137] In addition, the first player's current game context is related to the game state closest to the point in the game play at which the request was made. For example, the game context may provide information related to the first player's character, the assets owned by the character, the levels the character encounters in the game application, and the scenes in the levels. Any query or request for assistance by the first player will necessarily be related to the current game context. In this way, the game play of another player (e.g., a classified expert, friend, etc.) who has a game context that closely matches the first player's current game context may have knowledge of the game application that is helpful to the first player.

[0138] At 394, the method includes comparing the current game scenario of the first player (requesting help) with a plurality of historical expert game scenarios to view the matching degree of the expert with the first player, such as relative to the game scenario of the corresponding game player. That is, the comparison determines the matching degree of each expert's game playing process with the first player's game playing process. In one embodiment, for each game scenario captured for a specific expert, a comparison is made, and the game scenario closest to the first game scenario of the first player is used as the representative of the expert. In another embodiment, the game scenario information collected at various points during the game playing process of the specific expert can be combined and used to compare with the first game scenario of the first player. In an implementation, it is determined that at least one expert has a corresponding historical expert game scenario that matches the first game scenario.

[0139] At 395, the method includes assigning a first expert to the first player for assistance. That is, the first expert can then provide assistance to the first player regarding his or her game play. To select the first expert from the expert pool, various selection methods can be implemented. For example, the first expert is selected based on the game scenario of the first player and the experts in the expert group / pool.

[0140] In one embodiment, the first expert is selected based on the quality of the match between the game scenarios. For example, the set of matched expert game scenarios has a matching value indicating the quality of the match between the corresponding expert game scenario and the first game scenario, such as Figure 3BAs further described. For example, the method determines a first matching value having a highest value. The first matching value corresponds to an expert game scenario. In one case, the first matching value corresponds to a first expert game scenario for a first expert. Thus, based on the game scenario, a first expert is selected for the help session, wherein the first expert is the most suitable expert in the pool of experts to provide help to the first player.

[0141] In another embodiment, a first expert is selected from a pool of experts based on an availability factor. This provides a direct method of matching an expert to a player requesting help. In particular, this method may be beneficial when a gaming application is first released. Since it is recently released, there may be few experts registered, and due to lack of information, it may be difficult to make any comparisons between experts. In one implementation, the first available expert is selected and assigned to the first player for a help session.

[0142] In other embodiments, experts are selected based on response time, such as in a competition to obtain responses from qualified and / or available experts who are most likely to help, as will be shown in Figure 3B In another embodiment, multiple experts are polled at once to determine if they would like to provide assistance. During the polling process, the first expert who responds affirmatively is assigned to provide assistance, as will be described in Figure 3B Further described in .

[0143] In particular, at 396, the method includes generating a communication session connecting the first player and the first expert. In one embodiment, a communication session manager at the back-end server serves as an intermediary for establishing and managing a communication session. At least, a communication session is established between the device of the first expert and the device of the first player. The communication session is used to enable an expert to provide assistance to the player, such as providing assistance through a help session between the first expert and the first player. In one embodiment, the communication session is configured for text, audio, video, embedded audio and video, etc. For example, the method may include establishing one or more of a voice channel, may include establishing a text channel in the communication session, may include establishing a video channel (e.g., embedded video) configured for video chat. Moreover, the communication session manager may be used to create a new session to allow different forms of communication, such as providing a shared screen function, a shared game function, etc. In one embodiment, the communication session may be a peer-to-peer connection or may include a back-end server serving as an intermediate node. That is, once created by the communication session manager, the communication session is the direct communication path between the device of the first player and the first expert. In another embodiment, the communication session may flow through the back-end server.

[0144] In one embodiment, the first expert can share the screen of the first player, such as by sharing the screen function, as described above. By viewing the first player's game play process, the first expert can better understand the problems faced by the first player, thereby providing better help. In one embodiment, the sharing screen function is implemented through a communication session. The request for sharing the screen can be made by the first player or the first expert. For example, the first expert can request to share the video of the first player's game play process. In one implementation, the request is received by the help session controller at the backend server. The notification of the request is sent to the device of the first player. For example, the notification can be delivered from the help session controller. The help session controller receives authorization from the first player's device, wherein the first player provides authorization to share the video of the game play process with the expert. In this way, the first player's game play process is streamed to the first expert's device. For example, the help session controller can promote streaming through a communication session or through an independent streaming channel.

[0145] In another embodiment, the first expert can control the first player's game play process, such as by sharing the game function, as described above. Through sharing the game, the expert can take over the control of the game play process, such as to complete the goal that the first player cannot perform. The request for sharing the game can be made by the first player or the first expert. For example, a request from the first expert's device is received, wherein the request requires sharing the control of the first player's game play process. The request from the expert can be in the form of assistance suggestions from the expert to achieve the goal within the user's game play process. The request can be received by the help session controller at the back-end server. The notification of the request can be generated by the help session controller and passed to the first player's device from the help session controller. The help session controller receives authorization from the first player's device, wherein the first player provides authorization to share the control of the game play process with the expert. In this way, the expert can control the game play process by submitting game input commands. In one embodiment, the help session controller receives a set of input controls or commands from the first expert's device. Blocking of input commands from the first player's input controller is imposed so that the game engine (e.g., a local console or a backend game processor) blocks input commands originating from the first player's controller device and passes input commands originating from the first expert's controller device. For example, the help session controller may send instructions to a processor (e.g., a game engine) of a game application executing a game play process for the first player to block input controls associated with the first player. In this way, a set of input controls from the first expert's controller device is passed to a processor (e.g., a game processor) of a game application executing a game play process for the first player. In addition, control may be passed back to the first player at any time. For example, as previously described, the first player may have the ability to take back control of the game play process at any time (such as using a kill command).

[0146] Figure 3E is a data flow diagram illustrating the flow of data in a system or method for connecting a player to an expert through a communication session to provide real-time assistance during a player's game play of a game application according to one embodiment of the present disclosure.

[0147] As shown, player 1 (P1) is playing a game application. Player P1 may encounter roadblocks during his or her game play process and request information and / or assistance. For example, as previously described, queries from player P1 are made through user interface 351-P1 and passed back to help session controller 352 of the backend server via network 150. In particular, a matching engine 353 cooperating with help session controller 352 is configured to match game scenarios of player P1 with expert pool 354. The expert pool is obtained from multiple players 355, where the players are playing one or more game applications. The experts in pool 355 have all played the game application and are, for example, registered as experts of the game application. For example, pool 355 includes one or more experts E1, ..., E5, ..., E103, ..., E n .

[0148] The game scenarios 356 are input into the matching engine 123 for comparison. For example, the input includes a game scenario 356-P1 for player P1, a game scenario 356-E1 for expert E1, a game scenario 356-E103 for expert E103, a game scenario 356-E64 for expert E64, a game scenario 356-E5 for expert E5, ..., and a game scenario 356-En for expert En. The matching process performed by the matching engine is previously described. Basically, the game scenario 356-P1 of player P1 is compared with each game scenario associated with the expert pool 354. A matching vector is determined for each game scenario, wherein each matching vector has a corresponding matching value (e.g., a quality factor or Q factor) that indicates the matching quality of the corresponding expert game scenario with the game scenario 356-P1 of player P1.

[0149] The matching engine 353 is configured to select an expert from the expert pool 354. As shown, expert E5 is selected and provided as output 357 from the matching engine 353. For the purpose of generating and managing a help session to provide assistance to player P1, the output 357 is provided to the help session controller 352. As previously described, one or more methods can be implemented to select an expert. For example, the expert pool 354 can be further filtered by applying a threshold to the match value, wherein the expert associated with the match value that meets the threshold standard is considered to be selected. In one implementation, the highest quality match value is used to select the expert. That is, the highest match value is used for selection. In another example, the notification of the help session request is passed to the expert associated with the match value that meets the threshold standard. The expert who first responds to the notification can be selected for the help session. In another example, any expert associated with the match value that meets the threshold standard can be selected such as by random selection, first selection, etc.

[0150] In one embodiment, the matching engine 353 may select a friend who is an expert from the friend pool instead of matching the player P1 with an expert. For example, the friend may be a social network friend established through one or more social networks.

[0151] The help session controller is configured to establish and manage a help session to provide real-time assistance to player P1. For example, a communication session is generated between a device of player P1 (e.g., user interface 351-P1) and a device of expert E5 (e.g., user interface 351-E5). In one embodiment, a communication session is generated between a communication session manager of the help session controller 352, the device of player P1, and the device of expert E5. In another embodiment, a communication session is generated and establishes direct communication between the device of player P1 and the device of expert E5.

[0152] One or more communication channels can be established in a communication session. For example, one or more of a voice channel 361, a text channel 362, a screen sharing channel 363 and / or a shared game channel can be established. As shown, voice channel 361 is a two-way communication path so that player P1 and expert E5 can talk and listen to each other's voice communication. Moreover, text channel 362 is a two-way communication path so that player P1 and expert E5 can communicate with each other by texting. In addition, screen sharing channel 363 can be a one-way communication path so that the video of the game playing process from player P1 is delivered to the device of expert E5 for viewing. In addition, shared game channel 364 can be a two-way communication path so that input control can be transmitted from expert E5 to the game engine of player P1 locality, or transmitted to another game engine at the back-end server.

[0153] A separate control channel may be established to pass control or other information between the help session controller 352 and the user interface 351-P1 or the user interface 351-E5. For example, instructions may be passed to the user interface 351-P1 that block input control from the player P1 or send video via the screen sharing channel 363. Additionally, rating information may be passed on the channel 365 to provide rating information. For example, after a help session, the player P1 may provide a rating of the help session on the channel 365. Additionally, the player P1 may provide a rating of the overall performance of the expert E5 (e.g., personality, usefulness, ability to control spoiler release, depth of knowledge of the game application, etc.). Furthermore, the expert E5 may provide a rating of the player P1 (e.g., level of cooperation, ability to accept help, personality, gratitude, etc.).

[0154] In one embodiment, a help session is implemented concurrently with the user's game play on a second computing device associated with player P1. For example, in one embodiment, there may be two communication channels for communicating information, such as a first communication channel established to communicate data representing the user's game play to a first computing device of player P1, and a second communication channel established to communicate data associated with the help session to a second computing device of player P2. For example, the first computing device may be a local game console and / or display, and the second computing device may be a smartphone. In another embodiment, the help session may be communicated along with the data representing the user's game play, such as via a split screen including a first screen displaying the game play and a second screen displaying the help session.

[0155] 4-13 are diagrams of various examples of a player inputting a query or a sequence of related queries in association with one or more game applications. The player may be playing the corresponding game application, or may be acting independently of playing the game application (e.g., while traveling, or in a meeting, or in class, etc.).

[0156] Figure 4A A home screen 405 of a companion mobile device 11 is shown according to one embodiment of the present disclosure, the companion mobile device 11 being configured to provide gaming assistance to a player playing a gaming application executed on a local gaming console or a cloud gaming network. The companion mobile device 11 supports game play presented on a display different from the display on the device 11. The home screen 405 includes a background image that can change based on the current location of the game play in the gaming application. For example, for illustrative purposes, the back image can be a silhouette of a town found in the gaming application HorizonZeroDawn.

[0157] Icon 401 provides access to bookmarks. In one embodiment, bookmarks can include content pushed from a display presenting the gameplay. For example, content can be pushed from an on-screen display user interface accessible via a bookmark for "further reading."

[0158] Icon 410 shows a microphone feature that enables the player to enter a query by speaking. The game assist feature can be activated by a trigger word that indicates that at least one of the next few sentences may include a query. General field 407 indicates that the game assist feature is turned on. However, the game assist feature can be entered by any number of methods, including manually typing a query in text format into the search field (e.g., a "tap or speak" bubble). Typing can be performed when speaking is not possible (e.g., sitting in a classroom).

[0159] The quick search buttons 415 allow the player to select a query simply by selecting one of the quick search buttons 415. For example, the player can select between various buttons including queries, such as "I need help" or "What do I do now?" or "Did I miss anything?" For example, the quick search button 415D provides the query "nearest blood recovery point". The quick search buttons may gradually change with selection to help guide the player without providing text instructions or audio instructions. The quick search buttons can serve as an educational tool for shortcut operations that can be used in the user interface of a television or display that presents the game play process.

[0160] Additionally, the scenario cards 420 provide information about a group of themes or objects. For example, a scenario card may provide information about a particular location (e.g., a hunter's lodge), or may provide information about a particular character (e.g., Talanah Khane Padish). In another example, a scenario card 420C may provide information about a particular enemy beast, such as Redmaw, which is the Thunderjaw beast in the game application Horizon Zero Dawn.

[0161] Figure 4B An embodiment according to the present disclosure is shown Figure 4A Selection of a quick search button 415D on a companion mobile device 11 configured to provide gaming assistance. In particular, Figure 4B The interaction between the companion mobile device and the display 12 when displaying the player's response to the query is shown, wherein the display 12 is configured to present the player's game play process. In particular, a query is presented from the companion mobile device (e.g., selecting the quick search button 415D) to inquire about the location of the "nearest health point" so that the user's character can heal or gain additional life, etc. The response to the query of the quick search button is presented on the display 12 separate from the mobile device 11. For example, the screen of the mobile device 11 shows that the query 415D is "nearest health point", and also provides an instruction 435A indicating that the response (map) is presented on the display 12. In particular, the map 430 of the game environment of "Horizon Zero Dawn" shows the current location "Aloy" and a mark 435B showing the location of the nearest health point. In this way, the player can enter a query on one device and receive a response on another device.

[0162] Figure 4C An embodiment according to the present disclosure is shown Figure 4A The interaction between the companion mobile device configured to provide game assistance and the display 12 presents the game play progress of the player playing the game application and directs the player to the mobile device to obtain the requested information. In particular, the player can select Figure 4AThe query may be entered by pressing the quick search button 420C ("Redmaw") on the companion mobile device 11 of the game console. The query may be entered by any other method, such as through a standalone speaker or a speaker connected to the game console. In one embodiment, the query is entered through the speaker on the display 12 or through a search box on the display 12. Responses to the query may also be presented on the screen of the companion mobile device 11. For example, the image 450B shows a picture of Redmaw, while the window 450C provides accompanying information about Redmaw: "Redmaw is the legendary Thunderjaw..."

[0163] Additionally, pointer instructions may be provided on the display 12 showing the gameplay. For example, instruction 450A instructs that a response to a query be presented on the companion mobile device 11: "I sent your phone more information about Redmaw." In this way, if the player is absorbed in the gameplay, the player is gently reminded that the response is on the companion mobile device 11. In this way, the player can enter a query on one device and receive a response on another device.

[0164] Figure 4DA companion mobile device configured to provide game assistance to a user according to one embodiment of the present disclosure is shown, the player is not currently playing a game application, but can present queries related to the game application to a backend server for support. For example, the user may be at work or on the go. The user may need some general information about the game application, or information related to social network friends who play one or more game applications, or may want to schedule a game session with one or more friends. The user can also request other more specific information. For example, user River Hsu can initiate a help session with the game assistance server through the mobile device 11 by saying a trigger word such as "Hey PlayStation" (not shown). In this way, a game assistance application (e.g., PlayStation Assist) is started or restarted locally when a query is expected, where the game assistance application can provide docking between remote devices. The user can then enter a query 460A (e.g., verbally through a microphone on the mobile device 11) asking "When did I start playing God of War". The query is then received and passed to the backend game assistance server, which processes the query. For example, the backend game assistance server can implement a deep learning engine to match the query to an appropriate response given the current scenario. Because the query is displayed outside of the game playing application, the context may not be sufficient, and thus the query may require more complete information rather than just a general question. Because the query includes the title of the game application (e.g., God of War), sufficient context is provided to match the query. In this way, based on information collected from the user of the game application throughout the game playing process (e.g., global and game context), responses can be matched and / or generated. For example, response 460B can indicate that the user started playing the game application in December 2017.

[0165] Additionally, follow-up questions are entered as a second query 470A, asking how long the user has played and how much of the game the user has completed. A response 470B is matched and / or generated by the game assistance server, wherein the response indicates that the user has played for a total of approximately 16 hours, has completed approximately 27% of the missions in the game application, and has visited 55% of the game world (e.g., the open world environment).

[0166] Figure 4EA companion mobile device 11 according to one embodiment of the present disclosure is shown, which is configured to provide game assistance to the user to support the player's game play process. The user interface of the mobile device 11 includes a header 510, which indicates that the game play process is for the game application "Horizon Zero Dawn" and that the information provided comes from the PlayStation Assist application. The footer area 520 includes links to various features provided by PlayStation Assist, such as a head-up display feature (HUD), a link to a wizard, a link to a history of queries and responses, and a link to messaging. For example, a player may be playing a game at a specific location in the game environment of a game application and enter a query asking for the location of potions and battles. The back-end game assistance server processes the query using the current context of the game play process through a deep learning engine. A response including a map of the game environment can be generated, including showing the center location of the player character within the game environment. In addition, the location of potions and battles is also presented relative to the location of the player character.

[0167] FIG. 5A to FIG. 5B The use of a companion mobile device 11 for game assistance associated with gameplay of a game application presented on a separate display according to one embodiment of the present disclosure is shown. Specifically, the screenshot of the mobile device 11 includes a header 510 indicating that the gameplay is for the game application Horizon Zero Dawn and that the information provided is from the PlayStation Assist application; and includes a footer area 520 that includes links to various features provided by PlayStation Assist, such as a heads-up display feature (HUD), a link to a wizard, a link to a history of queries and responses, and a link to messaging. The header 510 and the footer area 520 can be viewed in FIGS. 6 to 63. Figure 7. As previously described, the player River Hsu has triggered the PlayStationAssist application, such as by a trigger word. In this way, the communication 515 from PlayStation Assist indicates that the application is ready to accept queries from the player. The player is playing a game application, where the game play process can be presented on a separate display. In this way, the query can support the game play process. In particular, the player's query 517 asks "How do I kill this dragon?" Since the query is input with context data (including game context and global context data), the query can be matched to an appropriate response. The game assistance server generates a response through the PlayStation Assist application, such as through a deep learning engine process. For example, a response indicating that information is available is introduced through communication 515 from PlayStation Assist, or by selecting button 525A to obtain information about the dragon's weakness, or by selecting button 527 to obtain information about strategies to defeat the dragon. In addition, given the game context of the game play process (e.g., the current location of the player character), PlayStation Assist may provide a "wizard" to help the player solve difficulties close to the time or place of the game play process (e.g., provide a warning about what is about to happen or provide some tips). For example, button 528 provides access to a wizard that explains how to obtain a steel sword.

[0168] The player River Hsu can select the weak point button 525A to get more information. Figure 5B As shown, title 525B shows the weaknesses of the Thunderjaw dragon. Additional information may also be provided to provide tips on how to fight the dragon. In addition, if the player wants more information related to the Thunderjaw dragon, button 530 provides a link to "Strategy" which provides tips on how to defeat the dragon, and "Resistance" provides information about how powerful the dragon is in certain areas, which may provide information on how to defeat the dragon.

[0169] FIG. 6A to FIG. 6GThe use of a companion mobile device for game assistance related to game play of a gaming application presented on a separate display is shown in accordance with one embodiment of the present disclosure to navigate through a series of connected layers of information provided in response to a query. In particular, as shown in each figure, the screenshots of the mobile device 11 include a header 510 indicating that the game play is for the gaming application Horizon Zero Dawn and that the information provided is from the PlayStation Assist application. As previously described, the footer area 520 includes links to various features provided by PlayStation Assist. Various categories of information are provided to the player, each of which can be accessed through, for example, Fig. 6A For example, button 630A provides access to a game strategy for a game application, and when selected, the button provides access to an information tree related to the game strategy, such as FIG. 6A to FIG. 6G provided.

[0170] Figure 6B Shows the selected Fig. 6A Button 630A is used to return the user to the one or more game guides on the screen shot of the mobile device 11. Button 630B is used to return the user to Fig. 6A As shown in the figure, the game guide 635 includes the main task game guide, the side task game guide (which can be selected by button 640A) and the mission game guide, each of which can be selected by a corresponding button. In another layer, Figure 6C Shows one or more side quests and their options. Figure 6B As shown, side quests 645 include the Companion side quest, the College of Winterhold side quest (selectable via button 650A), and the Thieves' Guild side quest, each of which can be selected via a corresponding button. Back button 640B returns the player to Figure 6B User interface.

[0171] On another level, Fig.6D A walkthrough for the College of Winterhold side quest is shown, where button 650B returns the player to Figure 6C In particular, a step-by-step process is provided in section 655, including First Lessons (completed), Under Saarthal (selectable via button 660A), etc., each of which can be selected via a corresponding button. In another layer, Fig. 6E A specific gameplay walkthrough for the "Under Saarthal" step is shown (after selecting button 660A), with button 660B returning the player to Fig.6D The user interface provided in . Provides general information, including how long it takes to complete this step and what reward is obtained after completing this step. Button 670A launches the game walkthrough.

[0172] On another level, Fig. 6F The various actions that need to be performed in the "Under Saarthal" step and its game walkthrough are shown. FIG. 1 shows a "Steps" 671 section for accessing step information. For example, step 1 of the game walkthrough (meeting Tolfir) is shown, and step 2 (finding Arnial) is also shown. Additional sections can be shown to the player upon request (e.g., scrolling down). In another layer, Figure 6G Shows the Fig. 6F 672 in the game to access options for viewing the game walkthrough. For example, the complete game walkthrough can be accessed via an envelope 673, which can be mailed to a separate device. For each step of the game walkthrough, specific steps or sub-steps can also be accessed. As shown, for step 1 (meet Tolfir), one or more sub-steps can be accessed via a mailed (e.g., digital) envelope, and one or more sub-steps of step 2 (find Arnial) can be accessed via a mailed envelope.

[0173] Figure 7 The use of a companion mobile device 11 for game assistance associated with a game play session of a player of a game application presented on a separate display is shown according to one embodiment of the present disclosure, wherein an option to connect the player to an expert via a communication session is presented. Specifically, the screenshot of the mobile device 11 includes a header 510 indicating that the game play session is for the game application Horizon Zero Dawn and that the information provided is from the PlayStationAssist application; and includes a footer area 520 that provides links to various features provided by PlayStation Assist, as previously described.

[0174] As previously described, the player River Hsu has triggered the PlayStation Assist application, such as by a trigger word. Thus, the communication 720 from PlayStation Assist indicates that the application is ready to accept queries from the player. The player is playing a game application, where the game play process can be presented on a separate display. Thus, the query can support the game play process. In particular, the player's query 735 asks for information on how to find the "Cauldron Zeta portal."

[0175] The PlayStation Assist application supported by the game assistance server is unable to match the query to a modeled or generated response. Communication 720 from the PlayStation Assist application indicates that no response is available. However, if the player wishes to receive real-time assistance, experts are available for a real-time assistance session. For example, the PlayStation Assist application can connect the player to expert Aspen Tokokhan (5-star rating) via button interface 740A, or connect the player to expert Rain-ah (3-star rating) via button interface 740B. Selecting an expert for presentation as an option has been described previously, and may involve matching the player's current situation (e.g., global and / or game situation) with a similar situation of an expert who has preferably played the game.

[0176] FIG. 8A to FIG. 8G The use of a display to present a player of a gaming application playing a game and browsing a series of connected information layers provided in response to a query to obtain an asset is shown according to one embodiment of the present disclosure. In particular, for example, FIG. 8A to FIG. 8G The acquisition intent process is shown, where the player is requesting information on how to obtain an asset. The information may be generated and / or matched by a game assistance server (such as through the PlayStation Assist application) and provided in a window on the display that also displays a video / image of the game play process, as previously described. As shown, the player is playing a game application, such as Skyrim. FIG. 8A to FIG. 8G The queries and responses shown in relate to the gaming application.

[0177] In particular, Fig. 8A A window presenting a Skyrim emblem (e.g., a flying dragon) is shown. The player's gameplay progress is also displayed in a different window (e.g., a character approaching a flying beast). A game assistance feature is available to the user, where the player can enter a query through a microphone, as shown by microphone icon 410. The player is shown triggering the PlayStation Assist application, such as through the trigger word "Hey PlayStation". The query entered relates to information about obtaining a steel sword. After matching the query to an appropriate response, for example, given a game application context that includes a global context, a game context, and a current game context, a game assistance server that supports the PlayStation Assist application can match the query with the response. Figure 8B The response is shown in , which shows information about how to obtain the steel sword. For example, the steel sword can be obtained through various techniques, including looting, through merchants, crafting, and by completing quests.

[0178] The player may make subsequent or supplemental requests that supplement the initial query. For example, a player may issue a query (e.g., verbally) asking how to loot a steel sword. Figure 8C A game assistance server is shown that supports the PlayStation Assist application to present responses to subsequent queries. The game assistance server can perform matching by applying a deep learning engine to match queries to appropriate responses. For example, in Figure 8C In the game, information about how to loot the sword may include locations where the sword can be found, such as in treasure chests throughout the levels. Figure 8C . Once selected, Fig.8D Specific loot locations are available in the game, including the Cairn or Alvor's Hearth.

[0179] Fig. 8E Responses presented in association with another subsequent or supplemental request based on an initial query or an intermediate query are shown. For example, a player may issue a query (e.g., verbally) asking how to purchase a steel sword. Information about merchants that sell swords is provided, and the information includes the nearest merchant (Annie Traveler in the town of Whiterun) and the price of the sword (100 coins). Other merchants are shown, including Adrianne Avenicci and Arnskar Ember-Master.

[0180] The player may make subsequent or supplemental requests that supplement the initial query. For example, a player may issue a query (e.g., verbally) asking how to make a steel sword. Figure 8F A game assistance server is shown supporting the PlayStation Assist application to present responses to subsequent queries, such as by matching with modeled responses implemented through deep learning. Information about how to make a sword includes the required materials and the location where the sword is built - at a forge.

[0181] Figure 8G A response is shown that is presented in association with another subsequent or supplemental request based on an initial query or an intermediate query. For example, a player may issue a query (e.g., verbally) asking how to obtain a steel sword through a mission. Information about one or more missions that provide a steel sword reward is provided, including the Wolf Queen Awakened mission and the Laid to Rest mission. As shown, a game walkthrough for each mission can be accessed through a link.

[0182] 9A to 9CThe present invention shows the use of a display to present a player of a gaming application playing a game and browsing a series of connected information layers provided in response to a query requesting information about how to defeat an opponent according to one embodiment of the present disclosure. In particular, for example, FIG. 8A to FIG. 8G A defeat intent process is shown, where a player is requesting information on how to defeat a beast or opponent. The information may be generated and / or matched by a game assistance server (such as through the PlayStation Assist application), as previously described. As shown, the player is playing a game application, such as Skyrim. 9A to 9C The queries and responses shown in relate to the gaming application.

[0183] In particular, Fig. 9A A window presenting the player's gameplay is shown (e.g., a character approaching a flying beast, the Mirmulni dragon). A game assistance feature is available to the user, where the player can enter queries through a microphone, as shown by microphone icon 410. The player is shown triggering the PlayStation Assist application, such as through the trigger words "Hey PlayStation." The query entered relates to information about defeating the dragon (Mirmulni) shown during gameplay. After matching the query to an appropriate response, a game assistance server supporting the PlayStation Assist application can match the query to a response, for example given a game application context that includes a global context, a game context, and a current game context. The response is in Fig. 9A , where information about how to defeat the Mirmulni dragon is shown. For example, two sections / categories of information may be accessed when learning about how to defeat the dragon, including a strategy category and an attack category.

[0184] The player may (e.g., verbally) select a strategy category. Information is shown regarding good strategies to use when battling Mirmulni dragons, and includes helpful hints (e.g., hiding to restore health, etc.). Additionally, Fig. 9B It is shown that the player can jump between sections by stating the name of the requested section (e.g., strategies or attacks). In addition, the player can close the PlayStation Assist application at any time by stating "Close help."

[0185] Fig. 9C Included below the attack section is information provided in response to the player selecting that category in a query (e.g., by verbal command). Information for attacks includes melee attacks (e.g., side attacks), using magic (e.g., avoiding fire spells), and information about bow weapons (shoot when the dragon lands).

[0186] FIG. 10A to FIG. 10D The present invention shows a display that presents a player of a game application playing a game and browsing a series of connected information layers provided in response to a query to find an object or location according to one embodiment of the present disclosure. In particular, for example, FIG. 10A to FIG. 10D The seek intent process is shown, where a player is requesting directions to a location. The information may be generated and / or matched by a game assistance server (such as by the PlayStation Assist application), as previously described. As shown, the player is playing a game application, such as Skyrim. FIG. 10A to FIG. 10D The queries and responses shown in relate to the gaming application.

[0187] In particular, Fig. 10A A window presenting the player's gameplay progress is shown (e.g., a character approaches a covered bridge over a river). A game assistance feature is available to the user, wherein the player can enter a query via a microphone, as indicated by a microphone icon. The player is shown triggering the PlayStation Assist application, such as via the trigger word "Hey PlayStation". The query entered relates to information regarding obtaining directions to a particular location (Bleak Falls Barrow).

[0188] After matching a query to an appropriate response, for example, given a game application context that includes a global context, a game context, and a current game context, a game assistance server supporting the PlayStation Assist application can match the query to the response. Fig. 10B The response is shown in , where directional information is shown. For example, step-by-step instructions are provided to get to the Bleak Falls Barrow from the current location displayed during game play. In addition, information about nearby treasure chests (e.g., in a tower) may also be provided.

[0189] The player may make a subsequent or supplemental request that supplements the initial query. For example, a player may make a query (e.g., verbally) asking how to exit the Bleak Falls Barrow area. Fig. 10C A gaming assistance server is shown that enables the PlayStation Assist application to present responses to subsequent queries, such as by matching with modeled responses implemented through deep learning. Fig. 10D. Specifically, step-by-step instructions are provided on how to exit Bleak Falls Barrow by climbing up the cave and entering by pulling a lever. Additional information is provided, including nearby enemies (e.g., Draugr) and information about nearby treasure. The PlayStation Assist application can have an ongoing conversation with the player, but the instructions provide a response. Additionally, the player can exit the PlayStation Assist application at any time by verbally instructing to stop.

[0190] FIG. 11A to FIG. 11B A display is shown according to one embodiment of the present disclosure for presenting a player of a gaming application playing the game and browsing a series of connected layers of information provided in response to a query having a primary intent and a secondary intent, such as a requirement to complete a job (e.g., meet General Tullius in Whiterun). For example, during a search for Bleak Falls Barrow, the player may also be required to perform a secondary task. This may be indicated by an icon. For example, one of the steps provided in the directions to Bleak Falls Barrow may require the character to approach General Tullius during gameplay. The player may enter a query asking who General Tullius is. A gaming assistance server supporting the PlayStationAssist application may indicate that a response has been provided. Additionally, the player may exit the directions or continue the directions to enter and exit a secondary intent (e.g., meet General Tullius). As shown in FIG. Fig. 11B As shown, information about General Tullius is provided, including biographical information and his current location. In addition, the player can request access to a map showing the location of Whiterun, more specifically the location of General Tullius.

[0191] FIG. 12A to FIG. 12C The use of a display to present a player of a gaming application playing a game and browsing a series of connected information layers provided in response to a query to learn how to complete a job is shown according to one embodiment of the present disclosure. In particular, for example, FIG. 12A to FIG. 12C The learning intention process is shown, where the player is requesting how to solve a puzzle. The information can be generated and / or matched by a game assistance server (such as through the PlayStation Assist application), as previously described. As shown, the player is playing a game application, such as Skyrim. FIG. 12A to FIG. 12C The queries and responses shown in relate to the gaming application.

[0192] In particular, Fig. 12A A window presenting the player's gameplay is shown (e.g., a character facing a keystone lock). A game assistance feature is available to the user, where the player can enter queries through a microphone, as indicated by a microphone icon. The player is shown triggering the PlayStation Assist application, such as through the trigger words "Hey PlayStation." The query entered relates to information about how to unlock a keystone displayed during gameplay. After matching the query to an appropriate response, a game assistance server supporting the PlayStation Assist application can match the query to a response, for example given a game application context that includes a global context, a game context, and a current game context. The response is in Fig. 12B , where information about how to unlock the keystone is shown. For example, a game assistance server supporting the PlayStation Assist application may indicate that a response was provided, and provide a shorthand clue – "Check Arvel's diary." A link to Arvel's diary is also provided, and when selected, the appropriate diary entry is displayed.

[0193] The player may make subsequent or supplemental requests that supplement the initial query. For example, the player may issue a query specifically asking what is important in Arvel's journal entry (e.g., verbal). Fig. 12C A game assistance server is shown that supports the PlayStation Assist application to present responses to subsequent queries, such as by matching with modeled responses implemented through deep learning. Information is provided that the golden claw is the solution. In addition, a gate code (e.g., bear-moth-owl) is provided.

[0194] FIG. 13A to FIG. 13B The use of a display to present a player of a gaming application playing the game and browsing a series of connected layers of information provided in response to a query to learn something about the gaming application (e.g., information about an enemy fighter) is shown in accordance with one embodiment of the present disclosure. In particular, for example, FIG. 13A to FIG. 13B The understanding intent process is shown, where the player is requesting information about an object. The information may be generated and / or matched by a game assistance server (such as by the PlayStation Assist application), as previously described. As shown, the player is playing a game application, such as Skyrim. FIG. 13A to FIG. 13B The queries and responses shown in relate to the gaming application.

[0195] In particular, Fig.13AA window is shown presenting the player's gameplay progress (e.g., a character encountering an unknown creature). A game assistance feature is available to the user, where the player can enter queries through a microphone, as indicated by a microphone icon. The player is shown triggering the PlayStation Assist application, such as through the trigger words "Hey PlayStation." The query entered relates to information about an unknown object - "What is that!". Since the creature and the player character are engaged, urgency is implied. After matching the query to an appropriate response, a game assistance server supporting the PlayStation Assist application can match the query to a response, for example given a game application context that includes a global context, a game context, and a current game context. The response is in Fig. 13B , where information about a creature (e.g., a Draugr) is shown. For example, the information includes general information about the creature and images of different types of Draugr, including a Draugr Thrall, a Restless Draugr, etc. Additional information may also be requested.

[0196] Fig.14 Components of an example device 1400 that may be used to perform aspects of various embodiments of the present disclosure are shown. For example, Fig.14 An exemplary hardware system according to an embodiment is shown, which is suitable for implementing a device that provides services to support users, such as providing game assistance in response to user queries related to game applications, wherein a deep learning engine is used to match queries with modeled responses. The block diagram shows a device 1400, which can be combined with or can be a personal computer, video game console, personal digital assistant or other digital device suitable for practicing the embodiments of the present disclosure. The device 1400 includes a central processing unit (CPU) 1402 for running software applications and optionally running an operating system. CPU 1402 may include one or more isomorphic or isomorphic processing cores. For example, CPU 1402 is one or more general-purpose microprocessors with one or more processing cores. Other embodiments can be implemented using one or more CPUs with a microprocessor architecture that is particularly suitable for highly parallel and computationally intensive applications, such as media and interactive entertainment applications, or applications configured to provide game assistance in response to user queries related to game applications, as described above. The device 1400 can be local to the player requesting assistance (e.g., a game console), or away from the player (e.g., a back-end game assistance processor).

[0197] Memory 1404 stores applications and data for use by CPU 1402. Storage device 1406 provides non-volatile storage and other computer-readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD or other optical storage devices and signal transmission and storage media. User input device 1408 transmits user input from one or more users to device 1400, and examples of user input devices may include keyboards, mice, joysticks, touch pads, touch screens, static or video recorders / cameras, tracking devices for identifying gestures, and / or microphones. Network interface 1414 allows device 1400 to communicate with other computer systems via an electronic communication network, and may include wired or wireless communications on a local area network and a wide area network such as the Internet. Audio processor 1412 is suitable for generating analog or digital audio output from instructions and / or data provided by CPU 1402, memory 1404 and / or storage device 1406. Components of device 1400 , including CPU 1402 , memory 1404 , data storage device 1406 , user input device 1408 , network interface 1410 , and audio processor 1412 , are connected via one or more data buses 1422 .

[0198] The graphics subsystem 1414 is also connected to the data bus 1422 and the components of the device 1400. The graphics subsystem 1414 includes a graphics processing unit (GPU) 1416 and a graphics memory 1418. The graphics memory 1418 includes a display memory (e.g., a frame buffer) for storing pixel data for each pixel of an output image. The graphics memory 1418 can be integrated in the same device as the GPU 1416, connected to the GPU 1416 as a separate device, and / or implemented in the memory 1404. The pixel data can be provided directly from the CPU 1402 to the graphics memory 1418. Alternatively, the CPU 1402 provides the GPU 1416 with data and / or instructions defining the desired output image, and the GPU 1416 generates pixel data for one or more output images from the data and / or instructions. The data and / or instructions defining the desired output image can be stored in the memory 1404 and / or the graphics memory 1418. In an embodiment, GPU 1416 includes 3D rendering capabilities for generating pixel data of an output image from instructions and data to define the geometry, lighting, shading, textures, motion and / or camera parameters of a scene. GPU 1416 may also include one or more programmable execution units capable of executing shader programs.

[0199] Graphics subsystem 1414 periodically outputs pixel data of an image from graphics memory 1418 for display on display device 1410 or for projection by projection system 1440. Display device 1410 may be any device capable of displaying visual information in response to a signal from device 1400, including CRT, LCD, plasma, and OLED displays. For example, device 1400 may provide an analog or digital signal to display device 1410.

[0200] Although specific embodiments have been provided to demonstrate providing real-time assistance during a player's gameplay of a game application through a real-time help session (e.g., connecting the player to an expert through a communication session) or through a recorded help session (e.g., connecting the player to a recorded help session transmitted through a communication session), these are described by way of example and not limitation. Those skilled in the art who have read this disclosure will recognize additional embodiments that fall within the spirit and scope of the present disclosure.

[0201] It should be noted that access services delivered over a wide geographic area (such as providing access to games of the current embodiment) typically use cloud computing. Cloud computing is a form of computing in which dynamically scalable and usually virtualized resources are provided as services over the Internet. Users do not need to be experts in the technical infrastructure in the "cloud" that supports them. Cloud computing can be divided into different services, such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services typically provide general-purpose applications, such as video games, that are accessed online from a web browser, while the software and data are stored on servers in the cloud. Based on the way the Internet is depicted in computer network diagrams, the term "cloud" is used as a metaphor for the Internet and is an abstract concept for the complex infrastructure that it hides.

[0202] Game clients use a game processing server (GPS) (or simply "game server") to play single-player and multi-player video games. Most video games played on the Internet are run via a connection to a game server. Typically, the game uses a dedicated server application that collects data from the player and distributes it to other players. This is more efficient and effective than a peer-to-peer arrangement, but it requires a separate server to host the server application. In another embodiment, the GPS establishes communication between the player and their corresponding game playing device to exchange information without relying on a centralized GPS.

[0203] Dedicated GPS are servers that run independently of the clients. Such servers are usually run on dedicated hardware located in a data center, providing more bandwidth and dedicated processing power. For most PC-based multiplayer games, dedicated servers are the preferred method of hosting game servers. Massively multiplayer online games run on dedicated servers, which are usually hosted by the software company that owns the game title, allowing them to control and update the content.

[0204] The user accesses the remote service with a client device, which includes at least a CPU, a display, and an I / O. The client device can be a PC, a mobile phone, a netbook, a PDA, etc. In one embodiment, the network executed on the game server identifies the type of device used by the client and adjusts the communication method used. In other cases, the client device uses a standard communication method (such as html) to access the application on the game server through the Internet.

[0205] Embodiments of the present disclosure may be practiced with various computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. The present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices linked by wired or wireless networks.

[0206] It should be understood that a given video game or game application may be developed for a specific platform and a specific associated controller device. However, when such a game is made available via the game cloud system presented herein, the user may be accessing the video game with a different controller device. For example, a game may have been developed for a game console and its associated controller, and the user may be accessing a cloud-based version of the game from a personal computer using a keyboard and mouse. In this case, the input parameter configuration may define a mapping of inputs generated from a user's available controller device (in this case, a keyboard and mouse) to inputs acceptable for executing the video game.

[0207] In another example, the user can access the cloud gaming system via a tablet computing device, a touch screen smartphone, or other touch screen driven device. In this case, the client device and the controller device are integrated together in the same device, wherein the input is provided by means of a detected touch screen input / gesture. For such devices, the input parameter configuration can define a specific touch screen input corresponding to the game input of the video game. For example, during the operation of the video game, a button, a direction block, or other types of input elements may be displayed or superimposed to indicate the position on the touch screen where the user can touch to generate the game input. Gestures such as sliding in a specific direction or specific touch actions can also be detected as game inputs. In one embodiment, a guide indicating how to provide input via a touch screen for the game process can be provided to the user, for example, before starting the game process of the video game, so as to adapt the user to operate the controls on the touch screen.

[0208] In some embodiments, the client device is used as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection to transmit input from the controller device to the client device. The client device can process these inputs, and then transmit the input data to the cloud game server via a network (e.g., accessed via a local networking device such as a router). However, in other embodiments, the controller itself can be a networked device with the ability to transmit input to the cloud game server directly via a network, without first transmitting such inputs through a client device. For example, a controller can be connected to a local networking device (such as the above-mentioned router) to send data to and receive data from a cloud game server. Therefore, although it may still be necessary for the client device to receive video output from a cloud-based video game and present it on a local display, input latency can be reduced by allowing the controller to bypass the client device and send input directly to the cloud game server over the network.

[0209] In one embodiment, the networked controller and the client device can be configured to send certain types of input directly from the controller to the cloud game server, and to send other types of input via the client device. For example, inputs whose detection does not depend on any additional hardware or processing other than the controller itself can be sent directly from the controller to the cloud game server via the network, bypassing the client device. Such inputs may include button inputs, joystick inputs, embedded motion detection inputs (e.g., accelerometers, magnetometers, gyroscopes), etc. However, inputs that utilize additional hardware or require processing by the client device can be sent by the client device to the cloud game server. These may include video or audio captured from the game environment, which the client device can process before sending to the cloud game server. In addition, the input from the motion detection hardware of the controller can be processed by the client device in conjunction with the captured video to detect the position and motion of the controller, which is then transmitted to the cloud game server by the client device. It should be understood that the controller device according to various embodiments can also receive data (e.g., feedback data) from the client device or directly from the cloud game server.

[0210] It should be understood that the embodiments described herein can be executed on any type of client device. In some embodiments, the client device is a head mounted display (HMD) or a projection system. FIG. 9 is a diagram showing components of a head mounted display 102 according to an embodiment of the present disclosure. The HMD 102 can be configured to provide and receive game assistance in response to user queries related to a game application, wherein a deep learning engine associated with a backend game assistance server is used to match queries with modeled responses.

[0211] The head mounted display 102 includes a processor 1500 for executing program instructions. A memory 1502 is provided for storage purposes and may include volatile and non-volatile memory. A display 1504 is included that provides a visual interface that a user may view. A battery 1506 is provided as a power source for the head mounted display 102. A motion detection module 1508 may include any of various types of motion sensitive hardware, such as a magnetometer 1510, an accelerometer 1512, and a gyroscope 1514.

[0212] An accelerometer is a device used to measure acceleration and gravity-induced reaction forces. Single-axis and multi-axis models can be used to detect the magnitude and direction of acceleration in different directions. Accelerometers are used to sense tilt, vibration, and shock. In one embodiment, three accelerometers 1512 are used to provide the direction of gravity, which gives an absolute reference for two angles (world-space pitch and world-space roll).

[0213] The magnetometer measures the strength and direction of the magnetic field near the head mounted display. In one embodiment, three magnetometers 1510 are used within the head mounted display to ensure an absolute reference for the world space yaw angle. In one embodiment, the magnetometer is designed to span the earth's magnetic field of ±80 microtesla. The magnetometer is affected by metal and provides a yaw measurement that varies monotonically with the actual yaw. The magnetic field may be distorted by metal in the environment, which results in a distortion of the yaw measurement. If necessary, information from other sensors such as gyroscopes or cameras can be used to calibrate this distortion. In one embodiment, an accelerometer 1512 is used together with the magnetometer 1510 to obtain the tilt and azimuth of the head mounted display 102.

[0214] Gyroscopes are devices for measuring or maintaining orientation based on the principle of angular momentum. In one embodiment, three gyroscopes 1514 provide information about movement across corresponding axes (x, y, and z) based on inertial sensing. Gyroscopes help detect rapid rotations. However, gyroscopes can drift over time without an absolute reference. This requires periodic resetting of the gyroscopes, which can be done using other available information, such as position / orientation determination based on visual tracking of the object, accelerometers, magnetometers, etc.

[0215] A camera 1516 is provided for capturing images and image streams of the real environment. More than one camera may be included in the head mounted display 102, including a rear-facing camera (pointed away from the user when the user is viewing the display of the head mounted display 102), and a front-facing camera (pointed toward the user when the user is viewing the display of the head mounted display 102). In addition, a depth camera 1518 may be included in the head mounted display 102 for sensing depth information of objects in the real environment.

[0216] In one embodiment, a camera integrated on the front of the HMD can be used to provide safety warnings. For example, if the user is approaching a wall or an object, the user can be warned. In one embodiment, a user can be provided with a contour view of physical objects in the room to warn the user of their presence. The contour can be, for example, an overlay in the virtual environment. In some embodiments, the HMD user can be provided with a view of a reference marker that is superimposed on, for example, the floor. For example, the marker can provide the user with a reference to the center of the room in which the user is playing a game. For example, this can provide the user with visual information about where the user should move to avoid hitting a wall or other object in the room. Haptic warnings and / or audio warnings can also be provided to the user to provide greater safety when the user wears and plays games or navigates content with the HMD.

[0217] The head mounted display 102 includes a speaker 1520 for providing audio output. Furthermore, a microphone 1522 may be included for capturing audio from the real environment, including sounds from the surrounding environment, speech uttered by the user, etc. The head mounted display 102 includes a tactile feedback module 1524 for providing tactile feedback to the user. In one embodiment, the tactile feedback module 1524 can cause movement and / or vibration of the head mounted display 102 to provide tactile feedback to the user.

[0218] LED 1526 is provided as a visual indicator of the status of the head mounted display 102. For example, the LED may indicate battery charge, power on, etc. A card reader 1528 is provided to enable the head mounted display 102 to read information from and write information to a memory card. A USB interface 1530 is included as an example of an interface for enabling connection of peripheral devices or connection with other devices such as other portable devices, computers, etc. In various embodiments of the head mounted display 102, any of various types of interfaces may be included to enable greater connectivity of the head mounted display 102.

[0219] A WiFi module 1532 is included to enable connection to the Internet via wireless networking technology. Furthermore, the head mounted display 102 includes a Bluetooth module 1534 for enabling wireless connection to other devices. A communication link 1536 may also be included for connecting to other devices. In one embodiment, the communication link 1536 utilizes infrared transmission for wireless communication. In other embodiments, the communication link 1536 may utilize any of a variety of wireless or wired transmission protocols to communicate with other devices.

[0220] An input button / sensor 1538 is included to provide an input interface for the user. Any of various types of input interfaces may be included, such as buttons, touch pads, joysticks, trackballs, etc. An ultrasonic communication module 1540 may be included in the head mounted display 102 to facilitate communication with other devices via ultrasonic technology.

[0221] Biosensor 1542 is included to enable detection of physiological data from the user. In one embodiment, biosensor 1542 includes one or more dry electrodes for detecting bioelectrical signals of the user through the user's skin.

[0222] The photoelectric sensor 1544 is included to respond to signals from a transmitter (e.g., an infrared base station) placed in the three-dimensional physical environment. The game console analyzes information from the photoelectric sensor 1544 and the transmitter to determine position and orientation information related to the head mounted display 102.

[0223] In addition, a gaze tracking system 1565 is included and configured to enable tracking of the user's gaze. For example, the system 1565 may include a gaze tracking camera that captures an image of the user's eyes, which is then analyzed to determine the user's gaze direction. In one embodiment, information about the user's gaze direction can be used to influence video rendering. Video rendering in the gaze direction can be prioritized or emphasized, such as by providing greater detail, higher resolution through foveal rendering, higher resolution particle system effects displayed in the foveal area, lower resolution particle system effects displayed outside the foveal area, or faster updates in the area where the user is looking.

[0224] The foregoing components of the head mounted display 102 are merely described as exemplary components that may be included in the head mounted display 102. In various embodiments of the present disclosure, the head mounted display 102 may include or may not include some of the various foregoing components. Embodiments of the head mounted display 102 may additionally include other components that are not currently described but are known in the art for the purpose of facilitating various aspects of the present disclosure as described herein.

[0225] Those skilled in the art will recognize that in various embodiments of the present disclosure, the aforementioned head mounted device can be utilized in conjunction with an interactive application displayed on a display to provide various interactive functions. The exemplary embodiments described herein are provided by way of example only and not limitation.

[0226] Fig.16 1 is a block diagram of a gaming system 1600 according to various embodiments of the present disclosure. The gaming system 1600 is configured to provide a video stream to one or more clients 1610 via a network 1615, such as in a single player mode or a multiplayer mode. The gaming system 1600 typically includes a video server system 1620 and an optional gaming server 1625. The video server system 1620 is configured to provide a video stream to one or more clients 1610 with a minimum quality of service. For example, the video server system 1620 may receive a game command that changes the state of a video game or the viewing angle within the video game, and provide an updated video stream reflecting this state change to the client 1610 with a minimum lag time. The video server system 1620 may be configured to provide a video stream in a variety of alternative video formats (including formats that have not yet been defined). In addition, the video stream may include video frames that are configured to be presented to a user at a variety of frame rates. Typical frame rates are 30 frames per second, 80 frames per second, and 820 frames per second. However, higher or lower frame rates are included in alternative embodiments of the present disclosure.

[0227] Client 1610 (referred to as 1610A, 1610B, etc., respectively, in this article) can include head-mounted display, terminal, personal computer, game console, tablet computer, phone, set-top box, kiosk, wireless device, numeric keypad, independent device, handheld game playing device, etc. Usually, client 1610 is configured to receive encoded video stream (for example, compressed), decode the video stream, and present the resulting video to the user, such as a game player. The process of receiving encoded video stream and / or decoding the video stream generally includes storing each video frame in the receiving buffer of the client. The video stream can be presented to the user on a display integrated with client 1610 or on a separate device such as a monitor or a television. Client 1610 is optionally configured to support more than one game player. For example, a game console can be configured to support two, three, four or more simultaneous players. Each of these players can receive a separate video stream, or a single video stream can include a frame area generated specifically for each player, such as a frame area generated based on the perspective of each player. Client 1610 is optionally geographically dispersed. The number of clients included in the gaming system 1600 can vary widely from one or two to thousands, tens of thousands, or greater. As used herein, the term "game player" is used to refer to a person playing a game, and the term "game playing device" is used to refer to a device for playing a game. In some embodiments, a game playing device may refer to a plurality of computing devices that collaborate to deliver a gaming experience to a user. For example, a game console and an HMD may collaborate with a video server system 1620 to deliver a game viewed through an HMD. In one embodiment, a game console receives a video stream from a video server system 1620, and the game console forwards the video stream or an update to the video stream to the HMD for rendering.

[0228] Client 1610 is configured to receive the video stream via network 1615. Network 1615 can be any type of communication network, including a telephone network, the Internet, a wireless network, a power line network, a local area network, a wide area network, a dedicated network, etc. In a typical embodiment, the video stream is transmitted via a standard protocol (such as TCP / IP or UDP / IP). Alternatively, the video stream is transmitted via a proprietary standard.

[0229] A typical example of a client 1610 is a personal computer including a processor, non-volatile memory, a display, decoding logic, network communication capabilities, and input devices. The decoding logic may include hardware, firmware, and / or software stored on a computer-readable medium. Systems for decoding (and encoding) video streams are well known in the art and vary depending on the specific encoding scheme used.

[0230] Client 1610 may but need not further include a system configured to modify the received video. For example, the client may be configured to perform further rendering, superimpose a video image on another video image, crop the video image, etc. For example, client 1610 may be configured to receive various types of video frames, such as I frames, P frames, and B frames, and process these frames into images to display to the user. In some embodiments, members of client 1610 are configured to perform other rendering, coloring, conversion to 3D, or similar operations on the video stream. Members of client 1610 are optionally configured to receive more than one audio or video stream. The input device of client 1610 may include, for example, a one-handed game controller, a two-handed game controller, a gesture recognition system, a line of sight recognition system, a voice recognition system, a keyboard, a joystick, a pointing device, a force feedback device, a motion and / or position sensing device, a mouse, a touch screen, a neural interface, a camera, an input device yet to be developed, etc.

[0231] The video stream (and optional audio stream) received by the client 1610 is generated and provided by the video server system 1620. As further described elsewhere herein, the video stream includes video frames (and the audio stream includes audio frames). Video frames are configured (e.g., they include pixel information in appropriate data structures) to make a meaningful contribution to the image displayed to the user. As used herein, the term "video frame" is used to refer to a frame that primarily includes information configured to contribute to (e.g., implement) the image shown to the user. Most of the teachings herein about "video frames" can also be applied to "audio frames."

[0232] Client 1610 is usually configured to receive input from the user. These inputs may include game commands that are configured to change the state of the video game or otherwise affect the process of playing the game. Game commands can be received using input devices and / or can be automatically generated by the computing instructions executed on client 1610. The received game commands are transmitted to video server system 1620 and / or game server 1625 via network 1615 from client 1610. For example, in some embodiments, game commands are transmitted to game server 1625 via video server system 1620. In some embodiments, a separate copy of the game command is transmitted to game server 1625 and video server system 1620 from client 1610. The transmission of game commands optionally depends on the identification of the command. Optionally, game commands are transmitted from client 1610A by different routes or communication channels for providing audio or video streams to client 1610A.

[0233] The game server 1625 is optionally operated by an entity different from the video server system 1620. For example, the game server 1625 may be operated by the publisher of a multi-player game. In this example, the video server system 1620 is optionally viewed as a client by the game server 1625, and is optionally configured to be a prior art client that executes a prior art game engine from the perspective of the game server 1625. The communication between the video server system 1620 and the game server 1625 is optionally performed via the network 1615. Therefore, the game server 1625 may be a prior art multi-player game server that sends game state information to multiple clients, one of which is the game server system 1620. The video server system 1620 may be configured to communicate with multiple instances of the game server 1625 at the same time. For example, the video server system 1620 may be configured to provide multiple different video games to different users. Each of these different video games may be supported by a different game server 1625 and / or published by a different entity. In some embodiments, several geographically distributed instances of the video server system 1620 are configured to provide game videos to multiple different users. Each of these instances of the video server system 1620 can communicate with the same instance of the game server 1625. The communication between the video server system 1620 and one or more game servers 1625 optionally occurs via a dedicated communication channel. For example, the video server system 1620 can be connected to the game server 1625 via a high-bandwidth channel that is dedicated to communication between the two systems.

[0234] The video server system 1620 includes at least a video source 1630, an I / O device 1645, a processor 1650, and a non-transitory storage device 1655. The video server system 1620 may include one computing device or be distributed among multiple computing devices. These computing devices are optionally connected via a communication system such as a local area network.

[0235] Video source 1630 is configured to provide a video stream, such as a stream video or a series of video frames forming a dynamic picture. In some embodiments, video source 1630 includes a video game engine and rendering logic. The video game engine is configured to receive game commands from the player, and maintains a copy of the video game state based on the commands received. The game state includes the position of the object in the game environment, and generally includes the viewing angle. The game state can also include the characteristics, image, color and / or texture of the object.

[0236] Usually based on game rules and such as moving, turning, attacking, setting focus, interacting, using and other game commands to maintain game state. A part of the game engine is optionally arranged in the game server 1625. Game server 1625 can maintain a copy of game state based on the game commands received from multiple players using geographically dispersed clients. In these cases, game state is provided to video source 1630 by game server 1625, wherein a copy of game state is stored and rendering is performed. Game server 1625 can directly receive game commands from client 1610 via network 1615, and / or can receive game commands via video server system 1620.

[0237] The video source 1630 typically includes rendering logic, such as hardware, firmware, and / or software stored on a computer-readable medium such as a storage device 1655. The rendering logic is configured to create video frames of a video stream based on the game state. All or part of the rendering logic is optionally disposed in a graphics processing unit (GPU). The rendering logic typically includes a processing stage that is configured to determine the three-dimensional spatial relationship between objects and / or apply appropriate textures, etc. based on the game state and the viewing angle. The rendering logic generates a raw video, which is then typically encoded before being transmitted to the client 1610. For example, the raw video may be encoded according to the following: Adobe Acrobat®, Adobe Premiere Pro, Adobe Illustrator, and Adobe Illustrator. Standard, .wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith, MPG-x, Xvid, FFmpeg, x264, VP6-8, realvideo, mp3, etc. The encoding process produces a video stream, which is optionally encapsulated for delivery to a decoder on a remote device. The video stream is characterized by frame size and frame rate. Typical frame sizes include 800×600, 1280×720 (e.g., 720p), 1024×768, but any other frame size can be used. The frame rate is the number of video frames per second. A video stream can include different types of video frames. For example, the H.264 standard includes "P" frames and "I" frames. I frames include information for refreshing all macroblocks / pixels on a display device, while P frames include information for refreshing a subset thereof. The data size of a P frame is typically smaller than an I frame. As used herein, the term "frame size" refers to the number of pixels within a frame. The term "frame data size" is used to refer to the number of bytes required to store a frame.

[0238] In an alternative embodiment, video source 1630 includes a video recording device, such as a camera. The camera can be used to generate delayed video or real-time video, which can be included in the video stream of a computer game. The resulting video stream optionally includes both rendered images and images recorded using a static or video camera. Video source 1630 can also include a storage device that is configured to store previously recorded videos to be included in the video stream. Video source 1630 can also include a motion or positioning sensing device configured to detect the motion or position of an object (e.g., a person), and logic configured to determine a game state or generate a video based on the motion and / or position detected.

[0239] Video source 1630 is optionally configured to provide overlays configured to be placed on other videos. For example, these overlays may include command interfaces, login instructions, messages to gamers, images of other gamers, video feeds of other gamers (e.g., webcam videos). In embodiments of client 1610A including a touch screen interface or a line of sight detection interface, the overlays may include a virtual keyboard, a joystick, a touchpad, etc. In one example of an overlay, the player's voice is overlaid on an audio stream. Video source 1630 optionally also includes one or more audio sources.

[0240] In embodiments where the video server system 1620 is configured to maintain a game state based on input from more than one player, each player may have a different viewing angle including the position and direction of viewing. The video source 1630 is optionally configured to provide a separate video stream for each player based on the viewing angle of each player. In addition, the video source 1630 may be configured to provide different frame sizes, frame data sizes, and / or encodings to each of the clients 1610. The video source 1630 is optionally configured to provide 3D video.

[0241] I / O devices 1645 are configured to enable the video server system 1620 to send and / or receive information, such as video, commands, information requests, game status, line of sight information, device motion, device location, user motion, client identification, player identification, game commands, security information, audio, etc. I / O devices 1645 typically include communication hardware, such as a network card or a modem. I / O devices 1645 are configured to communicate with game servers 1625, network 1615, and / or clients 1610.

[0242] Processor 1650 is configured to execute logic, such as software, included within the various components of the video server system 1620 discussed herein. For example, processor 1650 can be programmed with software instructions to perform the functions of video source 1630, game server 1625, and / or client qualifier 1660. Video server system 1620 optionally includes more than one instance of processor 1650. Processor 1650 can also be programmed with software instructions to execute commands received by video server system 1620, or to coordinate the operation of the various elements of the game system 1600 discussed herein. Processor 1650 can include one or more hardware devices. Processor 1650 is an electronic processor.

[0243] Storage device 1655 includes non-temporary analog and / or digital storage devices. For example, storage device 1655 may include an analog storage device configured to store video frames. Storage device 1655 may include a computer-readable digital storage device, such as a hard disk drive, an optical disk drive, or a solid-state storage device. Storage device 1655 is configured to store video frames, artificial frames, video streams including both video frames and artificial frames, audio frames, audio streams, etc. (for example, through an appropriate data structure or file system). Storage device 1655 is optionally distributed among multiple devices. In some embodiments, storage device 1655 is configured to store software components of video source 1630 discussed elsewhere herein. These components can be stored in a format that is ready to be provided when needed.

[0244] The video server system 1620 optionally also includes a client qualifier 1660. The client qualifier 1660 is configured to remotely determine the capabilities of a client such as client 1610A or 1610B. These capabilities may include the capabilities of the client 1610A itself and the capabilities of one or more communication channels between the client 1610A and the video server system 1620. For example, the client qualifier 1660 may be configured to test the communication channel through the network 1615.

[0245] The client qualifier 1660 can manually or automatically determine (e.g., discover) the capabilities of the client 1610A. Manual determination includes communicating with the user of the client 1610A and asking the user to provide capabilities. For example, in some embodiments, the client qualifier 1660 is configured to display images, text, etc. in the browser of the client 1610A. In one embodiment, the client 1610A is an HMD including a browser. In another embodiment, the client 1610A is a game console with a browser, which can be displayed on the HMD. The displayed object requests the user to input information such as the operating system, processor, video decoder type, network connection type, display resolution, etc. of the client 1610A. The information entered by the user is transmitted back to the client qualifier 1660.

[0246] Automatic determination can occur, for example, by executing an agent on client 1610A and / or by sending a test video to client 1610A. The agent may include computing instructions embedded in a web page or installed as a post-add-on, such as a java script. Client qualifier 1660 optionally provides an agent. In various embodiments, the agent can discover the processing power of client 1610A, the decoding and display capabilities of client 1610A, the latency reliability and bandwidth of the communication channel between client 1610A and video server system 1620, the display type of client 1610A, the firewall present on client 1610A, the hardware of client 1610A, the software executed on client 1610A, the registry entries within client 1610A, etc.

[0247] Client Qualifier 1660 includes hardware, firmware, and / or software stored on a computer-readable medium. Client Qualifier 1660 is optionally disposed on a computing device separate from one or more other elements of Video Server System 1620. For example, in some embodiments, Client Qualifier 1660 is configured to determine characteristics of a communication channel between Client 1610 and one or more instances of Video Server System 1620. In these embodiments, the information discovered by Client Qualifier can be used to determine which instance of Video Server System 1620 is best suited to deliver streaming video to one of Clients 1610.

[0248] Although specific embodiments have been provided to demonstrate the prediction and updating of a target landing point on a display so that the movement of the user's eyes is consistent with the presentation of the foveal area on the display at the updated target landing point, these are described by way of example and not limitation. Those skilled in the art who have read this disclosure will recognize additional embodiments that fall within the spirit and scope of the present disclosure.

[0249] It should be understood that the various embodiments defined herein can be combined or compiled into specific implementations using the various features disclosed herein. Therefore, the examples provided are only some possible examples and are not limited to various implementations that are possible by combining various elements to define more implementations. In some examples, some implementations may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0250] Embodiments of the present disclosure may be practiced with a variety of computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where work is performed by remote processing devices that are linked through wired or wireless networks.

[0251] In view of the above embodiments, it should be understood that the embodiments of the present disclosure can adopt various computer-implemented operations involving data stored in a computer system. These operations are those operations that require physical manipulation of physical quantities. Any operation that forms a part of the embodiments of the present disclosure described herein is a useful machine operation. Embodiments of the present invention also relate to devices or equipment for performing these operations. The equipment can be specially constructed for the desired purpose, or the equipment can be a general-purpose computer selectively activated or configured by a computer program stored in a computer. Specifically, various general-purpose machines can be used together with computer programs written according to the teachings of this article, or it may be more convenient to construct more specialized equipment to perform the desired operations.

[0252] The present disclosure may also be embodied as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data, which can thereafter be read by a computer system. Examples of computer readable media include hard drives, network attached storage devices (NAS), read-only memories, random access memories, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer readable medium may include a computer readable tangible medium distributed on a network coupled computer system so that the computer readable code is stored and executed in a distributed manner.

[0253] Although the method operations are described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be adjusted so that they occur at slightly different times, or the operations may be distributed in a system that allows processing operations to occur at various intervals associated with the processing as long as the processing of the superimposed operations is performed in the desired manner.

[0254] Although the foregoing disclosure has been described in some detail for the purpose of clarity of understanding, it will be apparent that some changes and modifications may be practiced within the scope of the appended claims. Therefore, the present embodiments are to be considered as illustrative rather than restrictive, and the embodiments of the present disclosure are not to be limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.

Claims

1. A method for providing game assistance, comprising: determining a current game scenario of a game application, the current game scenario including at least a game state and user information related to the game application; sending a query from the game application including the current game scenario; and receiving, by the game application, a response to the query, the response being generated based on a comparison of the current game scenario with the playing processes of multiple users.

2. The method according to claim 1, wherein, the query further includes audio or video input of a user from the game application.

3. The method according to claim 1, wherein, the current game scenario includes a level indication.

4. The method according to claim 1, wherein, the current game scenario indicates a specific scene in the game application for a given game situation.

5. The method according to claim 1, wherein, the method further includes: receiving, by a user device that did not send the query, the response to the query.

6. The method according to claim 1, wherein, the current game scenario includes a game level situation of the playing process of the game application.

7. The method according to claim 1, wherein, the current game scenario includes an operating system level situation of the playing process of the game application.

8. The method according to claim 7, wherein, the operating system level situation includes an indication of an actuated button or actuation speed.

9. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors of a system, cause the system to perform operations including the following: determining a current game scenario of a game application, the current game scenario including at least a game state and user information related to the game application; sending a query from the game application including the current game scenario; and receiving, by the game application, a response to the query, the response being generated based on a comparison of the current game scenario with the playing processes of multiple users.

10. The one or more non-transitory computer-readable storage media according to claim 9, wherein, the current game scenario includes the duration of a player playing the game application.

11. The one or more non-transitory computer-readable storage media according to claim 9, wherein, the current game scenario includes the time when the player of the game application last played the game application.

12. The one or more non-transitory computer-readable storage media according to claim 9, wherein, the current game scenario includes the frequency of a player of the game application requesting assistance.

13. The one or more non-transitory computer-readable storage media according to claim 9, wherein, the current game scenario indicates at least one of the following: the position of a controlled character in the game application, what the controlled character has done, what assets and skills the controlled character has accumulated, or what tasks or jobs have been presented to the controlled character.

14. One or more non-transitory computer-readable storage media as recited in claim 9, wherein, the current game scenario indicates at least one of the following: a set of game characters, a set of game objects, a set of game object attributes, a set of game object states, a set of graphical overlays, a set of positions of characters in the game, the scene of the game application, a set of character assets, character types or races, a set of tasks presented to the player, equipment, a set of character skills, a set of character attributes, character positions, remaining number of lives, total possible number of available lives, a set of armors or a set of trophies.

15. A system, comprising: one or more storage media storing first instructions; and one or more processors configured to execute the instructions to cause the system to perform the operations of any one of claims 1-14 above.

Images