System and method for enabling prediction assistance during game play

Abstract

SOLUTION: A method for providing assistance during gameplay is described. The method includes accessing a profile model associated with a user account of a user. The profile model is used to generate one or more predictive indicators on the basis of a plurality of game contexts of one or more games. The method further includes receiving a request for accessing a game for a game session via the user account, and generating assistance input for the user in response to the one or more predictive indications that the user will be unable to complete a task in the game. The task is associated with a game context. The assistance input is provided before the user performs the task.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to systems and methods for enabling prediction assistance during game play. [Background technology]

[0002] Conventional games may offer a cooperative multiplayer mode in which players use their in-game avatars, which are virtual representations within the game environment, to help each other complete tasks. In a multiplayer mode, typically two separate players work together to achieve a goal.

[0003] In contrast, in single-player game modes, the objective of the game is to provide challenges to an individual player within the game world, and it is up to this player to accomplish the in-game objectives. However, occasionally, single or multiplayer games provide challenges that utilize more skills than a player possesses.

[0004] Traditionally, to address the challenges presented by single or multiplayer games, some players detail their game progress as a text document or a series of videos, but these solutions can be inadequate, especially when a player is unable to overcome a particular challenge.

[0005] It is against this background that embodiments of the present invention arise. Summary of the Invention

[0006] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure provide systems and methods that enable prediction assistance during game play.

[0007] In one embodiment, a method is described for processing a user's gameplay over a period of time to learn the user's tendencies, skill level, input accuracy, and general ability. The gameplay may be processed by capturing a gameplay history of a user, such as a player, for one or more games. The gameplay history may include game state data for each of the one or more games that identify the context of the game at different instances and the user's reaction to the particular game context. Over time, it is possible to classify different types of inputs and game contexts to build a profile model that describes the user's game tendencies and potential weaknesses. Using the profile model, during a current gameplay, one or more processors, such as an artificial intelligence processor, may predict where the user is going to move or achieve in a particular context. If the user is attempting a game input or combination of inputs in a particular context where the player is weakened, the one or more processors may assist the user in the form of button input or tactical suggestions for the particular context.

[0008] In some embodiments, a user who is unable to consistently provide correct inputs to a particular game control may be assisted by one or more processors that automatically press buttons on the user's behalf. This button pressing on the user's behalf is based on the user's predicted intent for a particular game context. The predicted intent is determined by the one or more processors.

[0009] In some embodiments, the one or more processors may execute multiple button presses or combinations of button presses on behalf of the user if the user foresees difficulty or inability to complete a particular level, combination of inputs, or achieve a result after some misstep or failure.

[0010] In embodiments, predictive assistance may be provided automatically by the one or more processors when the assistance mode is activated by a user. When the user is aware that the assistance mode is activated, the one or more processors may apply one or more inputs on behalf of the user based on the user's predicted intent or weaknesses in performing a particular task.

[0011] In one embodiment, the assisted mode may be a form of autopilot in which the user allows the gaming system to complete one or more game tasks, sequences, levels, or achieve outcomes that are difficult for the user. Once the user feels comfortable resuming control, the assisted mode can be deactivated.

[0012] In an embodiment, the one or more processors learn the user's gaming style and use it to press buttons on behalf of the user based on the predicted intent. For example, the one or more processors perform one or more tests to determine the user's gaming style, such as informing the user to quickly complete in-game tasks, informing the user to jump over virtual objects, etc. In this example, the one or more processors store the gaming style in a user profile to build a profile model and use the gaming style to adjust the user's game.

[0013] In one embodiment, the method includes determining a mood or enthusiasm of a user, such as a cool mood, an intense mood, etc., and adjusting a mode of gameplay, for example, to an easy mode or a difficult mode based on the mood or enthusiasm indicated by the user.

[0014] In an embodiment, the method includes controlling button illumination and haptic feedback to train the user.

[0015] In one embodiment, for new or novice users, the assistance mode can be an easy mode, for moderately skilled users, the assistance mode can be a medium mode, and for experienced users, the assistance mode can be a hard mode.

[0016] In an embodiment, a method of providing assistance during game play is described. The method includes accessing a profile model associated with a user account of a user. The profile model is used to generate one or more predictive indicators based on a plurality of game contexts of one or more games. The method further includes receiving a request to access a game for a game session via the user account, and generating an assistance input for the user in response to the one or more predictive indicators that the user will not be able to complete a task in the game. The task is associated with the game context. The assistance input is provided before the user performs the task.

[0017] In one embodiment, a computer system for providing assistance during game play is described. The computer system includes a processor. The processor accesses a profile model associated with a user account of a user. The profile model is used to generate one or more predictive indicators based on a plurality of game contexts of one or more games. The processor receives a request to access a game for a game session via the user account, and generates an assistance input for the user in response to the one or more predictive indicators that the user will not be able to complete a task in the game. The task is associated with the game context. The processor provides the assistance input before the user performs the task. The computer system includes a memory device coupled to the processor.

[0018] In an embodiment, a system for providing assistance during game play is described. The system includes a client device and a server coupled to the client device via a computer network. The server accesses a profile model associated with a user account of a user. The profile model is used to generate one or more predictive indicators based on a plurality of game contexts of one or more games. The server receives a request to access a game for a game session via the user account. The server generates an assistance input for the user in response to the one or more predictive indicators that the user will not be able to complete a task in the game. The task is associated with the game context. The server transmits the assistance input before the user performs the task. The client device receives the assistance input from the server via the computer network.

[0019] Some advantages of the systems and methods described herein include providing predictive assistance to a user while playing a game. Providing predictive assistance increases network traffic efficiency and server efficiency. When the predictive assistance is provided, the user interacts with the game and progresses through the game. Without predictive assistance, the user is unable to progress through the game and continues to manipulate the controller in an improper manner. Manipulating the controller in an improper manner generates improper inputs. The improper inputs are then sent to the server, which increases network traffic between the server and the controller. Also, the load on the server increases. Providing predictive assistance to a user increases server efficiency because the likelihood of generating improper inputs is reduced. Also, the likelihood of network traffic congestion is reduced. If the likelihood of network traffic congestion is reduced, the efficiency of the network is increased.

[0020] Other aspects of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the embodiments described in the present disclosure. [Brief description of the drawings]

[0021] The various embodiments of the present disclosure are best understood by referring to the following description taken in conjunction with the accompanying drawings.

[0022] [Figure 1] FIG. 1 illustrates an embodiment of a game context showing multiple activities presented to a user to enable the user to play a game.

[0023] [Diagram 2] 1 is a graph showing the activity level during each activity undertaken by a user and the time it took the user to complete the activity.

[0024] [Diagram 3] 1 is a graph illustrating the collection of state data by one or more processors of a gaming system as a user participates in an activity within a gaming context.

[0025] [Figure 4] FIG. 1 illustrates an embodiment of a system illustrating generation of predictive indicators based on state data determined from a user's engagement in multiple game contexts.

[0026] [Diagram 5] FIG. 1 is a diagram of an embodiment of a system illustrating the use of a handheld controller (HHC) and a gaming system to provide assistance input data to a user via the HHC based on a predictive indicator.

[0027] [Figure 6] FIG. 1 is a diagram of an embodiment of a system illustrating that assistance input data is generated and transmitted to one or more different areas of an HHC.

[0028] [Figure 7] 1 illustrates components of an exemplary device that can be used to implement aspects of various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] A system and method for enabling prediction assistance during game play is described. It should be noted that various embodiments of the present disclosure may be practiced without some or all of these specific details. In other instances, well known process operations have not been described in detail so as not to unnecessarily obscure various embodiments of the present disclosure.

[0030] 1 is a diagram of an embodiment of a game context 100 illustrating multiple activities A1, A2, A3, and A4 presented to a user 102 to enable the user 102 to play a game. An example of a game context described herein is one or more virtual scenes, such as a virtual reality (VR) scene having one or more activities. Illustratively, the game context 100 is a virtual scene that includes multiple virtual objects, such as a virtual character 106, another virtual character 108, a virtual giant 110, a virtual game chest 112, and a virtual valley 114, as well as the positions and orientations of the virtual objects relative to one another within the virtual scene. Illustratively, each virtual object is identified by the shape, size, and color of the virtual object.

[0031] A user 102 plays a game using a handheld controller (HHC) 104. The HHC 104 is an example of a game controller. By using the HHC 104, the user 102 logs into a user account 1 assigned to the user 102 and accesses a game session of the game from a game system, such as a cloud system or a computing device, through the user account 1. In one example, the game system includes one or more processors and one or more memory devices. The one or more processors are coupled to the one or more memory devices. Examples of the processor include an application specific integrated circuit (ASIC), a programmable logic device (PLD), a microcontroller, or a microprocessor. Examples of the memory device include a read only memory (ROM), or a random access memory device (RAM), or a combination thereof. In one example, the memory device is a flash memory, or a hard disk, or a redundant array of independent disks (RAID). Examples of the computing device include a desktop computer, a laptop computer, a smart TV, a tablet, a head mounted display (HMD), a game console, and a smartphone. An example of a cloud system includes one or more servers.

[0032] It should be noted that the HHC 104 and / or the computing devices are examples of client devices and the cloud system is an example of a server system. The client devices are coupled to the server system via a computer network, such as the Internet, an intranet, or a combination thereof.

[0033] A game session is accessed when a game program, such as game code, is executed by one or more processors of the game system. Upon accessing the game session, a game context 100 is displayed on a display device of the client device. During the game session, the user 102 encounters multiple activities, such as activities A1-A4, within the game context 100. For example, the user 102 uses the HHC 104 to control a virtual character 106 and fight a virtual character 108. In this example, the fight between the virtual character 106 and the virtual character 108 is an example of activity A1 within the game context 100. Further, in this example, the user 102 then uses the HHC 104 to control the virtual character 106 and fight a virtual giant 110. In this example, the fight with the virtual giant 110 is an example of activity A2. Also in this example, after participating in activity A2, for example, the virtual character 106 is controlled by the user 102 via the HHC 104 to fight the virtual giant 110, and after defeating the virtual giant 110, the virtual character 106 finds a virtual game chest 112. In this example, the user 102 controls the HHC 104 to open the virtual game chest 112 and obtains a virtual item, such as a virtual gun or virtual ammunition, from the virtual game chest 112. In this example, the opening of the virtual game chest 112 by the virtual character 106 and the retrieval of the virtual item from the virtual game chest 112 are examples of activity A3. Further, in this example, after activity A3, the user 102 controls the virtual character 106 via the HHC 104 to cross the virtual valley 114. In this example, the action of the virtual character 106 crossing the virtual valley 114 is an example of activity A4. In this example, game context data for generating the game context 100 is generated by one or more processors of the game system for display on a display device of the client device. At the end of the game session, user 102 uses HHC 104 to log out of user account 1. Logging out of user account 1 ends the game session.

[0034] Similarly, user 104 accesses additional game sessions of this game or other games through user account 1. As user 102 accesses additional game sessions, one or more processors of the gaming system create game contexts 116, and user 102 engages in a number of activities provided by game contexts 116 in each of the additional game sessions. In one example, one or more of game contexts 116 are for the same game as game context 100. Illustratively, game context 100 is for a game with game title Fortnite™, and one or more of game contexts 116 are for the same game with game title Fortnite™. In another example, one or more of game contexts 116 are for a different game than game 100. Illustratively, game context 100 is for a game with game title Fortnite™, and one or more of game contexts 116 are for a different game with game title Apex Legends™. Note that each of the additional game sessions is accessed by user 102 after logging into user account 1, and each of the additional game sessions ends after user 102 logs out of user account 1.

[0035] In an embodiment, a game session ends when a user 102 achieves a final outcome in the game session.

[0036] In one embodiment, a game session of a game ends when the game ends or is controlled to end by the user 102. After the game ends, the game session of the same game or a different game is accessed from one or more processors of the gaming system for game play by the user 102.

[0037] In one embodiment, the one or more virtual scenes include one or more augmented reality (AR) scenes.

[0038] In an embodiment, the user 102 uses multiple handheld controllers instead of the HHC 104 to play a game. For example, the user 102 holds a first handheld controller in his left hand and a second handheld controller in his right hand to play a game. Each of the handheld controllers may be referred to herein as a game controller.

[0039] In one embodiment, the game context includes any other number of activities, for example, the game context includes one activity, or two activities, or ten activities.

[0040] In one embodiment, instead of the HHC 104, an HMD is used as the controller.

[0041] In an embodiment, the HMD is used as a controller in addition to the HHC 104. In this embodiment, the HHC 104 and the HMD are referred to herein as the controller.

[0042] FIG. 2 is a graph 200 illustrating activity levels, such as difficulty levels between each of activities A1-A4, and the time t taken by user 102 to complete the activities. Graph 200 plots activity levels on the y-axis and time t on the x-axis. Time t ranges from time t0 to time t22. Note that the duration between any two consecutive times between times t0-t22 is equal. For example, the duration from time t0 to t1 is equal to the duration from time t1 to t2. Activity levels range from 0 to 8, increasing from 0 to 8. 0 is an easy activity level, 8 is a difficult or strenuous activity level, and activity levels from 1 to 7 range between easy and difficult levels. Activity levels are assigned by a game program executed by one or more processors of the game system.

[0043] As shown in graph 200, one or more processors of the gaming system assign activity level 4 to activity A1 and determine that a user 102 (FIG. 1) takes a first time period from time t0 to time t6 to complete activity A1. For example, the one or more processors of the gaming system may include or have access to a clock source and determine, based on a clock signal generated by the clock source, that a user 102, via user account 1, battles a virtual character 108 (FIG. 1) and takes a first time period to defeat the virtual character 108. Similarly, the one or more processors of the gaming system may assign activity level 8 to activity A2 and determine that a user 102 takes a time period from time t6 to time t24 to complete activity A2, assign activity level 1 to activity A3 and determine that a user 102 takes a time period from time t14 to time t17 to complete activity A3, and assign activity level 3 to activity A4 and determine that a user 102 takes a time period from time t17 to time t22 to complete activity A4. For example, the one or more processors of the gaming system determine, based on the clock signal, that it takes a second time period for the user 102 to battle the virtual giant 110 (FIG. 1) via user account 1. In this example, the second time period is the time it takes the virtual character 106 (FIG. 1) to battle and defeat the virtual giant 110. In this example, the one or more processors of the gaming system further determine that the second time period is greater than the first time period. Similarly, the one or more processors of the gaming system determine an activity level for an activity completed by the user 102 during the game context 116 (FIG. 1).

[0044] 3 is a graph 300 illustrating the collection of state data 302 by one or more processors of the gaming system as the user 102 (FIG. 1) participates in activities A1-A4 within the game context 100. Examples of state data are shown below: The one or more processors of the gaming system collect state data 302 during each of the activities A1-A4. For example, as the user 102 participates in the activities A1-A4, the one or more processors of the gaming system obtain, such as determine, identify, or access, the state data 302 and store the state data 302 in one or more memory devices of the gaming system.

[0045] It should be noted that the state data 302 is obtained and stored on the fly. For example, immediately after the virtual character 106 fights the virtual character 108 (FIG. 1) and defeats the virtual character 108, the one or more processors of the game system determine that the virtual character 106 has finished fighting the virtual character 108 based on the posture and orientation of the virtual character 106 relative to the posture and orientation of the virtual character 108, and identify a number of virtual achievements, such as a number of virtual points, or a number of virtual kills, or a combination thereof, assigned to the user account 1 as soon as the posture and orientation of the virtual character 106 relative to the posture and orientation of the virtual character 108 is obtained. In this example, the posture of the virtual character 108 is a sideways posture, such as a dead position or a posture lying flat on the virtual ground in the game context 100 (FIG. 1). Also, in this example, the posture of the virtual character 106 is an upright posture, such as a standing position. Furthermore, in this example, the one or more processors of the game system identify the activity level as 4 for activity A1 and calculate a first period. In this example, the pose and orientation of virtual character 106, the pose and orientation of virtual character 108, the number of virtual achievements, the activity level, and the first time period are examples of state data 302, and the one or more processors store the number of virtual achievements, the activity level, and the first time period in one or more memory devices of the gaming system. Further, in this example, the one or more processors of the gaming system identify a skill level assigned to user 102 within user account 1 while engaged in activities A1-A4. For example, the skill level is beginner, average, or expert. In this example, the skill level is assigned to user 102 by the one or more processors based on the performance level of user 102 during gameplay preceding gameplay using context 100 and during gameplay using context 100. In this example, the skill level is an example of state data 302.

[0046] Further, in this example, the one or more processors of the gaming system track the time taken to press or move each of the buttons of the HHC 104 (FIG. 1) to engage in activity A1, or the time taken to move one of a set of buttons, such as a joystick, or the time taken to move the HHC 104 between two of a plurality of consecutive positions and orientations of the HHC 104, or a combination thereof. In this example, the time taken to move each of the buttons of the HHC 104 is an example of state data 302. Also, in this example, the time taken to move the set of buttons of the HHC 104 between two of a plurality of consecutive positions and orientations is an example of state data 302. Further, in this example, the one or more processors of the gaming system track multiple movements, such as pressing or moving a button of the HHC 104, to engage in activity A1, and store the time and the number of button movements in one or more memory devices of the gaming system. In this example, the number of movements is an example of state data 302. Illustratively, the one or more processors of the gaming system may determine that it took 10 seconds from the time activity A1, such as virtual characters 106 and 108, to be displayed on the display device for user 102 to press the X button on HHC 104 to initiate a jump and thereby avoid being shot by virtual character 108 (FIG. 1). In the illustrative example, the one or more processors of the gaming system may determine that it took 15 seconds for user 102 to press the R1 button on HHC 104 after pressing the X button to enable virtual character 106 to shoot and defeat virtual character 108. In the illustrative example, each of the X button and the R1 button is an example of one of the buttons on HHC 104.

[0047] Further by way of example, the one or more processors of the gaming system determine a plurality of successive positions and orientations of one of a set of buttons, such as a joystick set, of the HHC 104 while moving the virtual character 106 to defeat the virtual character 108 based on inertial sensor data received from inertial sensors, such as magnetometers, accelerometers, and gyroscopes, of the buttons of the HHC 104. By way of further example, the plurality of successive positions and orientations of one of the set of buttons include a first position (x1, y1, z1), a first orientation (θ1, φ1, γ1), a second position (x2, y2, z2), and a second orientation (θ2, φ2, γ2). In a further example, each of x1 and x2 is a respective distance along the x-axis from a reference coordinate of the body of the HHC 104, each of y1 and y2 is a respective distance along the y-axis from the reference coordinate, each of z1 and z2 is a respective distance along the z-axis from the reference coordinate, θ1 is an angle between the x-axis and a first position, θ2 is an angle between the x-axis and a second position, φ1 is an angle between the y-axis and the first position, φ2 is an angle between the y-axis and the second position, γ1 is an angle between the z-axis and the first position, and γ2 is an angle between the z-axis and the second position. Further, in this example, the one or more processors determine a plurality of consecutive positions and orientations of one of the set of buttons of the HHC 104.

[0048] In this example, one or more processors of the gaming system identify each moved button of the HHC 104 and distinguish it from other buttons of the HHC 104 that have been moved or not moved. Also, in this example, one or more processors of the gaming system determine a sequence, such as the order in which the buttons of the HHC 104 are moved by the user 102 while engaging in activity A1. In this example, the multiple successive positions and orientations of one of the set of buttons of the HHC 104, the identity of each of the buttons of the HHC 104 that are moved, and the sequence are examples of state data 302.

[0049] Similarly, state data 302 relating to activities A2-A4 of game context 100 is obtained by one or more processors of the game system and stored in one or more memory devices of the game system. Similarly, state data relating to game context 116 (FIG. 1) is obtained by one or more processors of the game system and stored in one or more memory devices of the game system.

[0050] 4 is a diagram of an embodiment of a system 400 illustrating generation of a predictive indicator 426, such as a predicted outcome, based on state data 418 determined from a user's 102 engagement with the game context 100 and the game context 116. The system 400 includes a metadata processor 402, a context labeler 404, an action labeler 406, a metric labeler 408, a context classifier 410, an action classifier 412, a metric classifier 414, and a profile model 416. In one example, each of the context labeler 404, the action labeler 406, the metric labeler 408, the context classifier 410, the action classifier 412, the metric classifier 414, and the profile model 416 is a hardware or software component. For example, each of the context labeler 404, the action labeler 406, the metric labeler 408, the context classifier 410, the action classifier 412, the metric classifier 414, and the profile model 416 is a software program or part of software executed by an artificial intelligence (AI) processor. For further example, the profile model 416 is a machine learning model, or a neural network, or an artificial intelligence model. For another example, each of the context labeler 404, the action labeler 406, the metric labeler 408, the context classifier 410, the action classifier 412, the metric classifier 414, and the profile model 416 is a hardware circuit part of an ASIC or PLD. The AI ​​processor and metadata processor 402 are examples of one or more processors of a gaming system. The system 400 further includes state data 418, game context 420, user interactions 422, and performance metrics 424. Examples of state data 418 include a combination of state data 302 (FIG. 3) and state data generated based on the user's 102 engagement with the game context 116 .

[0051] The one or more processors of the game system collect the state data 418 on the fly. For example, the one or more processors of the game system do not pause a game program being executed to generate one of the game contexts 420 while obtaining the state data 418 from one or more of the user interactions 422 with one of the game contexts 420.

[0052] Examples of game contexts 420 include game context 100 and game context 116 (FIG. 1). Examples of user interactions 422 include the time it takes to move a button of the HHC 104, the time it takes to move each of the set of buttons of the HHC 104 between two of a plurality of successive positions and orientations of the buttons of the HHC 104, the number of button movements, the identity of the button moved, the sequence in which the buttons of the HHC 104 are moved, and a plurality of successive positions and orientations of each of the set of buttons of the HHC 104 during user interactions 422 with each of the game contexts 420. Additionally, examples of performance metrics 424 include a number of virtual achievements collected during each of the game contexts 420, or a value determined based on the skill level of the user 102 during each of the game contexts 420, or a combination thereof during each of the game contexts 420. For example, the one or more processors of the gaming system generate a first performance metric based on a weighted combination of the average skill level of the user 102 during activities A1-A4 of the game context 100 and a first number of virtual achievements achieved by the user 102 via user account 1 during the game context 100. In this example, the one or more processors of the gaming system generate a second performance metric based on a weighted combination of the expert skill level of the user 102 during one of the game contexts 116 and a second number of virtual achievements achieved by the user 102 via user account 1 during one of the game contexts 116. In this example, the second performance metric is a value that is greater than or less than the value of the first performance metric. Further, in this example, the one or more processors of the gaming system assign a first identifier PM1 to the first performance metric and assign a second identifier PM2 to the second performance metric. In this example, the first identifier and the second identifier PM1 and PM2 are part of the state data 418.

[0053] The one or more processors of the gaming system assign different identifiers to the game context 420 than to the user interactions 422 and the performance metrics 424. For example, the one or more processors of the gaming system assign identifiers to the game context 420, each of the identifiers having the term GC. Further, in this example, the one or more processors of the gaming system assign identifiers to the user interactions 422, each of the identifiers having the term UI. Also, in this example, the one or more processors of the gaming system assign identifiers to the performance metrics 424, each of the identifiers having the term PM. The identifiers for the game context 420, the user interactions 422, and the performance metrics 424 are part of the state data 418.

[0054] The one or more processors of the game system also assign different identifiers to each of the game contexts 420. For example, the game context 100 is assigned an identifier such as GC1, and one of the game contexts 116 is assigned an identifier such as GC2 to distinguish the game context 100 from one of the game contexts 116.

[0055] The one or more processors of the gaming system further assign different identifiers to each button of the HHC 104 and each different type of button movement. For example, the one or more processors of the gaming system assign identifier BT1 to a first button of the HHC 104 and identifier BT2 to a second button of the HHC 104. Furthermore, in this example, the one or more processors of the gaming system assign identifier 1BP to a first button press and identifier 2BPU to a second button press. As another example, the one or more processors of the gaming system assign identifier 2BPUDU to a button press up, another identifier 2BPUDD to a button press down, yet another identifier 2BPUDR to a button press to the right, and yet another identifier 2BPUDL to a button press to the left. In this example, up, down, right, and left movements are types of button movements. Identifiers of the buttons of the HHC 104 and identifiers of the types of button movements are part of the state data 418.

[0056] Additionally, the one or more processors of the gaming system provide correspondence identifiers between the buttons of the HHC 104, the types of button movement, and the game context 420. For example, the one or more processors of the gaming system assign a correspondence identifier CI1 to identify a unique relationship between an identifier of the game context 100, one or more identifiers of one or more buttons of the HHC 104 moved during the user's 102 engagement with the game context 100, and one or more identifiers of one or more movement types of the one or more buttons during engagement. The correspondence identifier is also part of the state data 418.

[0057] The metadata processor 402 is coupled to a context labeler 404, an action labeler 406, and a metric labeler 408. The context labeler 404 is also coupled to a context classifier 410, the action labeler 406 is coupled to an action classifier 412, and the metric labeler 408 is coupled to a metric classifier 414. The context classifier 410, the action classifier 412, and the metric classifier 414 are coupled to a profile model 416. The context labeler 404 is coupled to the action labeler 406 and the metric labeler 408.

[0058] The metadata processor 402 accesses state data 418 from one or more memory devices of the gaming system and analyzes the state data 418 to identify game context 420, user interactions 422, and performance metrics 424. For example, the metadata processor 402 reads each row of the state data 418 to distinguish between the game context 420, the user interactions 422, and the performance metrics 424. By way of example, the metadata processor 402 distinguishes between the game context 420, the user interactions 422, and the performance metrics 424 based on the identifiers GC, UI, and PM. By way of further example, the metadata processor 402 determines that the row of state data 418 having the identifier GC is one of the game contexts 420 and that the row of state data 418 having the identifier UI is one of the user interactions 422.

[0059] The context labeler 404 receives the game contexts 420 from the metadata processor 402 and identifies and labels each of the game contexts 420. For example, the context labeler 404 determines that the game context 100 includes a set of virtual objects that is different from the set of virtual objects of one of the game contexts 116. By way of example, the context labeler 404 identifies each virtual object in the game context based on the size, shape, color, or a combination thereof of the virtual objects, and further determines from the identifiers of the virtual objects in the game context that the game context 100 includes a set of virtual objects that is different from the set of virtual objects in one of the game contexts 116. By way of further example, the context labeler 404 identifies that the game context 100 includes a virtual character 106, a virtual character 108, a virtual giant 110, a virtual game chest 112, and a virtual valley 114. In a further example, the context labeler 404 identifies that one of the game contexts 116 includes a virtual avatar, a virtual desktop monitor, a virtual pencil holder, and a virtual desk. In a further example, the set of the virtual character 106, the virtual character 108, the virtual giant 110, the virtual game chest 112, and the virtual valley 114 is different from the set of the virtual avatar, the virtual desktop monitor, the virtual pencil holder, and the virtual desk. As another example, the context labeler 404 determines that a different identifier, such as GC1, is assigned to the game context 100 by one or more processors of the game system to distinguish the game context 100 from one of the game contexts 116, as distinct from an identifier, such as GC2, assigned to one of the game contexts 116. The respective labeling of the game contexts 420 distinguishes one of the game contexts 420 from another one of the game contexts 420. The context labeler 404 provides the labels of the game contexts 420 to the action labeler 406 and the metric labeler 408.

[0060] The action labeler 406 receives the user interactions 422 from the metadata processor 402 and the labels of the game contexts 420 from the content labeler 404, and identifies and labels each of the user interactions 422 during each of the game contexts 420. For example, the action labeler 406 distinguishes a button from the rest of the buttons of the HHC 104 based on an identifier of the button of the HHC 104, and distinguishes a type of movement of a button from another type of movement of a button of the HHC 104 during one of the game contexts 420 based on an identifier of the type of movement of the button.

[0061] The metric labeler 408 receives the performance metrics 424 from the metadata processor 402 and the labels of the game contexts 420 from the content labeler 404, identifies each of the performance metrics 424 during each of the game contexts 420, and labels each of the performance metrics 424. For example, the metric labeler 408 distinguishes a first performance metric from a second performance metric based on the identifier of the performance metric during one of the game contexts 420.

[0062] Each label is an identifier, i.e., a sequence of alphanumeric characters that distinguishes one label from another.

[0063] The context classifier 410 receives the labels of the game contexts 420 from the context labeler 404 and classifies each of the game contexts 420 to output a game context classification. For example, the context classifier 410 determines that the game context 100 includes a predetermined number of activities, such as A1 and A2, and each activity has an activity level that exceeds a predetermined level, such as 3.5. In this example, the context classifier 410 classifies the context 100 as a difficult context. In this example, the difficult context is one example of a context classification. As another example, the context classifier 410 determines that one of the game contexts 116 does not have a predetermined number of activities, and each activity has an activity level that exceeds a predetermined level. In this example, the context classifier 410 determines that one of the game contexts 116 is an easy context. In this example, the easy context is one example of a context classification.

[0064] Similarly, the action classifier 412 receives the labels of the user interactions 422 from the action labeler 406, classifies each of the user interactions 422, and outputs an action classification. For example, the action classifier 412 determines that the user 102 took longer than a predetermined time to engage in multiple user interactions 420 with multiple game contexts 420, such as pressing the X button of the HHC 104. In this example, after the virtual characters 106 and 108 (FIG. 1) are displayed in the multiple game contexts 420, one of the user interactions 420 occurs to control the virtual character 106 to start jumping to dodge a virtual bullet fired by the virtual character 108. Furthermore, in this example, the action classifier 412 classifies each of the multiple user interactions 422 as a difficult interaction. In this example, the number of game contexts 420 is greater than a preset number, and the number of user interactions 422 is equal to the number of game contexts 420. As another example, the action classifier 412 determines that the user 102 is engaged in multiple user interactions 422, such as pressing the square button used to reload a virtual gun instead of the R1 button used to shoot the virtual giant 110 (FIG. 1). In this example, the multiple user interactions 422 occur due to involvement in multiple game contexts 420, such as the game context 100 (FIG. 1). In this example, the action classifier 412 classifies each of the multiple user interactions 422 of pressing the square button as a difficult interaction. In this example, the number of game contexts 420 is greater than a preset number, and the number of user interactions 422 is equal to the number of game contexts 420. In this example, the R1 button for shooting is a predetermined button to be pressed to shoot. As yet another example, the action classifier 412 determines that the user 102 is engaged in multiple game contexts 420 through the multiple user interactions 422.In this example, an example of each of the plurality of user interactions 422 is selecting, such as pressing, a sequence of buttons on the HHC 104 that differs from a predetermined sequence of buttons on the HHC 104 in order to battle and defeat the virtual giant 110 in each of the plurality of game contexts 420. Further, in this example, the action classifier 412 classifies each of the plurality of user interactions 422 as a difficult interaction. In this example, the number of game contexts 420 is greater than the preset number, and the number of user interactions 422 is equal to the number of game contexts 420.

[0065] In the three preceding examples, the difficult interaction is an example of the action classification. As yet another example, the action classifier 412 determines that the time taken for the user 102 to press the X button button of the HHC 104 to start a jump after the virtual character 108 is displayed in the game context 100 (FIG. 1) is less than or equal to a predetermined time, and thus classifies one of the user interactions 422, pressing a button, as an easy interaction. As another example, the action classifier 412 determines that the user 102 presses the R1 button to shoot the virtual giant 110, and thus classifies one of the user interactions 422, pressing the predetermined button, R1 button, as an easy interaction. As yet another example, the action classifier 412 determines that the user 102 selects, such as pressing a predetermined sequence of buttons on the HHC 104 to fight the virtual giant 110, and thus classifies one of the user interactions 422, as an easy interaction. In the three preceding examples, the easy interaction is an example of the action classification.

[0066] Further, the metric classifier 414 receives the labels of the performance metrics 424 from the metric labeler 408 and classifies each of the performance metrics 424 to output a metric classification. For example, the metric classifier 414 determines that a first performance metric is above a predefined threshold to determine that the user 102 has a high performance metric while interacting with the activities A1-A4 of the game context 100. In this example, the metric classifier 414 determines that a second performance metric is below a predefined threshold to determine that the user 102 has a low performance metric while interacting with the activities A1-A4 of the game context 100. Further, in this example, the metric classifier 414 determines that the second performance metric is above a predefined threshold to determine that the user 102 has an average or low performance metric while interacting with the activities A1-A4 of the game context 100. In this example, the high, low, and average performance metrics are examples of metric classifications.

[0067] Each classification is a level. For example, a difficult context is a classification level and an easy context is another classification level. As another example, a difficult interaction is a classification level and an easy interaction is a classification level. As yet another example, a high performance metric is a classification level and an average or low performance metric is a classification level.

[0068] The action classifier 412 provides a label of the user interaction 422 to the profile model 416. For example, the action classifier 412 provides a label indicating which button of the HHC 104 was moved by the user 102, the position and orientation of a set of buttons, such as a joystick of the HHC 104, and a type of movement, such as a button press, a button advance, or a button press in a direction. The profile model 416 also receives a context classification from the context classifier 410, an action classification from the action classifier 412, and a metric classification from the metric classifier 414 to generate a predictive indicator 426. For example, the profile model 416 identifies from the metric classification that the metric classification is a high-performance metric in a number of game contexts 420 that are greater than a preset threshold. Furthermore, in this example, the profile model 416 identifies from the context classification received from the context classifier 410 that each of the multiple game contexts 420 is a difficult context. Further, in this example, the profile model 416 identifies interactions by the user 102 as easy interactions while engaged in the plurality of game contexts 420. In this example, the correspondence between high performance metrics, difficult contexts, and easy interactions are examples of relationships. In this example, the profile model 416 generates a favorable predictive indicator for the user 102 indicating that the user 102 will achieve high performance metrics in future game sessions in which game contexts similar to one or more of the plurality of game contexts 420 are displayed.

[0069] In one example, each of the predictive indicators 426 is an indicator of the gaming skill of the user 102. In another example, each of the predictive indicators 426 indicates whether the user 102 will achieve a goal of a game context similar to one or more of the plurality of game contexts 420. In an example, the goal is to complete one or more of the activities A1-A4 of the similar game context. In an example, if the goal is not achieved, the user 102 will not progress or advance in the game having the similar game context. In a further example, if the user 102 is unable to complete the activities A1-A4 of the similar game context of the game, the one or more processors of the game system may not display another game context of the game. In a further example, the other game contexts may follow the similar game contexts in succession according to the game. In another example, the goal is to earn a predetermined number of virtual points by interacting with the similar game context. In an example, if the goal is not achieved, the user 102 will not progress in the game having the similar game context. To further illustrate, if the user 102 fails to earn a predetermined number of virtual points in a similar game context, the one or more processors of the gaming system may not display another game context of the game.

[0070] As another example, the profile model 416 identifies from the metric classification that in a number of the game contexts 420 greater than a preset threshold, the metric classification is an average or low performance metric. Further, in this example, the profile model 416 identifies from the context classification received from the context classifier 410 that each of the plurality of game contexts 420 is a difficult or easy context. Further, in this example, the profile model 416 identifies that an interaction by the user 102 while engaged in the plurality of game contexts 420 is a difficult interaction. In this example, the correspondence between the average or low performance metric, the difficult or easy context, and the difficult or easy interaction are examples of relationships. In this example, the profile model 416 generates an unfavorable predictive indicator for the user 102 indicating that the user 102 achieves a low or average performance metric during a game session in which a game context similar to one or more of the plurality of game contexts 420 is displayed. In this example, the unfavorable predictive indicator indicates a lack of one or more game skills of the user 102. Illustratively, the unfavorable predictive indicator may indicate that user 102 will press a different sequence of buttons on HHC 104 than the predetermined sequence to achieve a goal in a future game context similar to one or more of the plurality of game contexts 420. As another example, the unfavorable predictive indicator may indicate that user 102 will press the joystick of HHC 104 in an upward direction rather than a downward direction in a future game context similar to one or more of the plurality of game contexts 420. In the illustrative example, the downward direction is the predetermined direction for achieving a goal in a game context similar to one or more of the plurality of game contexts 420.

[0071] As yet another example, the profile model 416 identifies from the metric classifications that in a number of game contexts 420 that are equal to or less than a pre-defined threshold, the metric classification is a high performance metric. In this example, the number of game contexts 420 is equal to the number of metric classifications. Further, in this example, the profile model 416 identifies from the context classifications received from the context classifier 410 that each of the plurality of game contexts 420 is a difficult context. Further, in this example, the profile model 416 identifies that a plurality of user interactions 422 by the user 102 while engaged in the plurality of game contexts 420 are easy interactions. In this example, the number of user interactions is equal to, less than, or more than the number of game contexts 420. In this example, the profile model 416 generates an unfavorable predictive indicator for the user 102 indicating that the user 102 will achieve a low performance metric during a game session in which a game context similar to one or more of the game contexts 420 is displayed.

[0072] As yet another example, the profile model 416 identifies from the metric classifications that a plurality of metric classifications are high performance metrics in a number of game contexts 420 that is equal to or less than a preset threshold. In this example, the number of game contexts 420 is greater than the preset number, and the number of metric classifications is equal to the number of game contexts 420. Furthermore, in this example, the number of metric classifications is equal to the number of performance metrics 424 instead of the number of game contexts 420. Furthermore, in this example, the profile model 416 identifies from the context classifications received from the context classifier 410 that each of the plurality of game contexts 420 is an easy context. Furthermore, in this example, the profile model 416 identifies that an interaction by the user 102 while engaged in the plurality of game contexts 420 is a difficult interaction. In this example, the profile model 416 generates an unfavorable predictive indicator for the user 102 indicating that the user 102 will achieve low performance metrics during a game session in which a game context similar to one or more of the game contexts 420 is displayed. It should be noted that generating the predictive index 426 based on the context classification received from the context classifier 410, the action classification received from the action classifier 412, and the metric classification received from the metric classifier 414 is an example of training the profile model 426.

[0073] In one embodiment, the metadata processor 402 does not access the state data 418 from one or more memory devices of the game system. Rather, in this example, the metadata processor 402 accesses the state data 418 directly from one or more processors of the game system that obtain the state data 418 during user interaction 422 with the game context 420.

[0074] In embodiments, the terms game context and selection context are used interchangeably herein.

[0075] In one embodiment, a high performance metric is generated by one or more processors of the gaming system when a predetermined number of activities of a preset number of game contexts 420 are completed by user 102 via user account 1 and HHC 104. In this embodiment, a low performance metric is generated by one or more processors of the gaming system when a predetermined number of activities of a preset number of game contexts 420 are not completed by user 102 via user account 1 and HHC 104.

[0076] In embodiments, difficult interactions may be referred to herein as inappropriate interactions, and easy interactions may be referred to herein as appropriate interactions.

[0077] 5 is a diagram of an embodiment of a system 500 illustrating the use of a gaming system 502 and an HHC 104 to provide assistance input data 506 to a user 102 via the HHC 104 based on a predictive indicator 426. The system 500 includes a gaming system 502, such as a computing device or a cloud system. The system 500 further includes a display device 514, which is an example of a display device of a client device.

[0078] The gaming system 502 includes state data 418, metadata processor 402, performance metrics 424, profile model 416, assistance data generator 504, assistance input data 506, game programs 510 for the games, current game play data 508, and assistance input features 512. In one example, the game having the game program 510 may be the same as or different from one or more of the games based on which the predictive indicators 426 are generated. An example of the assistance data generator 504 is one or more processors of the gaming system 502.

[0079] Examples of the auxiliary input functionality 512 are hardware or software. As an example, the auxiliary input functionality 512 is a software program executed by one or more processors of the gaming system 502. As another example, the auxiliary input functionality 512 is an ASIC, or a PLD, or another integrated circuit of the gaming system 502.

[0080] Similarly, examples of the assistance data generator 504 are hardware or software. As an example, the assistance data generator 504 is a software program executed by one or more processors of the gaming system 502. As another example, the assistance data generator 504 is an ASIC, or a PLD, or another integrated circuit of the gaming system 502.

[0081] The profile model 416 is coupled to an assist input function 512, which is coupled to the game program 510. The profile model 416 provides the predictive indicators 426 to the assist input function 512. For example, one or more processors of the game system 502 executing the assist input function 512 receive the predictive indicators 426 from the profile model 416. The assist data generator 504 is also coupled to the profile model 416 and the assist input function 512. The assist input function 512 and the game program 510 are coupled to the HHC 104. The assist data generator 504 is coupled to the game program 510.

[0082] During or after the occurrence of the game session for which the predictive indicators 426 are generated, the user 102 logs into user account 1 and uses the HHC 104 to generate and transmit a request to access a game having the game program 510. Once the user 102 logs into user account 1 and access to the game is provided by one or more processors of the game system 502, the one or more processors of the game system 502 transmit a request to a client device to display a prompt on the display device 514 to determine whether the user 102 wishes to use the assistive input features 512 while executing the game program 510. The user 102 uses the HHC 104 to indicate that the user 102 wishes to use the assistive input features 512, and the HHC 104 transmits the indication to the game system 502. Once the indication is received from the HHC 104, the game program 510, along with the assistive input features 512, are executed by the one or more processors of the game system 502 to initiate a current game session of the game on the display device 514. The assistive input function 512 receives the predictive indicators 426 from the profile model 416 and applies the predictive indicators 426. If the user 102 receives another indication that he does not want to apply the assistive input function 512, the game program 510 runs without applying the assistive input function 512.

[0083] During a current game session, as the game program 510 executes, one or more processors of the game system 502 generate current game play data 508. Illustratively, the current game play data 508 includes a current game context of a game having the game program 510, a plurality of user interactions of the user 102 with the current game context, and performance metrics determined by the one or more processors of the game system 502 based on the user interactions with the current game context.

[0084] During or prior to the current game session based on which the current game play data 508 is generated, the assistance data generator 504 accesses the profile model 416 to receive the predictive indicators 426 from the profile model 416. For example, the assistance data generator 504 requests the predictive indicators 426 from the profile model 416. In this example, the profile model 416 sends the predictive indicators 426 to the assistance data generator 504 in response to the request. The assistance data generator 504 then processes, e.g., analyzes, the predictive indicators 426 to generate or output the assistance input data 506. For example, the assistance data generator 504 receives unfavorable predictive indicators generated based on one or more of the user interactions 422 with one or more of the game contexts 420. In this example, the assistance input data 506 is generated to modify the unfavorable predictive indicators to favorable predictive indicators. In this example, the unfavorable predictive indicator indicates that the user 102 will not be able to complete a task in a game having a current game context, such as killing a virtual character, or achieving a predetermined number of virtual points, or crossing a virtual valley, or climbing a virtual mountain, or killing a virtual character 108, or killing a virtual giant 110, or a combination thereof.

[0085] In this example, the assistance data generator 504 communicates with the game program 510 to generate the assistance input data 506 to modify the unfavorable predictive indicators to favorable predictive indicators. To illustrate, the assistance data generator 504 identifies that the user 102 selected the square button instead of the R1 button to shoot one or more virtual objects, such as the virtual giant 110 (FIG. 1), during one or more of the pre-set amount of game contexts 420 based on the labels of the user interactions 422 received from the action classifier 412. In the example, the assistance data generator 504 communicates with the game program 510 to determine that the R1 button should be pressed instead of the square button to modify one or more unfavorable predictive indicators over one or more of the pre-set amount of game contexts 420 to one or more favorable predictive indicators. Further, in the example, the assistance data generator 504 generates the assistance input data 506 to indicate, e.g., suggest, to the user 102 that the user 102 selects the R1 button instead of the square button during or immediately prior to the current game context. In the example, the selection of the R1 button is an example of an input type. In the illustrated example, the assistance input data 506 is light data for highlighting the R1 button or press input data for controlling the pressing of the R1 button. In the illustrated example, the press input data is an example of controller input data.

[0086] As another example, the assistance data generator 504 identifies, based on the labels of the user interactions 422 received from the action classifier 412, that the user 102 operates the HHC 104 to select a set of buttons of the HHC 104 according to a first sequence or a first sequential order. In the example, the assistance data generator 504 communicates with the game program 510 to determine that a set of buttons of the HHC 104 is selected according to a second sequence or a second sequential order to modify one or more unfavorable predictive indicators over one or more of the pre-set amount of game contexts 420 to one or more favorable predictive indicators. In the example, the one or more unfavorable predictive indicators indicate that the user 102 is unable to complete a task in a game having a current game context. In the example, the assistance data generator 504 generates assistance input data 506 during or immediately prior to the current game context to indicate, such as suggest, to the user 102 that the user 102 selects a set of buttons according to a second sequence. In the example, the selection of a set of buttons according to the second sequence is an example of an input type. In the illustrated example, the assistance input data 506 is light data for highlighting a set of buttons in the order of the second sequence, or press input data for depressing a set of buttons according to the second sequence. In the illustrated example, the press input data is an example of controller input data.

[0087] As another example, upon receiving a preset amount of favorable predictive indicators 426 for the game context 420 , the assistance data generator 504 does not output assistance input data 506 for providing to the assistance input functionality 512 .

[0088] During the current game session, the assistance data generator 504 provides the assistance input data 506 to the assistance input functionality 512. The assistance input functionality 512 is executed by one or more processors of the game system 502 to receive the assistance input data 506 and apply the assistance input data 506 to modify user interactions during execution of the game program 510 for game play in the current game session.

[0089] While the user 102 is playing the game based on which the current gameplay data 508 is generated, the assistive input function 512 analyzes, e.g., processes, the predictive indicators 426 and the current gameplay data 508 to determine whether to transmit the assistive input data 506 to the HHC 104. For example, the assistive input function 512 receives the unfavorable predictive indicators generated based on one or more of the user interactions 422 with one or more of the game contexts 420. In this example, the assistive input function 512 obtains the gameplay data 508 from the game program 510, identifies, e.g., detects from the gameplay data 508 a current game context with which the user 102 is interacting, and determines whether the current game context is similar to the preset amount of the game context 420 based on which the unfavorable predictive indicator is generated. By way of example, the assistive input function 512 determines whether a preset amount of virtual objects in the current game context matches a preset amount of virtual objects in each of the preset amounts of the game contexts 420 based on which the unfavorable predictive indicator is generated. In the example, the predetermined amount of virtual objects are identified based on the size, shape, and color of the virtual objects. Further, in the example, if the assistive input function 512 determines that the predetermined amount of virtual objects in the current game context match the predetermined amount of virtual objects in the predetermined amount of game context 420, the assistive input function 512 determines that the current game context is similar to the predetermined amount of game context 420 and determines to send the assistive input data 506 to the HHC 104. On the other hand, in the example, if the assistive input function 512 determines that the predetermined amount of virtual objects in the current game context do not match the predetermined amount of virtual objects in the predetermined amount of game context 420, the assistive input function 512 determines that the current game context is not similar to the predetermined amount of game context 420 and does not send the assistive input data 506 for the current game context to the HHC 104.

[0090] Continuing with the example, when the assistance input function 512 determines that the current game context is similar to the preset amount of the game context 420 based on which the predictive indicator 426 is generated, the assistance input function 512 activates an assistance cue, such as sending the assistance input data 506 to the HHC 104. In this example, the assistance input data 506 is sent to modify a potentially unfavorable predictive indicator to a favorable predictive indicator for the current game context. To illustrate, the assistance data generator 504 identifies, based on the label of the user interaction 422 received from the action classifier 412, that the user 102 selected the square button instead of the R1 button during the preset amount of the game context 420 based on which the unfavorable predictive indicator is generated. In the example, the assistance data generator 504 provides the assistance input data 506 to the assistance input function 512. In an example, before the user 102 attempts to interact with the current game context, such as performing a task of fighting a virtual character 108 or a virtual giant 110 (FIG. 1), the assistive input function 512 transmits assistive input data 506 to the HHC 104 to indicate to the user 102 that the user 102 selected the R1 button instead of the square button. In an example, the assistive input data 506 is light data that causes the R1 button to light up, or press input data that controls the R1 button to be automatically pressed, or a combination thereof. In an example, the assistive input data 506 is transmitted after the current game context is generated but before the user 102 interacts with the current game context. In an example, the assistive input data 506 is transmitted from the game system 502 to the HHC 104 via a computer network if the game system 502 is a cloud system. Furthermore, in an example, the assistive input data 506 is transmitted from the game system 502 to the HHC 104 via a wired or wireless communication medium if the game system 502 is a computing device. An example of a wired communication medium is a cable, and an example of a wireless communication medium is a medium that applies a wireless protocol for communication, such as Bluetooth™ or Wi-Fi™.

[0091] As another example, the assistance data generator 504 identifies that the user 102 selects the square button on the HHC 104 during one or more of the game contexts 420 that are similar to the current game context based on the labels of the user interactions 422 received from the action classifier 412. In the example, the assistance data generator 504 identifies that one of the predictive indicators 426 corresponding to the selection of the square button is unfavorable. In the example, the assistance data generator 504 provides the assistance input data 506 to the assistance input function 512. Further, in the example, before the user 102 attempts to interact with the current game context, the assistance input function 512 sends the assistance input data 506 to the HHC 104 to indicate to the user 102 that the user 102 selected the R1 button.

[0092] As yet another example, the assistance data generator 504 identifies, based on the labels of the user interactions 422 received from the action classifier 412, that the user 102 manipulated the HHC 104 to select a set of buttons on the HHC 104 according to a first sequence or a first sequential order during one or more of the game contexts 420 that are similar to the current game context. In the example, the assistance data generator 504 identifies that one of the predictive indicators 426 corresponding to the selection of the set of buttons according to the first sequence is unfavorable. In the example, the assistance data generator 504 provides the assistance input data 506 to the assistance input function 512. In the example, the assistance input function 512 transmits the assistance input data 506 to the HHC 104 to indicate, e.g., indicate to the user 102 that the user 102 selected a set of buttons according to a second sequence before the user 102 attempts to interact with the current game context. When the assistance input data 506 is transmitted to the HHC 104 , predictive gaming assistance is provided to the user 102 via user account 1 and the HHC 104 .

[0093] At any time while a game context, such as the current game context or the next game context, is displayed on the display device 514, the one or more processors of the gaming system 502 determine whether the assistive input function 512 is suspended. For example, the predictive indicators 426 are modified or additional predictive indicators are generated based on the user's 102 interaction with one or more game contexts, such as the current game context and the next game context. In this example, the user's 102 interaction occurs after the assistive input function 512 is applied to modify one or more of the predictive indicators 426 from unfavorable to favorable. Further, in this example, the assistive input function 512 determines whether at least a predetermined number of the predictive indicators 426 are modified from unfavorable to favorable based on the user interaction with the one or more game contexts. In this example, upon such determination, the one or more processors deactivate the assistive input function 512 such that the assistive input function 512 is not applied to the gaming program 510 during a game context, such as the current game context or the next game context. In one example, the next game context occurs during a next game session that occurs after the current game session in which the current game play data 508 is generated. As another example, the one or more processors determine whether a predetermined number of virtual points are achieved during the one or more game contexts, or a predetermined result is achieved during the one or more game contexts, or a predetermined number of game tasks are achieved during the one or more game contexts, or a combination thereof. In this example, upon such a determination, the one or more processors deactivate the assistive input functionality 512.

[0094] In an embodiment, the user 102 uses the HHC 104 to deactivate the assistive input functionality 512 at any point in a game context, such as the current game context or the next game context.

[0095] In one embodiment, the auxiliary input functionality 512 is integrated with the game program 510. For example, one or more processors of the game system 502 execute a program including the auxiliary input functionality 512 and the game program 510 in a time-shared manner.

[0096] In an embodiment, two or more of the assistance data generator 504, the assistance input data 506, the assistance input functionality 512, and the game program 510 are integrated with one another. For example, one or more processors of the game system 502 execute programs including the assistance data generator 504, the assistance input functionality 512, and the game program 510 in a time-shared manner.

[0097] In one embodiment, the one or more processors of the gaming system 502 learn the gaming style of the user 102 during a training or calibration session of the game prior to the current gaming context to determine one or more of the predictive indicators 426, and apply one or more of the predictive indicators 426 to control, such as pressing or highlighting, one or more buttons of the HHC 104 for the user 102 during the current gaming context. Further, in this embodiment, in addition to or as an alternative to controlling one or more buttons of the HHC 104, the one or more processors of the gaming system 502 control the gaming program 510 based on the one or more predictive indicators 426 generated based on the training or calibration session. For example, the one or more processors of the gaming system 502 perform one or more tests, such as jumping over a virtual object, that inform the user 102 to quickly accomplish a task in the training or calibration session to determine the gaming style of the user 102. In this example, the one or more processors of the gaming system 502 store the gaming style in a user profile of the user 102 to build a profile model 416, and use the gaming style to adjust the game of the user 102. For example, the one or more processors of the game system 502 modify one or more portions of the current game context to adjust the game program 510 downward before the current game context is displayed on the display device 514. In this example, the one or more portions of the current game context are modified to output one or more modified current game contexts, thereby enabling the user 102 to increase a performance metric while interacting with the current game context. Illustratively, the one or more processors of the game 502 modify the current game context to delete an activity, such as activity A2, or modify a function of one or more virtual objects, such as modifying a function of the virtual giant 112 (FIG. 1), to attack the virtual character 106 (FIG. 1) less than a predetermined number of times, or to replace the activity with another easier activity.In a further example, activity has an activity level above a predetermined level, such as 3.5, and easy activity has an activity level below the predetermined level. As another example, one or more processors of game 502 modify the current game context to slow down the speed at which one or more virtual objects move within the current game context.

[0098] In one embodiment, one or more processors of the gaming system 502 determine a mood or enthusiasm of the user 102, such as a cool mood or an intense mood, and adjust the mode of gameplay of the game based on the mood or enthusiasm. For example, the mode is adjusted to an easy mode or a hard mode based on the mood or enthusiasm expressed by the user 102. Illustratively, the one or more processors of the gaming system 502 receive audio data via one or more microphones of the HHC 104 based on sounds made by the user 102 during a calibration or training session. Illustratively, the one or more processors of the gaming system 502 compare the audio data to predetermined audio data to determine whether the user 102 is agitated or relaxed. Illustratively, upon determining that the user 102 is agitated, the one or more processors of the gaming system 502 adjust the game downward to display the current game context in the manner described above. Further, in the example, upon determining that the user 102 is relaxed, the one or more processors of the game system 502 do not adjust the game downward or adjust the game upward to display the current game context. By way of further example, the one or more processors of the game system 502 modify one or more portions of the current game context to adjust the game program 510 upward before the current game context is displayed on the display device 514. In a further example, the one or more processors of the game 502 modify the current game context to output one or more modified current game contexts, thereby challenging the user 102 during interaction with the current game context. In a further example, the one or more processors of the game 502 modify the current game context to add an activity to the activities A1-A4 or modify a function of one or more virtual objects, such as modifying a function of the virtual giant 112 (FIG. 1), thereby attacking the virtual character 106 (FIG. 1) more times than a predetermined number of times or replacing one or more of the activities A1-A4 with a more difficult activity.In an example, a more difficult activity has an activity level greater than 8. As another further example, one or more processors of the game 502 modify the current game context to challenge the user 102 by increasing the speed at which one or more virtual objects move within the current game context.

[0099] In an embodiment, the one or more processors of the gaming system 502 adjust the game downward if they determine, based on the user profile, that the user 102 is a beginner player. In this embodiment, the one or more processors of the gaming system 502 adjust the game upward if they determine, based on the user profile, that the user 102 is an experienced player. Further, in this embodiment, the one or more processors of the gaming system 502 do not adjust the game upward or downward if they determine, based on the user profile, that the user 102 is an intermediate player. In this embodiment, an intermediate player has a level intermediate between an experienced player and a beginner player.

[0100] FIG. 6 is a diagram of an embodiment of a system 600 illustrating that assistance input data 506 is generated and transmitted to multiple different areas of the HHC 104. The system 600 includes a display device 602 and an HHC 104. The display device 602 is an example of the display device 514 (FIG. 5). Examples of the display device 602 include a display device of a desktop computer, a display device of a laptop computer, a display device of an HMD, a display device of a smart television, and a display device of a smartphone. The HHC 104 also includes a display device 616 having a display screen. The HHC 104 further includes a processor and a graphics processing unit (GPU) of the display device of the HHC 104. The processor of the HHC 104 is coupled to the GPU of the HHC 104, and the GPU is coupled to the display screen of the HHC 104.

[0101] The HHC 104 includes a left joystick 602 and a right joystick 604. The HHC 104 further includes a directional pad 606 including directional buttons such as an up button, a down button, a right button, and a left button. The HHC 104 also includes an X button, an O button, a triangle button, and a square button. The HHC 104 has an L1 button and an R1 button. Each of the left joystick 602 and the right joystick 604 is an example of a button of the HHC 104. Additionally, each button of the HHC 104 is an example of a component of the HHC 104, and each component of the HHC 104 is an example of a region of the HHC 104. The HHC 104 includes a body 608, such as a housing, from which the left joystick 602 and the right joystick 604 extend vertically upward.

[0102] The processor of the HHC 104 is coupled to a respective light system associated with each component of the HHC 104 via a respective driver. For example, the processor of the HHC 104 is coupled to a first light system associated with an X button of the HHC 104 via a first driver, and is coupled to a second light system associated with an O button of the HHC 104 via a second driver. In this example, the processor of the HHC 104 is coupled to a third light system associated with a left joystick 602 via a third driver, and is coupled to a fourth light system associated with a right joystick 604 via a fourth driver. Further, in this example, the processor of the HHC 104 is coupled to a fifth light system associated with an L1 button via a fifth driver, and is coupled to a sixth light system associated with an R1 button via a sixth driver. Examples of drivers include one or more transistors coupled to each other. Examples of light systems include one or more light emitters, such as light emitting diodes (LEDs), or light sources, or LED strips, or strings of LED lights, or laser lights, or light bulbs. Each light emitter emits visible light that is visible to a user.

[0103] In one example, a light system is associated with a button of the HHC 104 where light emitted from the light system highlights the button to the user 102 to motivate the user 102 to press the button. In one example, the X button is manufactured from a transparent or translucent material such as plastic, and the first light system is attached to the bottom surface of the X button. In one example, the first light system includes one or more LED lights attached to the bottom surface of the X button and disposed below the top surface of the transparent or translucent material. In another example, the third light system includes one or more LED lights surrounding the head or body of the left joystick 602 to highlight the left joystick 602. Further, in one example, the third light system is disposed within an enclosure of the body 608 of the HHC 104 and forms the shape of a circle having a diameter larger than that of the left joystick 602. In one example, the left joystick 602 has its body located below the head and integrated at the joint with the head.

[0104] Also, the processor of the HHC 104 is coupled to each of the buttons of the HHC 104 via a respective driver and a respective drive system. For example, the processor of the HHC 104 is coupled to the X button of the HHC 104 via a primary driver and a primary drive system, and is coupled to the O button of the HHC 104 via a secondary driver and a secondary drive system. In this example, the processor of the HHC 104 is coupled to the L1 button of the HHC 104 via a tertiary driver and a tertiary drive system, and is coupled to the R1 button of the HHC 104 via a quaternary driver and a quaternary drive system. An example of a drive system includes a coil and a magnet. In this example, the coil is proximate to the magnet at one end and proximate to the driver at the opposite end. In this example, the coil is proximate to the magnet when a magnetic field induced by a current flowing through the coil interferes with the magnetic field of the magnet. Furthermore, in this example, the magnet is proximate to the button of the HHC 104 to control, such as pressing the button. Illustratively, a magnet is attached or fixed to the underside of the button to control the button.

[0105] The processor of the HHC 104 is also coupled to the speaker of the HHC 104 via an audio driver and an audio driving system. For example, the processor of the HHC 104 is coupled to a first speaker of the HHC 104 via a first audio driver and a first audio driving system, and is coupled to a second speaker of the HHC 104 via a second audio driver and a second audio driver system. An example of an audio driver includes one or more transistors. An example of an audio driving system includes an arrangement of one or more coils and one or more magnets. An example of a speaker includes a diaphragm. In this example, one or more magnets of the speaker are proximate to the diaphragm of the speaker.

[0106] The processor of the HHC 104 receives the assist input data 506 (FIG. 5) from the game system 502 (FIG. 5). The assist input data 506 is received by the processor of the HHC 104 before the user 102 attempts to interact with the game context 610. The assist input data 506 instructs the processor of the HHC 104 to highlight the X button to indicate to the user 102 to press the X button. Upon receiving the assist input data 506 indicating the X button is pressed, the processor of the HHC 104 sends a control signal to the first driver. Upon receiving the control signal, the first driver generates a current signal and sends the current signal to the first light system. Upon receiving the current signal, the first light system associated with the X button is illuminated to highlight the X button. When the user 102 makes a selection, such as by pressing the highlighted X button, the virtual character 612 is controlled by the user 102 to jump to prevent the virtual character 612 from being shot by another virtual character 614 of the game context 610.

[0107] In addition to or as an alternative to highlighting the X button, the assistive input data 506 includes notification data, such as screen text data, for displaying a notification 618, such as "Press X," to motivate the user 102 to press the X button and interact with the game context 610. The assistive input data 506 instructs the processor of the HHC 104 to control the display device 602 to generate the notification 618 instructing the user 102 to press the X button. Upon receiving the assistive input data 506, the processor of the HHC 104 sends the assistive input data 506 to the GPU of the HHC 104. Upon receiving the assistive input data 506, the GPU controls the display screen of the HHC 104 to display the notification 618 on the display screen of the display device 602.

[0108] In addition to or as an alternative to displaying the notification 618 and in addition to or as an alternative to highlighting the X button, the assistance input data 506 includes audio data for generating a sound, such as "press X," to motivate the user 102 to press the X button within the game context 610. The assistance input data 506 instructs the processor of the HHC 104 to control a speaker of the HHC 104 to generate a sound instructing the user 102 to press the X button. Upon receiving the assistance input data 506, the processor of the HHC 104 generates a control signal and sends the control signal to an audio driver of the HHC 104. Upon receiving the control signal, the audio driver outputs a current signal to an audio driving system. The audio driving system generates a magnetic field to control a diaphragm of the speaker, thereby outputting a sound, such as "press X," to motivate the user 102 to press the X button.

[0109] In addition to or as an alternative to displaying the notification 618 and in addition to or as an alternative to highlighting the X button, the assistance input data 506 instructs the processor of the HHC 104 to control the X button to automatically select, such as by pressing the X button. Upon receiving the assistance input data 506 indicating pressing the X button, the processor of the HHC 104 sends a control signal to the primary driver. Upon receiving the control signal, the primary driver generates a current signal and sends the current signal to a coil of the primary drive system. Upon receiving the current signal, the coil of the primary drive system generates a magnetic field. The magnetic field interacts with a magnetic field of a magnet of the primary drive system to control, such as by pressing the X button downward toward the body 608.

[0110] Similarly, the assistive input data 506 is received by the HHC 104 before the user 102 attempts to interact with the game context 620. The assistive input data 506 instructs the processor of the HHC 104 to move the left joystick 602. Selection of the left joystick 602 helps the user 102 advance the game context 620. For example, selection of the left joystick 602 increases the number of virtual points earned through user account 1. As another example, selection of the left joystick 602 helps the virtual character 612 stay alive in the game context 620 and causes the virtual character 616 to advance the game context 622. Upon receiving the assistive input data 506 instructing the user 102 to select the left joystick 602, the processor of the HHC 104 sends a control signal to the third driver. Upon receiving the control signal, the third driver generates a current signal and sends the current signal to the fourth light system. Upon receiving the current signal, a third light system coupled to the left joystick 602 is turned on and emits a light to highlight the left joystick 602. When the user 102 moves the left joystick 602, the virtual character 612 is controlled by the user 102 to move left to prevent the virtual character 612 from being shot by the virtual character 614 of the game context 616.

[0111] In addition to or as an alternative to highlighting the left joystick 602, the assistive input data 506 includes notification data, such as screen text data, to display a notification 623, such as “left stick,” to motivate the user 102 to move the left joystick 602 and interact with the game context 620. The assistive input data 506 instructs the processor of the HHC 104 to control the display device 602 to generate the notification 623 instructing the user 102 to push the left joystick 602. Upon receiving the assistive input data 506, the processor of the HHC 104 sends the assistive input data 506 to the GPU of the HHC 104. Upon receiving the assistive input data 506, the GPU controls the display screen of the HHC 104 to display the notification 623 on the display screen.

[0112] Also, the assistive input data 506 is received by the HHC 104 before the user 102 attempts to interact with the game context 622. The assistive input data 506 instructs the processor of the HHC 104 to highlight the R1 button to indicate, e.g., suggest, the user 102 to select, e.g., press, the R1 button. Upon receiving the assistive input data 506 indicating the selection of the R1 button, the processor of the HHC 104 sends a control signal to the sixth driver. Upon receiving the control signal, the sixth driver generates a current signal and sends the current signal to the sixth light system. Upon receiving the current signal, the sixth light system coupled to the R1 button is turned on, e.g., receives power from a voltage source, to illuminate the R1 button. When the user 102 selects the R1 button, the virtual character 612 is controlled by the user 102 to fire a virtual bullet at the virtual character 614 to kill the virtual character 615 of the game context 622.

[0113] In addition to or as an alternative to highlighting the R1 button, the assistive input data 506 includes notification data, such as screen text data, for displaying a notification 624, such as "press R1," to motivate the user 102 to press the R1 button within the game context 622. The assistive input data 506 instructs the processor of the HHC 104 to control the display device 602 to generate a notification 618 instructing the user 102 to press the X button. Upon receiving the assistive input data 506, the processor of the HHC 104 sends the assistive input data 506 to the GPU of the HHC 104. Upon receiving the assistive input data 506, the GPU controls the display screen of the HHC 104 to display the notification 624 on the display screen.

[0114] In addition to or instead of displaying the notification 624, the assistance input data 506 includes audio data for generating a sound, such as "press R1," to motivate the user 102 to press the R1 button to interact with the game context 610. The assistance input data 506 instructs the processor of the HHC 104 to control a speaker of the HHC 104 to generate a sound instructing the user 102 to press the R1 button. Upon receiving the assistance input data 506, the processor of the HHC 104 generates a control signal and sends the control signal to an audio driver of the HHC 104. Upon receiving the control signal, the audio driver outputs a current signal to an audio driving system. The audio driving system generates a magnetic field to control a diaphragm of the speaker, thereby outputting a sound, such as "press R1," to motivate the user 102 to press the R1 button.

[0115] In addition to or as an alternative to highlighting the R1 button, and in addition to or as an alternative to displaying a notification 624, and in addition to or as an alternative to outputting a sound to indicate to the user 102 to press the R1 button, the assistance input data 506 instructs the processor of the HHC 104 to control the R1 button to automatically select, such as by pressing the R1 button. Upon receiving the assistance input data 506 indicating that the R1 button is to be pressed, the processor of the HHC 104 sends a control signal to the quaternary driver. Upon receiving the control signal, the quaternary driver generates a current signal and sends the current signal to a coil of the quaternary drive system. Upon receiving the current signal, the coil of the quaternary drive system generates a magnetic field. The magnetic field interacts with a magnetic field of a magnet of the quaternary drive system to control, such as by pressing the R1 button downward toward the body 608 of the HHC 104.

[0116] It should be noted that state data generated from a current game context, such as one of game contexts 610, 620, and 624, user interactions during the current game context, and performance metrics generated based on user interactions with the current game context become part of state data 418 (FIG. 4) to dynamically update, such as constantly or continuously, profile model 416. For example, metadata processor 402 analyzes state data 418 to distinguish game contexts 610, 620, and 624 from user interactions with game contexts 610, 620, and 624, and from performance metrics achieved by the user interactions. Game contexts 610, 620, and 624, user interactions with the game contexts, and performance metrics achieved by the user interactions are then labeled by labelers 404-408 and then classified by classifiers 410-414 to provide classifications to profile model 416 (FIG. 4). The profile model 416 generates additional predictive indices to add to or update the predictive indices 426 based on the classification and one or more of the labels.

[0117] In one embodiment, the HMD is controlled in the same manner as the HHC 104 based on the assistance input data 506. For example, a particular area may be illuminated, such as illuminating the right side of the HMD, or illuminating the left side of the HMD, or illuminating the top side of the HMD, or illuminating the bottom side of the HMD, based on the assistance input data 506. In this example, illumination of one of the four sides instructs the user 102 to look in the direction of one of the four sides to interact with the game context displayed on the HMD.

[0118] In an embodiment, in addition to or instead of providing feedback to the user 102 via a light, or a press input, or a notification, or a combination thereof, haptic feedback is provided to the user 102 by providing haptic feedback data to one or more buttons of the HHC 104.

[0119] In one embodiment, the embodiments described herein apply to one or more games, however, the embodiments apply to multimedia contexts of one or more interactive spaces, such as the Metaverse, as well.

[0120] FIG. 7 illustrates components of an exemplary device 700 that can be used to implement aspects of various embodiments of the present disclosure. The block diagram illustrates device 700, which can incorporate or be a personal computer, video game console, personal digital assistant, server, or other digital device suitable for implementing embodiments of the present disclosure. Device 700 includes a central processing unit (CPU) 702 for executing software applications and optionally an operating system. CPU 702 includes one or more homogeneous or heterogeneous processing cores. For example, CPU 702 is one or more general-purpose microprocessors having one or more processing cores. Further embodiments can be implemented using one or more CPUs with a microprocessor architecture that is particularly adapted for highly parallel and computationally intensive applications, such as interpreting queries, identifying contextually relevant resources, and immediately implementing and rendering contextually relevant resources within a video game. Device 700 can be one of many servers that are localized to a player playing a game segment (e.g., a game console) or that are remote from the player (e.g., a back-end server processor), or that use virtualization in a game cloud system for remote streaming of game play to clients.

[0121] Memory 704 stores applications and data used by CPU 702. Storage 706 provides non-volatile storage and other computer-readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, compact disc ROM (CD-ROM), digital versatile disc ROM (DVD-ROM), Blu-ray, high definition DVD (HD-DVD), or other optical storage devices, as well as signal transmission and storage media. User input devices 708 communicate user input from one or more users to device 700. Examples of user input devices 708 include a keyboard, mouse, joystick, touchpad, touch screen, still or video recorder / camera, gesture-recognizing tracking device, and / or microphone. Network interface 714 enables device 700 to communicate with other computer systems over an electronic communications network, which may include wired or wireless communications over local area networks and wide area networks such as the Internet. The audio processor 712 is adapted to generate analog or digital audio output from instructions and / or data provided by the CPU 702, memory 704, and / or data storage 706. The components of the device 700 including the CPU 702, memory 704, data storage 706, user input device 708, network interface 710, and audio processor 712 are connected via a data bus 722.

[0122] A graphics subsystem 720 is further coupled to a data bus 722 and the components of the device 700. The graphics subsystem 720 includes a graphics processing unit (GPU) 716 and a graphics memory 718. The graphics memory 718 includes a display memory (e.g., a frame buffer) used to store pixel data for each pixel of an output image. The graphics memory 718 may be integrated into the same device as the GPU 716, may be coupled as a separate device from the GPU 716, and / or may be implemented within the memory 704. The pixel data may be provided directly to the graphics memory 718 from the CPU 702. Alternatively, the CPU 702 provides data and / or instructions defining a desired output image to the GPU 716, which generates therefrom the pixel data for one or more output images. The data and / or instructions defining the desired output image may be stored in the memory 704 and / or the graphics memory 718. In one embodiment, GPU 716 includes three-dimensional (3D) rendering capabilities that generate pixel data for output images from instructions and data that define the geometry, lighting, shading, texturing, motion, and / or camera parameters for a scene. GPU 716 may further include one or more programmable execution units capable of executing shader programs.

[0123] Graphics subsystem 714 periodically outputs pixel data of an image from graphics memory 718 for display on display device 710. Display device 710 may be any device capable of displaying visual information in response to signals from device 700, including a cathode ray tube (CRT) display, a liquid crystal display (LCD), a plasma display, and an organic light emitting diode (OLED) display. Device 700 may provide analog or digital signals to display device 710, for example.

[0124] It should be noted that access services distributed over a wide geography, such as providing access to games in the current embodiment, often use cloud computing. Cloud computing is a computing paradigm in which dynamically scalable, often virtualized resources are provided as a service over the Internet. Users do not need to be experts in the technical infrastructure of the "cloud" that supports them. Cloud computing can be categorized into different services such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services often provide common applications, such as video games, online, accessed through a web browser, but the software and data are stored on servers in the cloud. The term cloud is used as a metaphor for the Internet based on how the Internet is depicted in computer network diagrams, and is an abstract concept that hides a complex infrastructure.

[0125] A game server may be used in some embodiments to perform the operation of a persistent information platform for video game players. Most video games played on the Internet operate through a connection to a game server. Typically, the game uses a dedicated server application that collects data from the player and distributes the collected data to other players. In other embodiments, the video game may be executed by a distributed game engine. In these embodiments, the distributed game engine may run on multiple processing entities (PEs), such that each PE executes a functional segment of the given game engine on which the video game is executed. Each processing entity is viewed by the game engine as simply a computational node. A game engine typically performs a functionally diverse set of operations to execute a video game application along with additional services experienced by the user. For example, the game engine implements game logic and performs game calculations, physics, geometry transformations, rendering, lighting, shading, audio, and additional in-game or game-related services. The additional services may include, for example, messaging, social utilities, audio communication, game play playback functions, help functions, and the like. The game engine may run on an operating system virtualized by a hypervisor on a particular server, but in other embodiments the game engine itself may be distributed across multiple processing entities, each of which may reside on a different server unit in a data center.

[0126] According to this embodiment, each processing entity for the execution of operations may be a server unit, a virtual machine, or a container, depending on the needs of each game engine segment. For example, if a game engine segment is responsible for camera transformation, since it will be performing a lot of relatively simple mathematical operations (e.g., matrix transformations), that particular game engine segment may be provisioned with a virtual machine associated with a GPU. Other game engine segments, requiring fewer but more complex operations, may be provisioned with processing entities associated with one or more higher-powered CPUs.

[0127] By distributing the game engine, the game engine has elastic computational characteristics that are not bound by the capabilities of a physical server unit. Instead, the game engine is provisioned with more or fewer computation nodes as needed to meet the demands of the video game. From the perspective of the video game and the video game player, a game engine that is distributed across multiple computation nodes is indistinguishable from a non-distributed game engine running on a single processing entity, since a game engine manager or supervisor distributes the workload and seamlessly integrates the results to provide the video game output component to the end user.

[0128] A user accesses a remote service using a client device, which includes at least a CPU, a display, and an input / output (I / O) interface. The client device can be a personal computer (PC), a mobile phone, a netbook, a personal digital assistant (PDA), etc. In one embodiment, a network running on the game server recognizes the type of device used by the client and adjusts the communication method to employ. In another case, the client device accesses an application on the game server over the Internet using a standard communication method such as HTML. It should be understood that a given video game or game application may be developed for a particular platform and a particular associated controller device. However, when making such games available through a game cloud system as presented herein, users may access the video game with different controller devices. For example, a game may have been developed for a game console and its associated controller, but a user may access a cloud-based version of the game from a personal computer utilizing a keyboard and mouse. In such a scenario, the input parameter configuration may define a mapping from inputs that can be generated by the user's available controller device (in this case, a keyboard and mouse) to inputs that are acceptable for execution of the video game.

[0129] In another example, a user may access the cloud gaming system via a tablet computing device, a touchscreen smartphone, or other touchscreen driven device. In this case, the client device and the controller device are integrated together in the same device, and input is provided by detected touchscreen input / gestures. For such devices, the input parameter configuration may define a particular touchscreen input that corresponds to game input for the video game. For example, a button, a directional pad, or other type of input element may be displayed or overlaid during the execution of the video game to indicate locations on the touchscreen that the user may touch to generate game input. Gestures such as swipes in a particular orientation, or particular touch motions may also be detected as game input. In one embodiment, to familiarize the user with control operations on the touchscreen, a tutorial may be provided to the user showing how to input gameplay via the touchscreen, for example, before beginning gameplay of the video game.

[0130] In some embodiments, the client device serves as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection and transmits inputs from the controller device to the client device. The client device may then process these inputs and then transmit the input data to the cloud gaming server over a network (e.g., a network accessed via a local network device such as a router). However, in other embodiments, the controller itself may be a networked device that has the ability to communicate inputs directly to the cloud gaming server over a network, without the need for such inputs to be communicated through the client device first. For example, the controller may connect to a local network device (such as the aforementioned router) to send and receive data from the cloud gaming server. Thus, input latency may be reduced by allowing the controller to send inputs directly over the network to the cloud gaming server, bypassing the client device, while the client device may still be required to receive video output from the cloud-based video game and render it on a local display.

[0131] In one embodiment, the networked controller and client devices can be configured to transmit certain types of inputs directly from the controller to the cloud gaming server and other types of inputs via the client device. For example, inputs that are not dependent on any additional hardware or processing, apart from the controller itself, can be transmitted directly from the controller to the cloud gaming server over the network, bypassing the client device. Such inputs can include button inputs, joystick inputs, embedded motion detection inputs (e.g., accelerometers, magnetometers, gyroscopes), and the like. However, inputs that utilize additional hardware or require processing by the client device can be transmitted by the client device to the cloud gaming server. These can include video or audio captured from the gaming environment that can be processed by the client device before transmission to the cloud gaming server. In addition, inputs from the controller's motion detection hardware can be processed by the client device in conjunction with the captured video to detect the position and movement of the controller, which is then communicated by the client device to the cloud gaming server. It should be understood that the controller device according to various embodiments can also receive data (e.g., feedback data) from the client device or directly from the cloud gaming server.

[0132] In one embodiment, various examples of the technology can be implemented using a virtual environment via an HMD. An HMD may also be referred to as a virtual reality (VR) headset. As used herein, the term "virtual reality" (VR) generally refers to user interaction with a virtual space / environment, including viewing the virtual space through an HMD (or VR headset) in a way that responds in real time to the (user-controlled) movements of the HMD to provide the user with the sensation of being in the virtual space or metaverse. For example, a user can see a three-dimensional (3D) view of the virtual space when facing a given direction, and when the user turns to the side, thereby similarly changing the orientation of the HMD, a view of that side of the virtual space is rendered on the HMD. The HMD can be worn in a manner similar to glasses, goggles, or helmets, and is configured to display video games or other metaverse content to the user. The HMD can provide a display mechanism in close proximity to the user's eyes, resulting in a highly immersive experience for the user. Thus, an HMD can provide each of the user's eyes with a display area that occupies a large portion, or even the entirety, of the user's field of view, and can also provide viewing with three-dimensional depth and perspective.

[0133] In one embodiment, the HMD can include an eye-tracking camera configured to capture images of the user's eyes while the user interacts with the VR scene. The gaze information captured by the eye-tracking camera(s) can include information related to the user's gaze direction and particular virtual objects and content items in the VR scene that the user is focusing on or is interested in interacting with. Thus, based on the user's gaze direction, the system can detect particular virtual objects and content items, e.g., game characters, game objects, game items, etc., that may be potential focal points for the user if the user is interested in interacting and engaging with them.

[0134] In some embodiments, the HMD may include an outward-facing camera(s) configured to capture images of the user's real-world space, such as the user's body movements, and images of any real-world objects that may be located in the real-world space. In some embodiments, images captured by the outward-facing cameras may be analyzed to determine the position / orientation of the real-world objects relative to the HMD. With the known position / orientation of the HMD, the real-world objects and inertial sensor data from the user's gestures and movements may be continuously monitored and tracked while the user interacts with the VR scene. For example, while interacting with an in-game scene, the user may perform various gestures, such as pointing at or walking toward a particular content item in the scene. In one embodiment, the gestures may be tracked and processed by the system to generate a prediction of an interaction with a particular content item in the game scene. In some embodiments, machine learning may be used to facilitate or assist in the above predictions.

[0135] Various types of single-handed and two-handed controllers may be used during use of the HMD. In some implementations, the controller itself may be tracked by tracking lights included in the controller or by tracking shape, sensor, and inertial data associated with the controller. These various controllers, or even simple hand gestures performed and captured by one or more cameras, may be used to interface, control, manipulate, interact with, and participate in a virtual reality environment or metaverse rendered on the HMD. In some cases, the HMD may be wirelessly connected to a cloud computing and gaming system via a network. In one embodiment, the cloud computing and gaming system maintains and executes the video game being played by the user. In some embodiments, the cloud computing and gaming system is configured to receive inputs from the HMD and interface objects via the network. The cloud computing and gaming system is configured to process the inputs to affect the game state of the running video game. Outputs, such as video data, audio data, and haptic feedback data from the running video game are sent to the HMD and interface objects. In other implementations, the HMD can communicate with the cloud computing and gaming system wirelessly via alternative mechanisms or channels, such as a cellular network.

[0136] Additionally, although embodiments of the present disclosure may be described with respect to a head-mounted display, it will be understood that in other embodiments, a non-head-mounted display may be used instead, including, but not limited to, a portable device screen (e.g., tablet, smartphone, laptop, etc.) or any other type of display that may be configured to render video and / or provide a display of an interactive scene or virtual environment in accordance with the present embodiments. It will be appreciated that the various embodiments defined herein may be combined or assembled into specific implementations using the various features disclosed herein. Thus, the examples provided are only some of the possible examples and are not intended to limit the various embodiments that may be combined to define many more embodiments by combining various elements. In some examples, an embodiment may include fewer elements without departing from the spirit of the disclosed or equivalent embodiments.

[0137] Embodiments of the present disclosure may be practiced with a variety of computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0138] Although the operations of the method have been described in a particular order, it should be understood that other housekeeping operations may be performed between operations, or operations may be coordinated to occur at slightly different times, or operations may be distributed within the system to allow processing operations to occur at various intervals relative to processing, so long as the processing of telemetry and game state data to generate modified game state is performed in the desired manner.

[0139] One or more embodiments may also be fabricated as computer readable code on a computer readable medium. The computer readable medium may be any data storage device that can store data. The data may then be read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical and non-optical data storage devices. The computer readable medium may include computer readable tangible media distributed across network connected computer systems such that the computer readable code is stored and executed in a distributed fashion.

[0140] In one embodiment, the video game is executed locally on a game console, personal computer, or server. In some cases, the video game is executed by one or more servers at a data center. When the video game is executed, some instances of the video game may be simulations of the video game. For example, the video game may be executed by an environment or server that generates a simulation of the video game. The simulation, in some embodiments, is an instance of the video game. In other embodiments, the simulation may be generated by an emulator. In either case, when the video game is represented as a simulation, the simulation may be executed to render interactive content that can be interactively streamed, executed, and / or controlled by user input.

[0141] It should be noted that in various embodiments, one or more features of some of the embodiments described herein may be combined with one or more features of one or more of the remaining embodiments described herein.

[0142] Although the foregoing embodiments have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Thus, the present embodiments should be considered as illustrative and not restrictive, and the present embodiments should not be limited to the details set forth herein, but may be modified within the scope of the appended claims and their equivalents.

Claims

1. 1. A method of providing assistance during game play, comprising: accessing an artificial intelligence model associated with a user account of the user, the artificial intelligence model being used to generate one or more predictive indicators that identify the user as being unable to complete a task during a game based on levels of game contexts of one or more games, levels of user interactions with the game contexts, and levels of performance metrics in the game contexts; receiving a request to access the game for a game session via the user account; providing, in response to the one or more predictive indicators that the user will be unable to complete the task in the game context, assistance input via a computer network to a handheld controller operated by the user during the game context of the game; Including, The method, wherein the assistance input is provided before the user performs the task.

2. 2. The method of claim 1 , wherein the artificial intelligence model includes a neural network trained by one or more game plays conducted via the user account, the one or more game plays being used to label and classify the plurality of game contexts, label and classify the plurality of user interactions, and label and classify the plurality of performance metrics.

3. 3. The method of claim 2, wherein the providing of the assistance input is performed upon determining that the game context of the game session is similar to one or more of the plurality of game contexts used to train the artificial intelligence model.

4. generating state data for play of the one or more games; analyzing the state data to identify the game contexts of the one or more games, the user interactions with the game contexts, and the performance metrics generated based on the user interactions and the game contexts; The method of claim 2 , further comprising:

5. labeling the plurality of game contexts to identify the plurality of game contexts; labeling the plurality of user interactions to identify the plurality of user interactions; labeling the plurality of performance metrics to identify the plurality of performance metrics; categorizing the plurality of game contexts to provide the level of the plurality of game contexts; categorizing the plurality of user interactions to provide the level of the plurality of user interactions; categorizing the plurality of performance metrics to provide the levels of the plurality of performance metrics; The method of claim 4 further comprising:

6. The method of claim 5 , wherein the one or more predictive indicators are determined based on the levels of the plurality of game contexts, the levels of the plurality of user interactions, and the levels of the plurality of performance metrics.

7. 2. The method of claim 1 , wherein the auxiliary input includes controller input data, light data for controlling one or more lights on the handheld controller used to play the game, screen text data for presentation on a display device used during the game session, and audio data for outputting sounds during the game session.

8. 1. A computer system for providing assistance during game play, comprising:

1. A processor comprising: accessing an artificial intelligence model associated with a user account of the user, the artificial intelligence model being used to generate one or more predictive indicators that identify the user as being unable to complete a task during a game based on levels of game contexts of one or more games, levels of user interactions with the game contexts, and levels of performance metrics in the game contexts; receiving a request to access the game for a game session via the user account; providing, in response to the one or more predictive indicators that the user will be unable to complete the task in the game context, assistance input via a computer network to a handheld controller operated by the user during the game context of the game; The processor is configured to: a memory device coupled to the processor; A computer system comprising:

9. 9. The computer system of claim 8, wherein the artificial intelligence model includes a neural network trained by one or more game plays conducted via the user account, the one or more game plays being used to label and classify the plurality of game contexts, label and classify the plurality of user interactions, and label and classify the plurality of performance metrics.

10. 10. The computer system of claim 9, wherein the assistance input is generated when the artificial intelligence model determines that the game context of the game session is similar to one of the multiple game contexts used to train the artificial intelligence model.

11. The processor, generating state data for play of the one or more games; analyzing the state data to identify the game contexts of the one or more games, the user interactions with the game contexts, and the performance metrics generated based on the user interactions and the game contexts; 10. The computer system of claim 9 configured to:

12. The processor, labeling the plurality of game contexts to identify the plurality of game contexts; labeling the plurality of user interactions by the user to identify the plurality of user interactions; labeling the plurality of performance metrics to identify the plurality of performance metrics; categorizing the plurality of game contexts to provide the level of the plurality of game contexts; categorizing the plurality of user interactions to provide the level of the plurality of user interactions; categorizing the plurality of performance metrics to provide the levels of the plurality of performance metrics; 12. The computer system of claim 11 configured to:

13. The computer system of claim 12 , wherein the one or more predictive indicators are determined based on the levels of the plurality of game contexts, the levels of the plurality of user interactions, and the levels of the plurality of performance metrics.

14. 9. The computer system of claim 8, wherein the auxiliary input includes controller input data, light data for controlling one or more lights on the handheld controller used to play the game, screen text data for presentation on a display device used during the game session, and audio data for outputting sounds during the game session.

15. 1. A system for providing assistance during game play, comprising: A client device; a server coupled to the client device via a computer network, accessing an artificial intelligence model associated with a user account of the user, the artificial intelligence model being used to generate one or more predictive indicators that identify the user as being unable to complete a task during a game based on levels of game contexts of one or more games, levels of user interactions with the game contexts, and levels of performance metrics in the game contexts; receiving a request to access the game for a game session via the user account; providing, in response to the one or more predictive indicators that the user will be unable to complete the task in the game context, assistance input via a computer network to a handheld controller operated by the user during the game context of the game; and the server configured to: The system, wherein the client device is configured to receive the assistance input from the server via the computer network.

16. 16. The system of claim 15, wherein the assistance input is generated when the artificial intelligence model determines that the game context of the game session is similar to one of the multiple game contexts used to train the artificial intelligence model.

17. 16. The system of claim 15, wherein the artificial intelligence model includes a neural network trained by one or more game plays conducted via the user account, the one or more game plays being used to label and classify the plurality of game contexts, label and classify the plurality of user interactions, and label and classify the plurality of performance metrics.

18. The server, generating state data for play of the one or more games; analyzing the state data to identify the game contexts of the one or more games, the user interactions with the game contexts, and the performance metrics generated based on the user interactions and the game contexts; The system of claim 17 configured to:

19. The server, labeling the plurality of game contexts to identify the plurality of game contexts; labeling the user interactions with the game contexts to identify the user interactions; labeling the performance metrics derived based on the user interactions to identify the performance metrics; and categorizing the plurality of game contexts to provide the level of the plurality of game contexts; categorizing the plurality of user interactions to provide the level of the plurality of user interactions; categorizing the plurality of performance metrics to provide the levels of the plurality of performance metrics; The device is configured to: The system of claim 18 , wherein the one or more predictive indicators are determined based on the levels of the plurality of game contexts, the levels of the plurality of user interactions, and the levels of the plurality of performance metrics.

20. The system of claim 15 , wherein the client device is a handheld controller or a head-mounted display.

21. The method of claim 1, wherein the one or more predictive indicators identify that the user will be unable to complete the game while playing the game based on the amount of time it takes the user to interact with virtual objects in the game context.

Images