Controller State Management for Client-Server Networking

Abstract

Described herein are techniques for transmitting state data between a controller and another electronic device. In embodiments, such techniques may include receiving input at a user device related to a first action to be taken by an avatar, providing a first set of state data generated from the received input to a second electronic device over a first communication channel, and identifying a user intent related to a second action to be taken by the avatar. Based on the user intent, a second set of state data may be generated that is different from the first set of state data and includes less than all available state data. The second set of state data may be provided to the second device over a second communication channel to cause the avatar to take a modified action.

Description

Background Art

[0001] Background

[0001] Modern computer controller systems such as those used by computers, video games, and general-purpose operating systems employ various techniques to network controller data within requirements between client machines and server machines for online games. As described in U.S. Patent No. 8,632,409 and U.S. Patent No. 8,823,701, there are complex networking solutions for computer or console / server communication for first-person perspective games (e.g., the DESTINY and DESTINY 2 series). However, the inherent problems in implementing the networking of controller data are different when considering isometric game engines as opposed to first-person perspectives, and when considering computing environments where a larger share of computing power is on the server side as opposed to the client side. Therefore, an improved networking implementation for supporting single-client games and games with various perspectives is required.

Summary of the Invention

[0002] Summary

[0002] Techniques are presented herein for facilitating communication between a controller and another electronic device (e.g., a server or console). In such techniques, when user input is received, a first set of state data can be provided by the controller via a first communication channel. The controller can clarify the predicted intention of the user based on the received user input. Based on the predicted intention of the user, a second set of state data is provided by the controller via a second communication channel.

[0003]

[0003] In one embodiment, a method is disclosed as being executed by a user device, the method comprising receiving, at a controller device, a touch input from a user of the controller device; providing, on a first communication channel, a first set of state data generated from the received touch input to a second electronic device; ascertaining a user's intention based on the received touch input; generating a second set of state data based on the user's intention; and providing, on a second communication channel different from the first communication channel, a second set of state data generated from the received touch input to the second electronic device.

[0004]

[0004] One embodiment is directed to a computing system including a touch screen display, a processor, and a memory that, when executed by the processor, causes the computing device to at least receive a touch input from a user of the user device; provide, on a first communication channel, a first set of state data generated from the received touch input to a second electronic device; ascertain a user's intention based on the received touch input; generate a second set of state data based on the user's intention; and provide, on a second communication channel different from the first communication channel, a second set of state data generated from the received touch input to the second electronic device.

[0005]

[0005] One embodiment is directed to a non-transitory computer-readable medium collectively storing computer-executable instructions that, when executed, collectively cause one or more computing devices to perform acts including receiving, at a controller device, a touch input from a user of the controller device; providing, on a first communication channel, a first set of state data generated from the received touch input to a second electronic device; ascertaining a user's intention based on the received touch input; generating a second set of state data based on the user's intention; and providing, on a second communication channel different from the first communication channel, a second set of state data generated from the received touch input to the second electronic device.

[0006]

[0006] One embodiment is directed to a method for receiving, at a user device, an input from a user related to a first action performed by an avatar, providing, on a first communication channel, a first set of state data generated from the received input to a second electronic device, and clarifying a user's intention related to a second action performed by an avatar different from the first action based on the received input. The method may further include generating, based on the user's intention, a second set of state data that is different from the first set of state data and includes less state data than all available state data, providing the second set of state data to the second electronic device on a second communication channel different from the first communication channel, and causing the avatar to perform a modified action different from the first action based on the first and second sets of state data. The user's intention may include any of an interruption that impedes the first action being performed by the avatar, a modification to the first action being performed by the avatar, or a second action that is performed after the first action being performed by the avatar. The first set of state data can include a set of predetermined data values, the second set of state data is inferred from the received input, and can include a set of data values selected based on the user's intention that may require less bandwidth to transmit than the first set of state data.

[0007]

[0007] The above content will become more apparent when referring to the following specification, the appended claims, and the accompanying drawings, along with other features and embodiments. The embodiments of the invention that this patent encompasses as a scope are defined by the appended claims rather than this summary. This summary is a high-level overview of various aspects of the invention and introduces some of the concepts further described in the following detailed description sections. This summary is not intended to identify important or essential features of the content recited in the claims, nor is it intended to be used separately to determine the scope of the content recited in the claims. This content should be understood by referring to the entire specification of this patent, any or all of the drawings, and the appropriate portions of each claim.

[0008] Brief Description of the Drawings

[0008] A detailed description will be given with respect to the accompanying drawings. In the figures, the leading digit of a reference number identifies the figure in which that reference number first appears. The use of the same reference number in different figures indicates the same or identical item or feature.

Brief Description of the Drawings

[0009]

Figure 1

[0009] FIG. 1 is a simplified system diagram showing a service environment in which a virtual controller can be used according to various embodiments of the present disclosure.

Figure 2

[0010] FIG. 2 shows an example for explaining an environment including a virtual controller, which can implement a method for networking controller data between a client machine and a server machine for an online game according to an embodiment.

Figure 3

[0011] FIG. 3 is a block diagram showing various components of a computing system architecture that supports the implementation of a virtualized physical controller according to an embodiment.

Figure 4

[0012] FIG. 4 shows a block diagram showing a process for executing a modified action in response to receiving a touch input according to an embodiment.

Figure 5

[0013] FIG. 5 is a process flow diagram showing an example of a process for networking controller data between a client machine and a server machine for an online game.

Figure 6

[0014] FIG. 6 shows a flow diagram showing an example of a process flow for transmitting status data between electronic devices according to an embodiment.

Best Mode for Carrying Out the Invention

[0010] Detailed Description

[0015] The following description sets forth various embodiments. In this description, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it will also be apparent to one skilled in the art that the embodiments may be practiced without these specific details. In addition, well-known features may be omitted or simplified in order not to obscure the described embodiments.

[0011]

[0016] Embodiments herein are directed to techniques for implementing a communication protocol for use with a controller (e.g., a virtual controller). In particular, the techniques are directed to transmitting additional state data between a controller and a server while minimizing the bandwidth used. In such techniques, typical movement data is transmitted over a first communication channel. In addition, the "intent" of the user is revealed by the controller in order to identify the state data most likely to be relevant to the next user action. Based on the predicted intent, the controller can transmit a set of state data related to that intent over a second communication channel.

[0012]

[0017] In a conventional client-server networking model (see, e.g., the published Unreal 3 networking literature based on the Tribes Networking Model (https: / / www.gamedevs.org / uploads / tribes-networking-model.pdf) at https: / / docs.unrealengine.com / udk / Three / NetworkingOverview.html), for a given set of machines that want to experience a synchronous multiplayer experience together, there is one machine ("the server") that has the official (authoritative) networked "state" (e.g., the position and health of an avatar, the score of the game, etc.) that is transmitted to the "client" machines.

[0013]

[0018] For a "client" machine to run its avatar on a "server" that sends the officially updated state of the avatar it controls to all clients connected in a multiplayer experience, a "movement request" is constructed from the player's input (e.g., the state of the controller and buttons), and the "movement request" is immediately executed on the client machine (referred to as "client-side prediction", which allows the client to informally change the state of the avatar it controls locally to make the input feel responsive).

[0014]

[0019] Part of the state resulting from the "movement request" is captured (e.g., the end position of the avatar after the movement) and transmitted to the "server" along with the "movement request". The server then executes the "movement request" against the official state of the avatar and compares the resulting state. If the resulting state matches the client's transmission, the movement is "permitted" and the server sends an affirmative response. If the movement is not permitted, the server sends a "correction" to the client that includes the official state of the avatar.

[0015]

[0020] Embodiments of the present disclosure provide several advantages over conventional systems. In particular, in many standard implementations of the above networking model, the state of the "movement request" includes the "raw" analog input state sufficient to perform basic operations (e.g., movement), but generally lacks the raw data sufficient to perform any advanced operations. In many cases, the state data transmitted between the client and the server is often limited based on the "game engine" being implemented. As an example, an implementation of Unreal Engine 4 includes the acceleration vector of the avatar, and such acceleration vector is derived from the state of the analog thumbstick, but states other than the acceleration vector are often lost. Embodiments of the present disclosure provide the ability to transmit additional state data while respecting bandwidth limitations.

[0016]

[0021] Making control accessible to users across multiple platforms typically involves separating the implementation from any individual controller so that it can be easily converted into multiple types of controllers, customizable controllers, or even virtual controllers (e.g., touch screen devices embedded in or replacing the controller). For example, in a game implemented on a flat screen without physical buttons, the player's intent can be associated with a nearly indeterminate range of potential inputs. To make certain avatar actions and movements feel responsive to the player's desires, player input (e.g., quickly swiping in the direction opposite to the direction of the movement of one's avatar) regarding, for example, how strongly and in which direction a gesture is being made, is important from the perspective of "intent". However, state data containing such information is often not included in the communication between the client device and the server. In that case, the system cannot accurately respond to user input in the way it would be possible if complete state data were provided. In some cases, a wide variety of different state data may be made available to the client for transmission to the server, but the amount of data that can be sent may be limited based on bandwidth.

[0017]

[0022] Embodiments of the present disclosure may be advantageous over conventional systems in that additional state data can be provided from the client to the server to enable a greater level of responsiveness to user input. As examples, some non-limiting examples of additional state data that may be transmitted may include input vectors, input offsets, input changes, or any other suitable state data.

[0018]

[0023] The following terms are defined in the present disclosure.

[0019]

[0024] Input vector - The "raw" virtual joystick input value that is converted into a 2D vector and processed with internal and external "dead zones" to smooth the lossy analog values.

[0020]

[0025] Input offset - magnitude of the input vector.

[0021]

[0026] Input change - By remembering past inputs, certain patterns or rapid changes in the input can be tracked over time. For example, if a "U-turn" is detected on a virtual controller based on a rapid change in user input, the first input and the second input can be tracked and used to calculate the parameters of the input change that can be tracked as an input itself. The input change, such as the direction and magnitude of the change between the first input and the second input, can be stored as an additional vector. Patterns of input change other than direction can also be tracked. For example, the rapid change between pressing one button and pressing another button can be tracked as an additional input, or the behavior of the second button press can be modified based on the button input or pattern of button inputs registered in the past.

[0022]

[0027] In contrast to a physical controller, a virtual controller can be any suitable electronic device on which at least a part of the control input mechanism is virtually instantiated. For example, a virtual controller can include a touch screen display configured to present buttons or other input mechanisms. A user's touch on the touch screen display can be detected during the use of the virtual controller. If the detected user touch position corresponds to the position of the presented button, that button can be activated.

[0023]

[0028] Receiving additional states enables the server to adjust movement actions based on the perceived user intent. For example, a determination can be made regarding "whether the user wants to grab the avatar for a trip?". Conventional techniques for determining this type of user intent may use data such as the direction of the avatar's movement or the vector of the orientation, but such techniques may misread the player's intent, especially when state data is lost. Using the context-based technique of the present disclosure, the actual state of the virtual joystick (e.g., "the player just snapped with their thumb towards the trip") is used to infer that the player wants to grab the trip for their avatar, creating a more intuitive, responsive, comfortable, and forgiving operation scheme for the player.

[0024]

[0029] FIG. 1 is a simplified system diagram showing a service environment 100 in which a virtual controller can be used, according to various embodiments of the present disclosure. The service environment 100 includes at least one server 101, and the at least one server 101 includes at least one processor 103 and a non-transitory memory 105 storing instructions as software for facilitating the operation of the service environment. The server 101 is typically connected via a network 121 (e.g., the Internet or a local network) to any suitable number of user-owned client devices 133, 143 operating with respective local user networks 131, 141 (e.g., consumer or commercial local area networks, WIFI networks, etc.).

[0025]

[0030] Server 101 can also be connected to any suitable number of control services 111 that monitor the network between server 101 and client devices 133, 143, such as a network connection computing system having its own processor 113 and memory 115. In some embodiments, server 101 can be one or more servers operating on a commercial scale, such as a data center or server farm. Client devices 133, 143 can include, but are not limited to, consumer personal computers, video game consoles, sink client devices operable to stream video content from server 101 for presentation on a local screen, or mobile devices such as smartphones and tablets. Client devices 133, 143 can be connected to any suitable number of controllers, such as controllers 135, 137, 145, 147.

[0026]

[0031] Each controller (e.g., controller 135) can be a hardware device (e.g., a console-specific controller, a cross-corresponding controller, or a virtual controller) having connectivity hardware and protocols for communicating with its respective client device 133. According to some embodiments, controller 135 can be a virtualized controller operating on a sink client device or a touch screen device, such as a touch screen smartphone, tablet, or a console-like controller having a touch-responsive panel. For example, in some further embodiments where client device 133 is a sink client device or a mobile device, controller 135 can be a touch screen having a virtualized control incorporated into the client device. Alternatively, even if client device 133 is a sink client device, controller 135 can be a hardware controller configured to be physically or wirelessly connected to the client device. According to some embodiments, client device 133 and server 101 can operate on the same hardware, such as the client device being executed as a virtual instance on the server.

[0027]

[0032] The method described in this specification can be implemented on a client device in conjunction with a service environment such as the service environment 100 described in FIG. 1. The method can further function in the context of an arbitrary arrangement of virtual controllers that manipulate both the orientation and movement of an avatar on the screen.

[0028]

[0033] For clarity, a certain number of components are shown in FIG. 1. However, it will be understood that embodiments of the present disclosure may include multiple of each component. Additionally, some embodiments of the present disclosure may include fewer or more components than all of the components shown in FIG. 1. Additionally, the components of FIG. 1 may communicate via any suitable communication medium (including the Internet) using any suitable communication protocol.

[0029]

[0034] FIG. 2 shows an example for the description of an environment 200 including a virtual controller that can implement a technique for networking controller data between a client machine and a server machine for an online game according to an embodiment. Any suitable service environment or system, such as the service environment 100 of FIG. 1, can be used to enable an example of this technique.

[0030]

[0035] According to various embodiments, player input can be received by manipulating a controller by the player (201). The client device can begin to simulate the movement or action of an avatar in the game and generate resulting information such as a final position (i.e., “movement”) in response to the player input (202). Along with simulating responsive actions, the system also stores the player input and / or the resulting “movement” specified by the input (203).

[0031]

[0036] Subsequent movements can be made after the initial movement, and the subsequent movements are also repeatedly saved in the same way. The service monitors the intensity of the input, such as "transmission time" or "send dT", and if the intensity is below a threshold, the service can put the input parameters into the cache as variables, i.e., pending movements (204). If subsequent movements also fall below the threshold, the pending movement variables can be added or replaced.

[0032]

[0037] If the intensity of the input exceeds the threshold, the service queries whether there is an old movement in the memory, and if so, transmits the old movement for reception by the server (208) (205). If the old movement is not in the memory, the service queries whether there is a pending movement in the memory. If so, the service transmits both the new movement and the pending movement for reception by the server (208) (206). Otherwise, the service transmits the new movement to the server (207).

[0033]

[0038] When receiving a movement instruction from the server (208), the server-side service executes the player input (201) to reveal the official state of the avatar in the player's game, and compares that official state with the "movement" (i.e., the client-generated information (203) regarding the result of the player input (201)). If the official state matches the movement (i.e., if the movement is valid), the server positively responds to the "movement" (209), and upon receiving the positive response, the client device saves the last positively responded movement (211) and removes the movement preceding the last positively responded movement from the memory (212).

[0034]

[0039] If the official state does not match the client-generated state of the avatar in the game, the server calculates an adjustment for the client-generated state (210) and transmits that adjustment to the client device (213). Then the client device replays the movement so that the official state and the client-generated state match (214).

[0035]

[0040] FIG. 3 is a block diagram showing various components of a computing system architecture that supports the implementation of a virtualized physical controller according to an embodiment. This system architecture may include at least one controller 302. In some embodiments, the controller 302 can communicate with one or more servers 304, which may be an example of the server 101 described with respect to FIG. 1. In some embodiments, one or more servers 101 may provide backend support for the controller 302. For example, in some cases, at least a portion of the processing described as being performed by the controller 302 may be performed instead by the server 101. In some embodiments, the controller 302 can communicate with a client device 306. The client device 306 may be an example of the client device 133 or 143 described above with respect to FIG. 1. In some embodiments, the client device 306 can further communicate with a display device 308. Each of the components described herein can communicate by means of connections on a network 310.

[0036]

[0041] The controller 302 may include any suitable computing device configured to perform at least a portion of the operations described herein and configured to enable a user to interact with a software application. In some embodiments, the controller may be a mobile device (e.g., a smartphone or tablet) having a touchscreen function. The controller 302 may include a communication interface 312, one or more processors 314, a memory 316, and hardware 318. The communication interface 312 may include wireless and / or wired communication components that enable the controller 302 to send and receive data with other networked devices. The hardware 318 may include additional user interface, data communication, or data storage hardware. For example, the user interface may include at least one output device 320 (e.g., a visual display, an audio speaker, and / or a tactile feedback device) and one or more data input devices 322. The data input devices 322 may include, but are not limited to, one or more combinations of a keypad, a keyboard, a mouse device, a touchscreen that accepts gestures, a microphone, a voice or language recognition device, and any other suitable device.

[0037]

[0042] Memory 316 can be implemented using a computer-readable medium such as a computer storage medium. The computer-readable medium includes at least two types of computer-readable media, namely computer storage media and communication media. The computer storage media includes any appropriate volatile and non-volatile removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. The computer storage media includes, but is not limited to, RAM, DRAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical storage areas, magnetic cassettes, magnetic tapes, magnetic disk storage areas or other magnetic storage devices, or any other non-transmission media that can be used to store information for access by a computing device. In contrast, communication media, such as a carrier wave or other transmission mechanism, can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal.

[0038]

[0043] One or more processors 314 and memory 316 of the controller can implement functions including one or more software modules and data stores. Such software modules can include routines, program instructions, objects, and / or data structures that are executed by processor 314 to perform specific tasks or implement specific data types. More specifically, memory 316 can include a module (such as state selection module 324) configured to predict possible intentions of a user and then identify additional state data to be provided to another device.

[0039]

[0044] In addition, the memory 316 can include various data stores. For example, the memory 316 can hold a prediction model 326 as well as past user input / action data (such as usage data 328). In this example, the prediction model can be a trained machine learning model trained to correlate received user input with possible user intentions. In some cases, such a prediction model can be trained based on past user input / action data regarding the user.

[0040]

[0045] The state selection module 324, together with the processor 314, can be configured to predict possible user intentions. Based on the predicted intention, the state selection module 324 can be configured to identify additional state data (such as state data not yet provided) related to the predicted intention. The state selection module 324 can then be configured to transmit the additional state data to another electronic device via a communication channel. In some cases, this can include creating a communication channel separate from an existing communication channel over which initial state data is provided.

[0041]

[0046] The server 304 can include any computing device configured to perform at least a portion of the operations attributable to the server 304. The server 304 can be configured as one or more general-purpose computers, dedicated server computers (including, for example, PC (personal computer) servers, UNIX (registered trademark) servers, midrange servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other suitable configuration and / or combination. The server 304 can include other computing architectures involving virtualization, such as one or more virtual machines executing a virtual operating system, or one or more flexible pools of virtualizable logical storage devices for maintaining virtual storage devices for computers. For example, the server 304 can include a virtual machine hosted in the cloud or a virtual computing device in the form of a software container.

[0042]

[0047] Server 304 may include one or more processors 330 and a memory 332. One or more processors 330 and the memory 332 of the controller may implement functions including one or more software modules and data stores. Such software modules may include routines, program instructions, objects, and / or data structures that are executed by the processor 330 to perform a specific task or implement a specific data type. More specifically, the memory 332 may include a game engine 334 configured to track game state data and provide the game state data to the controller 302.

[0043]

[0048] The game engine 334 may include software instructions configured to facilitate interaction between the avatars of one or more players and the virtual environment. In some cases, the software instructions may be composed of several different components including at least an input system 336 and an action management module 338.

[0044]

[0049] The input system 336 may be a component of the game engine configured to receive a set of predetermined state data from the controller and convert the set of predetermined state data into an action (e.g., movement) of the avatar associated with the controller for performing the set of predetermined state data. In some embodiments, a first active communication channel 340 over which a set of predetermined state data may be transmitted may be maintained between the controller 302 and the server 304.

[0045]

[0050] The action management module 338 can be a component of a game engine configured to receive additional state data from a controller and determine one or more actions to be completed based on that state data. The additional state data can be received from the controller via a second communication channel 342. In some embodiments, the action management module can be configured to change or enhance an action instructed to be performed by the input subsystem 336. For example, the type of action performed by an avatar (e.g., movement, attack, etc.) can be determined based on a set of predetermined state data received by the input subsystem, while the degree or amount of that action can be determined based on additional state data received by the action management module 338.

[0046]

[0051] As an example, consider an example where a set of predetermined state data provided to the input subsystem includes an input vector resulting from movement performed by an avatar. In this example, the additional state data can indicate the amount of pressure applied to a directional pad and / or the speed of interaction. In this example, the movement indicated by the input subsystem can be adjusted based on the speed or velocity of the movement.

[0047]

[0052] The client device 306 may include any suitable computing device configured to receive an input from the controller 302 and perform an action based on the input. In some embodiments, the client device can be a game system such as a gaming console, and such a game system can receive inputs from several controllers that can each be used to manipulate an avatar or character within a software application (e.g., a computer game). Although several components (e.g., game engine 334) are described as being included within the server 304, it should be noted that such components may actually be included in the client device instead. In that case, functions attributed to the server may instead be performed by the client device, and those skilled in the art will understand that the client device becomes the equivalent of the server in those instances.

[0048]

[0053] FIG. 4 shows a block diagram illustrating a process for performing a modified action in response to receiving a touch input, according to an embodiment. Process 400 can be executed by the system described above in FIG. 3. For example, the first part of this process may be executed by a controller (e.g., controller 302), and the second part of this process may be executed by a server (e.g., server 304). In some cases, the controller can be a user device on which a virtual physical controller is implemented.

[0049]

[0054] At 402, process 400 may include receiving a touch input from a user. The touch input may include the selection of a button or other input mechanism. In some cases, the degree or amount related to the touch input can be detected. For example, the speed of the user's touch, the amount of pressure, and / or the distance from the center of the input mechanism can be detected.

[0050]

[0055] At 404, process 400 may include providing a set of predetermined state data to a server. In some embodiments, the set of predetermined state data may be transmitted to the server over a first communication channel. For example, in some cases, the set of predetermined state data may include an input vector that includes a direction.

[0051]

[0056] At 406, process 400 may include determining the type of action to be performed by the avatar. In some cases, the type of action may be selected from a set of available actions to be performed by the avatar based on the type of touch input received. In some cases, the type of action may depend on a set of predetermined state data. For example, the set of predetermined state data may include an input vector indicating a direction of movement. In this example, the type of action may be identified as movement in the indicated direction.

[0052]

[0057] At 408, process 400 may include predicting a possible intention of the user. In some cases, based on factors (such as degree or amount) associated with the received user input, the possible intention of the user that can be predicted can be clarified. For example, each of the intentions of several different users may correspond to a value range for various factors. In some embodiments, the intention of the user can be clarified by providing the received touch input to a machine learning model trained to correlate the user's touch input with the user's intention. In some cases, the intention of the user may be to perform an enhanced or modified version of an action. In some cases, the intention of the user may be to perform a subsequent action following the currently performed action. In such cases, the subsequent action can be predicted based on the tendency of the user to perform that action after the user provides the received touch input.

[0053]

[0058] At 410, process 400 may include identifying additional state data related to the predicted intent of the user. For example, if the user's intent is determined to be an intent to perform an action, this process may include identifying one or more data values indicated as being related to the action associated with that intent. The data values may represent state data generated from received touch inputs that are not provided to the server at 404. For example, when detecting a touch input provided by the user, various state data can be generated from that touch input. Some non-limiting examples of such state data may include the direction indicated by the touch input, the speed of the touch input, the change in speed across the touch input (e.g., deceleration or acceleration within a swipe), the change in direction across the touch input, or any other suitable factor related to the touch input.

[0054]

[0059] At 412, process 400 may include modifying the action determined at 406. For example, if at 406 of process 400 it is determined that a certain movement action is to be performed by an avatar, at 412, that movement action can be modified based on additional received state data. For example, if it is determined (e.g., at 406) to perform a typical movement, additional state data may instead indicate that a special movement should be performed instead. In this example, the special movement can replace the typical movement while being performed in the same direction as when the typical movement would be performed. At 414, process 400 may include executing the modified action.

[0055]

[0060] FIG. 5 is a process flow diagram showing an example of a process 500 for networking controller data between a client machine and a server machine for an online game. Process 500 may be executed under the control of a service environment or system such as the service environment 100 described with respect to FIG. 1.

[0056]

[0061] According to various embodiments, process 500 includes, at 501, receiving a first player input by a player manipulating a controller on a game client device. Then at 502, the game system typically simulates a first action by an avatar in the game based on the first player input without first synchronizing with a server. Then at 503, the system can receive a second player input by manipulating the controller.

[0057]

[0062] When the input is received within a predetermined period at 504, at 506 the system can generate a predicted input based on the first input and the second input. This predicted input is different from the result of any individual input in that rapid changes in the raw input values received by the controller are tracked as if they were an input and stored in memory. Then at 507, the system can simulate a second action by an avatar in the game based on the predicted input, and the second action can be different from an action simulated based only on sequentially executing the first input and the second input. However, if the first input and the second input are not received within the defined period at 504, at 505 the system can process the inputs separately and sequentially.

[0058]

[0063] FIG. 6 shows a flowchart illustrating an example of a process flow 600 for communicating state data between electronic devices according to an embodiment. Process 600 can be executed by a computing device configured to generate activation data based on user input. For example, process 600 can be executed by a controller that can facilitate interaction between a user and a software application, such as controller 302 described above with respect to FIG. 3. In some embodiments, such a software application is a video game that a user plays, and a controller is used to manipulate an avatar within the video game. In some cases, the controller is a virtual controller having a touch screen display.

[0059]

[0064] At 602, process 600 includes receiving a touch input from a user. In some embodiments, the touch input is related to an action performed by an avatar. In some embodiments, the touch input is related to one or more input mechanisms implemented on a controller.

[0060]

[0065] At 604, process 600 includes providing a first set of state data to a second electronic device over a first communication channel, where the first set of state data is generated from the received touch input. In some embodiments, the first set of state data includes a set of predetermined data values. The second electronic device can be any suitable electronic device such as a server computing device or a gaming console.

[0061]

[0066] At 606, process 600 includes determining the user's intent based on the received touch input. In some embodiments, the user's intent includes a modification to an action performed by an avatar. In some embodiments, the user's intent includes a second action to be performed after an action performed by an avatar. In some embodiments, the user's intent is determined based on past usage data regarding the user of the user device. In some cases, the user's intent is determined by providing the touch input to a trained machine learning model.

[0062]

[0067] At 608, process 600 includes generating a second set of state data based on the determined user's intent. In some embodiments, the second set of state data includes a set of data values selected based on the user's intent. In some embodiments, the second set of state data includes some combination of at least one of an input vector, an input offset, or an input change.

[0063]

[0068] At 610, process 600 includes providing a second set of state data to a second electronic device over a second communication channel different from the first communication channel. In some embodiments, the second set of state data is used by the second electronic device to determine an action or a modification of an action performed by an avatar.

[0064]

[0069] The methods described herein are directed to virtual controllers, i.e., controllers that use a touch screen or touch screen-like functionality to provide an easily customizable layout of controller buttons. According to some embodiments, the touch screen is at least part of a physical handheld controller that interfaces with a gaming device such as a gaming console, a personal computer, a tablet, a smartphone, a sinklient device (e.g., a USB or HDMI device plugged into a screen), etc. According to some embodiments, the touch screen is a prominent feature of a controller that interfaces with a gaming device such as a gaming console, a personal computer, a tablet, a smartphone, a sinklient device (e.g., a USB or HDMI device plugged into a screen), etc. According to some embodiments, the controller is composed of a mobile device or a tablet together with enabling software that connects the mobile device or the tablet to a gaming device such as a gaming console, a personal computer, a sinklient device (e.g., a USB or HDMI device plugged into a screen), etc. or other suitable gaming devices. According to some further embodiments, the touch screen is a touch-responsive screen of a gaming device such as a gaming console, a personal computer, a tablet, or a smartphone.

[0065]

[0070] The present specification and drawings should be considered in an illustrative rather than a restrictive sense. However, it will be apparent that various modifications and changes can be made without departing from the broader spirit and scope of the present disclosure as set forth in the appended claims.

[0066]

[0071] Other modifications are within the spirit of the present disclosure. Thus, while the techniques disclosed are susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown and have been described in detail above. It is not intended, however, to limit the invention to the particular forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention as defined by the appended claims.

[0067]

[0072] The terms "a," "an," and "the" and similar referents used in the context of describing the disclosed embodiments (especially in the context of the appended claims) are to be construed to include both the singular and the plural unless otherwise indicated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "containing" are to be construed as open-ended terms (i.e., meaning "including, but not limited to") unless otherwise specified. The term "connected" is to be construed as being attached, joined, or connected, either partially or wholly, with or without intervening elements. The recitation of ranges of values herein is merely intended to serve as a convenient method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated herein as if it were individually recited herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The use of any and all examples, or exemplary language (e.g., "such as") provided herein is merely intended to better illuminate embodiments of the invention and does not pose a limitation on the scope of the invention unless otherwise claimed. No language in this specification should be construed as indicating any non-claimed element as essential for the practice of the invention.

[0068]

[0073] The preferred embodiments of the disclosure are described herein, including the best mode known to the inventors for carrying out the invention. Modifications of those preferred embodiments may become apparent to those of ordinary skill in the art upon reading the above description. The inventors expect those of ordinary skill in the art to employ such modifications as appropriate, and intend for the invention to be practiced otherwise than as specifically described herein. Accordingly, the invention includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Further, unless otherwise indicated herein or clearly contradicted by context, any combination of the above-described elements in all possible variations thereof is included within the scope of the invention.

Example

[0069]

[0074] Additional examples are described below to facilitate understanding of aspects of the invention.

[0070]

[0075] Example A. Receiving, at a user device, an input from a user related to a first action performed by an avatar; Providing, on a first communication channel, a first set of state data generated from the received input to a second electronic device; Revealing, based on the received input, a user's intent related to a second action performed by an avatar different from the first action; Generating, based on the user's intent, a second set of state data that includes less state data than all of the available state data and is different from the first set of state data; Providing, on a second communication channel different from the first communication channel, the second set of state data to the second electronic device; and Causing the avatar to perform a modified action different from the first action based on the first and second sets of state data comprising a method.

[0071]

[0076] Example B. The method according to the preceding example, wherein the user's intention includes an interruption that prevents the first action from being performed by the avatar.

[0072]

[0077] Example C. The method according to any one of the preceding examples, wherein the user's intention includes a modification to the first action to be performed by the avatar.

[0073]

[0078] Example D. The method according to any one of the preceding examples, wherein the user's intention includes a second action to be performed after the first action performed by the avatar.

[0074]

[0079] Example E. The method according to any one of the preceding examples, wherein the first set of state data includes a set of predetermined data values.

[0075]

[0080] Example F. The method according to any one of the preceding examples, wherein the second set of state data includes a set of data values selected based on the user's intention.

[0076]

[0081] Example G. The method according to any one of the preceding examples, wherein the second set of state data includes some combination of at least one of an input vector, an input offset, or an input change.

[0077]

[0082] Example H. The method according to any one of the preceding examples, wherein the user's intention is revealed based on past usage data regarding the user of the user device.

[0078]

[0083] Example I. The method according to any one of the preceding examples, wherein the user's intention is revealed by providing an input to a trained machine learning model.

[0079]

[0084] Example J. A processor and, When executed by the processor, at least Receiving, in the user device, an input from a user of the user device related to the first action to be performed by the avatar Providing a first set of state data generated from received input to a second electronic device over a first communication channel; Revealing a user's intention related to a second action performed by an avatar different from a first action based on the received input; Generating, based on the user's intention, a second set of state data that is different from the first set of state data and includes less state data than all available state data; Providing the second set of state data to the second electronic device over a second communication channel different from the first communication channel; and Causing the avatar to perform a modified action different from the first action based on the first and second sets of state data A memory including instructions for causing a user device to perform the above; and A user device including the above.

[0080]

[0085] Example K. The user device according to the preceding example, wherein the user device includes a controller used to operate an avatar in a video game.

[0081]

[0086] Example L. The user device according to any one of the preceding examples, wherein the controller includes a virtual controller having a touch screen display.

[0082]

[0087] Example M. The user device according to any one of the preceding examples, wherein the input is related to one or more input mechanisms implemented on the controller.

[0083]

[0088] Example N. The user device according to any one of the preceding examples, wherein the second electronic device includes a gaming machine.

[0084]

[0089] Example O. The user device according to any one of the preceding examples, wherein the second electronic device includes a server computing device.

[0085]

[0090] Example P. The user device according to any one of the preceding embodiments, wherein the second set of state data is used by a second electronic device to cause an avatar to perform a second action after performing a first action.

[0086]

[0091] Example Q. The user device according to any one of the preceding embodiments, wherein the second set of state data is used by a second electronic device to modify a first action performed by an avatar.

[0087]

[0092] Example R. Receiving, at a user device, an input from a user of the user device related to a first action performed by an avatar, providing, on a first communication channel, a first set of state data generated from the received input to a second electronic device, clarifying, based on the received input, the intention of a user related to a second action performed by an avatar different from the first action, generating, based on the intention of the user, a second set of state data that, unlike the first set of state data, includes less state data than all available state data, providing the second set of state data to the second electronic device on a second communication channel different from the first communication channel, and causing the avatar to perform a modified action different from the first action based on the first and second sets of state data A non-transitory computer-readable medium storing, collectively, computer-executable instructions that, when executed, cause one or more computing devices to collectively perform acts including the above.

[0088]

[0093] Example S. The non-transitory computer-readable medium according to the preceding embodiments, wherein the intention of the user includes one of an interruption that prevents a first action being performed by an avatar, a modification to a first action being performed by an avatar, or a second action being performed after a first action being performed by an avatar.

[0089]

[0094] Example T. A non-transitory computer-readable medium according to any one of the preceding examples, wherein the first set of state data includes a set of predetermined data values and the second set of state data includes a set of data values selected based on the user's intention.

[0090] Conclusion

[0095] Although the subject matter has been described in language specific to features and methodological acts, it should be understood that the claimed subject matter is not necessarily limited to the specific features or acts described herein. Rather, the specific features and acts are disclosed as example forms of implementing the claimed subject matter.

Claims

1. Receiving, at a user device, game play input related to a first action performed by an avatar; Providing, on a first communication channel, to a computing device, first set of state data including a set of predetermined data values generated from the received input; Predicting an action intention of an action related to a second action performed by the avatar different from the first action, wherein predicting the action intention is based on a prediction model trained based on action data of past game play; Generating, a second set of state data including state data less than all available state data and different from the first set of state data, the second set of state data including a set of data values selected based on the predicted action intention; Providing, on a second communication channel different from the first communication channel, the second set of state data to the computing device; and Modifying the first action based on the first set of state data and the set of data values selected based on the predicted action intention A method for modifying an action of an avatar, comprising:

2. The first action is modified by an interruption that prevents the first action from being performed by the avatar, or Modifying the first action includes causing the avatar to perform a second action after the first action, the method according to claim 1.

3. The method according to claim 1, wherein the second set of state data includes at least one of an input vector, a magnitude of the input vector, or an input change.

4. The action data of past game play is associated with a user of the user device, or The prediction model is trained by machine learning to correlate game play input with game play actions for the avatar, the method according to claim 1.

5. A user interface for receiving game play input related to a first action performed by an avatar; A communication interface communicating via a communication network for providing, on a first communication channel, to a computing device, first set of state data including a set of predetermined data values generated from the received input; A processor that executes instructions stored in a memory, wherein the processor predicts the intention of an action related to a second action performed by the avatar that is different from the first action, and predicting the intention of the action is based on a prediction model trained based on action data of past game play, the predicting generating a second set of state data that includes state data that is less than all available state data and different from the first set of state data, and includes a set of data values selected based on the predicted intention of the action providing, via the communication interface, the second set of state data on a second communication channel different from the first communication channel to the computing device, and modifying the first action based on the first set of state data and the set of data values selected based on the predicted intention of the action A user device that executes instructions including the above to modify the action of the avatar.

6. The user device according to claim 5, further including a controller used to operate the avatar during the game play of the video game.

7. The user device according to claim 6, wherein the controller includes a virtual controller presented within a touch screen display.

8. The user device according to claim 6, wherein the game play input is related to one or more input mechanisms implemented on the controller.

9. The user device according to claim 5, wherein the computing device includes a game console or a game server computing device.

10. The second set of state data is used by the computing device to cause the avatar to perform the second action after performing the first action, or The first action is modified by an interruption that prevents the first action from being performed by the avatar, the user device according to claim 5.

11. Receiving, in a user device, game play input related to a first action performed by an avatar Providing, on a first communication channel, to a computing device, a first set of state data including a set of predetermined data values generated from the received input Predicting the intention of an action related to a second action performed by the avatar that is different from the first action, wherein predicting the intention of the action is based on a prediction model trained based on action data of past gameplay; Generating a second set of state data that includes a set of data values selected based on the predicted intention of the action, the second set of state data being different from the first set of state data and including less state data than all available state data; Providing the second set of state data to the computing device over a second communication channel that is different from the first communication channel; and Modifying the first action based on the first set of state data and the set of data values selected based on the predicted intention of the action A non-transitory computer-readable recording medium collectively storing computer-executable instructions that cause one or more computing devices to collectively perform, at runtime, a method for modifying an avatar's action, the method including the above steps.

12. The non-transitory computer-readable recording medium according to claim 11, wherein modifying the first action includes interrupting the first action being performed by the avatar based on the set of data values selected based on the intention of the action, or including one or more of a second action performed after the first action.

Images