METHOD AND USER DEVICE FOR STATE-BASED ACTIVATION - Patent application

Abstract

Described herein are techniques for implementing a virtualized physical controller. The techniques may include receiving a first touch input from a user via a touch screen display corresponding to at least one input mechanism of a set of input mechanisms, determining a currently active first state, and generating first activation data based on the first state including an indication of a first action. The techniques may further include receiving a second touch input from a user via the touch screen display corresponding to the at least one input mechanism, determining a currently active second state, and generating second activation data based on the second state including an indication of a second action different from the first action.

Description

[Background technology]

[0001]

[0001] Modern computer controller systems, used for example by computer and video games, as well as general purpose operating systems, employ various techniques to direct the movement of objects displayed on a screen. Known techniques include the use of an external manipulation device, such as a mouse, directional nub, touchpad, pen, game controller, or joystick, to create a directional vector, specify a location for moving an on-screen object, such as a pointer or reticle, or cause a movement of the user's eye point. Some techniques may employ an additional layer of refinement by measuring the speed of movement of an external device to improve the movement of an on-screen object by varying the behavior of the on-screen object depending on a parameter of the input (e.g., the acceleration of a pointer based on the speed at which the external device is moved). Touch-enabled devices may also be configured to accept inputs to simulate the behavior of an external manipulation device. However, the manipulation schemes of touch-enabled devices tend to fall short of the tactile feel and responsiveness that have been achieved with physical controllers, and further developments in this field are needed. For example, some touch-enabled manipulation schemes have been demonstrated in existing games, but existing manipulation schemes cannot take advantage of the flexibility afforded by virtualization. Summary of the Invention

[0002]

[0002] Presented herein are techniques for implementing a virtual controller in which input mechanisms may be dynamically updated based on a current state as the state is updated. In embodiments, actions associated with one or more buttons of the controller may change when different buttons are pressed and / or different actions are performed.

[0003] In one embodiment, a method is disclosed that is executed by a user device, the method including receiving a first touch input from a user via a touch screen display corresponding to at least one input mechanism of a set of input mechanisms, determining a currently active first state, and generating first activation data including an indication of a first action based on the first state. The method further includes receiving a second touch input from a user via the touch screen display corresponding to the at least one input mechanism, determining a currently active second state, and generating second activation data including an indication of a second action different from the first action based on the second state.

[0004]

[0004] One embodiment is directed to a computing system comprising a processor and memory containing instructions that, when executed with the processor, cause the computing device to at least: receive a first touch input from a user via the touch screen display corresponding to at least one input mechanism of a set of input mechanisms; determine a currently active first state; generate first activation data including an indication of a first action based on the first state; receive a second touch input from a user via the touch screen display corresponding to the at least one input mechanism; determine a currently active second state; and generate second activation data including an indication of a second action different from the first action based on the second state.

[0005]

[0005] Certain embodiments are directed to a non-transitory computer-readable medium that collectively stores computer-executable instructions that, when executed, cause one or more computing devices to perform acts including receiving a first touch input from a user via a touch screen display corresponding to at least one input mechanism of a set of input mechanisms, determining a currently active first state, and generating first activation data including an indication of a first action based on the first state. The acts further include receiving a second touch input from a user via the touch screen display corresponding to the at least one input mechanism, determining a currently active second state, and generating second activation data including an indication of a second action different from the first action based on the second state.

[0006]

[0006] The above, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings. The embodiments of the invention covered by this patent are defined by the following claims, not this summary. This summary is a high-level overview of various aspects of the invention and introduces some of the concepts that are further described in the Detailed Description section below. This summary is not intended to identify key or essential features of the subject matter defined in the claims, nor is it intended to be used alone to determine the scope of the subject matter defined in the claims. The subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim. [Brief description of the drawings]

[0007]

[0007] The detailed description will now be described with reference to the accompanying drawings, in which the left-most digit of a reference number identifies the figure in which the reference number first appears. Use of the same reference number in different figures indicates similar or identical items or features.

[0008] [Figure 1]

[0008] FIG. 1 is a simplified system diagram showing a service environment in which a virtual controller may be used according to various embodiments of the present disclosure. [Diagram 2]

[0009] 1 illustrates an illustrative example of an environment including a virtual controller in communication with a video game system according to various embodiments. [Diagram 3]

[0010] FIG. 2 is a block diagram illustrating various components of a computing system architecture that supports the implementation of a virtualized physical controller according to an embodiment. [Figure 4]

[0011] FIG. 2 is a block diagram illustrating a process for implementing a dynamic state-based controller according to an embodiment. [Diagram 5]

[0012] 1 illustrates a graphical representation of a process for dynamically adjusting input mechanisms (i.e., buttons) on a virtual controller according to an embodiment. [Figure 6]

[0013] FIG. 1 shows a block diagram illustrating a process for providing dynamic state changes of buttons used to control an avatar, according to some embodiments. [Figure 7]

[0014] FIG. 1 illustrates a flow diagram showing an exemplary process flow for generating and implementing a virtualized physical controller according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009]

[0015] In the following description, various embodiments are described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it will be apparent to one skilled in the art that the embodiments may be practiced without the specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the described embodiments.

[0010]

[0016] Embodiments herein are directed to techniques for providing a virtual controller configuration that can be dynamically adapted based on state. For example, when a virtual controller is implemented in a video game, this may include dynamically changing the appearance and functionality of virtual buttons based on the state and interaction history of different virtual buttons. This method uses user input (if the user is currently touching a button), tracked state of past actions, and the state of the user's "avatar" being controlled within the game.

[0011]

[0017] Embodiments of the present disclosure offer several advantages over conventional systems: For portable flat screen devices that may be used as virtual controllers (such as mobile phones), screen real estate is at a premium. For games that are intended to be played with the device held horizontally (or "landscape"), there is limited space to provide two-handed input to the game, but ample space to view actual in-game gameplay (for mobile games) or to view control schemes and auxiliary information (for console or PC games played with a virtual controller).

[0012]

[0018] This space limitation means that games need to pay attention to the number and size of UI elements on the screen. When combined with the physical constraints of the human hand (e.g., the size of the player's thumb, the length of the thumb nail, the orientation and angle of the thumb knuckle), there are tough constraints and tradeoffs around the desire to provide more means to input information (more buttons for more complex input actions), the size and location of those buttons (which affects how easy it is for the user to physically touch the buttons), and the "screen real estate" remaining for viewing the in-game action. Incidentally, a typical modern console game traditionally uses eight buttons for input, often with the expectation that the player will use multiple buttons simultaneously (e.g., right thumb + right index finger simultaneously).

[0013]

[0019] Finally, the goal of input design is to enable "muscle memory" based input. After a short training / instruction on how to operate, given a request ("I want to make my in-game avatar jump"), the user should not need to physically look at where the on-screen button is located to know which button to press. The user's thumb should automatically and instinctively go to the correct place to touch the virtual button. Most mobile games approach these problems by reducing the number of buttons (often by reducing the complexity of the game), adding many on-screen buttons that are transparent (so as not to block the in-game action), or dynamically changing the text of the on-screen buttons based on the game state.

[0014]

[0020] In contrast, the present disclosure leverages the current state of the virtual button (pressed or not pressed), the previous state of the virtual button (pressed a short time ago), and the state of the player's avatar (e.g., "Can my character do X right now?") to minimize the number of buttons without reducing complexity or requiring changes to the game state.

[0015]

[0021] 1 is a simplified system diagram illustrating a service environment 100 in which a virtual controller may be used, according to various embodiments of the present disclosure. The service environment 100 comprises at least one server 101, which comprises at least one processor 103 and a non-transitory memory 105 that stores instructions as software to facilitate operation of the service environment. The server 101 is connected via a network 121 (e.g., the Internet or a local network) to any suitable number of user-owned client devices 133, 143 that typically operate on corresponding local user networks 131, 141 (e.g., consumer or commercial local area networks, WIFI networks, etc.).

[0016]

[0022] The server 101 may also be connected to any suitable number of control services 111, e.g., networked computing systems with their own processors 113 and memory 115, that monitor the network between the server 101 and the client devices 133, 143. In some embodiments, the server 101 may be one or more servers running on a commercial scale, e.g., in a data center or server farm. The client devices 133, 143 may include consumer personal computers, video game consoles, thin client devices operable to stream video content from the server 101 for presentation on a local screen, or mobile devices such as smartphones, tablets, etc. The client devices 133, 143 may be connected to any suitable number of controllers, e.g., controllers 135, 137, 145, 147.

[0017]

[0023] Each controller (e.g., controller 135) may be a hardware device (e.g., console-specific controller, mutually compatible controller, or virtual controller) with connection hardware and protocols for communicating with its corresponding client device 133. According to some embodiments, controller 135 may be a virtualized controller running on a thin client device or touch screen device, such as a touch screen smartphone, a controller simulated on a tablet, or a console-like controller with a touch-enabled panel. According to some further embodiments, for example when client device 133 is a thin client device or a mobile device, controller 135 may be a touch screen with virtualized control means built into the client device. Alternatively, even if client device 133 is a thin client device, controller 135 may be a hardware controller configured to connect to the client device physically or wirelessly. According to some embodiments, client device 133 and server 101 may run on the same hardware, e.g., client device runs as a virtual instance on the server.

[0018]

[0024] The methods described herein may be implemented on a client device associated with a service environment, such as the service environment 100 illustrated in Figure 1. The methods may further function in conjunction with any arrangement of virtual controllers that control both the orientation and movement of an avatar on the screen.

[0019]

[0025] For clarity, a certain number of components are shown in Figure 1. However, it is understood that embodiments of the present disclosure may include more than one of each component. Also, some embodiments of the present disclosure may include fewer or more than all of the components shown in Figure 1. Also, the components of Figure 1 may communicate over any suitable communications medium (including the Internet) and using any suitable communications protocol.

[0020]

[0026] 2 illustrates an illustrative example of an environment 200 including a virtual controller 235 in communication with a video game system (e.g., gaming console) 233, according to various embodiments. The virtual controller 235 includes a touch screen 251, a frame 253, and virtualized controls, e.g., 255 and 257. Imagine, for example, two virtual buttons ("dash" being the first virtual button 259 and "jump" being the second virtual button 261), which can be triggered independently (the user can press and hold either button with their thumb).

[0021]

[0027] When the "sprint" button is pressed in 257b (e.g., touch 263 appears on the border of the "sprint" button), the next action may be to change the "sprint" button's appearance (icon texture), haptic feedback, sound, and if the controlled avatar can perform a "long jump", the character's state may change to "sprinting", and a successful touch may be indicated by one or more of the following: the text on the "jump" button 261 may change to "long jump" alone or in combination with a change in the button's color or texture. In response to the "sprint" command, the player's avatar begins sprinting. In 257c, as the user slides their thumb from "sprint" towards "long jump", the appearance (icon texture) of the "sprint" button changes to give the user immediate feedback indicating that they are no longer touching (returning to default state), but that they are no longer touching "sprint", but that a timer is started to track when they stopped pressing the button. When the user's thumb presses the "long jump" button at 257d, a successful touch is indicated by one or more of changing the "long jump" button's appearance (icon texture), haptic feedback, or sound, and that the player's avatar now performs a "long jump" in response to a command from the second button. Thus, the behavior of the second button changes dynamically based on a combination of recent button presses and the player avatar's status without changing the overall game state.

[0022]

[0028] When the "sprint" timer expires, the character may return from the "sprinting" state, the text on the "long jump" button may revert back to "jump", and the player's avatar stops sprinting. When the user removes their thumb from the "jump" button in 257e, the appearance (icon texture) of the "jump" button similarly reverts to a default state, showing the default state of both buttons. In this way, the user can now provide input that causes the avatar to "jump" or "long jump" without explicitly requesting an additional button for the "long jump" and without deviating from the expected control scheme (i.e., related tasks like "jump" and "long jump" may be contextually dependent on whether the character is sprinting). These methods differ from combination-based context cues in that in traditional games, button presses are simultaneous (e.g., directional control with the left hand plus button control with the right hand) or the change in button behavior cannot be signaled by changing the virtual button when the contextual requirements are met (e.g., "chord" or "combo" input with the right hand).

[0023]

[0029] FIG. 3 is a block diagram illustrating various components of a computing system architecture supporting implementation of a virtualized physical controller according to an embodiment. The system architecture may include at least one controller 302. In some embodiments, the controller 302 may be in communication with one or more servers 304, which may be an example of the server 101 described with respect to FIG. 1. In some embodiments, the one or more servers 101 may provide back-end support to the controller 302. For example, at least a portion of the processing described as being performed by the controller 302 may be performed instead by the server 101, in some cases. In some embodiments, the controller 302 may be in communication with a client device 306. The client device 306 may be an example of the client device 133 or 143 described above with respect to FIG. 1. In some embodiments, the client device 306 may further be in communication with a display device 308. Each of the components described herein may be in communication via a connection through a network 310.

[0024]

[0030] The controller 302 may include any suitable computing device configured to perform at least some of the operations described herein and to allow a user to interact with a software application. In some embodiments, the controller may be a mobile device (e.g., a smartphone or tablet) with touch screen capabilities. The controller 302 may include a communication interface 312, one or more processors 314, memory 316, and hardware 318. The communication interface 312 may include wireless and / or wired communication components that enable the controller 302 to transmit and receive data to and from other network devices. The hardware 318 may include additional user interface, data communication, or data storage hardware. For example, the user interface may include at least one output device 320 (e.g., a visual display, an audio speaker, and / or a haptic feedback device) and one or more data input devices 322. The data input device 322 may include one or more combinations of a keypad, a keyboard, a mouse device, a touch screen that accepts gestures, a microphone, a speech or voice recognition device, and any other suitable devices, and the like.

[0025]

[0031] The memory 316 may be implemented using a computer readable medium such as a computer storage medium. Computer readable media includes at least two types of computer readable media: computer storage media and communication media. Computer storage media includes any suitable volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data. Computer storage media includes RAM, DRAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that may be used to store information accessed by a computing device. In contrast, communication media may embody computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism.

[0026]

[0032] The controller's one or more processors 314 and memory 316 may implement functionality including one or more software modules and data stores. Such software modules may include routines, program instructions, objects, and / or data structures executed by the processor 314 to perform particular tasks or implement particular data types. More specifically, the memory 316 may include modules (e.g., state management module 324) configured to determine a current state of the controller and implement input mechanism configurations based on such state, and modules (e.g., configuration management module 326) configured to maintain and implement configuration information for the input mechanisms of the virtualized controller.

[0027]

[0033] Memory 316 may also include various data stores. For example, memory 316 may maintain data (e.g., configuration data 328) regarding the virtualized controller configuration based on the context. In some embodiments, such configuration data may include an indication of one or more aspects of the input mechanisms to be implemented based on the state. For example, the configuration data may include an indication of the size, position, shape, appearance (e.g., color, shading, and / or text) of each input mechanism associated with a particular state. In some examples, the configuration data may indicate which input mechanisms should or should not be presented during a particular state.

[0028]

[0034] The state management module 324, in cooperation with the processor 314, may be configured to implement one or more input mechanisms based on the detected state and configuration data. In some examples, the state management module is configured to monitor for state changes. Such state changes may occur when an input mechanism is actuated or when some other condition is met. Upon detecting a state change, the state management module may be configured to determine the current state(s) of the controller or a software application in communication with the controller and implement each appropriate input mechanism based on the state. This may include providing an indication of which input mechanisms should be presented and how each of those input mechanisms should be presented.

[0029]

[0035] The configuration management module 326, in cooperation with the processor 314, may be configured to generate and manage configuration information regarding the placement of one or more input mechanisms within a user interface presented on the controller 302. In some embodiments, the configuration management module facilitates customization of input mechanism layouts in accordance with some embodiments. It should be noted that such customization is described in related Patent Cooperation Treaty (PCT) Application No. US2022 / 019240 by Gregory Peng, entitled "Virtualized Physical Controller," which is incorporated herein by reference in its entirety.

[0030]

[0036] In some embodiments, the configuration data may be customized by the user to indicate the layout / appearance of the input mechanisms presented by the virtual controller, and such layout / appearance information stored in the configuration data for each input mechanism may be customized by the user on a per-state basis. In other words, how each input mechanism is presented for each state (its appearance, position, size, shape, or other suitable factors) may be determined, at least in part, by the input provided by the user.

[0031]

[0037] Server 304 may comprise any computing device configured to perform at least a portion of the operations that occur thereby. Server 304 may be comprised of one or more general purpose computers, specialized server computers (including, by way of example, PC (personal computer) servers, UNIX servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other suitable configuration and / or combination. Server 304 may comprise one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization, such as one or more flexible pools of logical storage devices that may be virtualized to maintain the virtual storage devices of the computer. For example, server 304 may comprise a virtual computing device in the form of a virtual machine or software container hosted in the cloud.

[0032]

[0038] Client device 306 may include any suitable computing device configured to receive input from controller 302 and perform actions based on the input. In some embodiments, a client device may be a gaming system, such as a gaming console, that may receive input from multiple controllers, each of which may be used to control an avatar or character within a software application (e.g., a computer game).

[0033]

[0039] 4 shows a block diagram illustrating a process for implementing a dynamic state-based controller according to an embodiment. Process 400 may be executed on a user device on which a virtual physical controller, such as controller 302 described above with respect to FIG.

[0034]

[0040] At 402, process 400 may include detecting a current state applicable to a virtual controller being operated. In some embodiments, state data may be provided to the virtual controller by a software application interacting through the virtual controller. In some embodiments, multiple states may be applicable at one time. For example, if the software application is a video game, one state may correspond to a character currently being played, while a second state may correspond to an action currently being performed by that character.

[0035]

[0041] At 404, process 400 may include presenting a button (or other input mechanism) based on the detected current state. In some embodiments, a set of configuration data for the virtual controller may include an indication of how each button should be presented based on various states. In the configuration data, each button may be associated with a default appearance. Also, each button may be assigned different appearances based on various detected states. In some examples, each button may be assigned a different size, position, and / or shape based on the current decision state. In some embodiments, one or more of the buttons may be implemented (e.g., visible and interactable) only during certain states.

[0036]

[0042] At 406, process 400 may include monitoring for a change in state. In some embodiments, the change in state may be detected upon actuation of an input mechanism. For example, when a player presses a button intended to effect an action, a state corresponding to that action may be initiated. In some embodiments, one or more states may be temporary, in that they are set to a current state for a predetermined period of time, in which case the state may end at the end of the predetermined period of time.

[0037]

[0043] At 408, process 400 may include determining whether a change in state has occurred (e.g., as detected at 406). If such a change is detected in the current state, the process returns to 404 and the rendering of one or more buttons of the virtual controller may be implemented for the newly detected state. In some examples, the appearance of one or more buttons may be updated based on the newly detected state. For example, text on a button may be changed or otherwise updated based on information included in the configuration data.

[0038]

[0044] If a change in state is not detected, the process 400 may include monitoring for a user input (e.g., touch input) at 410. In an embodiment, a user input may be detected upon detecting a user touch originating within one or more areas associated with the implemented buttons. In particular, when a user touch input is detected, a determination may be made as to whether the touch input is within one or more areas associated with one or more of the currently implemented buttons. Based on whether touch input is detected from a user and whether the touch input is within one or more areas of the currently implemented buttons, a determination may be made as to whether an actuation of at least one button is detected at 412. If no actuation is detected, the process may include continuing to monitor for a change in state at 406.

[0039]

[0045] At 414, if actuation of at least one button is detected, the process 400 may include generating activation data based on the button actuation. The activation data may include an indication of the one or more buttons whose actuation was detected. A determination may be made as to which action should be initiated by the button actuation based on which one or more states are currently active. In some embodiments, configuration data for the virtual controller may maintain information regarding which action should be initiated in which state upon each button actuation. The generated activation data may include an indication of the action associated with the button actuation in the current detected state.

[0040]

[0046] In some examples, the activation data may further include information regarding the degree or amount associated with the actuation, such as the length of time the user touched the button, the direction and / or length of a swipe action made by the user in relation to the button, the length of time the user touched the button, or any other suitable information. Once the activation data is generated, the process may return to monitoring for changes in state at 406.

[0041]

[0047] 5 shows a graphical representation of a process for dynamically adjusting input mechanisms (i.e., buttons) on a virtual controller according to an embodiment. The process 500 is illustrated in a series of images 502 (AC) of a user device on which the virtual controller may be implemented. As shown, the virtual controller may be implemented via a graphical user interface (GUI) 504 presented on a touch screen display.

[0042]

[0048] At 502(A), a first state may be detected for the virtual controller. In the illustrated example, configuration data associated with the virtual controller may indicate information for implementing one or more input mechanisms on the virtual controller in each state. In some embodiments, if the implementation of a particular input mechanism is conflicting for each of two different states determined to be active, the configuration data may further indicate which implementation should be prioritized when the two states are detected.

[0043]

[0049] In some embodiments, a user may provide a touch input to the input mechanism shown at 506. Upon detecting the touch input, a determination may be made to initiate a first action based on the detected first state, and activation data may be generated to initiate the action. Once the action has been initiated, a second state may be detected.

[0044]

[0050] As an example, consider a scenario in which a virtual controller is implemented on a mobile device running a video game. In this scenario, a character in the video game may be in a "standing" state (i.e., a first state). A user may touch a button (e.g., an input mechanism) that corresponds to performing a "jump" action. In this example, when the user touches the jump button, the character may initiate a jump action and a second "jump" state (i.e., a second state) may be detected. In this scenario, if there is a conflict between the implementation of a particular button in the configuration data for the "standing" state and the "jump" state, it may be resolved in favor of the "jump" state.

[0045]

[0051] At 502(B), a second state may be detected by the virtual controller. In some examples, this state may be additive to the first state (i.e., both states may be active simultaneously). In other examples, the states may be mutually exclusive in that activation of one state may supersede the current state. Upon detecting the second state, the virtual controller may determine whether one or more of the input mechanisms require updating. To do this, it may consult the configuration data to determine implementation data for each input mechanism in the configuration data. Based on such implementation data, the virtual controller may update the position, size, shape, appearance (e.g., text), and / or action associated with the input mechanism.

[0046]

[0052] As shown, upon detecting the second state, the virtual controller may update the implemented input mechanisms. In some examples, this may include adding or showing input mechanisms (e.g., input mechanism 508) that may be active while the second state is active. In some examples, this may include removing, disabling, or hiding input mechanisms (e.g., input mechanism 510) that may not be active while the second state is active. In some examples, this may include changing the appearance of one or more input mechanisms (e.g., input mechanism 512).

[0047]

[0053] At 502(C), a determination may be made that the second state is no longer active. In some examples, this may result from another state replacing the second state. In some examples, this may result from the second state terminating after a predetermined time associated with the second state has elapsed (e.g., if the second state is a temporary state). In some embodiments, the second state may persist as long as the user continues to provide touch input to the button. For example, a "dash" state may persist as long as the user presses the "dash" button.

[0048]

[0054] As an example, consider a continuation of the exemplary scenario described above. Upon detecting that the "jump" state is active, the button configuration may be updated by the virtual controller such that each of the buttons in the current button configuration are implemented to be associated with the jump state. When one or more of those buttons are touched, an action may be performed that is different from the action taken in the standing state. In this example, the jump state may persist as long as the character is jumping, which may depend on a certain amount of time the character spends jumping, how much the user has pressed the jump button, or any other suitable factor. Continuing with this exemplary scenario, once the jump is completed, the state may return to the first "standing" state. Each of the buttons may then be reimplemented according to the first state.

[0049]

[0055] FIG. 6 illustrates a block diagram showing a process for effecting dynamic state changes of buttons used to manipulate an avatar according to some embodiments. Process 600 may be performed in any suitable service environment, such as the service environment 100 illustrated in FIG. 1. According to various embodiments, process 600 includes sensing 601 a first touch input to a touch screen device at a location corresponding to a first button, where the system may cause the player avatar to perform a first action based on the first touch input at 602, and assign 603 a status to the player avatar based on the first touch input that persists for a non-zero time after release of the first touch input, e.g., the player's avatar is "sprinting" on the screen while the "sprint" button is assigned a status "sprinting" that persists for a period of time after the "sprint" button is released.

[0050]

[0056] In process 600, the behavior of the second button is temporarily modified from a default behavior to a modified behavior in response to the status assigned in 604, e.g., "jump" is changed to "long jump", "attack" is changed to "charge attack", etc. The displayed state of the second button, i.e., the name displayed to the user, may also be changed from an initial state to a modified state during the status assigned in 605, thus changing not only the behavior of the button but also its visible properties.

[0051]

[0057] When the system senses a second touch input at 606 at a second location corresponding to the second button, provided the behavior of the second button remains modified (i.e., within a period of time after the release of the first button), the system will read the second input and cause the player avatar to perform the modified action of the second button instead of the default action based on the second touch input at 607. In contrast, if the modified behavior has timed out, the system will instead cause the player avatar to perform the default action corresponding to the second button.

[0052]

[0058] 7 illustrates a flow diagram illustrating an example process flow 700 for generating and implementing a virtualized physical controller according to an embodiment. Process 700 may be performed by a computing device configured to generate and provide a product strategy for a product. For example, process 700 may be performed by a controller that can facilitate interaction between a user and a software application, such as controller 302 described above with respect to FIG. 3. In some embodiments, such a software application is a video game played by a user.

[0053]

[0059] At 702, process 700 includes receiving a first touch input corresponding to one input mechanism of a set of input mechanisms implemented in the controller. As described below, the controller may include a touchscreen display, and the first touch input (and the second touch input) may be received from a user via the touchscreen display. In some embodiments, the user device is one of a smartphone or a tablet device.

[0054]

[0060] At 704, the process 700 includes determining a first state currently applicable to the controller. In some embodiments, the first state may be a default state or a state in which the software application is launched.

[0055]

[0061] At 706, the process 700 includes generating first activation data based on the actuation of the input mechanism and the first state. The first activation data may include information regarding one or more actions triggered by the input mechanism.

[0056]

[0062] At 708, the process 700 includes receiving a second touch input corresponding to an input mechanism implemented in the controller. Like the first touch, the second touch may be received via a touch input provided on a touch screen display of the controller.

[0057]

[0063] At 710, process 700 includes determining a second state currently applicable to the controller. In some embodiments, the second state may be initiated upon actuation of a different input mechanism of the set of input mechanisms. For example, a user may press or touch a different button, which initiates the second state. In some examples, the states may be additive in that both the first state and the second state are active at the same time. In some examples, the states may be exclusive in that only one of the first state or the second state is active at the same time, with the second state replacing the first state.

[0058]

[0064] In some embodiments, the controller may detect when the second state is initiated. Additionally, when determining that the second state is currently active, the process may include updating at least one of a size, a position, a shape, or an appearance of the input mechanism. In some embodiments, the process may further include, when determining that the second state is currently active, updating the set of input mechanisms by adding at least one additional input mechanism to the set of input mechanisms during the time period that the second state is active. For example, one or more additional buttons (or other suitable input mechanisms) may be added to the virtual controller while the second state is active. In some embodiments, the process may further include, when determining that the second state is currently active, updating the set of input mechanisms by removing at least one additional input mechanism from the set of input mechanisms during the time period that the second state is active. For example, one or more buttons (or other suitable input mechanisms) may be removed or otherwise deactivated while the second state is active. In some embodiments, each of the first state and the second state is associated with a status of an avatar played by the user in the video game.

[0059]

[0065] At 712, the process 700 includes generating second activation data based on the actuation of the input mechanism and the second state. In an embodiment, one or more actions indicated in the second activation data are different from one or more actions indicated in the first activation data.

[0060]

[0066] In some embodiments, the second state may be temporary in that it lasts for a non-zero amount of time. In some examples, the second state lasts for a predetermined amount of time. In some examples, the second state lasts as long as a condition is met (e.g., as long as the user continues to press a button). When the second state ends, the virtual controller may deactivate the second state and revert to activating the first state. In such examples, the process may further include receiving a third touch input, determining that the second state is no longer active, and generating third activation data that includes an indication of the first action.

[0061]

[0067] The methods described herein are directed to virtual controllers, i.e. controllers that use touch screens or touch screen-like features to provide easily customizable controller button layouts. According to some embodiments, the touch screen is at least a portion of a physical handheld controller that interfaces with a gaming device, such as a gaming console, personal computer, tablet, smartphone, thin client device (e.g., USB or HDMI device plugged into a screen). According to some embodiments, the touch screen is a primary feature of the controller that interfaces with a gaming device, such as a gaming console, personal computer, tablet, smartphone, thin client device (e.g., USB or HDMI device plugged into a screen). According to some embodiments, the controller comprises a mobile device or tablet in conjunction with executable software that connects the mobile device or tablet to a gaming device, such as a gaming console, personal computer, thin client device (e.g., USB or HDMI device plugged into a screen). According to some further embodiments, the touch screen is a touch-enabled screen of a gaming device, such as a gaming console, personal computer, tablet, or smartphone.

[0062]

[0068] The specification and drawings are to be regarded in an illustrative rather than a restrictive sense, however it will be apparent that various modifications and changes thereto may be made without departing from the broader spirit and scope of the present disclosure as set forth in the appended claims.

[0063]

[0069] Other variations are within the spirit of the present disclosure. Thus, while the disclosed technology is susceptible to various modifications and alternative constructions, several illustrated embodiments of the disclosed technology are shown in the drawings and have been described above in detail. It should be understood, however, that there is no intention to limit the invention to the particular form(s) disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions and equivalents falling within the spirit and scope of the invention as defined in the appended claims.

[0064]

[0070] Use of the terms "a," "an," and "the," and similar referents in the context of describing the disclosed embodiments (especially in the context of the claims that follow) should be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "containing" should be construed as open-ended terms (i.e., meaning "including, but not limited to"), unless otherwise stated. The term "connected" should be construed as partially or wholly contained within, attached to, or joined together, even if there is something intervening. Recitations of ranges of values ​​herein are intended to serve merely as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated herein as if it were individually recited herein. All methods described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. Any and all examples provided herein, or the use of exemplary language (e.g., "such as"), are intended merely to better illuminate embodiments of the invention, and do not impose limitations on the scope of the invention unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the invention.

[0065]

[0071] Preferred embodiments of the present disclosure are described herein, including the best mode for carrying out the invention known to the inventors. Variations of these preferred embodiments may become apparent to those skilled in the art upon reading the foregoing description. The inventors expect that those skilled in the art will adopt such variations as appropriate, and the inventors intend to practice the invention otherwise than as specifically described herein. Accordingly, this invention includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the invention unless otherwise indicated herein or otherwise clearly contradicted by context.

[0066]

[0072] In the following, further examples are provided to facilitate understanding of aspects of the present invention.

[0067]

[0073] Example A. Receiving a first touch input from a user via a touch screen display, the first touch input corresponding to at least one input mechanism of a set of input mechanisms; Determining a currently active first state; generating first activation data based on the first state, the first activation data including an indication of a first action; receiving a second touch input from a user via the touch screen display corresponding to the at least one input mechanism; Determining a currently active second state; and generating second activation data based on the second state, the second activation data including an indication of a second action different from the first action; A method comprising:

[0068]

[0074] Example B. The method of example A, further including updating at least one of a size, a position, a shape, or an appearance of the at least one input mechanism when determining that the second state is currently active.

[0069]

[0075] Example C. The method of Examples A or B, further including, when determining that the second state is currently active, updating the set of input mechanisms by adding at least one additional input mechanism to the set of input mechanisms while the second state is active.

[0070]

[0076] Example D. The method of any of Examples A to C, further including, when determining that the second state is currently active, updating the set of input mechanisms by excluding at least one additional input mechanism from the set of input mechanisms while the second state is active.

[0071]

[0077] Example E. The method of any of examples A through D, wherein the first state and the second state are both determined to be active simultaneously.

[0072]

[0078] Example F. The method of any of Examples A through E, wherein the second state replaces the first state such that only one of the first state or the second state is active at a time.

[0073]

[0079] Example G. The method of any of examples A through F, wherein the second state is initiated upon actuation of a different input mechanism of the set of input mechanisms.

[0074]

[0080] Example H. The method of any of Examples A through G, wherein the second condition persists for a predetermined period of time.

[0075]

[0081] Example I. Receiving tertiary touch input determining that the second state is no longer active; and generating third activation data including an indication of the first action; The method of any of Examples A through H, further comprising:

[0076]

[0082] Example J. A user device, comprising: A processor; When executed by the processor, the user device receiving a first touch input from a user corresponding to at least one input mechanism of a set of input mechanisms; Determining a currently active first state; generating first activation data based on the first state, the first activation data including an indication of a first action; receiving a second touch input from a user corresponding to the at least one input mechanism; Determining a currently active second state; and generating second activation data based on the second state, the second activation data including an indication of a second action different from the first action; a memory including instructions to at least: A user device comprising:

[0077]

[0083] Example K. A device further comprising a touch screen display; The user device of example J, wherein the first touch input and the second touch input are received from the user via the touch screen display.

[0078]

[0084] Example L. The user device of examples J or K, where the user device is one of a smartphone or a tablet device.

[0079]

[0085] Example M. The user device of any of Examples J to L, wherein the instructions further cause the user device to initiate a second state upon receiving a third touch input corresponding to a second input mechanism different from at least one input mechanism of the set of input mechanisms.

[0080]

[0086] Example N. The user device of any of Examples J-M, including a virtual controller where the instructions can facilitate interaction between the user and the software application.

[0081]

[0087] Example O. The user device of any of Examples J through N, wherein the software application includes a video game played by the user.

[0082]

[0088] Example P. The user device of any of Examples J-O, wherein each of the first state and the second state relates to a status of an avatar being played by the user in a video game.

[0083]

[0089] Example Q. The user device of any of Examples J to P, wherein the instructions further cause the user device to update at least one of a size, position, shape, or appearance of the at least one input mechanism when the instructions determine that the second state is currently active.

[0084]

[0090] Example R. Receiving a first touch input from a user via a touch screen display, the first touch input corresponding to at least one input mechanism of a set of input mechanisms; Determining a currently active first state; generating first activation data based on the first state, the first activation data including an indication of a first action; receiving a second touch input from a user via the touch screen display corresponding to the at least one input mechanism; Determining a currently active second state; and generating second activation data based on the second state, the second activation data including an indication of a second action different from the first action; A non-transitory computer-readable medium that collectively stores computer-executable instructions that, when executed, collectively cause one or more computing devices to perform actions that include:

[0085]

[0091] Example S. The non-transitory computer-readable medium of example R, further including updating at least one of a size, a position, a shape, or an appearance of the at least one input mechanism upon determining that the second state is currently active.

[0086]

[0092] Example T. The non-transitory computer-readable medium of example R or S, wherein the second state is initiated upon actuation of a different input mechanism of the set of input mechanisms.

[0087] Conclusion

[0093] Although the subject matter has been described in language specific to features and methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. Determining a first game state that is currently active; generating first activation data including an indication of a first action based on the determined first game state; receiving a first touch input on a touch screen display having an indication of the first action, the first touch input having a position corresponding to a position of at least one input mechanism of a set of input mechanisms presented on the touch screen display; determining a currently active second game state different from the first game state in response to the first touch input; generating second activation data based on the determined second game state, the second game state being different from the first game state, the second activation data including an indication of a second action different from the first action; receiving a second touch input on the touch screen display with an indication of the second action, the second touch input having a location corresponding to a same location of the at least one input mechanism; A method for state-based activation, including:

2. The method of claim 1, further comprising: updating at least one of a size, position, shape, or appearance of the at least one input mechanism based on the determined second game state; updating the set of input mechanisms by adding an input mechanism to the set of input mechanisms presented on the touch screen display while the second game state is active; or updating the set of input mechanisms by removing an input mechanism from the set of input mechanisms presented on the touch screen display while the second game state is active.

3. The method of claim 1 , wherein the first game state and the second game state are both determined to be active simultaneously.

4. the second game state replaces the first game state such that only the second game state is active; or the second game state is initiated upon actuation of a different input mechanism of the set of input mechanisms presented on the touch screen display; or The method of claim 1 , wherein the second game state lasts for a predetermined period of time.

5. determining that the second game state is no longer active; generating third activation data including an indication of the first action based on determining that the second game state is no longer active; receiving a third touch input on the touch screen display with an indication of the first action, the third touch input having a location corresponding to a same location of the at least one input mechanism; The method of claim 1 further comprising:

6. A user device comprising a processor and a memory, The memory includes: When executed by the processor, the user device is Determining a first game state that is currently active; generating first activation data including an indication of a first action based on the determined first game state; receiving a first touch input on a touch screen display having an indication of the first action, the first touch input having a position corresponding to a position of at least one input mechanism of a set of input mechanisms presented on the touch screen display; determining a currently active second game state different from the first game state in response to the first touch input; generating second activation data based on the determined second game state, the second game state being different from the first game state, the second activation data including an indication of a second action different from the first action; receiving a second touch input on the touch screen display with an indication of the second action, the second touch input having a location corresponding to a same location of the at least one input mechanism; A user device comprising instructions to at least:

7. The user device of claim 6, further comprising the touch screen display.

8. The user device of claim 6 , wherein the user device is one of a smartphone or a tablet device.

9. The user device of claim 6, wherein the touch screen display receives a third touch input at a position corresponding to a position of a second input mechanism different from the at least one input mechanism of the set of input mechanisms presented on the touch screen display.

10. The user device of claim 6 , wherein the instructions are associated with a software application specific virtual controller.

11. the software application comprises a video game; The user device of claim 10 , wherein each of the first game state and the second game state is associated with a status of an avatar in the video game.

12. 7. The user device of claim 6, wherein the instructions further cause the user device to update at least one of a size, a position, a shape, or an appearance of the at least one input mechanism when determining that the second game state is currently active.

13. Determining a first game state that is currently active; generating first activation data including an indication of a first action based on the determined first game state; receiving a first touch input on a touch screen display having an indication of the first action, the first touch input having a position corresponding to a position of at least one input mechanism of a set of input mechanisms presented on the touch screen display; determining a currently active second game state different from the first game state in response to the first touch input; generating second activation data based on the determined second game state, the second game state being different from the first game state, the second activation data including an indication of a second action different from the first action; receiving a second touch input on the touch screen display with an indication of the second action, the second touch input having a location corresponding to a same location of the at least one input mechanism; A non-transitory computer-readable medium that collectively stores computer-executable instructions that, when executed, collectively cause one or more computing devices to perform actions that include:

14. 14. The non-transitory computer-readable medium of claim 13, further comprising updating at least one of a size, a position, a shape, or an appearance of the at least one input mechanism upon determining that the second game state is currently active.

15. 14. The non-transitory computer-readable medium of claim 13, wherein the second game state is initiated upon actuation of a different input mechanism of the set of input mechanisms presented on the touch screen display.

Images