Virtualized Physical Controller

Abstract

Techniques are described herein for implementing a virtualized physical controller. These techniques may include receiving one or more user inputs on a touchscreen display of a user device, the user inputs including a first touch that may define a primary input area, and one or more secondary touches offset from the first that may define secondary input areas arranged adjacent to or multiple to the primary input area. The input areas are associated with respective input mechanisms or commands in configuration data. While the configuration data is implemented, the touchscreen display may monitor touches, detect touch input in the configured areas, and generate activation data for each input mechanism or command in response to the touches. The positions and shapes of the input areas may be iteratively redefined as touch input is monitored on the touchscreen display.

Description

[Background technology]

[0001] background

[0001] Modern computer controller systems, such as those used by computer and video games and general purpose operating systems, employ a variety of techniques to direct the movement of objects displayed on a screen. Known techniques include using an external control device, such as a mouse, directional nub, touchpad, pen, game controller, or joystick, to generate a directional vector or to specify a location to move an on-screen object, such as a pointer or reticle, or to cause movement of the user's viewpoint.

[0002]

[0002] Some techniques can employ an additional layer of refinement by measuring the speed of movement of an external device to enhance the movement of an on-screen object by modifying the behavior of the on-screen object in response to a parameter of the input (e.g., the acceleration of a pointer based on the speed at which the external device is moved). Touch-enabled devices can also be configured to accept input in a manner that simulates the behavior of an external control device. However, the control schemes of touch-enabled devices tend to fall short of the tactile feel and responsiveness achieved with physical controllers, and further development in this field is needed.

[0003]

[0003] Although some touch-enabled control schemes are offered in existing games, for many users the transition from playing games with a physical video game controller to playing games on a flat screen device with physical buttons is not satisfactory. Users can quickly learn how to provide input to a well-designed physical controller without having to look at their hands to see which buttons they are pressing.

[0004]

[0004] A physical controller is optimized for a "resting" hand and thumb position that makes it easy to trigger adjacent buttons by muscle memory without looking, and has a tactile feel that makes it easy to determine where the controls are and what inputs have been used. In the absence of mechanical controls, improved touch-enabled control schemes are needed. Summary of the Invention

[0005] overview

[0005] Techniques are provided herein for implementing a virtualized physical controller, that is, a controller that uses software to replicate what is typically done with a physical hardware controller. A virtualized physical controller can be implemented via a number of buttons that are displayed on a touchscreen display of a user device (such as a mobile phone).

[0006]

[0006] In one embodiment, a method is disclosed that is performed by a user device, the method including receiving input from a user indicating one or more locations on a touch screen display associated with one or more input mechanisms, storing in configuration data an association between one or more regions on the touch screen display corresponding to the one or more locations and each of the one or more input mechanisms, determining that the configuration data is to be implemented, and once it is determined that the configuration data is to be implemented, implementing the configuration data by monitoring touch input on the touch screen display, detecting touch input within the one or more regions, and generating activation data for each of the one or more input mechanisms.

[0007]

[0007] One embodiment is directed to a computing system comprising a processor and a memory containing instructions that, when executed with the processor, cause the computing device to at least: receive input from a user of a controller computing device indicating one or more locations on a touch screen display associated with one or more input mechanisms; store in configuration data an association between one or more regions corresponding to the one or more locations on the touch screen display and each of the one or more input mechanisms; determine that the configuration data is to be implemented; and, when it is determined that the configuration data is to be implemented, implement the configuration data by monitoring touch input on the touch screen display, detecting touch input within the one or more regions, and generating activation data for each of the one or more input mechanisms.

[0008]

[0008] One embodiment is directed to a non-transitory computer readable medium that collectively stores computer-executable instructions that, when executed, cause one or more computing devices to collectively perform acts including receiving input from a user indicating one or more locations on a touch screen display associated with one or more input mechanisms; storing in configuration data an association between one or more regions corresponding to the one or more locations on the touch screen display and each of the one or more input mechanisms; determining that the configuration data is to be implemented; and, once it has been determined that the configuration data is to be implemented, implementing the configuration data by monitoring touch input on the touch screen display, detecting touch input within the one or more regions, and generating activation data for each of the one or more input mechanisms.

[0009]

[0009] An embodiment is directed to receiving one or more user inputs on a touch screen display of a user device associated with an input mechanism or command, the user input including a first touch that can define a primary input region, and one or more secondary touches offset from the first. In the configuration data, one or more secondary input regions adjacent to or radially arranged around the primary input region are defined based at least in part on the relative positions of the one or more secondary touches with respect to the first, and may further be defined based on the distance between the touches, a preset pattern of input regions stored in memory, or a user configuration input. Thus, the position or center of the primary input region can be determined based on the first touch, the position of the secondary input region can be determined based on a subsequent touch, and the relative size and shape of the primary and secondary input regions can be determined by the relative positions of the touches within a context obtained from the initial data (e.g., a selected avatar or character). The input regions are associated in the configuration data with the respective input mechanism or command. While the configuration data is implemented, the touchscreen display can monitor touches, detect touch input in the configured areas, and generate activation data for each input mechanism or command in response to the touch. The location and shape of the input areas can be iteratively redefined as touch input is monitored on the touchscreen display.

[0010]

[0010] The above, together with other features and embodiments, will become more apparent by reference to the following specification, claims, and accompanying drawings. The embodiments of the invention covered by this patent are defined by the following claims, not this Summary. This Summary is a high-level summary of various aspects of the invention and introduces some of the concepts that are further described in the "Detailed Description" section below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to the entire patent specification, any or all drawings, and appropriate portions of each claim.

[0011] BRIEF DESCRIPTION OF THE DRAWINGS The detailed description will now be set forth with reference to the accompanying drawings, in which the leftmost digit or digits of a reference number identify the figure in which the reference number first appears. Use of the same reference number in different figures indicates similar or identical objects or features. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a simplified system diagram illustrating a service environment in which a virtual controller can be used, according to various embodiments of the present disclosure. [Figure 2A]

[0013] 1 illustrates a first exemplary button configuration that may be implemented in accordance with an embodiment. [Figure 2B]

[0014] 1 illustrates a second exemplary button configuration that may be implemented in accordance with an embodiment. [Figure 2C]

[0015] 1 illustrates a third exemplary button configuration that may be implemented in accordance with an embodiment. [Diagram 3]

[0016] FIG. 2 is a block diagram illustrating various components of a computing system architecture that supports the implementation of a virtualized physical controller according to an embodiment. [Figure 4]

[0017] 1 depicts a graphic illustration of a process for configuring and implementing a virtualized physical controller in accordance with an embodiment. [Diagram 5]

[0018] 10 depicts graphic illustrations of various examples of virtual controller configurations that may be implemented in accordance with some embodiments. [Figure 6]

[0019] 1 depicts a graphic illustration of a process for adjusting a virtual controller configuration according to an embodiment. [Figure 7]

[0020] 1 illustrates a virtual controller implemented on a user device that can be used as a controller for a separate client device, according to an embodiment. [Figure 8]

[0021] 1 depicts a flow diagram showing an exemplary process flow for detecting a button press in a virtual controller according to an embodiment. [Figure 9]

[0022] 1 depicts a flow diagram showing an example process flow for matching primary and secondary button locations of a virtual controller, according to various embodiments. [Figure 10]

[0023] 1 depicts a flow diagram illustrating an exemplary process flow for creating and implementing a virtualized physical controller in accordance with an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Detailed Description

[0024] In the following description, various embodiments are described. For the purpose of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments can be practiced without these specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the described embodiments.

[0014]

[0025] Embodiments herein are directed to techniques for simulating the feel and functionality of input mechanisms (e.g., buttons on a physical video game controller) on a flat screen device. The disclosed methods can also be used to customize the on-screen layout of buttons based on user preferences, the particular application (e.g., game) being operated, the actual usage situation by the user, or the particular character or avatar being operated by the user. These techniques can be further extended by allowing users to customize parameters of the button layout based on personal preference or comfort.

[0015]

[0026] The embodiments of the present disclosure provide many advantages over conventional systems. While many conventional systems have attempted to adapt input mechanisms to touch screen devices, many problems remain in this field. For example, without the change in physical sensation of sliding a thumb across a controller and feeling the raised edge of a button, how would a user know that a button was there without looking at the flat screen? Without the tactile "click" of a physical button when a user presses a button on a flat screen, how would the user know that they had pressed it? Physical controllers are optimized for a "rest" hand and thumb position that makes it easy to trigger adjacent buttons by muscle memory without looking. How do you achieve this in a flat screen device?

[0016]

[0027] Some games implemented on touchscreen devices attempt to solve some of these problems by having a virtual button layout that is accessible with the right thumb (e.g., one can simply move the right thumb to touch an on-screen button), but rely on looking at the button placement and using peripheral vision to verify that the thumb is contacting the intended button (or the user touches the screen with the intention of triggering a button press, only to realize that no input is actually being provided and adjust their hand).

[0017]

[0028] Some games allow players to customize the location and size of on-screen buttons, making it easier for users to configure a layout that suits their hand size, but still rely on the player to use peripheral vision to determine button locations and provide limited feedback that a button press has occurred.

[0018]

[0029] In contrast, embodiments of the present disclosure involve registering a button as "pressed" when a user provides input in a designated button area (as opposed to "pressed" when the user touches and releases the button area). When a button is pressed, the system can provide haptic feedback, change the graphical representation of an icon, and / or play an audio effect. This means that a user can slide their right thumb across a flat screen without looking at the position of their thumb and know when they have triggered a button by haptic feedback and audio. In conjunction with this notification scheme, the layout of the buttons can be arranged to be custom to the application (e.g., game) or character that the user is controlling.

[0019]

[0030] For example, in some embodiments, a "primary button" is positioned across the natural center of the touchscreen area and is at least partially surrounded by adjacent buttons over which the user can slide their thumb radially without looking. The primary button may be associated with the character's primary ability, i.e., the button on which the user is expected to rest their thumb or tap repeatedly, while the radially arranged secondary buttons may correspond to secondary or secondary skills that are used less frequently or in conjunction with the primary ability. When used, with or without lifting, as the user slides their thumb from the primary button to the adjacent button, a button press is triggered, the previous button is released, and the user gets physical, auditory, and visual feedback to know that they have triggered a button. Importantly, for games where multiple characters are available, the button scheme can automatically change when a different character is selected without altering the feel of the controls. For example, the primary skill can remain in the center and the secondary button can change in response to the selection of the character at the time of use.

[0020]

[0031] 1 is a simplified system diagram illustrating a service environment 100 in which a virtual controller can be used, according to various embodiments of the present disclosure. The service environment 100 includes at least one server 101, which includes at least one processor 103 and non-transitory memory 105 for storing software instructions that facilitate operation of the service environment. The server 101 is connected via a network 121 (e.g., the Internet or a local network) to any suitable number of user-owned client devices 133, 143 that typically operate in conjunction with respective local user networks 131, 141 (e.g., consumer or commercial local area networks, WIFI networks, etc.).

[0021]

[0032] The server 101 may also be connected to any suitable number of control services 111, e.g., a networked computing system with its own processor 113 and memory 115 that monitors the network between the server 101 and the client devices 133, 143. In some embodiments, the server 101 may be one or more servers operating on a commercial scale, e.g., a data center or a server farm. The client devices 133, 143 may include, but are not limited to, consumer personal computers, video game consoles, thin client devices operable to stream video content from the server 101 for presentation on a local screen, or mobile devices such as smartphones or tablets. The client devices 133, 143 may be connected to any suitable number of controllers, e.g., controllers 135, 137, 145, 147.

[0022]

[0033] Each controller (e.g., controller 135) may be a hardware device (e.g., a console-specific controller, a cross-compatible controller, or a virtual controller) with connectivity hardware and protocols for communicating with a respective client device 133. According to some embodiments, controller 135 may be a virtualized controller operating on a thin client device or a touch screen device, such as a controller simulated on a touch screen smartphone, tablet, or console-like controller with a touch-enabled panel. According to some further embodiments, for example where client device 133 is a thin client device or a mobile device, controller 135 may be a touch screen with virtualized controls built into the client device. Alternatively, even if client device 133 is a thin client device, controller 135 may be a hardware controller configured to connect to the client device physically or wirelessly. According to some embodiments, client device 133 and server 101 may operate on the same hardware, e.g., client device runs as a virtual instance on the server.

[0023]

[0034] The methods described herein can be implemented on a client device in conjunction with a service environment, such as the service environment 100 described in Figure 1. Furthermore, the methods can function in the context of an arbitrary on-screen placement of virtual controllers that control both the orientation and movement of an avatar.

[0024]

[0035] For clarity, a certain number of components are shown in Figure 1. However, it is understood that embodiments of the present disclosure may include two or more of each component. In addition, some embodiments of the present disclosure may include fewer or more than all of the components shown in Figure 1. In addition, the components of Figure 1 may communicate over any suitable communications medium (including the Internet) using any suitable communications protocol.

[0025]

[0036] 2A-2C show exemplary button configurations 200a-c corresponding to a control scheme employing a virtual controller 235 in communication with a video game system 233, according to various embodiments. The displayed button arrangement 257 can vary for different characters in a game utilizing a touch screen interface 251.

[0026]

[0037] 2A illustrates a first exemplary button configuration that may be implemented in accordance with an embodiment. In the first configuration 200a as shown in FIG. 2A, a movement area 255 is present on the left side of a screen 251, and a button array 257 is displayed on the right side of the screen, although it will be understood that the left and right configurations may be swapped.

[0027]

[0038] In this example, the primary button 259 is located in an ergonomically central location relative to the touch screen 251 with respect to the location of the user's thumb, especially in embodiments in which the virtual controller 235 includes a handle 253 or similar ergonomic feature. Primary button 259 refers to an input area that may include a visible cue (i.e., a projected button) that is defined at least in part by the input area of ​​the primary button, although the input area of ​​the primary button may extend beyond what is visible in any on-screen cues. In some embodiments, the primary button may not be visible on the screen.

[0028]

[0039] Secondary buttons 261, which correspond to secondary input regions, are arranged in a pattern around the primary button 259, but like the primary buttons, the visual cues of the secondary buttons may not directly correspond to their input regions. That is, the exact boundaries of the visual cues may match the corresponding input regions or may be slightly offset. The input regions of the secondary buttons 261 may be bounded by a border 263 that is not visible to the user. In some embodiments, the secondary input regions radiate outward from the location of the primary button. Additional input regions may be stacked further away from the primary button.

[0029]

[0040] Figure 2B illustrates a second example button configuration that may be implemented in accordance with an embodiment, showing the same numbered elements as Figure 2A, with slight variations in the number, sizing, and location of buttons.

[0030]

[0041] In some cases, the primary button 259 may be in the "middle" of several auxiliary buttons 261 (e.g., as shown in both FIGS. 2A and 2B). The particular location, size, and / or shape of the primary button 259 may be modified in accordance with the techniques described herein.

[0031]

[0042] 2C illustrates a third exemplary button configuration that may be implemented in accordance with an embodiment. As illustrated in FIG. 2C, in some other cases, the primary button may occupy a corner of the touchscreen 251 and have a radial secondary button 261 positioned toward the interior of the touchscreen 251. In any case, the location, sizing, and number of buttons may be varied based on the situation. For example, the location, sizing, and number of buttons may be varied to correspond to the move sets of individual game characters or to meet the needs of an entirely different game.

[0032]

[0043] FIG. 3 is a block diagram illustrating various components of a computing system architecture supporting the implementation of a virtualized physical controller according to an embodiment. The system architecture may include at least one controller 302. In some embodiments, the controller 302 may communicate with one or more servers 304, which may be an example of the server 101 described with respect to FIG. 1. In some embodiments, the one or more servers 101 may provide back-end support for the controller 302. For example, at least some of the processing described as being performed by the controller 302 may, in some cases, be performed by the server 101 instead. In some embodiments, the controller 302 may communicate with a client device 306, which may be an example of the client device 133 or 143, as described with respect to FIG. 1 above. In some embodiments, the client device 306 may further communicate with a display device 308. Each of the components described herein may communicate by a connection via a network 310.

[0033]

[0044] The controller 302 may include any suitable computing device configured to perform at least some of the operations described herein and to allow a user to interact with a software application. In some embodiments, the controller may be a mobile device (e.g., a smartphone or tablet) with touch screen capabilities. The controller 302 may include a communication interface 312, one or more processors 314, memory 316, and hardware 318. The communication interface 312 may include wireless and / or wired communication components that enable the controller 302 to transmit data to and receive data from other networked devices. The hardware 318 may include additional user interface, data communication, or data storage hardware. For example, the user interface may include at least one output device 320 (e.g., a visual display, an audio speaker, and / or a haptic feedback device) and one or more data input devices 322. The data input device 322 may include, but is not limited to, one or more combinations of a keypad, a keyboard, a mouse device, a touch screen that accepts gestures, a microphone, a voice or speech recognition device, and any other suitable device.

[0034]

[0045] The memory 316 may be implemented using computer readable media such as computer storage media. Computer readable media includes at least two types of computer readable media: computer storage media and communication media. Computer storage media includes any suitable volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, DRAM, ROM, EEPROM, flash memory, or other memory technology, CD-ROM, digital versatile disks (DVDs), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other non-transmission media that can be used to store information for access by a computing device. In contrast, communication media may embody computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transmission mechanism.

[0035]

[0046] The controller's one or more processors 314 and memory 316 may implement functionality including one or more software modules and data stores. Such software modules may include routines, program instructions, objects, and / or data structures executed by the processor 314 to perform particular tasks or implement particular data types. In particular, the memory 316 may include modules (e.g., configuration management module 324) configured to maintain and implement configuration information for the input mechanisms of the virtualization controller.

[0036]

[0047] In addition, the memory 316 may include various data stores. For example, the memory 316 may maintain data regarding the configuration of the virtualization controller based on a context (e.g., configuration data 326). In some embodiments, the memory may further include data regarding user interactions with the virtualization controller (e.g., usage data 328).

[0037]

[0048] The configuration management module 324, in cooperation with the processor 314, may be configured to generate and manage configuration information associated with an arrangement of one or more input mechanisms in a user interface presented on the controller 302. In some embodiments, the configuration management module may be configured to receive input from a user indicating a location on a touchscreen display associated with a particular input mechanism (e.g., a button). In some cases, the user may be prompted to hold the controller, move a thumb or other finger to a position that is comfortable for the user, and touch the touchscreen display at that point. Upon detecting the user's touch on the touchscreen display, the configuration management module may be configured to associate the location of the touch with a particular input mechanism. A similar process may be performed for each of several input mechanisms. Once the configuration management module has collected input location data for each input mechanism, the location data may be stored in the memory 316 as configuration data. In some cases, the configuration data stored in the controller may be specific to a particular user, software application, or avatar.

[0038]

[0049] In some embodiments, the configuration management module may be invoked (e.g., by an executed software application) via an application programming interface (API). In some cases, information specific to the implementation of a virtual controller configuration may be passed from the software application to the configuration management module 324 via a call to the API. Upon activating a controller for use with a software application, such as a game, the configuration management module 324 may be configured to identify and implement an appropriate input mechanism configuration. In some embodiments, such implementation may be specific to some combination of the software application being executed, the user currently using the controller (e.g., identified by a user login, etc.), and / or one or more conditions specific to the software application (e.g., a particular character or avatar is being played). To implement a configuration, the configuration management module 324 may assign areas within the touch screen to particular input mechanisms such that when a touch is detected at a location, the configuration management module generates input corresponding to the input mechanism or mechanisms assigned to that location. Such input may then be relayed (e.g., via the API) to the software application. In some cases, activation data corresponding to the activated input mechanism is generated, and the activation data is relayed to the software application.

[0039]

[0050] In some embodiments, the activation data generated in response to the selection of a particular input mechanism may include information regarding how or to what extent the user activated the input mechanism. For example, if a touch by the user occurs within a region assigned to the input mechanism and the touch is then determined to move, the configuration management module may determine that the touch is a swipe motion. The generated activation data may indicate the swipe motion, as well as the distance of the swipe motion, the direction of the swipe, where the swipe ended, or any other suitable information. In some embodiments, if a swipe is initiated within a region associated with the input mechanism, the swipe motion may remain associated with the input mechanism even if the swipe is continued outside of that region.

[0040]

[0051] In some embodiments, the configuration management module may be further configured to update the configuration data. In some cases, such updates may occur in response to a user request. In other cases, such updates may occur automatically. For example, once a particular virtual controller configuration has been implemented, the configuration management module may monitor touch inputs entered by a user operating the controller. These touch inputs may be aggregated over time to evaluate whether the current location of the input mechanism is suitable. This is described in more detail below with respect to FIG. 6.

[0041]

[0052] Server 304 may include any computing device configured to perform at least a portion of the operations attributed thereto. Server 304 may be comprised of one or more general purpose computers, specialized server computers (including, by way of example, PC (personal computer) servers, UNIX servers, mid-range servers, mainframe computers, rack-mounted servers, etc.), server farms, server clusters, or any other suitable arrangement and / or combination. Server 304 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization, such as one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices of the computer. For example, server 304 may include virtual computing devices in the form of cloud-hosted virtual machines or software containers.

[0042]

[0053] Client device 306 may include any suitable computing device configured to receive input from controller 302 and perform actions based on the input. In some embodiments, a client device may be a gaming system, such as a gaming console, that can receive input from a number of controllers, each of which can be used to control an avatar or character within a software application (e.g., a computer game).

[0043]

[0054] 4 depicts a graphic illustration of a process for configuring and implementing a virtualized physical controller in accordance with an embodiment. In particular, FIG. 4 depicts a graphic illustration related to entering configuration data for user device 402, generating configuration data by assigning entered locations to various input mechanisms on user device 404, and implementing a virtualized physical controller using the generated configuration data for user device 406.

[0044]

[0055] As depicted at 402, the configuration data may be generated from input received from a user of a user device. In some embodiments, the input may include an indication of an identifier (e.g., a button name) associated with an input mechanism to be added to the configuration. For example, a user may select a button from a list of buttons presented to the user on a screen of the user device. Once an input mechanism to be added to the configuration data is identified, location data may be assigned to the input mechanism based on the received user input. In some embodiments, the user is provided with a prompt 408 to select a location within a touch screen display. In these embodiments, when the prompt is displayed, the user may respond by touching the touch screen at a location 410 that the user wishes to associate with the identified input mechanism. In other embodiments, the user may be asked to drag and drop a graphic illustration of a particular button to a location on the touch screen where the user wishes to associate the button.

[0045]

[0056] As depicted in 404, each of several input mechanisms may be assigned to a region 412 located on the touch screen display. In some embodiments, a size and / or shape may also be assigned to each of the regions for the input mechanisms. Note that in some cases, an input mechanism may be assigned to multiple different regions such that a resulting configuration may include multiple instances of the same input mechanism. Additionally, two or more input mechanisms may be assigned to overlapping regions such that each of the two or more input mechanisms may be triggered by selecting an area within the overlapping regions in the implemented configuration.

[0046]

[0057] The generated configuration may be implemented for an executed software application, as depicted at 406. When a software application, such as a game, is executed, a virtual controller may be instantiated on the user device. In this example, the software application may call an API associated with the implemented virtual controller. The call to the API may include information used to identify the particular virtual controller configuration implemented. In some embodiments, this may involve instantiating a graphical overlay on the user device over the presentation of the software application. In some embodiments, as depicted at 406(A), the visual representation of the input mechanism or their corresponding area may be made invisible to the user. In other embodiments, as depicted at 406(B), the visual representation 414 of the input mechanism or their corresponding area may be made visible to the user.

[0047]

[0058] When implemented, input provided to the user device may be monitored to determine when one or more input mechanisms are activated. In some embodiments, this may involve monitoring touch data input from the user and determining whether the location of the touch data is within an area associated with one or more input mechanisms. In some cases, the user may touch the screen with a finger within one or more areas associated with the input mechanism. In some cases, the user may touch the screen with a finger and drag the finger across one or more areas associated with the input mechanism. Upon detecting that an input mechanism has been selected, the virtual controller may be configured to provide feedback to the user indicating the selection of the input mechanism. For example, the virtual controller may provide auditory (e.g., audible), visual, or haptic feedback to a user of the virtual controller. In some cases, the feedback provided to the user may differ based on the identity of the selected input mechanism. The virtual controller may further be configured to provide user input to a software application based on which input mechanisms have been selected and when those input mechanisms have been selected.

[0048]

[0059] 5 depicts graphical illustrations of various examples of virtual controller configurations that may be implemented according to some embodiments. In the depicted example, a first exemplary controller 502 is depicted implementing a virtual controller having multiple instances of one or more input mechanisms. In addition, a second exemplary controller 504 is depicted implementing a virtual controller having two regions associated with different overlapping input mechanisms.

[0049]

[0060] As previously mentioned, the stored virtual controller configuration data may include a mapping of correlations between certain regions located on the touchscreen display and certain input mechanisms. In some embodiments, one or more input mechanisms may be mapped to multiple different, separate regions on the touchscreen display. Thus, when a virtual control configuration is implemented as described herein, multiple instances (e.g., 506) of a single input mechanism may be displayed at different locations on the touchscreen display. In particular, each of the different instances of the input mechanism may be implemented at their corresponding locations. In the illustrated virtual controller example, a user may touch the touchscreen display at a first location 508(A) and drag a finger to a second location 508(B). In this example, each input mechanism between the first location 508(A) and the second location 508(B) may be activated when the user's finger enters the region associated with the respective input mechanism. When an input mechanism is activated in this manner, a signal specific to that input mechanism is generated and provided to a running software application, which may then interpret the signal.

[0050]

[0061] It should be noted that embodiments of a virtualized physical controller that can include multiple instances of a single input mechanism can provide significant advantages over traditional controllers. For example, a user who can place multiple instances of an input mechanism in a line can swipe across them instead of tapping on multiple instances. In another example, if a user places two different input mechanism instances along a line, swiping along the line will generate an input that would typically require tapping two different buttons in turn. In these examples, it should be noted that implementing multiple instances of an input mechanism in this manner can make the controller easier to operate and facilitate its use by individuals with disabilities.

[0051]

[0062] In some embodiments, the virtual controller configuration may include one or more regions associated with different input mechanisms that overlap. For example, an input region 510 associated with a first input mechanism may overlap an input region 512 associated with a second input mechanism. In this example, an area 514 may result from the overlap between the two regions. Upon detecting that a user has touched the touch screen display within area 514, each of the first and second input mechanisms may be activated simultaneously. Note that this also has an advantage over physical controllers, as such controllers typically require two buttons to be pressed simultaneously to achieve the same result.

[0052]

[0063] 6 depicts a graphic illustration of a process for adjusting a virtual controller configuration according to an embodiment. The process 600 is depicted on a series of images of user devices 602 (A-C) in which a virtual controller may be implemented. The virtual controller configuration data may include an indication of a region 604 associated with an input mechanism.

[0053]

[0064] In some embodiments, the virtual controller configuration data may be adjusted or updated at various times. In some embodiments, the virtual controller configuration data may be updated upon receiving an update request. In some embodiments, the virtual configuration data may be updated automatically (e.g., without human interaction) once it is determined that an update should occur.

[0054]

[0065] In some embodiments, usage data 606 may be maintained over a period of time that includes information about user touch location inputs provided by a user during manipulation of a virtualized physical controller on the user device 602(A). Such usage data 606 may include locations that correspond to locations on a touch screen display that a user touched when interacting with a particular input mechanism.

[0055]

[0066] In some embodiments, usage data may be aggregated to generate a usage distribution 608 that represents the distribution of users' touch probabilities across areas on the user device 602(B). Such usage distribution 608 may be centered around where users are most likely to touch the touchscreen display when attempting to activate an input mechanism.

[0056]

[0067] In some embodiments, a distance between the center of usage distribution 608 and the center of region 604 may be determined. In these embodiments, if the determined distance is greater than a threshold distance, a determination may be made that the virtual controller configuration data should be updated to move the location of region 604. If a determination is made that the location of the region should be updated, region 604 may be moved to be centered on usage distribution 610. It should be noted that other aspects of the regions indicated in the configuration data may be updated in this manner. For example, the size and / or shape of the regions may be updated in addition to or instead of the location of such regions.

[0057]

[0068] Using the techniques described herein, the locations of the regions associated with the input mechanisms stored in the virtual controller configuration data may be updated as the user's usage of the virtual controller changes. For example, as a user grows, the placement of the user's hands on the user device may change and the locations of various regions may need to be updated.

[0058]

[0069] 7 depicts an illustration of a virtual controller implemented on a user device that may be used as a controller for a separate client device according to an embodiment. As depicted in system 700, a user device 702 implementing a virtualized physical controller can communicate with a client device 704, such as a gaming console. In some embodiments, the client device 704 can further communicate with a display device 706, such as a television.

[0059]

[0070] In some embodiments, a device discovery process may be performed by the user device. Once such a process is performed, the user device may discover the client device and a communication session may be established between the user device 702 and the client device 704. In some embodiments, the client device 704 may instruct the user device to obtain a software application that facilitates interaction between the two. For example, upon establishing communication with the user device, the client device may instruct the user device to download a software application associated with the client device. Thereafter, communication between the user device and the client device may be facilitated by the software application.

[0060]

[0071] In embodiments, when the virtualized physical controller is instantiated on a user device, it may be used to communicate input received from a user via a touchscreen display to the client device, which may then be used to control a character in a video game running on the client device.

[0061]

[0072] 8 depicts a flow diagram showing an example process flow 800 for detecting a button press in the virtual controller of FIGURES 2A-2C. Process 800 may be performed in any suitable service environment, including but not limited to the service environment 100 shown in FIGURE 1.

[0062]

[0073] According to various embodiments, process 800 includes detecting a first user input in a primary button area of ​​a touch screen device by detecting a touch in the primary button area at 801. The system can then generate a first haptic, auditory, or visual signal (or any suitable combination thereof) in response to detecting the first user input at 802. A second user input in a secondary button area of ​​the touch screen device can be detected by detecting that a touch has moved from the primary button area to the secondary button area of ​​the touch screen device at 803, which can cause the system to generate a second haptic, auditory, or visual signal in response to detecting the second user input at 804. The first and second user inputs are registered as commands to be communicated to the gaming system at 805.

[0063]

[0074] According to various embodiments, the primary button area is arranged in a natural location for a user's thumb when the user is holding the virtualized controller, and the secondary button area includes one of a number of auxiliary button areas arranged radially around the primary button area, however, the exact positioning of the areas can be modified to fit the user's hand, as follows:

[0064]

[0075] FIG. 9 depicts a flow diagram showing an example process flow 900 for adapting a primary button location and an auxiliary button location of a virtual controller, according to various embodiments. The process 900 may be performed in any suitable service environment, including but not limited to the service environment 100 shown in FIG. 1. For example, the process 900 may include a system generating a first instruction for presentation to a user, at 901, requesting the user to generate a first training touch by placing a thumb in a natural location. In response to detecting the first training touch by the user, the system may set a location of the primary button area based on a natural location selected by the user, at 902. The system may generate a second instruction for presentation to the user, at 903, requesting the user to generate a second training touch by placing a thumb in a second location (e.g., a secondary button location) different from the natural location. In response to detecting the second training touch by the user, the system may then set a location of the secondary button area, at 904.

[0065]

[0076] The system can refine the position, size, and location of the auxiliary buttons (i.e., the "spread" of the control scheme) based on repeated calibration requests as described above until a control scheme that minimizes the need for the user to look is achieved. Thus, the positioning and sizing of the buttons can be determined in part (at least initially) via a training module in which the user is instructed to place a control thumb or finger on the screen in a natural location without looking at the screen, and then to move the control thumb or finger in one or more directions from the natural location in a pattern corresponding to the number of auxiliary buttons. One or more presses at the "natural" location instruct the module to define the center of the primary button, and one or more presses at the auxiliary locations instruct the module to define the edge of the primary button, and thus the area corresponding to the auxiliary buttons.

[0066]

[0077] In some embodiments, the user's touch input may be monitored over time during the user's operation of the virtualized controller. For example, each time a touch input is received in association with one or more of the input mechanisms, the location of the touch input is recorded. Samples of the touch input location associated with each input mechanism may be used to update and / or define the region associated with the respective input mechanism, as described with respect to FIG. 6 above. In some cases, the region associated with each input mechanism may be automatically updated (e.g., without user interaction) as the controller is used. Such use may occur during operation of the controller even when the controller is not being operated in a training state.

[0067]

[0078] 10 depicts a flow diagram illustrating an example process flow 1000 for generating and implementing a virtualized physical controller according to an embodiment. Process 1000 may be performed by a computing device configured to generate and provide a product strategy for a product. For example, process 1000 may be performed by an operations optimization platform, such as operations optimization platform 104 described with respect to FIG. 1 above.

[0068]

[0079] At 1002, process 1000 includes receiving input from a user indicating one or more locations on a touchscreen display, the one or more locations intended to be associated with an input mechanism. In some embodiments, the input received from the user includes a touch on the touchscreen display by a user holding the user device in a natural grip. In some embodiments, the input received from the user includes an indication that the user has dragged and dropped one or more input mechanisms onto a respective one or more locations on the touchscreen display. In some embodiments, in addition to information regarding the location stored in association with a region associated with the input mechanism, information regarding the size, orientation, and / or shape of the region may also be stored.

[0069]

[0080] At 1004, process 1000 includes defining a primary input area based on a first location of the first touch. In some embodiments, the first location of the first touch may be located at a location on the touch screen device that can be comfortably reached by a user's thumb or finger.

[0070]

[0081] At 1006, the process 1000 includes defining one or more secondary input regions based at least in part on relative positions of the one or more second touches with respect to the first touch. In some embodiments, the one or more secondary input regions may be arranged radially around the primary input region. In some embodiments, the one or more second touches may be arranged a predetermined distance from a center of the first position.

[0071]

[0082] At 1008, the process 1000 includes saving an association between the region corresponding to the indicated location and the corresponding input mechanism in the configuration data. In some embodiments, the configuration data includes multiple separate regions associated with a single input mechanism of the one or more input mechanisms. In some embodiments, one or more regions associated with at least two of the one or more input mechanisms in the configuration data may overlap. In such a case, when a touch input is detected to be located within the overlap, an input for each of the at least two input mechanisms is generated.

[0072]

[0083] At 1010, process 1000 includes determining that the configuration data is to be implemented. In some embodiments, the configuration data is determined to be implemented based at least in part on at least one of a software application being executed, an identity of a user, or a character being played in the software application. In some embodiments, determining that the configuration data is to be implemented includes receiving a request to implement the configuration data via an application programming interface by a software application executing on the user device. In these embodiments, the configuration data may be implemented via a graphical overlay presented on the software application executing on the user device.

[0073]

[0084] At 1012, the process 1000 includes implementing a virtualized physical controller according to the configuration data. This may involve monitoring touch input on a touch screen display, detecting touch input within one or more regions, and generating input corresponding to at least one respective input mechanism of the one or more input mechanisms. The activation data may be any data intended to be generated upon activation of a particular input mechanism. For example, when it is determined that a particular button has been pressed or otherwise activated by a user, activation data may be generated that includes an identifier for that button. In some embodiments, a determination may be made that the touch input includes a swipe motion based on the movement associated with the touch data. In these embodiments, the activation data may further include information regarding the distance of the swipe motion, the direction of the swipe, or the end location of the swipe.

[0074]

[0085] In some embodiments, process 1000 further includes providing feedback to the user upon detecting touch input within one or more regions. Such feedback may include any combination of haptic, auditory, or visual signals. In some cases, feedback may be provided as soon as a user's touch is detected within a region associated with the input mechanism (as opposed to when the user releases the touch).

[0075]

[0086] In some embodiments, process 1000 may further involve automatically updating the configuration data (e.g., without human interaction). In such embodiments, the touch input may be aggregated and / or stored as usage data. Updating the configuration data may involve generating distribution data based on the usage data, and updating one or more positions associated with the input mechanism based on the distribution data. In some embodiments, the one or more positions associated with the input mechanism are updated to be centered on the generated distribution data. In some embodiments, the configuration data is updated if the distance between the one or more positions and the center of the distribution data is greater than a threshold distance.

[0076]

[0087] The methods described herein are directed to virtual controllers, i.e., controllers that use a touch screen or touch screen-like functionality to provide an easily customized controller button layout. According to some embodiments, the touch screen is at least a part of a physical handheld controller that interfaces with a gaming device, such as a gaming console, personal computer, tablet, smartphone, thin client device (e.g., a USB or HDMI device plugged into a screen). According to some embodiments, the touch screen is a primary feature of a controller that interfaces with a gaming device, such as a gaming console, personal computer, tablet, smartphone, thin client device (e.g., a USB or HDMI device plugged into a screen). According to some embodiments, the controller is made on a mobile device or tablet along with enabling software that connects the mobile device or tablet to a gaming device, such as a gaming console, personal computer, thin client device (e.g., a USB or HDMI device plugged into a screen), or other suitable gaming device. According to some further embodiments, the touch screen is a touch-enabled screen of a gaming device, such as a gaming console, personal computer, tablet, or smartphone.

[0077]

[0088] The specification and drawings are to be regarded in an illustrative rather than a restrictive sense, but it will be apparent that various modifications and changes may be made thereto without departing from the broader spirit and scope of the disclosure as set forth in the appended claims.

[0078]

[0089] Other variations are within the spirit of the disclosure. Thus, while the disclosed techniques are susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and described in detail above. It is to be understood, however, that there is no intention to limit the invention to the particular form or forms disclosed, but on the contrary, the intention is to cover all modifications, alternative constructions, and equivalents falling within the spirit and scope of the invention as defined by the appended claims.

[0079]

[0090] In the context of describing the disclosed embodiments (particularly in the context of the claims that follow), the use of the terms "a," "an," and "the," and similar referents, are to be construed to cover both the singular and the plural, unless otherwise indicated herein or clearly contradicted by context. The terms "comprising," "having," "including," and "containing" are to be construed as open-ended terms (i.e., meaning "including, but not limited to"), unless otherwise indicated. The term "connected" is to be construed as being partly or wholly contained within, attached to, or joined to, even if there is something intervening. The recitation of ranges of values ​​herein, unless otherwise indicated herein, is intended merely to serve as a shorthand method of referring individually to each separate value falling within the range, and each separate value is incorporated herein as if it were individually set forth herein. All methods described herein can be performed in any suitable order unless otherwise indicated herein or clearly contradicted by context. Any examples provided herein, or the use of exemplary language (e.g., "such as"), are intended merely to facilitate understanding of embodiments of the invention and do not limit the scope of the invention unless otherwise claimed. No language in the specification should be construed as indicating any non-claimed element as essential to the practice of the invention.

[0080]

[0091] Preferred embodiments of the present disclosure are described herein, including the best mode known to the inventors for carrying out the invention. Variations of these preferred embodiments may become apparent to those skilled in the art upon reading the foregoing description. The inventors anticipate that such variations will be adopted by those skilled in the art as appropriate, and the inventors intend the invention to be practiced otherwise than as specifically described herein. Accordingly, this invention includes all modifications and equivalents of the subject matter recited in the claims appended hereto as permitted by applicable law. Additionally, any combination of the above-described elements in all possible variations thereof is encompassed by this invention unless otherwise indicated herein or clearly contradicted by context.

[0081]

[0092] In the following, further examples are set forth to facilitate understanding of aspects of the present invention.

[0082]

[0093] Example A. Receiving input from a user indicating a location on a touchscreen display of a user device associated with two or more input mechanisms, the input including a first touch on the touchscreen display and one or more second touches offset from the first touch; defining, in the configuration data, a primary input area based on a first position of the first touch; and defining, in the configuration data, one or more secondary input areas radially arranged around the primary input area based at least in part on relative positions of the one or more second touches with respect to the first touch; storing, in the configuration data, an association between (a) the primary input area and one or more secondary input areas, and (b) each input mechanism of the two or more input mechanisms; Determining that the configuration data is to be implemented; Once it has been decided that configuration data will be implemented, monitoring touch input on a touch screen display; Detecting touch input within one or more regions; generating activation data for each input mechanism of the two or more input mechanisms; implementing the configuration data by A method comprising:

[0083]

[0094] Example B. The method of the preceding example, further comprising providing feedback to a user when input is detected within one or more regions.

[0084]

[0095] Example C. The method of any preceding example, wherein the feedback includes at least one of a haptic signal, an auditory signal, or a visual signal.

[0085]

[0096] Example D. A method as in any one of the preceding examples, wherein the configuration data is determined to be implemented based at least in part on at least one of the software application being executed, the identity of a user, or a character being played in the software application.

[0086]

[0097] Example E. The method of any one of the preceding examples, wherein the configuration data includes a plurality of distinct regions associated with a single input mechanism of the one or more input mechanisms.

[0087]

[0098] Example F. The method of any one of the preceding examples, wherein one or more secondary input areas overlap the primary input area or an additional secondary input area.

[0088]

[0099] Example G. The method of any one of the preceding examples, wherein activation data for each of at least two of the input mechanisms is generated when a touch input is detected to be located within the overlap.

[0089]

[0100] Example H. The method of any one of the preceding examples, wherein the input received from the user includes a touch on the touch screen display by the user holding the user device in a natural grip.

[0090]

[0101] Example I. The method of any one of the preceding examples, wherein the input received from the user includes an indication that the user has dragged and dropped one or more input mechanisms onto a respective one or more locations on the touch screen display.

[0091]

[0102] Example J. A processor; When executed by a processor, the method comprises at least receiving, from a user of a user device, input indicating two or more locations on a touch screen display associated with two or more input mechanisms, the input including a first touch on the touch screen display and one or more second touches offset from the first touch; defining, in the configuration data, a primary input area based on a first location of the first touch; defining, in the configuration data, one or more secondary input areas radially arranged about the primary input area based at least in part on relative positions of the one or more second touches with respect to the first touch; storing, in the configuration data, (a) the primary and secondary input regions, and (b) an association between each primary input mechanism and each of one or more secondary input mechanisms of the two or more input mechanisms; Determining that the configuration data is to be implemented; Once it has been decided that configuration data will be implemented, monitoring touch input on a touch screen display; Detecting touch input within one or more regions; generating activation data for each input mechanism of the one or more input mechanisms; implementing the configuration data by a memory containing instructions for causing the user device to A user device comprising:

[0092]

[0103] Example K. A user device as described in any one of the preceding examples, wherein the user device is configured to communicate with a gaming console, and the activation data is sent to the gaming console.

[0093]

[0104] Example L. The user device of any one of the preceding examples, wherein the instructions further cause the touch input to be stored in the user device as usage data, and the configuration data is updated based on the usage data.

[0094]

[0105] Example M. A user device as described in any one of the preceding examples, wherein updating the configuration data includes generating distribution data based on the usage data and updating one or more positions associated with the input mechanism based on the distribution data, the distribution data being generated by repeatedly sampling touch input from a user of the user device.

[0095]

[0106] Example N. A user device as described in any one of the preceding examples, wherein updating one or more positions associated with the input mechanism includes updating a first location of the primary input area to be centered on the generated distribution data, and updating at least one second location of the secondary input area.

[0096]

[0107] Example O. The user device of any one of the preceding examples, wherein at least one of a size or shape of the one or more secondary regions is updated based on the distribution data.

[0097]

[0108] Example P. The user device of any one of the preceding examples, wherein the configuration data is updated if the distance between one or more locations and the center of the distribution data is greater than a threshold distance.

[0098]

[0109] Example Q. The user device of any one of the preceding examples, wherein the instructions further cause the user device to detect that the touch input includes a swipe motion based on a movement associated with the touch data.

[0099]

[0110] Example R. At runtime, receiving input from a user indicating two or more locations on a touch screen display associated with two or more input mechanisms; defining two or more input regions based on the input, the two or more input regions including at least a primary input region based on a first location of a first touch and one or more secondary input regions arranged radially around the primary input region based on a second location of a second touch; storing, in the configuration data, a respective association between each of the two or more regions and a respective input mechanism of the two or more input mechanisms; Determining that the configuration data is to be implemented; Once it has been decided that configuration data will be implemented, monitoring touch input on a touch screen display; Detecting touch input within one or more regions; generating activation data for each input mechanism of the one or more input mechanisms; implementing the configuration data by A non-transitory computer-readable medium that collectively stores computer-executable instructions that collectively cause one or more computing devices to perform actions including:

[0100]

[0111] Example S. A non-transitory computer-readable medium as described in any one of the preceding examples, wherein determining that the configuration data is to be implemented includes receiving a request to implement the configuration data via an application programming interface by a software application running on the user device.

[0101]

[0112] Example T. The non-transitory computer-readable medium of any one of the preceding examples, wherein the configuration data is implemented via a graphical overlay presented on a software application executing on a user device.

[0102] conclusion

[0113] Although the subject matter has been described in language specific to features and methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described herein. Rather, the specific features and acts are disclosed as example forms of implementing the set of claims.

Claims

1. 1. A method for simulating a physical controller, comprising: receiving touch inputs at different locations on a touch screen of a user device, the touch inputs including a first touch at a first location and at least one second touch at a second location offset from the first location; determining which of a plurality of different configurations should be implemented on the touchscreen, each of the plurality of different configurations determining an arrangement of input regions within the touchscreen and mapped to a respective set of input mechanisms; determining that the first location and the second location of the touch input are located within one or more of the input regions of an arrangement defined by the configuration; generating activation data based on an input mechanism mapped to an input region in which the first position and the second position are located; iteratively redefining a position and shape of at least one input area within the touch screen according to the touch input; A method comprising:

2. The method of claim 1 , further comprising generating feedback regarding activation of two or more of the input mechanisms mapped to the input area.

3. The method of claim 2 , wherein the feedback comprises at least one of a haptic signal, an auditory signal, or a visual signal.

4. The method of claim 1 , wherein the configuration is determined based at least in part on at least one of a software application being executed, a user's identity, or a character being played in the software application.

5. The method of claim 1 , wherein the arrangement defined by the determined configuration includes multiple distinct input regions mapped to the same input mechanism.

6. the input area defined by the determined configuration includes a primary input area and one or more secondary input areas; The method of claim 1 , wherein the secondary input area overlaps the primary input area or another secondary input area.

7. the first location or the second location of the touch input is located within the overlap; The method of claim 6 , wherein the activation data is generated based on the overlap.

8. The layout of the input area corresponds to a natural grip of a user; The method of claim 1 , further comprising calibrating the determined configuration for the user by refining a position, a size or an extent of the input area.

9. 10. The method of claim 1, further comprising mapping the determined configuration based on input including an indication of dragging and dropping one or more of the input mechanisms onto one or more input areas within the touch screen.

10. A user device for simulating a physical controller, comprising: a touch screen for receiving touch input at different locations; a processor for executing instructions stored in a memory; the touch input includes a first touch at a first location and at least one second touch at a second location offset from the first location; The processor executes the instructions to: determining which of a plurality of different configurations should be implemented on the touchscreen, each of the plurality of different configurations determining an arrangement of input regions within the touchscreen and mapping to a respective set of input mechanisms; determining that the first location and the second location of the touch input are located within one or more of the input regions of an arrangement defined by the configuration; generating activation data based on an input mechanism mapped to an input region in which the first position and the second position are located; A user device that iteratively redefines a position and shape of at least one input area within the touch screen according to the touch input.

11. The user device of claim 10, further comprising a communications interface that transmits the activation data to a game console over a communications network.

12. a memory for storing usage data corresponding to the touch input; The user device of claim 10 , wherein the determined configuration is updated based on the usage data.

13. The user device of claim 12 , wherein the processor further executes instructions to generate distribution data by repeatedly sampling touch input from a user of the user device based on the usage data, and to update the input region based on the distribution data.

14. 14. The user device of claim 13, wherein the processor updates the input area by centering a primary input area and updating a position of a secondary input area based on the generated distribution data.

15. The user device of claim 14 , wherein a size or shape of the one or more secondary regions is updated based on the distribution data.

16. 14. The user device of claim 13, wherein the processor further executes instructions to update the determined configuration based on identifying a distance between one or more of the first and second positions and a center of the distribution data that is greater than a threshold distance.

17. The user device of claim 10 , wherein the processor further executes instructions to detect that the touch input includes a swipe motion based on a movement associated with the touch input.

Images