Game controllers that can operate in Bluetooth® Low Energy (BLE) mode

Abstract

To provide a hand-held controller that avoids an optional operating system limit that is applied to a game controller input in the case where it cannot be operated.SOLUTION: A hand-held controller is paired with a BLE device that executes a client application used for a video game streaming when it is operated in a BLE mode. In a game play, a video game is executed on a host computer, a player provides a user input to a game controller to control an aspect of the video game. In responce to such a user input, the game controller transmits a controller input data to the BLE device via a radio device of the game controller. The player uses an input gesture on the game controller to operate the game controller in the BLE mode, and switches an operation of the game controller in a receiver mode.SELECTED DRAWING: Figure 1A

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a PCT application claiming priority to U.S. Patent Application No. 16 / 370,656, filed March 29, 2019, entitled "GAME CONTROLLER OPERABLE IN BLUETOOTH LOW ENERGY (BLE) MODE," which is hereby incorporated by reference in its entirety. [Technical Field]

[0002] [Background technology] In-home video game streaming allows a user to play a video game on one computer (the target device) while the video game is actually running on another computer (the host computer) elsewhere in the home environment. An exemplary setup involves a personal computer (PC) (the host computer) located in a home office running the video game, capturing video game data (e.g., audio and video data), and streaming the video game data over a home network to a living room television (the target device). In this scenario, a player can use a handheld video game controller to play the video game from a living room couch while the video game content is rendered on the television. If the living room television does not have a suitable receiver, a hub device can act as an intermediary, receiving video game data from the PC and providing the video game data to the living room television via an audio / video interface. The hub device can also receive controller input data from the game controller and provide the controller input data upstream to the PC. Another setup involves using a universal serial bus (USB) dongle as a wireless receiver that can be plugged into a target device with a suitable USB port, such as a laptop. The dongle receives controller input data provided by the game controller and enables gameplay on a target device that streams video game data from a PC over a home network. Using these and other setups, a player can play video games on multiple different devices located throughout the home while the video game is actually running on a host computer somewhere else in the environment.

[0003] Today, there is a ubiquitous consumer demand for games to be streamed to consumer electronic devices (e.g., tablet computers, smartphones, etc.). There is also a concurrent demand for players to be able to switch between devices, even during a single video game session. For example, a player may want to start playing a video game on a tablet computer and then later switch to playing the video game on the living room television. All the while, the user may want to carry around their preferred video game controller that can be used to play the video game on these various target devices.

[0004] For security reasons or other reasons, operating system (OS) vendors impose restrictions on some types of peripherals (e.g., game controllers) that are permitted to connect to devices running their OS. Apple's iOS is exemplary, as it only allows MFi-certified gamepads to connect to iOS devices. This means that players cannot use non-MFi-certified game controllers to stream video games on iOS devices. Instead, players are forced to use MFi-certified, often somewhat inferior, gamepads to stream video games on iOS devices and play the games with handheld video game controllers. Devices running other popular OSs, such as Android, may connect to certain game controllers, but their OSs may not support all of the game controller's features and functionality. For example, the OS may not be able to interpret guide / menu or back button inputs when these button inputs are activated on the game controller. Furthermore, plugging a USB dongle into a tablet or smartphone with thin sides is a less desirable solution because the device or the dongle is more susceptible to damage if the dongle is struck by an object. First of all, some devices do not support the use of dongles, and in addition, when users need to carry the dongle from one device to another to stream video games to different target devices, the dongle is often lost or misplaced.

[0005] In light of these considerations, today's in-home video game streaming environment architecture is limited to a limited subset of target devices that work with most video game controllers, forcing players to use less powerful game controllers with other target devices. The disclosure made herein is presented with respect to these and other considerations. [Brief explanation of the drawings]

[0006] The detailed description is set forth with reference to the accompanying drawings. In each drawing, the leftmost digit(s) of a reference number identifies the drawing in which the reference number first appears. Use of the same reference number in different drawings indicates similar or identical components or features.

[0007] [Figure 1A] FIG. 1 is a schematic diagram illustrating an example technique for streaming a video game to a first target device when operating a game controller in Bluetooth Low Energy (BLE) mode.

[0008] [Figure 1B] FIG. 1 is a schematic diagram illustrating an exemplary technique for streaming a video game to a second target device when operating a game controller in receiver mode.

[0009] [Figure 2] 1 is a diagram illustrating components of an exemplary client machine usable as a host computer, components of an exemplary game controller, and components of an exemplary BLE device.

[0010] [Figure 3] 10A-10C are diagrams illustrating exemplary gestures that may be provided by a user of a game controller to operate the game controller in different modes, including at least a BLE mode and a receiver mode.

[0011] [Figure 4] FIG. 1 is a flow diagram of an exemplary process for operating a handheld video game controller in one of multiple operating modes.

[0012] [Figure 5] FIG. 1 is a flow diagram of an example process for operating a BLE device to stream a video game running on a host computer and receive controller input data from a game controller via BLE.

[0013] [Figure 6] FIG. 1 is a flow diagram of an exemplary process for operating a host computer to stream a video game running on the host computer to a target device while receiving controller input data from a game controller separate from the host computer. DETAILED DESCRIPTION OF THE INVENTION

[0014] Among other things, techniques, devices, and systems are described herein for streaming video game data to Bluetooth Low Energy (BLE) devices and for enabling players to play video games on such BLE devices using BLE game controllers. Specifically, a handheld video game controller can operate in, among other modes, a BLE mode, enabling it to transmit controller input data to a target BLE device in a manner that circumvents any operating system (OS) limitations that might otherwise be imposed on game controller input. Because BLE radios are widespread in today's wireless consumer electronic devices, the ability to provide controller input data to a BLE device—using the BLE protocol as a means of circumventing known OS limitations on game controller input—expands the ecosystem of devices and platforms used for video game streaming. That is, BLE devices—previously unusable as target devices for video game streaming due to their OS limitations imposed on game controller input—can now be used as target devices for video game streaming using a BLE game controller.

[0015] According to embodiments disclosed herein, a handheld video game controller includes a radio and logic for operating in different modes. The different operating modes include at least a BLE mode and a receiver mode. When operating in BLE mode, the handheld video game controller establishes a wireless connection with a BLE device running a client application used for video game streaming. During gameplay, the video game runs on a host computer, and the player provides user input to the game controller to control aspects of the video game. In response to such user input, the game controller transmits controller input data to the BLE device via the radio. In this manner, the BLE interface can be used as a sideband communication path between the BLE-enabled game controller and the host computer to enable video game streaming (e.g., over a home network). This sideband communication path effectively circumvents any OS limitations that the BLE device's OS might otherwise impose on video game controller peripherals and their corresponding controller input data.

[0016] When operating in receiver mode, a handheld video game controller uses the game controller's radio to establish a wireless connection with a wireless receiver that utilizes a non-BLE protocol. During gameplay, in response to user input to control aspects of the video game, the game controller can transmit controller input data to the wireless receiver via the radio. This wireless receiver can be connected to or incorporated into any target device on which the player wishes to play the video game. If the wireless receiver is connected to or incorporated into a device different from the aforementioned BLE device, the player can switch between target devices by switching modes on the game controller, so that the video game can be played in different locations within the environment (e.g., different rooms in a house). In some embodiments, the wireless receiver can be connected to or incorporated into the same BLE device that is available as a target device when the game controller is in BLE mode. In the latter scenario, a player can play a video game on a single BLE device but can switch between using the game controller in BLE mode and using the game controller in receiver mode for performance or other reasons. For example, because streaming data over BLE is known to be less performant than certain other non-BLE wireless protocols, such as low-latency WiFi protocols, that are available in receiver mode, a player may switch to operating their game controller in receiver mode to take advantage of the performance gains of using receiver mode (e.g., to connect more game controllers to the same wireless receiver without experiencing input lag, or to play more latency-sensitive video games where the player must react quickly to perform well, etc.).On the other hand, a player may want to revert to operating the game controller in BLE mode, for example, if the player desires to play a video game using a different (e.g., more portable / mobile) device, such as a tablet or smartphone. Switching between modes can be enabled via gestural input to the handheld video game controller (e.g., using multi-button gestures requiring a particular combination of buttons).

[0017] Also disclosed herein is a client application (sometimes referred to herein as a “client app”) that can be downloaded to and executed on a BLE device to enable video game streaming from a host computer to the BLE device and to enable the passing of controller input data from a handheld video game controller to the host computer via the BLE device. That is, the client application supports the use of a handheld video game controller in BLE mode, providing full access to the game controller's features. This setup does not require the use of a dongle or a separate external wireless receiver, which can provide advantages such as eliminating the need for players to carry a dongle that can easily be lost or damaged. The client app executable on the BLE device can include a custom BLE feature that configures the BLE device to receive controller input data via a wireless connection from a handheld video game controller operating in BLE mode. A user can connect the BLE device to a local area network (LAN), so that the BLE device can transmit controller input data, which is received from the game controller by a host computer running a video game.

[0018] A host computer running a video game can be co-located in the environment with the BLE device and handheld video game controller, or the host computer can be located remotely in a geographically separate location (e.g., a server computer located “in the cloud”). If the host computer is local to the environment in which the BLE device and game controller are located, the host computer can be connected to the same LAN to which the BLE device is connected. The host computer can use a video game client to run the video game during a video game session. During gameplay, the host computer receives controller input data from an intermediary device (e.g., from a BLE device connected to a game controller operating in BLE mode), and the video game client of the host computer can convert the controller input data into video game input data that is then processed by the video game running on the host computer. The video game client can hook into an input library specific to the video game it is running to convert the controller input data into something usable by the video game. As each frame is drawn during gameplay, the video game client captures the video game data (e.g., audio and video data), and the host computer sends the video game data over the LAN to a target device, such as a BLE device. BLE devices output audio and video data (e.g., via speakers and displays) for consumption by players during gameplay.

[0019] The technology and system disclosed herein enable players to utilize BLE devices as target devices for video game streaming, regardless of the type of OS running on the BLE device. This is more flexible and versatile than having to use an OS-certified gamepad with a BLE device, which limits compatible game controllers to a much smaller set. The disclosed technology and system enable the use of virtually any type of BLE-enabled game controller to play video games streamed to a connected BLE device, allowing video game players to enjoy using their preferred game controller as long as the game controller includes the BLE-enabled radio and logic described herein for transmitting controller input data to the connected BLE device via a BLE interface. Furthermore, the player's ability to switch between operating a handheld video game controller in different modes—i.e., between BLE mode and receiver mode—enables users to play video games on any desired target device in an environment and switch between these devices even during a single game session. This opens the door to streaming games on BLE devices that do not have a USB port to accept a dongle, improving the flexibility and convenience of video game streaming. The continued ability to operate handheld video game controllers in receiver mode (e.g., by transmitting controller input data via a radio to a wireless receiver utilizing a non-BLE protocol) provides players with lower latency connection options compared to lower performance BLE connections, allowing players to, for example, connect more game controllers or play input latency sensitive video games.

[0020] FIG. 1A is a schematic diagram illustrating an exemplary technique for streaming a video game to a first target device when the game controller is operating in Bluetooth Low Energy (BLE) mode. FIG. 1B is a schematic diagram illustrating an exemplary technique for streaming a video game to a second target device when the game controller is operating in receiver mode. Accordingly, FIGS. 1A and 1B are shown to contrast different operational modes of a handheld video game controller 102. FIGS. 1A and 1B include a remote system 104 (sometimes referred to as a “remote computing system” 104). This remote system 104 serves as or has access to a platform for delivering (e.g., downloading) programs (and content) to a client machine, such as the client machine 106 shown in FIGS. 1A and 1B. These programs (and content) may include a video game (or video game program). The client machine 106 can communicate with the remote system 104 via a communication interface of the client machine 106 over a computer network 108. This computer network 108 may represent or include, but is not limited to, the Internet, other types of data and / or voice networks, wired infrastructures (e.g., coaxial cable, fiber optic cable, etc.), wireless infrastructures (e.g., radio frequency (RF), cellular, satellite, etc.), and / or other connection technologies. Thus, client machine 106 sends / receives data to remote system 104 via wireless access point (WAP) 110, which is part of a local area network (LAN), using any suitable communication protocol. Remote system 104 may, in some cases, be part of a network-accessible computing platform that is maintained and accessed via computer network 108.Such network-accessible computing platforms may be referred to using terms such as "on-demand computing," "software as a service (SaaS)," "platform computing," "network-accessible platform," "cloud services," "data center," and the like.

[0021] Although the client machine 106 may represent a personal computer (PC), the client machine is not limited to a PC configuration. That is, the client machine 106 may be implemented as any suitable type of computing device configured to run video games, including, but not limited to, a PC, a desktop computer, a laptop computer, a mobile phone (e.g., a smartphone), a tablet computer, a personal digital assistant (PDA), a wearable computer (e.g., a virtual reality (VR) headset, an augmented reality (AR) headset, smart glasses, etc.), an in-vehicle (e.g., in-car) computer, a television (smart TV), a set-top box (STB), a game console, and / or any similar computing device. The client machine 106 may be located within an environment 112 along with one or more additional devices. The environment 112 in which the client machine 106 is located may be a home or other premises, an automobile, or any similar environment. Such an environment may include, for example, the aforementioned handheld video game controller 102 (sometimes referred to herein as a "game controller" 102 or a "handheld controller" 102), a WAP 110, a BLE device 114, a television 116, and a wireless receiver 118, as well as other possible devices such as Internet of Things (IoT) devices, other consumer electronic devices, etc.

[0022] Generally, the client machine 106 shown in FIGS. 1A and 1B represents a computing device on which a player 120 can execute, as a “host computer,” a program (e.g., a video game) and other software on the client machine. That is, as used herein, “host computer” refers to a computer running a video game, regardless of whether a player 120 is playing the video game on that host computer or whether that host computer is streaming video game data to a target device. The terms “user 120,” “player 120,” and / or “gamer 120” may be used interchangeably herein and may refer to a user playing a video game using a computing device. Referring briefly to FIG. 2, the client machine 106 includes one or more processors 200, such as a central processing unit (CPU) and graphics processing unit (GPU), a display 202, memory 204 (or non-transitory computer-readable medium 204), and a communication interface 206, among other components. Although the exemplary client machine 106 of FIG. 2 suggests that the client machine 106 includes a display 202, the client machine 106 may in fact omit a display and / or be coupled to a peripheral display.

[0023] Memory 204 (or non-transitory computer-readable medium 204) may include volatile and non-volatile memory, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium that can be used to store desired information and that can be accessed by a computing device. Computer-readable medium 204 may be implemented as a computer-readable storage medium (“CRSM”), which may be any available physical medium that is accessible by processor 200 to execute instructions stored on memory 204. In one basic implementation, CRSM may include random access memory (“RAM”) and flash memory. In other embodiments, the CRSM may include, but is not limited to, read-only memory (“ROM”), electrically erasable programmable read-only memory (“EEPROM”), or any other tangible medium that can be used to store desired information and that can be accessed by the processor 200.

[0024] The memory 204 may include an operating system module 208 configured to manage hardware within the client machine 106 and to be coupled to the client machine for the benefit of the other modules. The client machine 106 may also have a video game client 210 installed in the memory 204. The video game client 210 may represent an executable client application configured to launch and run a program, such as a video game (or video game program). In other words, the video game client 210 may include game software usable to play a video game on the client machine 106. With the video game client 210 installed, the client machine 106 may have the capability to receive (e.g., download, stream, etc.) a video game from a remote system 104 via the computer network 108 and run the video game via the video game client 210. For this purpose, any type of content distribution model may be utilized, such as a direct purchase model in which a video game can be purchased separately for download and run on the client machine 106, a subscription-based model, a content distribution model in which a video game is rented or leased for a period of time, or the like. Thus, client machine 106 may include one or more video games, such as video game 212, in video game library 214. These video games may be retrieved and run by loading video game client 210. In one example, player 120 may select to play one of multiple video games they have purchased and downloaded from video game library 214 by loading video game client 210, selecting video game 212, and beginning execution of video game 212. Video game client 210 may allow a user, such as player 120, to log in to the video game service using credentials (e.g., user account, password, etc.).

[0025] The video game client 210 is also shown as including a streaming component 216 configured to stream the video game to a target device and to receive and process controller input data generated by a game controller, such as game controller 102. This may occur when a player 120 is playing a video game 212 streamed from a client machine 106 acting as a host computer to any target device within the environment 112. In some embodiments, if the player 120 wishes to play the video game 212 on a target device other than the client machine 106, a user, such as the player 120, provides user input to the client machine 106 and invokes the streaming component 216. It should be understood that the same or similar components (and their associated functionality) as shown with respect to the client machine 106 may be implemented in (or by) the remote system 104 to enable streaming of video game data directly from the remote system 104 to networked devices within the environment 112 without running the video game locally with respect to the environment 112. That is, remote system 104, in some embodiments, can function as a host computer that runs a video game and streams video game data to a target device over computer network 108.

[0026] The game controller 102 may include one or more input / output (I / O) devices 218, such as controls (e.g., joysticks, trackpads, triggers, push buttons, etc.), and potentially any other type of input or output device, as shown in FIG. 2 . For example, the I / O devices 218 may include one or more microphones for receiving audio input, such as user voice input. In some implementations, one or more cameras or other types of sensors may function as input devices for receiving gestural input, such as movement of the handheld controller 102. In some embodiments, additional input devices may be provided in the form of a keyboard, keypad, mouse, touchscreen, joystick, control buttons, etc. The input devices may further include controls such as basic volume control buttons for increasing / decreasing the volume, as well as power and reset buttons.

[0027] Output devices, on the other hand, may include displays, light elements (e.g., LEDs), vibrators that create tactile sensations, speakers (e.g., headphones), etc. There may also be simple light elements (e.g., LEDs) to indicate a status, such as when the power is on. While several examples have been provided, the handheld controller 102 may additionally or alternatively include any other type of output device. In some cases, output by one or more output devices may be based on input received by one or more of the input devices. For example, activation of a control may result in the output of a tactile response by a vibrator located adjacent to (e.g., directly below) the control or in any other location.

[0028] Additionally, the handheld controller 102 may include one or more communications interfaces 220 to facilitate wireless connection to a network and / or another device. The communications interface 220 may implement multiple types of radios or wireless technologies to support operation of the game controller 102 in different modes. For example, the communications interface 220 may implement a radio configured to operate in a BLE mode or a receiver mode and switch between the two modes of operation, as described herein. That is, a single radio may execute code that causes the radio to operate in a BLE mode or a receiver mode, as described herein. However, it should be understood that the communications interface 220 may implement multiple radios, such as a Wi-Fi radio, a BLE radio, and a cellular radio. In some embodiments, separate radios may be utilized to operate in each mode, e.g., a Wi-Fi radio may be utilized to operate in receiver mode and a BLE radio may be utilized to operate in BLE mode. It should be understood that the handheld controller 102 may further include a physical port that facilitates a wired connection to a plug-in network device that communicates with a network, connected peripheral devices, or other wireless networks.

[0029] In the illustrated embodiment, the handheld controller further includes one or more processors 222 and memory 224 (or computer-readable medium 224). In some implementations, the processor 222 may include a central processing unit (CPU), a graphics processing unit (GPU), both a CPU and a GPU, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or additionally, the functionality described herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), etc. Additionally, each of the processors 222 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0030] Memory 224 may include volatile and nonvolatile memory, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, RAID storage systems, or any other medium that can be used to store desired information and that can be accessed by a computing device. Memory 224 may be implemented as a computer-readable storage medium (“CRSM”), which may be any available physical medium that is accessible by processor 222 to execute instructions stored on memory 224. In one basic implementation, CRSM may include random access memory (“RAM”) and flash memory. In other embodiments, the CRSM may include, but is not limited to, read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), or any other tangible medium that can be used to store desired information and that can be accessed by the processor 222.

[0031] Some modules, such as instructions, data storage, etc., may be stored in memory 224 and configured to execute on processor 222. While some example functional modules are shown as stored in memory 224 and executed on processor 222, the same functionality may alternatively be implemented in hardware, firmware, or as a system-on-chip (SOC).

[0032] The operating system module 226 may be configured to manage hardware within and couple to the handheld controller 102, for the benefit of other modules. Additionally, the memory 224 may store a network communication module 228, which enables the handheld controller 102 to communicate with one or more other devices, such as the BLE device 114 or wireless receiver 118 introduced in FIGS. 1A and 1B, a game console, etc., via the communication interface 220. The memory 224 may further include a game session database 230 for storing data associated with games (or other applications) running on the handheld controller or on a computing device to which the handheld controller 102 couples. The memory 224 may also include a device record database 232 that stores data associated with devices to which the handheld controller 102 couples. This device record database 232 may maintain a history of previously connected devices ordered by most recent connection, such that the game controller 102 can determine the last known device to which the game controller 102 was last connected at any given instance. The memory 224 may further store game control instructions 234 that configure the handheld controller 102 to function as a game controller by transmitting controller input data to another device associated with a game, and universal control instructions 236 that configure the handheld controller 102 to function as a controller for other non-gaming devices.

[0033] The memory 224 may further store an operational mode component 238 configured to determine and implement an operational mode of the game controller 102 based on user input (e.g., based on a gesture provided by the player 120). For example, as described in further detail below, the player 120 may provide a first gesture, such as by pressing a first combination of buttons, to set the handheld controller 102 in BLE mode, and a second gesture, such as by pressing a second combination of buttons, to set the handheld controller 102 in receiver mode. The operational mode component 238 may also store the last mode used in the local memory 224 of the game controller 102. In this way, if a particular mode is not invoked by the player 120 via a gesture upon startup of the game controller 102, the operational mode component 238 may determine the last mode in which the game controller 102 operated and may operate in that mode upon startup.

[0034] The game controller 102 may further include one or more sensors 240, including, but not limited to, a touch sensor (e.g., a capacitive touch sensor, a resistive touch sensor, an infrared touch sensor, a touch sensor that utilizes acoustic sound waves to detect the proximity of a finger, etc.), a force-sensing resistor (FSR), a motion sensor (e.g., an inertial measurement unit (IMU) including one or more gyroscopes, and / or accelerometers, and / or magnetometers, and / or a compass, camera, or 3D sensor configured for use in feature tracking, etc.).

[0035] 2 , the BLE device 114 may include similar components as the client machine 106, such as a processor 242, a display 244, a memory 246, and a communication interface 248, which may be similar to the processor 200, the display 202, the memory 204, and the communication interface 206 described with reference to the client machine 206. The communication interface 248 of the BLE device 114 (as its name implies) includes a radio for receiving controller input data from the game controller 102 via the BLE interface. This radio may also be configured to send / receive data with the WAP 110 in the environment via WiFi, and / or the communication interface 248 may include a dedicated WiFi radio for sending / receiving data with the WAP 110 in the environment.

[0036] The memory 246 of the BLE device 114 may include an operating system 250 configured to manage the hardware therein and to be coupled to the BLE device 114 for the benefit of other modules. The BLE device 114 may also download a client application 252 (or “client app” 252) and store the client app 252 in the memory 246. The client app 252 includes a streaming component 254 that configures the BLE device 114 to act as a target device for video game streaming. In this manner, the BLE device 114 receives streamed video game data from a host computer (e.g., the client machine 106) running the video game 212 and outputs audio and video data via the BLE device 114's output devices (e.g., the display 244 and speakers). Using the client app 252, the BLE device 114 also serves as a receiver of controller input data transmitted by the game controller 102 during gameplay via wireless. The client app 252 may further maintain a device record database 256 that stores data associated with devices to which the BLE device 114 couples. This may allow the BLE device 114 to maintain security and / or authentication credentials and pair with previously connected devices, such as the game controller 102.

[0037] To illustrate how the game controller 102 can operate in BLE mode to enable video game streaming to the BLE device 114, refer again to FIG. 1A . First, the player 120 connects the client machine 106 and the BLE device 114 to a home network (e.g., by connecting these devices to the WAP 110). The player 120 also pairs the game controller 102 with the BLE device 114, which is running a client app 252. This pairing operation can be accomplished by the player 120 providing user input to the game controller 102, which places the game controller 102 in BLE pairing mode. The client app 252 running on the BLE device 114 may include custom BLE characteristics that configure the BLE device 114 to receive controller input data over the radio from the game controller 102 operating in BLE mode. The data used by BLE profiles and services is called a “characteristic.” Thus, profiles and applications interface with the Generic Attribute Profile (GATT) layer in the BLE protocol stack for characteristic communication. A BLE “profile” is a specification for an aspect of Bluetooth communication. The profile defines the configuration and capabilities that enable services provided using wireless communication. In some embodiments, the client app 252 can provide a notification in response to the game controller 102 entering BLE pairing mode, indicating to the player 120 that the game controller 102 wishes to connect to the BLE device 114. The player 120 can provide user input to the BLE device that accepts the connection request, pairing the two devices over the BLE interface.

[0038] The game controller 102 can operate in BLE mode by default in response to the player 120 pairing the game controller 102 with the BLE device 114 over BLE via the operational mode component 238. Alternatively, the player 120 can provide user input to the game controller 102 corresponding to a command to operate the game controller 102 in BLE mode. In either case, the game controller 102 can establish a wireless link with the BLE device 114 running the client app 252 using the game controller's 102 radio.

[0039] The player 120 can also initiate a video game session for a video game 212 running on a video game client 210 on the client machine 106. If there are multiple client machines running the video game client 210, the client app 252 can provide an additional notification to the player 120 inquiring as to which client machine the user would like to connect to as the host computer for streaming the video game 212. The player 120 can select a particular client machine, such as the client machine 106, and the streaming connection is set up at this point. Establishing a connection in this manner may include authentication or authorization procedures (e.g., digital handshaking, credential verification, etc.).

[0040] During gameplay, for each frame, a video game client 210 running on a client machine 106 (host computer) captures video game data 122 and transmits the video game data over a LAN (e.g., over a WAN 110) to a BLE device 114. This may include the host computer's video game client 210 capturing the state of a video game 212 window, encoding the video and audio data into bits, and transmitting the encoded bits to a BLE device 114, where a client app 252 running on the BLE device 114 decodes the bits to render the video game content for the frame and output the audio through its speakers (or through headphones connected to the BLE device 114). The encoding of the video data may include H.264 video encoding and transmitting the encoded data using a low-latency network protocol. In one example, the client machine 106 may be connected to the WAP 110 via a wired (e.g., Ethernet) connection for low latency, and the WAP 110 may transmit encoded data wirelessly to the BLE device 114, which in turn receives the encoded data via its radio using the WiFi protocol. The streaming resolution and bitrate may be configurable by the player 120. For example, over a 5 gigahertz (GHz) network, 1080p or higher (e.g., 4K) resolution may be available at a frame rate of 60 frames per second (FPS).

[0041] The player 120 consumes video game content on the BLE device 114, which functions as a target device. Notably, the BLE device 114 is not running the video game 212; that is, the host computer—in this case, the client machine 106—is the device running the video game 212. The BLE device 114 effectively functions as a thin client that simply receives video game data 122 (e.g., audio and video data) that is processed (e.g., decoded) and output via an appropriate output device of the BLE device 114 for consumption by the player 120. During gameplay, the player 120 provides video game input using a game controller 102 (connected to the BLE device 114 when the game controller is operating in BLE mode). Thus, in response to user input detected by the game controller 102 to control aspects of the video game 212, the game controller 102 transmits controller input data 124 to the BLE device 114 via the game controller's 102 radio. A client app 252 running on the BLE device 114 receives the controller input data 124 and causes the BLE device 114 to forward the controller input data 124 over a LAN (e.g., via WAP 110) to a host computer (e.g., client machine 106). The manner in which the controller input data 124 is routed over BLE to the client app 252 running on the BLE device 114 effectively circumvents any OS limitations on game controller input, so that these OS limitations do not inhibit the player 120's ability to stream the video game 212 on the BLE device 114 and / or use the game controller 102 as a gamepad to play the video game 212.

[0042] In response to receiving the controller input data 124, the video game client 210 running on the client machine 106 can determine, based on the currently running video game 212 and based on the preferences of the player 120, how to translate the received controller input data 124 into video game input that the video game 212 (i.e., the video game code) can process. By default, the game controller 102 (and / or its controller input data) may be “displayed” as a mouse and keyboard, regardless of its current operating mode. Given that the client machine 106 typically utilizes a mouse and keyboard as input devices, this may be the simplest way to provide the controller input data to the host computer (e.g., the client machine 106) running the video game 212. In some embodiments, the game controller 102 can transmit the raw state of the game controller 102 to the client machine 106 via the BLE device 114 and the LAN, and the video game client 210 can interpret the raw state of the game controller 102 and remap it into appropriate video game input that can be injected into the video game 212 code. In some embodiments, a video game developer may utilize an application programming interface (API) that allows the video game developer to specify game controller configurations that remap (or otherwise translate) controller input data 124 from various types of controllers into video game inputs suitable for the video game. The video game client 210 may also translate between various types of game controllers, such as those commercially available from various game controller manufacturers. In some embodiments, a series of abstractions are used to obtain the controller input data 124 and, transparently and unnoticeably to the user, translate that data 124 into packets that are sent over a LAN as if the packets were received from the mouse and keyboard of the client machine 106.In this way, the remoteness of the game controller 102 with respect to the host computer running the video game 212 remains transparent to the user. In some embodiments, the video game client 210 can hook into the input library (or input libraries) of the video game 212 running on the host computer to determine how to convert the controller input data 124 into data that the video game 212 can use. This allows the video game client 210 to intercept function calls from the video game 212 code to read the game controller state and return the controller state that the video game 212 expects to see, which may mean remapping the state of the game controller 102 to the state of a different type of game controller.

[0043] After converting the controller input data 124 into video game input, the video game input is injected into the video game 212 code to control aspects of the video game 212, thereby causing the next frame to incorporate the controlled video game aspect, which is again captured and streamed to the target device, which in this case is the BLE device 114. In this manner, video game streaming continues by forwarding the video game data 122 downstream towards the BLE device 114 and the controller input data 124 upstream towards the client machine 106, as described herein.

[0044] At any time, the player 120 may decide to switch the operating mode of the game controller 102. This may be done for a variety of reasons, including, but not limited to, playing a video game 212 on a different target device, such as a television 116, or enabling a different wireless protocol that may have lower latency than BLE. FIG. 1B illustrates a scenario in which a user has switched the game controller 102 to operate in receiver mode (sometimes referred to as “non-BLE mode,” “WiFi mode,” or “dongle mode”). In the example of FIG. 1B, when the game controller 102 operates in receiver mode, the wireless receiver 118 is used as a conduit through which controller input data 124 is transferred upstream toward the host computer and through which video game data 122 is transferred downstream toward the target device. Thus, in some embodiments, the wireless receiver 118 may act more like a hub, serving as a concentrating point where data arrives from one or more devices and where data is transmitted to one or more devices.

[0045] If the wireless receiver 118 has not previously been connected to the game controller 102, the player 120 can initiate a pairing operation to pair the game controller 102 with the wireless receiver 118. The player 120 can provide user input to the game controller 102 to set it into wireless receiver pairing mode, and the player 120 can be prompted to enter a verification code to complete the connection between the game controller 102 and the wireless receiver 118. For example, if the wireless receiver 118 is connected to the television 116 via an audio / video interface, when the television 116 is toggled to the appropriate input, the wireless receiver 118 can display a notification with the verification code on the screen of the television 116 to complete the setup.

[0046] In response to a player 120 pairing the game controller 102 over the radio using a wireless receiver 118 that utilizes a non-BLE protocol (e.g., WiFi), the game controller 102 can operate in receiver mode by default via the operational mode component 238. Alternatively, the player 120 can provide the game controller 102 with user input corresponding to a command to operate the game controller 102 in receiver mode. If the respective device is already known to the game controller 102, no additional pairing operation needs to be performed when switching between operational modes. Instead, upon detecting a user input corresponding to a command to operate the game controller 102 in receiver mode, the game controller 102 can consult its device record database 232 to determine the last known wireless receiver to which the game controller 102 was last connected before receiving the user input and set the controller in receiver mode. This also works in the opposite direction (e.g., consulting the device record database 232 to determine the last known BLE device when switching to BLE mode). 1B , the game controller 102 can establish a wireless link with the wireless receiver 118 using the game controller's 102 radio (e.g., via WiFi), and the transfer of data is similar to the example of operating the game controller in BLE mode, except that the controller input data 124 is transferred to the host computer via the wireless receiver 118 rather than the BLE device 114. As noted above, compared to operating the game controller 102 in BLE mode, the receiver mode can utilize a different wireless communication protocol to transfer the controller input data 124 from the game controller 102 to the wireless receiver 118. This non-BLE communication protocol, in some embodiments, can represent an Enhanced Shock Burst (ESB)-based protocol operating in the 2.4 GHz band.

[0047] 1A and 1B, the disclosed handheld video game controller 102 is configured to operate in multiple different modes utilizing respective wireless protocols that differ from one another. BLE mode may be a lower-performance mode, but may be a mode that offers greater flexibility regarding the types of devices that can be used as target devices for streaming a video game 212 from a host computer, such as client machine 106. This is due, at least in part, to the fact that receiver mode may not be usable with some BLE devices due to OS limitations on game controller input and / or the fact that these devices do not support connection to peripheral wireless receivers, such as wireless receiver 118. The ability of the game controller 102 to switch to receiver mode is useful in situations where the higher-performance wireless protocol used in receiver mode allows the player 120 to connect more game controllers 102 to the wireless receiver 118, compared to some game controllers that can connect to the BLE device 114 without introducing significant input lag. For example, in BLE mode, the operating system 208 of the client machine 106 may utilize a 10 millisecond update rate (the period for reading the controller input data 124) for the game controller 102, and if the player 120 attempts to connect a second game controller to the BLE device 114 to play with a friend, the operating system 208 may allocate a 22 millisecond update rate for the additional game controller because the above update rate does not have enough wireless bandwidth in BLE mode to accommodate the additional game controller at any faster update rate. This can lead to a situation where one of two players competing in a video game experiences input lag, preventing the players from competing against each other in performance.To address this inherent limitation of the BLE protocol, players can switch their game controllers into receiver mode and stream video games via wireless receiver 118, which may provide a lower latency connection with suitable bandwidth to accommodate multiple game controllers.

[0048] As described elsewhere herein, the wireless receiver 118 may be implemented as a dongle (e.g., a USB dongle) that plugs into a corresponding port on the target device. In some cases, the BLE device 114 may include such a port, which means that a player 120 (or multiple players) can switch between wireless protocols while playing a video game on the same target device, i.e., the BLE device 114. That is, the player 120 may start playing a video game on the BLE device 114 while operating the game controller 102 in BLE mode, and then switch to operate the game controller 102 in receiver mode. When switching modes, if a wireless receiver is connected to or incorporated into the BLE device 114, video game data 122 continues to be streamed from the host computer to the BLE device 114, but controller input data 124 is received via a non-BLE protocol (e.g., WiFi) via a wireless receiver 118 connected to or incorporated into the BLE device 114 instead of the controller input data 124 being received via a BLE protocol via a radio in the BLE device 114.

[0049] 3 is a diagram illustrating example gestures that may be provided by a game controller user to operate the game controller in different modes, including at least a BLE mode and a receiver mode. This example game controller 102 may include depressible buttons 300(1)-(4) on the front of the game controller 102, including an X button 300(1), a Y button 300(2), an A button 300(3), and a B button 300(4). A player 120 may provide user input including gestures 302(1)-(4), or different gestures. Gestures 302(1)-(4) represent multi-button gestures that include specific combinations of depressible buttons 300 and guide buttons 304 (sometimes referred to as “Steam® buttons” 304 or “menu buttons” 304). For example, the player 120 can provide a first gesture 302(1) (first multi-button gesture 302(1)) to set the handheld controller 102 into BLE pairing mode by pressing a first combination of buttons (e.g., the Y button 300(2) and the Guide button 304). The player 120 can provide this first gesture 302(1) if the game controller 102 is not yet paired with the BLE device 114 and if he / she wants to stream the video game 212 to the BLE device 114 and operate the game controller 102 in BLE mode to play the video game 212.

[0050] The player 120 can provide a second gesture 302(2) (second multi-button gesture 302(2)) to set the handheld controller 102 to BLE mode by pressing a second combination of buttons (e.g., the B button 300(4) and the guide button 304). The player 120 can provide this second gesture 302(2) at the start of a game session if the game controller 102 has previously been paired with a BLE device, such as BLE device 114. In a different scenario, such as when the player 120 is playing a video game in receiver mode, the player 120 can provide the second gesture 302(2) to switch, perhaps during a game session, from operating the game controller 102 in receiver mode to operating the game controller 102 in BLE mode. In yet another scenario, such as when the player 120 is playing a video game in BLE mode, the player 120 can provide a second gesture 302(2) to switch from a first BLE device (e.g., BLE device 114) to a second BLE device within the environment 112. For example, the device record database 232 of the game controller 102 can be used to search for different BLE devices known to the game controller 102 to which the game controller 102 has previously connected. In some embodiments, specific gestures can be used to invoke different BLE devices within the device record database 232. In this case, the player 120 can provide one gesture (e.g., the B button 300(4) and the guide button 304) to connect to a most recently used BLE device in the device record database 232 and another gesture (e.g., the start button and the guide button 304) to connect to the next most recently used BLE device in the device record database 232. The player 120 may need to re-invoke these gestures to avoid having to toggle through devices to reach the desired BLE device.

[0051] Player 120 can provide a third gesture 302(3) (third multi-button gesture 302(3)) to place handheld controller 102 into receiver pairing mode by pressing a third combination of buttons (e.g., X button 300(1) and Guide button 304). Player 120 can provide this third gesture 302(3) if game controller 102 is not already paired with wireless receiver 118 and if the player wants to stream video game 212 to wireless receiver 118 connected to a target device in environment 112 and operate game controller 102 in receiver mode to play video game 212.

[0052] The player 120 can provide a fourth gesture 302(4) (fourth multi-button gesture 302(4)) to place the handheld controller 102 in receiver mode by pressing a fourth combination of buttons (e.g., the A button 300(3) and the guide button 304). The player 120 can provide this fourth gesture 302(4) at the beginning of a game session if the game controller 102 has previously been paired with a wireless receiver, such as the wireless receiver 118. In a different scenario, such as when the player 120 is playing a video game in BLE mode, the player 120 can provide the fourth gesture 302(4) to switch, perhaps during a game session, from operating the game controller 102 in BLE mode to operating the game controller 102 in receiver mode. In yet another scenario, such as when the player 120 is playing a video game in receiver mode, the player 120 can provide a fourth gesture 302(4) to switch from a first wireless receiver (e.g., wireless receiver 118) to a second wireless receiver within the environment 112. For example, the device record database 232 of the game controller 102 can be used to search for different wireless receivers known to the game controller 102 to which the game controller 102 has previously connected. In some embodiments, specific gestures can be used to invoke different wireless receivers within the device record database 232. In this case, the player 120 can provide one gesture (e.g., the A button 300(3) and the guide button 304) to connect to a most recently used wireless receiver in the device record database 232 and another gesture (e.g., the back button and the guide button 304) to connect to the next most recently used wireless receiver in the device record database 232. The player 120 may need to invoke these gestures again to avoid having to toggle through the device to reach the desired wireless receiver.

[0053] In some embodiments, player 120 can set preferences to connect to a preferred device (e.g., a preferred BLE device and / or a preferred wireless receiver) at startup, and player 120 can toggle between devices using gesture 302(2) or 302(4) depending on the current operating mode (i.e., BLE mode or receiver mode). Additionally or alternatively, as described above, device record database 232 can be used to determine the last known device (e.g., the last known BLE device and / or the last known wireless receiver) that was last connected before game controller 102 received gesture 302. At startup, device record database 232 can maintain a history of connected devices in persistent (non-volatile) memory and connect to the last known device at startup. Device record database 232 can maintain addresses of previously connected devices as well as security and / or authentication information (e.g., keys, passcodes, credentials, etc.) used to establish a wireless link with these devices after a previous disconnection. Additionally or alternatively, the operational mode component 238 of the game controller 102 can persist the last operational mode used (e.g., BLE mode or receiver mode) in the local memory 224 and restore the last used operational mode at startup when the player 120 does not provide one of the gestures 302. In some embodiments, whenever the player 120 downloads the client app 252 to a new BLE device, the player 120 can receive notification (e.g., via the client app 252) of the availability to connect the player's game controller 102 to the BLE device (e.g., by providing the first gesture 302(1) to the game controller 102).Similarly, whenever a player 120 connects the wireless receiver 118 to a BLE device running the client app 252, the player 120 may receive notification (e.g., via the client app 252) of the availability to connect the player's game controller 102 to the wireless receiver (e.g., by providing a third gesture 302(3) to the game controller 102).

[0054] While specific multi-button gestures 302 are illustrated in FIG. 3 , these are merely exemplary gestures, and other gestures can be implemented to provide similar functionality. For example, a touch sensor associated with a particular control, such as a top bumper control, can detect a swipe gesture. In this manner, a swipe in one direction (e.g., from left to right) or a swipe on a first control (e.g., the left bumper) may correspond to a command to operate the game controller 102 in BLE mode, while a swipe in the opposite direction (e.g., from right to left) or a swipe on a second control (e.g., the right bumper) may correspond to a command to operate the game controller 102 in receiver mode. Other gestures are envisioned as possibilities for this type of functionality, including voice gestures detected by the microphone of the game controller 102. For example, the player 120 can say “BLE mode” to set the game controller 102 to BLE mode, or the player 120 can say “receiver mode” to set the game controller 102 to receiver mode.

[0055] The processes described herein are illustrated as a collection of logical flow diagrams, which represent sequences of operations that may be implemented in hardware, software, firmware, or combinations thereof (referred to herein as "logic"). In the software context, the blocks represent computer-executable instructions, which, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any some of the described blocks may be combined in any order and / or in parallel to implement a process.

[0056] 4 is a flow diagram of an exemplary process 400 for operating a handheld video game controller 102 in one of multiple operational modes. For purposes of discussion, process 400 will be described with reference to the previous figures.

[0057] At 402, the game controller 102 may be powered up by transitioning from a power-off state to a power-on state. In the power-on state, power is supplied from a power source (e.g., one or more batteries) to one or more electrical components of the game controller 102. Powering up the game controller 102 at block 402 may be based on (e.g., in response to) the game controller 102 receiving a first user input. For example, pressing a power button, such as the guide button 304 shown in FIG. 3, powers up (powers on or wakes up) the game controller 102.

[0058] At 404, logic of the game controller 102 may determine whether the first user input received at block 402 includes a gesture, such as a predefined gesture, stored in memory 224 of the game controller 102. If it is determined at 404 that the first user input received at block 402 does not include a gesture (e.g., the player 120 may have simply pressed a power button, such as the guide button 304, to power on the game controller 102), process 400 may follow a “no” route from block 404 to block 406.

[0059] At 406, logic of the game controller 102 may determine to operate the game controller 102 in a particular operational mode, among multiple operational modes, in which the game controller 102 was last operated. For example, the multiple operational modes may include a BLE mode and a receiver mode, as described herein. At block 406, logic of the game controller 102 may search data maintained by the operational mode component 238 to determine which of these modes was last (most recently) used to operate the game controller 102 before receiving the first user input and activating the game controller 102 at block 402. If the BLE mode is identified as the last mode in which the game controller 102 was operated, then the BLE mode is selected at block 406. If the receiver mode is identified as the last mode in which the game controller 102 was operated, then the receiver mode is selected at block 406.

[0060] At 408, logic in the game controller 102 may determine devices (e.g., BLE devices or wireless receivers) to which the game controller 102 has previously connected. Specifically, at block 408, the logic may identify, based at least in part on data stored in the memory 224 of the game controller 102 (e.g., from the device records database 232), the device to which the handheld video game controller was last connected before receiving the first user input and activating the game controller 102 at block 402. These devices stored in the memory 224 of the game controller 102 may be associated with a particular operating mode (e.g., BLE mode or at least one of wireless receiver mode), which may be used as a filtering criterion for selecting devices at block 408. That is, if the BLE mode was selected as the last mode used at block 406, a recently connected BLE device 114 may be identified and selected at block 408. Conversely, if the receiver mode was selected as the last mode used at block 406, a recently connected wireless receiver 118 may be identified and selected at block 408.

[0061] At 410, with an operating mode and device selected, the game controller 102 can establish a wireless connection with the selected device via the game controller 102's radio using an appropriate protocol. For example, when operating in BLE mode, the game controller 102 can establish a wireless connection with the BLE device 114 running the client application 252 via the game controller 102's radio using a BLE protocol. However, when operating in receiver mode, the game controller 102 can establish a wireless connection with the wireless receiver 118 via the game controller 102's radio using a non-BLE protocol (e.g., WiFi). Establishing a wireless connection can include establishing a radio link (e.g., authentication, handshaking, etc.) over which data can be transmitted. This can include identifying the device's radio signal (and, in some cases, the strength of the radio signal).

[0062] At 412, the game controller 102 may receive a second user input for controlling an aspect of a video game running on a host computer (e.g., client machine 106) different from the device to which the game controller 102 was connected in block 410, such as the BLE device 114 or wireless receiver 118. For example, the client machine 106 may be running the video game 212 and streaming video game data 122 to the target device. When operating in BLE mode, the target device may be the BLE device 114 to which the game controller 102 was connected in block 410. When operating in receiver mode, the target device may be the device (e.g., television 116) connected to the wireless receiver 118 to which the game controller 102 was connected in block 410. The player 120 may be playing the video game by providing user input to the game controller 102 (e.g., operating controls on the game controller 102, moving the game controller 102 as detected by a motion sensor, etc.).

[0063] At 414, the game controller 102 may transmit the controller input data 124 to the device to which the game controller 102 was connected at block 410. Also, depending on the mode of operation, the receiving device of the controller input data 124 may be the BLE device 114 or the wireless receiver 118. When operating in BLE mode, the controller input data 124 may be transmitted to the BLE device 114 via the radio of the game controller 102 using a BLE protocol. When operating in receiver mode, the controller input data 124 may be transmitted to the wireless receiver 118 via the radio of the game controller 102 using a non-BLE protocol (e.g., WiFi). The controller input data 124 may be based on the user input received at block 412 and may be packaged in any suitable format.

[0064] In some embodiments, when operating in BLE mode, logic in the game controller 102 uses custom BLE characteristics to generate the controller input data 124 so that the controller input data can be received by a client application 252 running on the BLE device 114. This may include a BLE profile that interfaces with the GATT layer in the BLE protocol stack to convey the characteristics in the controller input data 124.

[0065] Referring back to block 404, if it is determined that the first user input received in block 402 includes a gesture (e.g., a multi-button gesture as described with reference to FIG. 3), process 400 may follow a “Yes” route from block 404 to block 416.

[0066] At block 416, logic in the game controller 102 may determine the type of gesture detected. For example, the memory 224 of the game controller 102 may store a plurality of predefined gestures, such as gestures 302(1)-(4) described with reference to FIG. 3 . These example gestures 302 may be categorized as pairing gestures (e.g., gestures 302(1) and 302(3) are pairing gestures) and mode selection gestures (e.g., gestures 302(2) and 302(4) are mode selection gestures). If the gesture is determined as a pairing gesture at block 416, process 400 may proceed from block 416 to block 418, where logic in the game controller 102 determines to operate the game controller 102 in a particular mode associated with the pairing gesture. For example, if the gesture is pairing gesture 302(1) of FIG. 3, the logic may determine to operate in BLE mode, while if the gesture is pairing gesture 302(3) of FIG. 3, the logic may determine to operate in receiver mode.

[0067] At 420, based on the operating mode selected in block 418, logic in the game controller 102 may determine a new device for that mode (i.e., a new device) to which the game controller is not already connected. For the BLE mode, this may be a new BLE device 114 that is placed into BLE pairing mode through a client app 252 running on the BLE device 114. For the receiver mode, this may be a new wireless receiver 118 that is placed into receiver pairing mode, as described herein.

[0068] Once a new device has been identified and selected in block 420, blocks 410-414 may be performed as described above, but with respect to the newly identified device. That is, the game controller 102 may establish a wireless connection with the new device (block 410), receive a (second) user input for controlling an aspect of a video game running on the host computer (block 412), and, based on the (second) user input, transmit controller input data 124 to the connected device via the radio using an appropriate protocol (block 414).

[0069] Referring back to block 416, if the gesture is determined to be a mode selection gesture, process 400 may proceed from block 416 to block 422, where logic in the game controller 102 determines to operate the game controller 102 in a particular mode associated with the mode selection gesture. For example, if the gesture is mode selection gesture 302(2) in FIG. 3, the logic may determine to operate in BLE mode, while if the gesture is mode selection gesture 302(4) in FIG. 3, the logic may determine to operate in receiver mode.

[0070] At 424, logic in the game controller 102 may determine whether the gesture is associated with a particular device, among multiple devices, to which the game controller 102 has previously connected. For example, data stored in the memory 224 of the game controller 102 (e.g., the device record database 232) may maintain a list of devices to which the game controller 102 has previously connected. These devices may be filtered based on the operational mode selected at block 422. If the gesture is specific to one of the devices associated with the selected operational mode, the process 400 may follow the “Yes” route from block 424 to block 426, where the device associated with the gesture may be selected. For example, the player 120 may provide one gesture (e.g., the B button 300(4) and the guide button 304) to connect to a recently used BLE device in the device record database 232 and another gesture (e.g., the start button and the guide button 304) to connect to the next recently used BLE device in the device record database 232. Depending on which one of these gestures is detected, the device associated with that gesture may be selected as the identified device in block 426. Similar logic applies to the receiver mode, as described herein.

[0071] If, at block 424, the gesture is not device-specific (e.g., the gesture is not specific to one of the devices associated with the selected operational mode), process 400 may follow a “no” route from block 424 to block 428, where the logic selects the device that was last connected before the game controller 102 received the first user input at block 402 and woke up the game controller 102. That is, unless the gesture is a particularly device-specific gesture, the most recently connected device may be selected at block 428. In BLE mode, this device is the BLE device 114 that was last connected before the game controller 102 received the first user input at block 402 and woke up the game controller 102. In receiver mode, this device is the wireless receiver 118 that was last connected before the game controller 102 received the first user input at block 402 and woke up the game controller 102.

[0072] Following either block 428 or block 426, blocks 410-414 may be performed as described above, but with respect to the device identified and selected in block 428 or block 426. That is, the game controller 102 may establish a wireless connection with the selected device (block 410), receive a (second) user input for controlling an aspect of the video game running on the host computer (block 412), and, based on the (second) user input, transmit controller input data 124 via the radio using an appropriate protocol to the connected device (block 414).

[0073] 5 is a flow diagram of an example process 500 for operating a BLE device 114 to stream a video game running on a host computer and receive controller input data over BLE from a game controller 102. For purposes of discussion, process 500 will be described with reference to the previous figures.

[0074] At 502, the BLE device 114 (connected to the LAN of the environment 112) can execute a client app 252. This client app 252 can configure the BLE device 114 to act as a target device for video game streaming and as a hub device for controller input data provided by a connected game controller 102 (e.g., connected via BLE).

[0075] At 504, while a video game 212 is running on a host computer (e.g., a client machine 106 located in the environment 112 along with the BLE device 114), the BLE device 114 can receive video game data 122 streamed from the host computer (e.g., the client machine 106) over a LAN (e.g., via the WAP 110).

[0076] At 506, the BLE device 114 may output the audio and video data (e.g., the video game data 122 including the audio and video data) via output devices (e.g., the display 244 and the speaker) of the BLE device 114.

[0077] At 508, the BLE device 114—acting as a hub device—can receive controller input data 124 from the connected game controller 102 via the radio of the BLE device 114 using the BLE protocol. Custom BLE features available through the client app 252 can enable receiving the controller input data in a way that circumvents any OS limitations on game controller input.

[0078] At 510, the BLE device 114 may transmit the controller input data 124 that it received from the game controller 102 to a host computer (e.g., the client machine 106) running the video game 212. The controller input data 124 may be transmitted from the BLE device 114 to the host computer over a LAN (e.g., via the WAP 110).

[0079] 6 is a flow diagram of an exemplary process 600 for operating a host computer to stream a video game executing on the host computer to a target device while receiving controller input data from a game controller separate from the host computer. For purposes of discussion, process 600 will be described with reference to the previous figures.

[0080] At 602, a host computer (e.g., client machine 106, remote system 104, etc.) can run a video game 212 using a video game client 210. The video game client 210 can be configured to allow a player 120 to play the video game on the host computer itself and / or to allow the video game 212 to be streamed to a target device (e.g., via a streaming component 216).

[0081] At 604, the video game client 210 can hook into an input library (or multiple input libraries) of the video game 212 running on the host computer to determine how to convert the controller input data 124 into data usable by the video game 212. This allows the video game client 210 to intercept function calls from the video game 212 code to read the game controller state and return the controller state that the video game 212 expects to see, which may include remapping the state of the connected game controller 102 to the state of a different type of game controller.

[0082] At 606, the host computer can transmit the video game data 122 captured by the video game client 210 to the target device over the LAN. As described herein, depending on the operational mode of the game controller 102, this may include transmitting the video game data 122 to the BLE device 114 while in BLE mode or to a wireless receiver 118 while in receiver mode (a wireless receiver 118 connected to or incorporated into the target device). As described herein, the video game data 122 may include audio and video data, and this data may be encoded for transmission over the LAN.

[0083] At 608, the host computer can receive the controller input data 124 from the transmitting device over the LAN. Also, depending on the operational mode of the game controller 102, the transmitting device of the controller input data 124 can be the BLE device 114 in BLE mode or the wireless receiver 118 in receiver mode.

[0084] At 610, the video game client 210 can convert the controller input data 124 into video game input that the video game 212 code can understand / process. This conversion at block 610 can be based at least in part on a hooked input library for the video game. This conversion at block 610 can also be based on other factors, such as the player's 120 preferences known to the video game client 210 as to how much the player prefers the converted controller input data 124. In some embodiments, the conversion at block 610 can include remapping the controller input data 124 to appropriate video game input that can be injected into the video game 212 code. In some embodiments, video game developers can utilize APIs that allow them to instantiate game controller configurations and remap (or convert) the controller input data 124 from various types of controllers into video game input suitable for their video game. Thus, the video game client 210 can utilize such game controller configurations for the remapping at block 610. The video game client 210 can also convert between different types of game controllers, such as those commercially available from various game controller manufacturers, and in some cases, between mouse and keyboard input.

[0085] At 612, the video game client 210 can "inject" video game input (translated from the controller input data 124) into video game 212 code to control aspects of the video game 212 running on the host computer. For example, a video game input that causes a player-controlled character to jump can cause the player-controlled character to jump over the course of several subsequent frames streamed to the target device.

[0086] Although the subject matter has been described in language specific to structural features, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the particular features described. Rather, the particular features are disclosed as exemplary forms of implementing the claims. The inventions described in the original claims of this application are set forth below. [1] 1. A handheld video game controller, comprising: a processor; A radio and a memory storing computer-executable instructions that, when executed by the processor, cause the handheld video game controller to: based at least in part on a first user input received by the handheld video game controller; activating the handheld video game controller by providing power to one or more electrical components of the handheld video game controller; and determining to operate in a Bluetooth® Low Energy (BLE) mode; determining, based at least in part on the data stored in the memory, a BLE device to which the handheld video game controller has previously connected; and establishing a wireless connection via the radio with the BLE device running a client application; receiving a second user input for controlling an aspect of a video game running on a host computer different from the BLE device; and transmitting controller input data to the BLE device via the radio, the controller input data being based at least in part on the second user input. [2] The computer-executable instructions, when executed by the processor, cause the handheld video game controller to: receiving a third user input corresponding to a command to operate the handheld video game controller in a receiver mode; determining, based at least in part on the data stored in the memory, a wireless receiver to which the handheld video game controller previously connected, the wireless receiver being different from the BLE device; and establishing a wireless connection with the wireless receiver via the radio; receiving a fourth user input for controlling an additional aspect of the video game executing on the host computer; and [1] The handheld video game controller of [1] further causes the controller to transmit additional controller input data to the wireless receiver via the radio, the additional controller input data being based at least in part on the fourth user input. [3] The handheld video game controller of [2], wherein the third user input includes a gesture corresponding to the command to operate the handheld video game controller in the receiver mode. [4] [1] The handheld video game controller described in [1], wherein determining to operate in the BLE mode includes determining that the first user input includes a gesture corresponding to a command to operate in the BLE mode. [5] The computer-executable instructions, when executed by the processor, cause the handheld video game controller to: determining that the gesture is associated with the BLE device among a plurality of BLE devices to which the handheld video game controller has previously connected; [4] The handheld video game controller of [4], further comprising: identifying the BLE device among the plurality of BLE devices based at least in part on the gesture. [6] The handheld video game controller of [1], wherein determining the BLE device includes identifying the BLE device as the BLE device last connected before the handheld video game controller received the first user input. [7] 1. A method comprising: receiving a first user input by a handheld video game controller; based at least in part on the first user input; starting the handheld video game controller; and determining to operate the handheld video game controller in a Bluetooth Low Energy (BLE) mode; determining, based at least in part on data stored in a memory of the handheld video game controller, a BLE device to which the handheld video game controller previously connected; establishing, by the handheld video game controller via a radio in the handheld video game controller, a wireless connection with the BLE device running a client application; receiving, by the handheld video game controller, a second user input for controlling an aspect of a video game executing on a host computer different from the BLE device; transmitting, by the handheld video game controller via the radio, controller input data to the BLE device, the controller input data being based at least in part on the second user input. [8] receiving, by the handheld video game controller, a third user input corresponding to a command to operate the handheld video game controller in a receiver mode; determining a wireless receiver to which the handheld video game controller previously connected based at least in part on the data stored in the memory of the handheld video game controller, the wireless receiver being different from the BLE device; and establishing a wireless connection with the wireless receiver via the radio by the handheld video game controller; receiving, by the handheld video game controller, a fourth user input for controlling an additional aspect of the video game executing on the host computer; and transmitting, by the handheld via a game controller, additional controller input data via the wireless to the wireless receiver, the additional controller input data being based at least in part on the fourth user input; The method according to [7], further comprising: [9] The method described in [7], wherein determining to operate the handheld video game controller in the BLE mode includes determining that the first user input includes a gesture corresponding to a command to operate the handheld video game controller in the BLE mode.

[10] The method of [9], wherein the gesture includes a multi-button gesture detected by the handheld video game controller as pressing a combination of multiple buttons including a first button and a second button on the handheld video game controller.

[11] The determining the BLE device includes: determining that the gesture is associated with the BLE device among multiple BLE devices to which the handheld video game controller has previously connected; and identifying the BLE device among the plurality of BLE devices based at least in part on the gesture.

[12] [7] The method of [7], wherein the determining the BLE device includes identifying the BLE device as the BLE device last connected to the handheld video game controller before receiving the first user input.

[13] The determining to operate the handheld video game controller in the BLE mode includes:

[12] The method of

[12] , comprising identifying the BLE mode as the operating mode in which the handheld video game controller was last operated before receiving the first user input.

[14] One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by a processor of a handheld video game controller, cause the handheld video game controller to perform operations, the operations including: Receiving a first user input; based at least in part on the first user input; powering the handheld video game controller; and determining to operate the handheld video game controller in a Bluetooth Low Energy (BLE) mode; determining, based at least in part on data stored in a memory of the handheld video game controller, a BLE device to which the handheld video game controller previously connected; establishing a wireless connection via a radio in the handheld video game controller with the BLE device running a client application; receiving a second user input for controlling an aspect of a video game running on a host computer different from the BLE device; and transmitting, via the radio, controller input data to the BLE device, the controller input data being based at least in part on the second user input.

[15] The operation is receiving a third user input corresponding to a command to operate the handheld video game controller in a receiver mode; determining a wireless receiver to which the handheld video game controller previously connected based at least in part on the data stored in the memory of the handheld video game controller, the wireless receiver being different from the BLE device; and establishing a wireless connection with the wireless receiver via the radio; receiving a fourth user input for controlling an additional aspect of the video game executing on the host computer; and

[14] The one or more non-transitory computer-readable media described in

[14] further comprising: transmitting, via the radio, additional controller input data to the radio receiver, the additional controller input data being based at least in part on the fourth user input.

[16]

[14] The one or more non-transitory computer-readable media described in

[14] , wherein determining to operate the handheld video game controller in the BLE mode includes determining that the first user input includes a gesture corresponding to a command to operate the handheld video game controller in the BLE mode.

[17]

[16] One or more non-transitory computer-readable media described in

[16] , wherein the gesture includes a multi-button gesture detected by the handheld video game controller as pressing a combination of multiple buttons including a first button and a second button on the handheld video game controller.

[18] The determining the BLE device includes: determining that the gesture is associated with the BLE device among multiple BLE devices to which the handheld video game controller has previously connected; and identifying the BLE device among the plurality of BLE devices based at least in part on the gesture.

[19]

[14] The one or more non-transitory computer-readable media described in

[14] , wherein the determining the BLE device includes identifying the BLE device as the BLE device last connected to the handheld video game controller before the receipt of the first user input.

[20] The determining to operate the handheld video game controller in the BLE mode includes:

[14] The one or more non-transitory computer-readable media described in

[14] , including identifying the BLE mode as an operational mode in which the handheld video game controller was last operated before receiving the first user input.

Claims

1. A handheld controller, a processor; A radio and and a memory storing computer-executable instructions that, when executed by the processor, cause the handheld controller to: determining that the user input received by the handheld controller includes a gesture; determining that the gesture is a mode selection gesture associated with one of a plurality of operational modes of the handheld controller, the operational mode being associated with a second wireless protocol; based at least in part on determining that the gesture is a mode selection gesture associated with the operational mode; switching from using a first wireless protocol to using the second wireless protocol, the first wireless protocol being different from the second wireless protocol; using the radio to transmit controller input data to a device configured to use the second wireless protocol.

2. The computer-executable instructions, when executed by the processor, cause the handheld controller to: filtering a list of devices to which the handheld controller previously connected based at least in part on the operational mode to obtain the filtered list of devices associated with the operational mode; Selecting the device from the filtered list of devices; The handheld controller of claim 1 , further comprising: establishing a wireless connection with the device before transmitting the controller input data to the device.

3. Determining that the gesture is the mode selection gesture associated with the operating mode includes determining that the gesture is a first mode selection gesture of a plurality of mode selection gestures; The handheld controller of claim 1 , wherein the plurality of mode selection gestures includes at least the first mode selection gesture and a second mode selection gesture associated with a second operational mode of the plurality of operational modes.

4. The gesture is: a swipe gesture detected by a touch sensor of the handheld controller; a voice gesture detected by a microphone of the handheld controller; or 10. The handheld controller of claim 1, wherein the gesture comprises at least one of: a multi-button gesture detected by the handheld controller as pressing a combination of multiple buttons on the handheld controller.

5. The computer-executable instructions, when executed by the processor, cause the handheld controller to: determining that a second user input received by the handheld controller includes a second gesture; determining that the second gesture is a second mode selection gesture associated with a second operational mode of the plurality of operational modes, the second operational mode being associated with the first wireless protocol; 10. The handheld controller of claim 1, further configured to transmit, using the radio, second controller input data to a second device configured to use the first wireless protocol based at least in part on determining that the second gesture is the second mode selection gesture associated with the second operational mode.

6. The operating mode includes a Bluetooth® Low Energy (BLE) mode; The handheld controller of claim 1 , wherein the second wireless protocol includes a BLE protocol.

7. The handheld controller of claim 1, wherein the handheld controller is a game controller configured to control an aspect of a video game.

8. The method of claim 7, further comprising: determining, by a handheld controller, that a user input received by the handheld controller includes a gesture; determining, by the handheld controller, that the gesture is a mode selection gesture associated with one of a plurality of operational modes of the handheld controller, the operational mode being associated with a second wireless protocol; based at least in part on determining that the gesture is a mode selection gesture associated with the operational mode; switching, by the handheld controller, from using a first wireless protocol to using the second wireless protocol, the first wireless protocol being different from the second wireless protocol; transmitting, by the handheld controller, controller input data using a radio in the handheld controller to a device configured to use the second wireless protocol.

9. The method of claim 8, further comprising: determining, by the handheld controller, that the gesture is a device-specific gesture; selecting, by the handheld controller, the device based at least in part on an association of the device with the device-specific gesture; The method of claim 8 , further comprising establishing, by the handheld controller, a wireless connection with the device prior to transmitting the controller input data to the device.

10. The method of claim 10, further comprising: determining, by the handheld controller, that the gesture is not a device-specific gesture; selecting, by the handheld controller, the device based at least in part on the handheld controller's last connection to the device prior to receiving the user input; The method of claim 8 , further comprising establishing, by the handheld controller, a wireless connection with the device prior to transmitting the controller input data to the device.

11. Determining that the gesture is the mode selection gesture associated with the operational mode includes determining that the gesture is a first mode selection gesture of a plurality of mode selection gestures; The method of claim 8 , wherein the plurality of mode selection gestures includes at least the first mode selection gesture and a second mode selection gesture associated with a second operational mode of the plurality of operational modes.

12. The gesture is: a swipe gesture detected by a touch sensor of the handheld controller; a voice gesture detected by a microphone of the handheld controller; or and a multi-button gesture detected by the handheld controller as pressing a combination of multiple buttons on the handheld controller.

13. The method of claim 12, further comprising: determining, by the handheld controller, that a second user input received by the handheld controller includes a second gesture; determining, by the handheld controller, that the second gesture is a second mode selection gesture associated with a second operating mode of the plurality of operating modes, the second operating mode being associated with the first wireless protocol; 10. The method of claim 8, further comprising transmitting, by the handheld controller, using the radio, second controller input data to a second device configured to use the first wireless protocol based at least in part on determining that the second gesture is the second mode selection gesture associated with the second operational mode.

14. The operating mode includes a Bluetooth Low Energy (BLE) mode; The method of claim 8 , wherein the second wireless protocol comprises a BLE protocol.

15. The method of claim 14, further comprising: determining, by a handheld controller, that a user input received by the handheld controller includes a gesture; determining, by the handheld controller, that the gesture is a pairing gesture associated with one of a plurality of operational modes of the handheld controller, the operational mode associated with a second wireless protocol; based at least in part on determining that the gesture is a pairing gesture associated with the operational mode; switching, by the handheld controller, from using a first wireless protocol to using the second wireless protocol, the first wireless protocol being different from the second wireless protocol; transmitting, by the handheld controller, controller input data using a radio in the handheld controller to a device configured to use the second wireless protocol.

16. Determining that the gesture is the pairing gesture associated with the operating mode includes determining that the gesture is a first pairing gesture of a plurality of pairing gestures; The method of claim 15 , wherein the plurality of pairing gestures includes at least the first pairing gesture and a second pairing gesture associated with a second mode of operation of the plurality of modes of operation.

17. The method of claim 16, further comprising: identifying, by the handheld controller, the device based at least in part on the device being configured in a pairing mode associated with the second wireless protocol; The method of claim 15 , further comprising establishing, by the handheld controller, a wireless connection with the device prior to transmitting the controller input data to the device.

18. The method of claim 17, further comprising: determining, by the handheld controller, that a second user input received by the handheld controller includes a second gesture; determining, by the handheld controller, that the second gesture is a second pairing gesture associated with a second operating mode of the plurality of operating modes, the second operating mode being associated with the first wireless protocol; 16. The method of claim 15, further comprising transmitting, by the handheld controller, using the radio, second controller input data to a second device configured to use the first wireless protocol based at least in part on determining that the second gesture is the second pairing gesture associated with the second operational mode.

19. The method of claim 15, wherein the handheld controller is a game controller configured to control an aspect of a video game.

20. The operating mode includes a Bluetooth Low Energy (BLE) mode; The method of claim 15 , wherein the second wireless protocol comprises a BLE protocol.

Images