Game controller that can operate in Bluetooth Low Energy (BLE) mode
The game controller, which uses Bluetooth Low Energy (BLE) mode and receiver mode, solves the compatibility limitations of the operating system for game controllers, enabling flexible use on multiple devices and low-latency connection, and improving the convenience of video game streaming.
Patent Information
- Application Number
- CN202410147805.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-03-29
- Filing Date
- 2020-03-24
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2040-03-24
AI Technical Summary
Existing home video game streaming systems suffer from poor compatibility due to operating system limitations on game controllers, making them unsuitable for use on multiple devices. Furthermore, dongles are easily damaged and inconvenient to carry.
The game controller uses Bluetooth Low Energy (BLE) mode and receiver mode to transmit controller input data wirelessly, bypassing operating system limitations and supporting switching between different devices. It can also be used to run a client application on a BLE device to enable video game streaming.
It enables flexible use of game controllers on a variety of BLE devices, avoiding damage and inconvenience of carrying dongles, providing lower latency connection options, and enhancing the flexibility and convenience of video game streaming.
Smart Images

Figure CN117942553B_ABST
Abstract
Description
[0001] This application is a divisional application of the invention patent application filed on March 24, 2020, with application number No. 202080024817.6, entitled "Game Controller Operable in Bluetooth Low Energy (BLE) Mode".
[0002] Cross-reference to related applications
[0003] This PCT application claims priority to U.S. Patent Application Serial No. 16 / 370,656, filed on March 29, 2019, entitled “GAME CONTROLLER OPERABLE INBLUETOOTH LOW ENERGY (BLE) MODE”, the entire contents of which are incorporated herein by reference. Background Technology
[0004] Home video game streaming allows playing a video game on one computer (target device) while the game is actually executed on another computer (host computer) elsewhere in the home environment. An exemplary setup includes a personal computer (PC) in a home office (host computer) that executes the video game, captures video game data (e.g., audio and video data), and streams the video game data to a living room TV (target device) via the home network. In this scenario, the player can play the video game from the living room sofa using a handheld video game controller while the video game content is displayed on the TV. If the living room TV does not include a suitable receiver, a hub device may act as an intermediary, receiving video game data from the PC and providing it to the living room TV via an audio / video interface. The hub device may also receive controller input data from the game controller and provide it to the PC in the uplink direction. Other setups may use a Universal Serial Bus (USB) dongle as a wireless receiver, which plugs into a target device such as a laptop with a suitable USB port. The dongle will receive controller input data provided by the game controller, thus allowing gameplay on the target device where video game data is streamed from the PC via the home network. Using these and other settings, players are able to play video games on multiple different devices located throughout the house, while the video games are actually executed on host computers elsewhere in the environment.
[0005] Today, there is a need to allow game streaming to ubiquitous consumer electronics devices (e.g., tablets, smartphones, etc.) that are always around consumers. There is also a need to allow players to switch between devices even within a single video game session. For example, a player might start by wanting to play a video game on a tablet and then want to switch to playing it on their living room TV. Users have also historically wanted to carry their favorite video game controller with them, which could be used to play the game on these different target devices.
[0006] For security or other reasons, operating system (OS) vendors restrict the types of peripherals (e.g., game controllers) allowed to connect to devices running their OS. Apple's iOS is illustrative, as it only allows MFi-certified game controllers to connect to iOS devices. This means that players cannot use non-MFi-certified game controllers to stream video games on iOS devices. Instead, players are forced to use typically lower-performance MFi-certified game controllers to stream video games on iOS devices and play with handheld video game controllers. Other popular OSes (such as Android) can connect to some game controllers, but the OS may not support all the features and functions of the game controller. For example, the OS may not be able to interpret boot / menu button inputs or back button inputs when activating these buttons on the game controller. Furthermore, plugging a USB dongle into a slim tablet or smartphone is less than ideal, as devices or dongles are easily damaged by impacts. Some devices do not support dongles from the outset. Additionally, dongles are easily lost or misplaced when users have to carry them from one device to another to stream video games to different target devices.
[0007] Given these considerations, today's home video game streaming ecosystem is limited to a finite subset of target devices that can work with most video game controllers, and players are forced to use less powerful game controllers with other target devices. The disclosures made herein are made in light of these and other considerations. Attached Figure Description
[0008] Specific embodiments are described with reference to the accompanying drawings. In these drawings, the leftmost digit of the reference numeral indicates the drawing in which that reference numeral first appears. The same reference numerals are used in different drawings to indicate similar or identical parts or features.
[0009] Figure 1A This is a schematic diagram illustrating an exemplary technique for streaming a video game to a first target device when a game controller is operated in Bluetooth Low Energy (BLE) mode.
[0010] Figure 1B This is a schematic diagram illustrating an exemplary technique for streaming a video game to a second target device when the game controller is operated in receiver mode.
[0011] Figure 2 This is an illustration showing components of an exemplary client machine that can be used as a host computer, components of an exemplary game controller, and components of an exemplary BLE device.
[0012] Figure 3 This is an illustration showing exemplary gestures that can be provided by the user of the game controller to operate the game controller in different modes, including at least BLE mode and receiver mode.
[0013] Figure 4 This is a flowchart illustrating an exemplary process for operating a handheld video game controller in one of a variety of operating modes.
[0014] Figure 5 This is a flowchart of an exemplary process for operating a BLE device to stream a video game executed on a host computer and receiving controller input data from a game controller via BLE.
[0015] Figure 6 This is a flowchart of an exemplary process for operating a host computer to stream a video game executed thereon to a target device while receiving controller input data from a game controller separate from the host computer. Detailed Implementation
[0016] Among other things, this article describes technologies, devices, and systems for streaming video game data to Bluetooth Low Energy (BLE) devices and enabling players to play video games on such BLE devices using BLE game controllers. Specifically, handheld video game controllers can operate in BLE mode as well as other modes to allow controller input data to be sent to the target BLE device in a way that bypasses any operating system (OS) limitations that might otherwise be imposed on the game controller input. Because BLE radio is ubiquitous in today's wireless consumer electronics, the ability to provide controller input data to BLE devices (using the BLE protocol as a means of bypassing known OS limitations on game controller input) expands the ecosystem of devices and platforms for game video streaming. That is, BLE devices (which were previously unusable as target devices for video game streaming due to the OS limitations they impose on game controller input) can now be used as target devices for streaming video games with BLE game controllers.
[0017] According to the embodiments disclosed herein, a handheld video game controller includes radio and logic for operation in different modes. The different operating modes include at least a BLE mode and a receiver mode. When operating in BLE mode, the handheld video game controller establishes a wireless connection with a BLE device executing a client application for video game streaming. During gameplay, the video game executes on a host computer, and the player provides user input to the game controller to control one aspect of the video game. In response to such user input, the game controller transmits controller input data to the BLE device via radio. In this way, the BLE interface can be used as a sideband communication path between the BLE-enabled game controller and the host computer to enable video game streaming (e.g., via a home network). The sideband communication path effectively circumvents any OS limitations that the BLE device's OS would normally impose on the video game controller peripherals and their corresponding controller input data.
[0018] When operating in receiver mode, the handheld video game controller establishes a wireless connection using its radio to a wireless receiver utilizing a non-BLE protocol. During gameplay, in response to user input controlling one aspect of the video game, the game controller can transmit controller input data via radio to the wireless receiver. This wireless receiver can be connected to or embedded in any target device on which the player wishes to play the video game. If the wireless receiver is connected to or embedded in a device different from the aforementioned BLE device, switching modes on the game controller allows the player to switch between target devices, enabling gameplay in different locations within an environment (e.g., different rooms in a house). In some implementations, when the game controller is in BLE mode, the wireless receiver can be connected to or embedded in the same BLE device that can be used as a target device. In the latter scenario, the player can play the video game on a single BLE device, but may switch between using the game controller in BLE mode and receiver mode for performance or other reasons. For example, since streaming data via BLE is known to be less efficient than certain other non-BLE wireless protocols, such as low-latency Wi-Fi protocols available in receiver mode, players may switch to receiver mode to operate their game controllers to take advantage of the performance improvements offered by receiver mode (e.g., to connect more game controllers to the same wireless receiver without experiencing input lag, or to play more latency-dependent video games where players must react quickly to perform well, etc.). However, if, for example, a player wants to use a different (e.g., a more portable / mobile) device (such as a tablet or smartphone) to play the video game, they may wish to revert to operating the game controller in BLE mode. Switching between modes can be enabled via gesture input on handheld video game controllers (e.g., using multi-button gestures involving specific button combinations).
[0019] This document also discloses a client application (sometimes referred to herein as the "client application") that can be downloaded to and executed on a BLE device to enable video game streaming from a host computer to the BLE device and to allow controller input data to be passed from a handheld video game controller to the host computer via the BLE device. In other words, the client application supports the use of a handheld video game controller in BLE mode to provide full access to the game controller's features. This setup eliminates the need for a dongle or additional external wireless receiver, which can offer advantages such as eliminating the need for players to carry easily lost or damaged dongles. The client application, which can execute on the BLE device, can include custom BLE features that configure the BLE device to receive controller input data wirelessly from a handheld video game controller operating in BLE mode. Users can connect the BLE device to a local area network (LAN), allowing the BLE device to forward controller input data received from the game controller to the host computer running the video game.
[0020] The host computer running the video game can be co-located with the BLE device and handheld video game controller in an environment, or the host computer can be located remotely in a geographically remote location (e.g., a server computer in the "cloud"). When the host computer is located in the local area of the environment where the BLE device and game controller are located, the host computer can be connected to the same LAN as the LAN connected to the BLE device. The host computer can use a video game client to execute the video game during a video game session. During gameplay, the host computer receives controller input data from an intermediate device (e.g., from a BLE device connected to a game controller operating in BLE mode), and the host computer's video game client can convert this controller input data into video game input data that is subsequently processed by the video game executed on the host computer. The video game client can attach an input library specific to the video game it is executing in order to convert the controller input data into data usable by the video game. As each frame is rendered during gameplay, the video game client captures video game data (e.g., audio and video data), and the host computer sends this video game data via the LAN to a target device, such as the BLE device. The BLE device outputs audio and video data (e.g., via speakers and a display) for the player to use during gameplay.
[0021] Regardless of the type of OS running on the BLE device, the technologies and systems disclosed herein allow players to utilize BLE devices as target devices for video game streaming. This is more flexible and versatile than having to use an OS-licensed gamepad with a BLE device (which limits compatible game controllers to a smaller set). Because the disclosed technologies and systems allow the use of virtually any type of BLE-enabled game controller to play video games streamed to a connected BLE device, as long as the game controller includes the BLE-enabled radio and logic described herein for sending controller input data to the connected BLE device via a BLE interface, the player of the video game can enjoy using their favorite game controller. Furthermore, the ability for players to switch between different modes of operating a handheld video game controller (i.e., between BLE mode and receiver mode) allows users to play video games on any desired target device within the environment and allows users to switch between these devices even during a single game session. This provides opportunities for streaming games on BLE devices that lack a USB port for receiving a dongle, increasing the flexibility and convenience of video game streaming. The ability to retain the handheld video game controller's input data in receiver mode (e.g., by transmitting the controller's input data via radio to a wireless receiver using a non-BLE protocol) provides players with lower latency connectivity options (compared to the less efficient BLE connections), allowing players to connect more game controllers or play video games that rely on input latency.
[0022] Figure 1A This is a schematic diagram illustrating an exemplary technique for streaming a video game to a first target device when a game controller is operated in Bluetooth Low Energy (BLE) mode. Figure 1B This is a schematic diagram illustrating an exemplary technique for streaming a video game to a second target device when the game controller is operated in receiver mode. Therefore, Figure 1A and Figure 1B Different operating modes of the handheld video game controller 102 are shown for comparison. Figure 1A and Figure 1B This includes a remote system 104 (sometimes referred to as a "remote computing system" 104). The remote system 104 can act as a means of distributing programs (and content) to (e.g., downloading to) client machines (such as...). Figure 1A and Figure 1BThe client machine 106 shown is a platform or has access to that platform. These programs (and content) may include video games (or video game programs). The client machine 106 may communicate with the remote system 104 via a computer network 108 through its communication interface. The computer network 108 may represent and / or include, but is not limited to, the Internet, other types of data and / or voice networks, wired infrastructure (e.g., coaxial cable, fiber optic cable, etc.), wireless infrastructure (e.g., radio frequency (RF), cellular, satellite, etc.) and / or other connectivity technologies. Therefore, the client machine 106 sends / receives data to / from the remote system 104 via a wireless access point (WAP) 110, which is part of a local area network (LAN) using any suitable communication protocol. In some cases, the remote system 104 may be part of a network-accessible computing platform maintained and accessible via the computer network 108. Such network-accessible computing platforms may be referred to by terms such as “on-demand computing,” “Software as a Service (SaaS),” “platform computing,” “network-accessible platform,” “cloud service,” “data center,” etc.
[0023] While client machine 106 may represent a personal computer (PC), it is not limited to a PC configuration. That is, client machine 106 can be implemented as any suitable type of computing device configured to perform video games, including but not limited to PCs, desktop computers, laptops, mobile phones (e.g., smartphones), tablets, portable digital assistants (PDAs), wearable computers (e.g., virtual reality (VR) headsets, augmented reality (AR) headsets, smart glasses, etc.), in-vehicle (e.g., vehicle) computers, televisions (smart TVs), set-top boxes (STBs), game consoles, and / or any similar computing devices. Client machine 106 may be located within environment 112 along with one or more additional devices. Environment 112, where client machine 106 is located, can be a residence or other location, a vehicle, or any similar environment. Such an environment may include, for example, the aforementioned handheld video game controller 102 (sometimes referred to herein as "game controller" 102 or "handheld controller" 102), WAP 110, BLE device 114, television 116, and wireless receiver 118, as well as other possible devices such as Internet of Things (IoT) devices, other consumer electronics devices, etc.
[0024] Generally speaking, Figure 1A and Figure 1BThe client machine 106 shown may be referred to as a “host computer” as a computing device on which player 120 can use to execute programs (e.g., video games) and other software. That is, as used herein, “host computer” refers to a computer that is running a video game, whether or not player 120 is playing the video game on that host computer, or whether or not the host computer is streaming video game data to a target device. The terms “user 120,” “player 120,” and / or “gamer 120” are used interchangeably herein to refer to a user using a computing device to play a video game. Brief Reference Figure 2 Among other components, the client machine 106 includes one or more processors 200 such as a central processing unit (CPU) and a graphics processing unit (GPU), a display 202, memory 204 (or non-transitory computer-readable medium 204), and a communication interface 206. Although Figure 2 The exemplary client machine 106 indicates that the client machine 106 includes a display 202, but the client machine 106 may actually omit the display and / or may be coupled to an external display.
[0025] Memory 204 (or non-transitory computer-readable medium 204) may include volatile and non-volatile memory, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, RAID storage systems, or any other medium that can be used to store desired information and is accessible by a computing device. Computer-readable medium 204 may be implemented as a computer-readable storage medium (“CRSM”), which may be any available physical medium accessible to processor 200 to execute instructions stored on memory 204. In one basic embodiment, the CRSM may include random access memory (“RAM”) and flash memory. In other embodiments, the CRSM may include, but is not limited to, read-only memory (“ROM”), electrically erasable programmable read-only memory (“EEPROM”), or any other tangible medium that can be used to store desired information and is accessible by processor 200.
[0026] Memory 204 may include an operating system module 208 configured to manage hardware within and coupled to client machine 106 to facilitate other modules. Client machine 106 may also have a video game client 210 installed in memory 204. Video game client 210 may represent an executable client application configured to launch and execute programs such as video games (or video game programs). In other words, video game client 210 may include game software that can be used to play video games on client machine 106. After installing video game client 210, client machine 106 may then have the ability to receive (e.g., download, stream, etc.) video games from remote system 104 via computer network 108 and execute video games via video game client 210. Any type of content distribution model can be used for this purpose, such as a direct purchase model where video games can be purchased individually for download and execution on client machine 106, a subscription-based model, a content distribution model in which video games are rented or leased for a period of time, etc. Therefore, client machine 106 may include one or more video games, such as video game 212, within video game library 214. These video games can be retrieved and executed by loading video game client 210. In one example, player 120 can select to play one of multiple video games they have purchased and downloaded to video game library 214 by loading video game client 210 and selecting video game 212 to begin playing video game 212. Video game client 210 can allow users such as player 120 to log in to the video game service using credentials (e.g., user account, password, etc.).
[0027] The video game client 210 is also shown to include a streaming component 216 configured to stream the video game to a target device and to receive and process controller input data generated by a game controller, such as game controller 102. This may occur when player 120 is playing video game 212 streamed from client machine 106, which acts as a host computer, to any target device in environment 112. In some embodiments, when player 120 wants to play video game 212 on a target device other than client machine 106, user such as player 120 provides user input to client machine 106 to invoke streaming component 216. It should be understood that the same or similar components shown relative to client machine 106 (and its associated functions) may be implemented at (or by) remote system 104 to enable direct streaming of video game data from remote system 104 to networked devices in environment 112, without executing the video game locally relative to environment 112. In other words, in some implementations, the remote system 104 can act as a host computer that executes the video game and streams the video game data to the target device via the computer network 108.
[0028] like Figure 2 As shown, the game controller 102 may include one or more input / output (I / O) devices 218, such as controls (e.g., joysticks, touchpads, triggers, pressable buttons, etc.), and any other possible type of input or output device. For example, I / O devices 218 may include one or more microphones to receive audio input, such as user voice input. In some embodiments, one or more cameras or other types of sensors may be used as input devices to receive gesture input, such as movement of the handheld controller 102. In some embodiments, additional input devices may be provided in the form of a keyboard, keypad, mouse, touchscreen, joystick, control buttons, etc. Input devices may also include control mechanisms, such as basic volume control buttons for increasing / decreasing volume, as well as power and reset buttons.
[0029] Simultaneously, the output devices may include a display, a light-emitting element (e.g., an LED), a haptic vibrator, a speaker (e.g., headphones), etc. Simple light-emitting elements (e.g., LEDs) may also be present to indicate a state, such as, for example, when powered on. While some examples have been provided, the handheld controller 102 may, in addition to or alternatively, include any other type of output device. In some cases, the output of the one or more output devices may be based on input received by one or more of the input devices. For example, actuation of the control may cause a vibrator located near (e.g., below) or at any other location to output a haptic response.
[0030] In addition, the handheld controller 102 may include one or more communication interfaces 220 to facilitate wireless connectivity with a network and / or another device. The communication interface 220 may implement various types of wireless or radio technologies to support operation of the game controller 102 in different modes. For example, the communication interface 220 may implement a radio configured to operate in BLE mode or receiver mode as described herein and switch between these two operating modes. That is, a single radio may execute code that causes the radio to operate in BLE mode or receiver mode, as described herein. However, it should be understood that the communication interface 220 may implement multiple radios, such as WiFi radio, BLE radio, cellular radio, etc. In some embodiments, a single radio may be used to operate in the corresponding mode, such as using a WiFi radio to operate in receiver mode and using a BLE radio to operate in BLE mode. It should be understood that the handheld controller 102 may also include a physical port to facilitate wired connection to a network, connected peripherals, or plug-in network devices communicating with other wireless networks.
[0031] In the illustrated embodiment, the handheld controller also includes one or more processors 222 and memory 224 (or computer-readable medium 224). In some embodiments, processor 222 may include a central processing unit (CPU), a graphics processing unit (GPU), both a CPU and a GPU, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or additionally, the functions described herein may be performed at least in part by one or more hardware logic components. For example, but not limited to, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc. In addition, each processor in processor 222 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.
[0032] Memory 224 may include volatile and non-volatile memory, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Such memory includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, RAID storage systems, or any other medium that can be used to store desired information and is accessible by a computing device. Memory 224 may be implemented as a computer-readable storage medium (“CRSM”), which may be any available physical medium accessible to processor 222 to execute instructions stored on memory 224. In one basic embodiment, the CRSM may include random access memory (“RAM”) and flash memory. In other embodiments, the CRSM may include, but is not limited to, read-only memory (“ROM”), electrically erasable programmable read-only memory (“EEPROM”), or any other tangible medium that can be used to store desired information and is accessible by processor 222.
[0033] Several modules, such as instructions and data storage, may be stored in memory 224 and configured to execute on processor 222. Some exemplary functional modules are shown as being stored in memory 224 and executed on processor 222, but the same functionality may alternatively be implemented using hardware, firmware, or a system-on-a-chip (SoC).
[0034] Operating system module 226 can be configured to manage the hardware within and coupled to the handheld controller 102, to facilitate other modules. Furthermore, memory 224 can store network communication module 228, which enables the handheld controller 102 to communicate with one or more other devices (such as...) via communication interface 220. Figure 1A and Figure 1BThe memory 224 may communicate with BLE devices 114 or wireless receivers 118, game consoles, etc. as described herein. The memory 224 may also include a game session database 230 to store data associated with a game (or other application) running on the handheld controller or a computing device coupled to the handheld controller 102. The memory 224 may also include a device record database 232 that stores data associated with devices coupled to the handheld controller 102. In any given instance, the device record database 232 may retain a history of previously connected devices ordered by the recentity of connection, allowing the game controller 102 to determine the last known device to which it was last connected. The memory 224 may also store game control instructions 234 and general control instructions 236, which configure the handheld controller 102 as a game controller by sending controller input data to another device associated with the game, and which configure the handheld controller 102 as a controller for other non-game devices.
[0035] The memory 224 may also store an operation mode component 238 configured to determine and implement the operation mode of the game controller 102 based on user input (e.g., based on gestures provided by player 120). For example, as will be described in more detail below, player 120 may provide a first gesture (such as by pressing a first combination of multiple buttons) to put the handheld controller 102 into BLE mode, and may provide a second gesture (such as by pressing a second combination of multiple buttons) to put the handheld controller 102 into receiver mode. The operation mode component 238 may also store the last used mode in the local memory 224 of the game controller 102. In this way, when the game controller 102 is started, if player 120 has not invoked a specific mode by gesture, the operation mode component 238 may determine the last mode for operating the game controller 102 and may operate in that mode upon startup.
[0036] The game controller 102 may also include one or more sensors 240, including but not limited to touch sensors (e.g., capacitive touch sensors, resistive touch sensors, infrared touch sensors, touch sensors that use acoustic waves to detect the proximity of a finger, etc.), force-sensing resistors (FSRs), motion sensors (e.g., inertial measurement units (IMUs) including one or more gyroscopes and / or accelerometers and / or magnetometers and / or compasses, cameras or 3D sensors configured for feature tracking, etc.).
[0037] like Figure 2As shown, BLE device 114 may include components similar to client machine 106, such as processor 242, display 244, memory 246, and communication interface 248, which may be similar to processor 200, display 202, memory 204, and communication interface 206 described with reference to client machine 206. Communication interface 248 of BLE device 114 (as the name suggests) includes a radio for receiving controller input data from game controller 102 via the BLE interface. This radio may also be configured to send / receive data to / from WAP 110 in the environment via WiFi, and / or communication interface 248 may include a dedicated WiFi radio for sending / receiving data to / from WAP 110 in the environment.
[0038] The memory 246 of the BLE device 114 may include an operating system 250 configured to manage the hardware within and coupled to the BLE device 114 for the benefit of other modules. The BLE device 114 may also have a client application 252 (or “client application” 252) downloaded and stored in the memory 246. The client application 252 includes a streaming component 254 that configures the BLE device 114 as a target device for video game streaming. In this way, the BLE device 114 receives video game data streamed from a host computer (e.g., client machine 106) executing the video game 212 and outputs audio and video data via output devices of the BLE device 114 (e.g., display 244 and speakers). The BLE device 114 using the client application 252 also functions as a receiver for controller input data transmitted wirelessly by the game controller 102 during gameplay. The client application 252 can also maintain a device record database 256, which stores data associated with devices coupled to the BLE device 114. This allows the BLE device 114 to retain security and / or authentication credentials for pairing with previously connected devices, such as the game controller 102.
[0039] To illustrate how the game controller 102 can operate in BLE mode to enable video game streaming to the BLE device 114, refer again to... Figure 1AInitially, player 120 connects client machine 106 and BLE device 114 to a home network (e.g., by connecting these devices to WAP 110). Player 120 also pairs game controller 102 with BLE device 114, which executes client application 252. This pairing operation is accomplished by player 120 providing user input to game controller 102 to put game controller 102 into BLE pairing mode. Client application 252, executing on BLE device 114, may include custom BLE features that configure BLE device 114 to receive controller input data from game controller 102 operating in BLE mode via radio. The data used by profiles and services in BLE is referred to as “features.” Therefore, profiles and applications are connected to the Generic Attribute Profile (GATT) layer in the BLE protocol stack for transmitting features. A BLE “profile” is a specification regarding Bluetooth communication. Profiles define the configurations and functions that enable the use of services provided by wireless communication. In some implementations, client application 252 may provide a notification to player 120 in response to game controller 102 entering BLE pairing mode, instructing game controller 102 to connect to BLE device 114. Player 120 may provide user input to the BLE device to accept the connection request, thereby pairing the two devices via the BLE interface.
[0040] In response to player 120 pairing game controller 102 with BLE device 114 via BLE, game controller 102 can operate in BLE mode by default via operation mode component 238. Alternatively, player 120 can provide user input to game controller 102 corresponding to commands for operating game controller 102 in BLE mode. In either case, game controller 102 can establish a radio link with BLE device 114 executing client application 252 using game controller 102's radio.
[0041] Player 120 may have also initiated a video game session for video game 212 running on video game client 210 on client machine 106. In cases where multiple client machines are running video game client 210, client application 252 may provide additional notification to player 120, asking which client machine he / she wants to connect to as the host computer for streaming video game 212. Player 120 may select a specific client machine, such as client machine 106, and a streaming connection is established at this point. Establishing a connection in this manner may include authentication or authorization procedures (e.g., digital handshake, credential verification, etc.).
[0042] During gameplay, for each frame, a video game client 210 executing on client machine 106 (host computer) captures video game data 122 and transmits it to BLE device 114 via a LAN (e.g., via WAN 110). This may involve the host computer's video game client 210 capturing the state of the video game 212 window, encoding video and audio data into bits, transmitting the encoded bits to BLE device 114, and a client application 252 executing on BLE device 114 decoding these bits to present the video game content of that frame and outputting audio via its speakers (or via headphones connected to BLE device 114). Encoding of the video data may include H.264 video encoding and transmitting the encoded data using a low-latency network protocol. In one example, client machine 106 may connect to WAP 110 via a wired (e.g., Ethernet) connection for low latency, and WAP 110 may wirelessly transmit the encoded data to BLE device 114, allowing BLE device 114 to receive the encoded data via its radio using the WiFi protocol. Player 120 may be able to configure streaming resolution and bit rate. For example, over a 5 GHz network, it can utilize 1080p or higher (e.g., 4K) resolution at a frame rate of 60 frames per second (FPS).
[0043] Player 120 consumes video game content on BLE device 114, which acts as the target device. It is important to note that BLE device 114 does not execute video game 212; the host computer (in this case, client machine 106) is the device executing video game 212. BLE device 114 effectively acts as a thin client, receiving only video game data 122 (e.g., audio and video data) processed (e.g., decoded) and output by the appropriate output device of BLE device 114 for use by player 120. During gameplay, player 120 provides video game input using game controller 102 (which is connected to BLE device 114 when operating in BLE mode). Therefore, in response to user input detected by game controller 102 to control one aspect of video game 212, game controller 102 transmits controller input data 124 to BLE device 114 via its radio. The client application 252 executing on the BLE device 114 receives controller input data 124 and causes the BLE device 114 to forward the controller input data 124 to a host computer (e.g., client machine 106) via a LAN (e.g., via WAP 110). The way the controller input data 124 is routed via BLE to the client application 252 executing on the BLE device 114 effectively bypasses any OS restrictions on game controller input, ensuring that these OS restrictions do not inhibit the player 120's ability to stream video game 212 on the BLE device 114 and / or use the game controller 102 as a gamepad for playing video game 212.
[0044] In response to receiving controller input data 124, the video game client 210 executing on client machine 106 can determine how to translate the received controller input data 124 into video game input that the video game 212 (i.e., video game code) can process, based on the currently executing video game 212 and the preferences of player 120. By default, regardless of the current operating mode of game controller 102, the game controller (and / or its controller input data) can be "displayed" as a mouse and keyboard. Considering that client machine 106 typically uses a mouse and keyboard as input devices, this is likely the most straightforward way to provide controller input data to the host computer executing video game 212 (e.g., client machine 106). In some implementations, game controller 102 can send its raw state to client machine 106 via BLE device 114 and LAN, and video game client 210 can interpret the raw state of game controller 102 and remap it to appropriate video game input, which can then be added to the video game 212 code. In some implementations, video game developers may utilize an application programming interface (API) that allows them to specify game controller configurations that remap (or otherwise transform) controller input data 124 from various types of controllers to video game input suitable for their video game. The video game client 210 may also perform conversions between different types of game controllers, such as those from various game controller manufacturers on the market. In some implementations, a series of extraction methods are used to acquire the controller input data 124 and transparently convert it into data packets sent over a LAN, as if those packets were received from the mouse and keyboard of the client machine 106. In this way, the remote nature of the game controller 102 remains transparent relative to the host computer executing the video game 212. In some implementations, the video game client 210 may hook up with one or more input libraries of the video game 212 executing on the host computer to determine how to convert the controller input data 124 into data usable by the video game 212. This allows the video game client 210 to intercept function calls from the video game 212 code for reading the game controller state and return the controller state that the video game 212 expects to see, which may involve remapping the state of the game controller 102 to different types of game controller states.
[0045] After the controller input data 124 is converted into video game input, this video game input is added to the video game 212 code to control one aspect of the video game 212. This causes the next frame to contain the controlled aspect of the video game, and the video game data is again captured and streamed to the target device, in this case, the BLE device 114. In this way, the video game streaming continues by downlinking the video game data 122 to the BLE device 114 and uplinking the controller input data 124 to the client machine 106, as described herein.
[0046] At any time, player 120 may decide to switch the operating mode of game controller 102. This may be done for a variety of reasons, including but not limited to playing video game 212 on different target devices (such as TV 116), or enabling different wireless protocols with lower latency than BLE. Figure 1B This illustrates a scenario where the user has switched to operating the game controller 102 in receiver mode (sometimes referred to as "non-BLE mode," "WiFi mode," or "donkey mode"). Figure 1B In the example, when the game controller 102 operates in receiver mode, the wireless receiver 118 acts as a conduit through which controller input data 124 is transmitted upstream to the host computer, and video game data 122 is transmitted downstream to the target device. Therefore, in some embodiments, the wireless receiver 118 may function more like a hub acting as a convergence point, through which data arrives from one or more devices and is transmitted from the convergence point to one or more devices.
[0047] If the wireless receiver 118 has not been previously connected to the game controller 102, the player 120 can initiate a pairing process to pair the game controller 102 with the wireless receiver 118. The player 120 can provide user input to the game controller 102 to put it into wireless receiver pairing mode, and the player 120 may be prompted to enter a verification code to complete the connection between the game controller 102 and the wireless receiver 118. For example, if the wireless receiver 118 is connected to a television 116 via an audio / video interface, when the television 116 is switched to the appropriate input, the wireless receiver 118 can display a notification with a verification code for completing the setup on the screen of the television 116.
[0048] In response to player 120 pairing game controller 102 with a wireless receiver 118 utilizing a non-BLE protocol (e.g., WiFi) via radio, game controller 102 can operate in receiver mode by default via operation mode component 238. Alternatively, player 120 can provide user input to game controller 102 corresponding to a command to operate game controller 102 in receiver mode. No additional pairing operation is required when switching between operation modes when game controller 102 is aware of the corresponding device. Instead, when user input corresponding to a command to operate game controller 102 in receiver mode is detected, game controller 102 can consult its device record database 232 to determine the last known wireless receiver to which game controller 102 last connected before receiving user input to place the controller in receiver mode. This is done in a similar manner in the reverse direction (e.g., when switching to BLE mode, querying device record database 232 to determine the last known BLE device). In any case, as Figure 1B As shown, game controller 102 can establish a radio link with wireless receiver 118 using its own radio (e.g., via WiFi), and the data transfer is similar to the example of operating the game controller in BLE mode, except that controller input data 124 is transmitted to the host computer via wireless receiver 118 instead of BLE device 114. As mentioned above, receiver mode can utilize a different wireless communication protocol to transmit controller input data 124 from game controller 102 to wireless receiver 118 compared to operating game controller 102 in BLE mode. In some embodiments, this non-BLE communication protocol may represent an enhanced burst (ESB) based protocol operating in the 2.4 GHz spectrum.
[0049] Therefore, as Figure 1A and Figure 1BAs shown, the disclosed handheld video game controller 102 is configured to operate in several different modes utilizing corresponding wireless protocols that differ from each other. BLE mode may be a lower-performance mode, but it offers greater flexibility in terms of the type of device that can be used as a target device for streaming video game 212 from a host computer (such as client machine 106). This is at least partly due to the fact that some BLE devices may not be able to use receiver mode due to OS limitations on game controller input; and / or these devices do not support connection to peripheral wireless receivers (such as wireless receiver 118). The ability of game controller 102 to switch to receiver mode is useful in situations where the higher-performance wireless protocol used in receiver mode allows player 120 to connect more game controllers 102 (compared to many game controllers that can be connected to BLE device 114 without introducing significant input latency) to wireless receiver 118. For example, in BLE mode, the operating system 208 of client machine 106 can use a 10-millisecond update rate (the interval for reading controller input data 124) for game controller 102, and if player 120 attempts to connect a second game controller to BLE device 114 to play with a friend, the operating system 208 can allocate a 22-millisecond update rate for the additional game controller because the operating system does not have sufficient radio bandwidth in BLE mode to accommodate the additional game controller at a faster update rate. This could result in one of the two players competing in a video game experiencing input lag, thus inhibiting that player's game performance. To address this inherent limitation of the BLE protocol, players can switch their game controllers to receiver mode to stream video games via wireless receiver 118, which provides a low-latency connection with appropriate bandwidth to accommodate multiple game controllers.
[0050] As described elsewhere in this document, the wireless receiver 118 may be implemented as a dongle (e.g., a USB dongle) plugged into a corresponding port on the target device. In some cases, the BLE device 114 may include such a port, meaning that player 120 (or multiple players) can switch between wireless protocols while playing a video game on the same target device (i.e., the BLE device 114). That is, player 120 may start playing a video game on the BLE device 114 while operating the game controller 102 in BLE mode, and then switch to operating the game controller 102 in receiver mode. When switching modes, if the wireless receiver is connected to or embedded in the BLE device 114, the video game data 122 continues to be streamed from the host computer to the BLE device 114, but the controller input data 124 is received via a non-BLE protocol (e.g., WiFi) via the wireless receiver 118 connected to or embedded in the BLE device 114, rather than via the BLE protocol through the radio of the BLE device 114.
[0051] Figure 3 This is an illustration of exemplary gestures that can be provided by a user of a game controller to operate the game controller in different modes, including at least BLE mode and receiver mode. An exemplary game controller 102 may include pressable buttons 300(1)-(4) on the front side of the game controller 102, including an X button 300(1), a Y button 300(2), an A button 300(3), and a B button 300(4). A player 120 may provide user input including gestures 302(1)-(4) or different gestures. Gestures 302(1)-(4) represent multiple pressable buttons 300 and a guide button 304 (sometimes referred to as "..."). A multi-button gesture consisting of a specific combination of buttons (“button” 304 or “menu button” 304). For example, player 120 may provide a first gesture 302(1) (first multi-button gesture 302(1)) to put handheld controller 102 into BLE pairing mode by pressing a first combination of multiple buttons (e.g., Y button 300(2) and guide button 304). If game controller 102 is not already paired with BLE device 114, and if player 120 wants to stream video game 212 to BLE device 114 and operate game controller 102 in BLE mode to play video game 212, the player may provide this first gesture 302(1).
[0052] Player 120 can provide a second gesture 302(2) (second multi-button gesture 302(2)) to put handheld controller 102 into BLE mode by pressing a second combination of multiple buttons (e.g., B button 300(4) and guide button 304). If game controller 102 has been previously paired with a BLE device such as BLE device 114, player 120 can provide this second gesture 302(2) at the start of a game session. In different scenarios, such as when player 120 is playing a video game in receiver mode, player 120 can provide the second gesture 302(2) to switch from operating game controller 102 in receiver mode to operating game controller 102 in BLE mode, which may occur midway through a game session. In yet another different scenario, such as when player 120 is playing a video game in BLE mode, player 120 can provide the second gesture 302(2) to switch from a first BLE device (e.g., BLE device 114) to a second BLE device 112 in the environment. For example, the device log database 232 of the game controller 102 can be used to find different BLE devices known to the game controller 102 that the game controller 102 has previously connected to. In some embodiments, specific gestures can be used to invoke different BLE devices in the device log database 232. In this case, the player 120 can provide one gesture (e.g., the B button 300(4) and the guide button 304) to connect to the most recently used BLE device in the device log database 232, and another gesture (e.g., the start button and the guide button 304) to connect to the next most recently used BLE device in the device log database 232. The player 120 may have to recall these gestures to avoid having to switch between devices to reach the desired BLE device.
[0053] Player 120 can provide a third gesture 302(3) (third multi-button gesture 302(3)) to put handheld controller 102 into receiver pairing mode by pressing a third combination of multiple buttons (e.g., X button 300(1) and guide button 304). The third gesture 302(3) can be provided by player 120 if game controller 102 is not already paired with wireless receiver 118 and if player 120 wants to stream video game 212 to wireless receiver 118 of a target device connected to environment 112 and operate game controller 102 in receiver mode to play video game 212.
[0054] Player 120 can provide a fourth gesture 302(4) (fourth multi-button gesture 302(4)) to place handheld controller 102 into receiver mode by pressing a fourth combination of multiple buttons (e.g., A button 300(3) and guide button 304). Player 120 can provide this fourth gesture 302(4) at the start of a game session if game controller 102 has been previously paired with a wireless receiver such as wireless receiver 118. In different scenarios, such as when player 120 is playing a video game in BLE mode, player 120 can provide the fourth gesture 302(4) to switch from operating game controller 102 in BLE mode to operating game controller 102 in receiver mode, which may occur midway through a game session. In yet another different scenario, such as when player 120 is playing a video game in receiver mode, player 120 can provide the fourth gesture 302(4) to switch from a first wireless receiver (e.g., wireless receiver 118) to a second wireless receiver 112 in the environment. For example, the device log database 232 of the game controller 102 can be used to locate different wireless receivers known to the game controller 102 that the game controller 102 has previously connected to. In some embodiments, specific gestures can be used to invoke different wireless receivers in the device log database 232. In this case, the player 120 can provide one gesture (e.g., A button 300(3) and Guide button 304) to connect to the most recently used wireless receiver in the device log database 232, and another gesture (e.g., Back button and Guide button 304) to connect to the next most recently used wireless receiver in the device log database 232. The player 120 may have to recall these gestures to avoid having to switch between devices to reach the desired wireless receiver.
[0055] In some implementations, player 120 may set a preference for connecting to preferred devices (e.g., preferred BLE devices and / or preferred wireless receivers) upon startup, and player 120 may switch between devices using gestures 302(2) or 302(4) depending on the current operating mode (i.e., BLE mode or receiver mode). Alternatively or otherwise, as previously described, device record database 232 may be used to determine the last known device (e.g., the last known BLE device and / or the last known wireless receiver) to which game controller 102 was last connected before receiving gesture 302. Upon startup, device record database 232 may maintain a history of connected devices in persistent (non-volatile) memory for connecting to the last known device upon startup. Device record database 232 may maintain addresses of previously connected devices, as well as security and / or authentication information (e.g., keys, passwords, credentials, etc.) for establishing radio links with these devices after previous disconnections. Alternatively, the operation mode component 238 of the game controller 102 may retain the last used operation mode (e.g., BLE mode or receiver mode) in local memory 224, and may revert to the last used operation mode at startup when the player 120 does not provide one of the gestures 302. In some embodiments, whenever the player 120 downloads the client application 252 to a new BLE device, the player 120 may be notified (e.g., via the client application 252) to connect their game controller 102 to the BLE device (e.g., by providing a first gesture 302(1) to the game controller 102). Similarly, whenever the player 120 connects the wireless receiver 118 to the BLE device running the client application 252, the player 120 may be notified (e.g., via the client application 252) to connect their game controller 102 to the wireless receiver (e.g., by providing a third gesture 302(3) to the game controller 102).
[0056] although Figure 3The image shows a specific multi-button gesture 302, but these are merely exemplary gestures, and other gestures can be implemented to provide similar functionality. For example, a touch sensor associated with a specific control, such as a top bumper control, can detect a swipe gesture. In this way, a swipe in one direction (e.g., from left to right) or on a first control (e.g., the left bumper) could correspond to a command to operate the game controller 102 in BLE mode, while a swipe in the opposite direction (e.g., from right to left) or on a second control (e.g., the right bumper) could correspond to a command to operate the game controller 102 in receiver mode. Other gestures are envisioned to be possible for this type of functionality, including voice gestures detected by the microphone of the game controller 102. For example, player 120 could say “BLE mode” to put the game controller 102 into BLE mode, or player 120 could say “receiver mode” to put the game controller 102 into receiver mode.
[0057] The processes described herein are shown as a set of boxes in a logic flowchart, representing a sequence of operations that can be implemented using hardware, software, firmware, or a combination thereof (referred to herein as "logic"). In a software environment, these boxes represent computer-executable instructions that, when executed by one or more processors, perform the enumerated operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific abstract data type. The order in which the operations are described is not intended to be construed as limiting, and any number of the described boxes can be combined in any order and / or in parallel to implement these processes.
[0058] Figure 4 This is a flowchart of an exemplary process 400 for operating a handheld video game controller 102 in one of a variety of operating modes. For the purposes of discussion, process 400 is described with reference to the preceding figures.
[0059] At 402, the game controller 102 can be activated by transitioning from a power-off state to a power-on state. In the power-on state, power is supplied to one or more electronic components of the game controller 102 from a power source (e.g., one or more batteries). Activating the game controller 102 at block 402 may be based on (e.g., in response to) the game controller 102 receiving a first user input. For example, a power button (such as...) may be pressed. Figure 3 The guide button 304 shown is used to activate the game controller 102 (powering it on or turning it on).
[0060] At 404, the logic of game controller 102 can determine whether the first user input received at box 402 includes a gesture, such as a predefined gesture stored in memory 224 of game controller 102. At box 404, if it is determined that the first user input received at box 402 does not include a gesture (e.g., player 120 may simply press a power button such as guide button 304 to power on game controller 102), process 400 can follow a "no" route from box 404 to box 406.
[0061] At block 406, the logic of game controller 102 can determine which of a plurality of operating modes is used to operate game controller 102, wherein game controller 102 is the last to be operated. For example, the plurality of operating modes may include BLE mode and receiver mode, as described herein. At block 406, the logic of game controller 102 can query data maintained by operating mode component 238 to determine which of these modes was last (most recently) used to operate game controller 102 before receiving the first user input to start game controller 102 at block 402. If BLE mode is identified as the last mode for operating game controller 102, then BLE mode is selected at block 406. If receiver mode is identified as the last mode for operating game controller 102, then receiver mode is selected at block 406.
[0062] At 408, the logic of the game controller 102 can determine the devices (e.g., BLE devices or wireless receivers) previously connected to the game controller 102. Specifically, at block 408, the logic can identify, at least in part, the device last connected to the handheld video game controller before receiving the first user input to start the game controller 102 at block 402, based on data stored in the memory 224 of the game controller 102 (e.g., from the device record database 232). These devices stored in the memory 224 of the game controller 102 can be associated with specific operating modes (e.g., at least one of BLE mode or wireless receiver mode). This can be used as a filtering criterion for selecting devices at block 408. That is, if BLE mode is selected as the last mode used at block 406, the most recently connected BLE device 114 can be identified and selected at block 408. On the other hand, if receiver mode is selected as the last mode used at block 406, the most recently connected wireless receiver 118 can be identified and selected at block 408.
[0063] At 410, with the operating mode and device selected, the game controller 102 can establish a wireless connection with the selected device via its radio using an appropriate protocol. For example, if operating in BLE mode, the game controller 102 can establish a wireless connection with the BLE device 114 executing the client application 252 via its radio using the BLE protocol. However, if operating in receiver mode, the game controller 102 can establish a wireless connection with the wireless receiver 118 via its radio using a non-BLE protocol (e.g., WiFi). Establishing a wireless connection may include establishing a radio link through which data can be transmitted (e.g., authentication, handshake, etc.). This may involve identifying the device's radio signal (and possibly, the strength of the radio signal).
[0064] At 412, game controller 102 may receive second user input controlling one aspect of a video game executed on a different host computer (e.g., client machine 106) than the device (such as BLE device 114 or wireless receiver 118) to which game controller 102 is connected at box 410. For example, client machine 106 may be executing video game 212, streaming video game data 122 to a target device. If operating in BLE mode, the target device may be the BLE device 114 to which game controller 102 is connected at box 410. If operating in receiver mode, the target device may be a device (e.g., television 116) connected to the wireless receiver 118 to which game controller 102 is connected at box 410. Player 120 may play the video game by providing user input to game controller 102 (e.g., operating controls on game controller 102, moving game controller 102 (e.g., detected by motion sensors), etc.).
[0065] At 414, the game controller 102 can transmit controller input data 124 to the device to which it is connected at box 410. Similarly, depending on the operating mode, the receiving device for the controller input data 124 can be a BLE device 114 or a wireless receiver 118. In BLE mode operation, the controller input data 124 can be transmitted to the BLE device 114 via the game controller 102's radio using the BLE protocol. In receiver mode operation, the controller input data 124 can be transmitted to the wireless receiver 118 via the game controller 102's radio using a non-BLE protocol (e.g., WiFi). The controller input data 124 can be based on user input received at box 412 and can be packaged in any suitable format.
[0066] In some implementations, when operating in BLE mode, the logic of game controller 102 can use custom BLE features to generate controller input data 124, which can be received by client application 252 executing on BLE device 114. This may involve connecting a BLE profile to the GATT layer in the BLE protocol stack for transmitting features in controller input data 124.
[0067] Return to reference box 404 if it is determined that the first user input received at box 402 includes a gesture (e.g., a multi-button gesture, as shown in reference box 404). Figure 3 If the above is true, then process 400 can follow the "yes" route from box 404 to box 416.
[0068] At 416, the logic of game controller 102 can determine the type of gesture detected. For example, the memory 224 of game controller 102 can store multiple predefined gestures, such as reference gestures. Figure 3 The gestures 302(1)-(4) are described above. These exemplary gestures 302 can be categorized as paired gestures (e.g., gestures 302(1) and 302(3) are paired gestures) and mode selection gestures (e.g., gestures 302(2) and 302(4) are mode selection gestures). At block 416, if the gesture is determined to be a paired gesture, process 400 can proceed from block 416 to block 418, where the logic of game controller 102 determines to operate game controller 102 in a specific mode associated with the paired gesture. For example, if the gesture is Figure 3 If the paired gesture 302(1) is used, the logic can determine that the operation is in BLE mode, and if the gesture is... Figure 3 If the pairing gesture 302(3) is executed, the logic can determine that it operates in receiver mode.
[0069] At 420, based on the operating mode selected at box 418, the logic of game controller 102 can determine a new device (i.e., a new device) that the game controller is not yet connected to for that mode. For BLE mode, this could be a new BLE device 114 that enters BLE pairing mode via client application 252 executed on BLE device 114. For receiver mode, this could be a new wireless receiver 118 that enters receiver pairing mode, as described herein.
[0070] If a new device is identified and selected at box 420, boxes 410 to 414 can be performed as described above but relative to the newly identified device. That is, the game controller 102 can establish a wireless connection with the new device (box 410), receive (second) user input (box 412) controlling one aspect of the video game executed on the host computer, and transmit controller input data 124 to the connected device via radio using an appropriate protocol based on the (second) user input (box 414).
[0071] Returning to reference box 416, if the gesture is determined to be a mode selection gesture, process 400 can proceed from box 416 to box 422, where the logic of game controller 102 determines to operate game controller 102 in the specific mode associated with the mode selection gesture. For example, if the gesture is... Figure 3 If the mode selection gesture is 302(2), then the logic can determine that the operation is in BLE mode, and if the gesture is Figure 3 If the mode selection gesture 302(4) is used, then the logic can determine that the operation is in receiver mode.
[0072] At 424, the logic of game controller 102 can determine whether a gesture is associated with a specific device among a plurality of devices to which game controller 102 has previously been connected. For example, data stored in memory 224 of game controller 102 (e.g., device record database 232) can maintain a list of devices to which game controller 102 has previously been connected. These devices can be filtered based on the operating mode selected at box 422. If the gesture is specific to one of the devices associated with the selected operating mode, process 400 can follow a “yes” route from box 424 to box 426, at which the device associated with the gesture can be selected. For example, player 120 can provide one gesture (e.g., B button 300(4) and guide button 304) to connect to the most recently used BLE device in device record database 232, and another gesture (e.g., start button and guide button 304) to connect to the next most recently used BLE device in device record database 232. Depending on which of these gestures is detected, the device associated with that gesture can be selected as the identified device at box 426. Similar logic applies to the receiver pattern, as described in this article.
[0073] At box 424, if the gesture is not device-specific (e.g., the gesture is not specific to one of the devices associated with the selected operating mode), process 400 may follow a "No" route from box 424 to box 428, where at box 428, the logic selects the device to which game controller 102 was last connected before receiving the first user input to start game controller 102 at box 402. That is, if the gesture was not originally a device-specific gesture, the most recently connected device can be selected at box 428. In BLE mode, this device is the BLE device 114 to which game controller 102 was last connected before receiving the first user input to start game controller 102 at box 402. In receiver mode, this device is the wireless receiver 118 to which game controller 102 was last connected before receiving the first user input to start game controller 102 at box 402.
[0074] Following box 428 or box 426, boxes 410 through 414 may be performed as described above but relative to the device identified and selected at box 428 or box 426. That is, the game controller 102 may establish a wireless connection with the selected device (box 410), receive (second) user input (box 412) controlling one aspect of a video game executed on a host computer, and transmit controller input data 124 via radio to the connected device using an appropriate protocol based on the (second) user input (box 414).
[0075] Figure 5 This is a flowchart of an exemplary process 500 for operating BLE device 114 to stream a video game executed on a host computer and receiving controller input data from game controller 102 via BLE. Process 500 is described with reference to the preceding figures for discussion purposes.
[0076] At 502, BLE device 114 (which is connected to the LAN of environment 112) can execute client application 252. The client application 252 can configure BLE device 114 as a target device for video game streaming and also as a hub device for controller input data provided by a connected game controller 102 (e.g., via a BLE connection).
[0077] At 504, during the execution of video game 212 on a host computer (e.g., client machine 106 co-located with BLE device 114 in environment 112), BLE device 114 may receive video game data 122 streaming from the host computer (e.g., client machine 106) via a LAN (e.g., via WAP 110).
[0078] At 506, BLE device 114 may output audio and video data (e.g., video game data 122 including audio and video data) via output devices of BLE device 114 (e.g., display 244 and speaker).
[0079] At 508, BLE device 114, acting as a hub device, can receive controller input data 124 from the connected game controller 102 via radio using the BLE protocol. Custom BLE features available via client application 252 allow controller input data to be received in a manner that bypasses any OS restrictions on game controller input.
[0080] At 510, BLE device 114 can send controller input data 124, which it receives from game controller 102, to a host computer (e.g., client machine 106) executing video game 212. Controller input data 124 can be sent from BLE device 114 to the host computer via LAN (e.g., via WAP 110).
[0081] Figure 6 This is a flowchart of an exemplary process 600 for operating a host computer to stream a video game executed thereon to a target device while simultaneously receiving controller input data from a game controller separate from the host computer. Process 600 is described with reference to the preceding figures for discussion purposes.
[0082] At 602, a host computer (e.g., client machine 106, remote system 104, etc.) can use video game client 210 to execute video game 212. Video game client 210 can be configured to allow player 120 to play the video game on the host computer itself, and / or allow video game 212 to be streamed to a target device (e.g., via streaming component 216).
[0083] At 604, the video game client 210 may attach to one or more input libraries of the video game 212 executing on the host computer to determine how to convert the controller input data 124 into data usable by the video game 212. This allows the video game client 210 to intercept function calls from the video game 212 code for reading the game controller state and return the controller state that the video game 212 expects to see, which may involve remapping the state of the connected game controller 102 to different types of game controller states.
[0084] At 606, the host computer can transmit video game data 122 captured by the video game client 210 to the target device via LAN. Depending on the operating mode of the game controller 102, as described herein, this may involve transmitting the video game data 122 to the BLE device 114 in BLE mode, or to the wireless receiver 118 (connected to or embedded in the target device) in receiver mode. The video game data 122 may include audio and video data, and this data may be encoded for transmission over the LAN, as described herein.
[0085] At 608, the host computer can receive controller input data 124 from the transmitting device via LAN. Similarly, depending on the operating mode of the game controller 102, in BLE mode, the transmitting device for controller input data 124 can be BLE device 114, or in receiver mode, the transmitting device can be wireless receiver 118.
[0086] At box 610, video game client 210 can convert controller input data 124 into video game input that the video game 212 code can understand / process. This conversion at box 610 may be based at least in part on the input library of the attached video game. The conversion at box 610 may also be based on other factors, such as player 120 preferences known to video game client 210 regarding how player 120 prefers to convert controller input data 124. In some implementations, the conversion at box 610 may involve remapping controller input data 124 into appropriate video game input that can be injected into the video game 212 code. In some implementations, video game developers may utilize APIs that allow them to specify game controller configurations that remap (or otherwise convert) controller input data 124 from various types of controllers into video game input suitable for their video game. Therefore, at box 610, video game client 210 may utilize such game controller configurations for remapping. The video game client 210 can also switch between different types of game controllers (such as controllers from various game controller manufacturers on the market) and may switch between mouse and keyboard input.
[0087] At 612, the video game client 210 can "inject" video game input (derived from controller input data 124) into the video game 212 code to control one aspect of the video game 212 executing on the host computer. For example, video game input that causes a player-controlled character to jump may cause the player-controlled character to jump over the next few frames as it is streamed to the target device.
[0088] Although the subject matter has been described in language specific to structural features, it should be understood that the subject matter defined in the appended claims is not limited to the specific features described. Rather, specific features are disclosed as exemplary forms for implementing the claims.
[0089] Example Project
[0090] Project 1. A handheld video game controller, the handheld video game controller comprising:
[0091] processor;
[0092] Radio; and
[0093] The memory stores computer-executable instructions that, when executed by the processor, cause the handheld video game controller to:
[0094] Based at least in part on the first user input received by the handheld video game controller:
[0095] It is activated by powering one or more electronic components of the handheld video game controller; and
[0096] Confirm that it is operating in Bluetooth Low Energy (BLE) mode;
[0097] The BLE device to which the handheld video game controller was previously connected is determined, at least in part based on data stored in the memory.
[0098] A wireless connection is established between the BLE device executing the client application and the radio.
[0099] Receive second user input controlling one aspect of a video game executed on a host computer different from the BLE device; and
[0100] The controller input data is transmitted to the BLE device via the radio, and the controller input data is based at least in part on the second user input.
[0101] Project 2. The handheld video game controller according to Project 1, wherein the computer-executable instructions, when executed by the processor, further cause the handheld video game controller to:
[0102] Receive third user input corresponding to commands that operate the handheld video game controller in receiver mode;
[0103] Based at least in part on the data stored in the memory, the wireless receiver to which the handheld video game controller was previously connected is determined, and the wireless receiver is different from the BLE device;
[0104] A wireless connection is established between the radio and the wireless receiver;
[0105] Receive fourth user input controlling additional aspects of the video game executed on the host computer; and
[0106] Additional controller input data is transmitted to the wireless receiver via the radio, the additional controller input data being at least partially based on the fourth user input.
[0107] Project 3. A handheld video game controller according to Project 2, wherein the third user input includes gestures corresponding to the commands used to operate the handheld video game controller in the receiver mode.
[0108] Project 4. The handheld video game controller according to Project 1, wherein determining operation in the BLE mode includes determining that the first user input includes a gesture corresponding to a command operated in the BLE mode.
[0109] Project 5. The handheld video game controller according to Project 4, wherein the computer-executable instructions, when executed by the processor, further cause the handheld video game controller to:
[0110] Determine that the gesture is associated with one of the multiple BLE devices previously connected to the handheld video game controller; and
[0111] The BLE device among the plurality of BLE devices is identified at least in part based on the gesture.
[0112] Project 6. The handheld video game controller according to Project 1, wherein determining the BLE device includes identifying the BLE device as the last BLE device to which the handheld video game controller was connected before receiving the first user input.
[0113] Item 7. A method, the method comprising:
[0114] The first user input is received by a handheld video game controller;
[0115] Based at least in part on the first user input:
[0116] Activate the handheld video game controller; and
[0117] It was determined that the handheld video game controller was operated in Bluetooth Low Energy (BLE) mode;
[0118] The BLE device previously connected to the handheld video game controller is determined, at least in part, based on data stored in the memory of the handheld video game controller;
[0119] A wireless connection is established between the handheld video game controller and the BLE device executing the client application via the controller's radio.
[0120] The handheld video game controller receives second user input controlling one aspect of a video game executed on a host computer different from the BLE device; and
[0121] The handheld video game controller transmits controller input data to the BLE device via the radio, the controller input data being at least partially based on the second user input.
[0122] Project 8. According to the method of Project 7, the method further includes:
[0123] The handheld video game controller receives third user input corresponding to the command operated in receiver mode;
[0124] Based at least in part on the data stored in the memory of the handheld video game controller, a wireless receiver previously connected to the handheld video game controller, which is different from the BLE device, is determined;
[0125] The handheld video game controller establishes a wireless connection with the wireless receiver via the radio.
[0126] The handheld video game controller receives a fourth user input that controls additional aspects of the video game executed on the host computer; and
[0127] The handheld video game controller transmits additional controller input data to the wireless receiver via the radio, the additional controller input data being at least partially based on the fourth user input.
[0128] Project 9. The method of Project 7, wherein determining to operate the handheld video game controller in the BLE mode includes determining that the first user input includes a gesture corresponding to a command to operate the handheld video game controller in the BLE mode.
[0129] Item 10. The method of Item 9, wherein the gesture includes a multi-button gesture detected by the handheld video game controller as a combination of pressing a plurality of buttons, the combination of the plurality of buttons including a first button and a second button of the handheld video game controller.
[0130] Item 11. The method according to Item 9, wherein determining the BLE device comprises:
[0131] Determine that the gesture is associated with one of the multiple BLE devices previously connected to the handheld video game controller; and
[0132] The BLE device among the plurality of BLE devices is identified at least in part based on the gesture.
[0133] Project 12. The method of Project 7, wherein determining the BLE device includes identifying the BLE device as the last BLE device to which the handheld video game controller was connected before the first user input was received.
[0134] Item 13. The method of Item 12, wherein determining the operation of the handheld video game controller in the BLE mode comprises:
[0135] The BLE mode is identified as the operating mode of the last operation of the handheld video game controller before the first user input was received.
[0136] Item 14. One or more non-transitory computer-readable media storing computer-executable instructions, which, when executed by a processor of a handheld video game controller, cause the handheld video game controller to perform operations including:
[0137] Receive the first user input;
[0138] Based at least in part on the first user input:
[0139] Power on the handheld video game controller; and
[0140] It was determined that the handheld video game controller was operated in Bluetooth Low Energy (BLE) mode;
[0141] The BLE device previously connected to the handheld video game controller is determined, at least in part, based on data stored in the memory of the handheld video game controller;
[0142] A wireless connection is established between the handheld video game controller and the BLE device executing the client application via a radio.
[0143] Receive second user input controlling one aspect of a video game executed on a host computer different from the BLE device; and
[0144] The controller input data is transmitted to the BLE device via the radio, and the controller input data is based at least in part on the second user input.
[0145] Item 15. According to one or more non-transitory computer-readable media of Item 14, the operation further includes:
[0146] Receive third user input corresponding to commands that operate the handheld video game controller in receiver mode;
[0147] Based at least in part on the data stored in the memory of the handheld video game controller, a wireless receiver previously connected to the handheld video game controller, which is different from the BLE device, is determined;
[0148] A wireless connection is established between the radio and the wireless receiver;
[0149] Receive fourth user input controlling additional aspects of the video game executed on the host computer; and
[0150] Additional controller input data is transmitted to the wireless receiver via the radio, the additional controller input data being at least partially based on the fourth user input.
[0151] Item 16. One or more non-transitory computer-readable media according to Item 14, wherein determining to operate the handheld video game controller in the BLE mode includes determining that the first user input includes a gesture corresponding to a command to operate the handheld video game controller in the BLE mode.
[0152] Item 17. One or more non-transitory computer-readable media according to Item 16, wherein the gesture includes a multi-button gesture detected by the handheld video game controller as a combination of pressing a plurality of buttons, the combination of the plurality of buttons including a first button and a second button of the handheld video game controller.
[0153] Item 18. One or more non-transitory computer-readable media according to Item 16, wherein determining the BLE device comprises:
[0154] Determine that the gesture is associated with one of the multiple BLE devices previously connected to the handheld video game controller; and
[0155] The BLE device among the plurality of BLE devices is identified at least in part based on the gesture.
[0156] Item 19. One or more non-transitory computer-readable media according to Item 14, wherein determining the BLE device includes identifying the BLE device as the BLE device to which the handheld video game controller was last connected before the first user input was received.
[0157] Item 20. One or more non-transitory computer-readable media according to Item 14, wherein determining to operate the handheld video game controller in the BLE mode comprises:
[0158] The BLE mode is identified as the operating mode of the last operation of the handheld video game controller before the first user input was received.
Claims
1. A handheld controller, comprising: processor; radio; and The memory stores computer-executable instructions that, when executed by the processor, cause the handheld controller to: The user input received by the handheld controller is determined to include gestures; The gesture is determined to be a mode selection gesture associated with an operating mode among multiple operating modes of the handheld controller, wherein the operating mode is associated with a second wireless protocol; and At least in part, this is based on determining that the gesture is a mode selection gesture associated with the operation mode: Switching from using a first wireless protocol to using a second wireless protocol, wherein the first wireless protocol is different from the second wireless protocol; and The radio is used to send controller input data to a device configured to use the second wireless protocol.
2. The handheld controller of claim 1, wherein the computer-executable instructions, when executed by the processor, further cause the handheld controller to: The list of devices previously connected to the handheld controller is filtered, at least in part, based on the operating mode to obtain a filtered list of devices associated with the operating mode; Select the device from the list of filtered devices; as well as A wireless connection with the device is established before the controller input data is sent to the device.
3. The handheld controller according to claim 1, wherein: Determining that the gesture is the mode selection gesture associated with the operation mode includes determining that the gesture is a first mode selection gesture among a plurality of mode selection gestures; and The plurality of mode selection gestures include at least a first mode selection gesture and a second mode selection gesture associated with a second operation mode among the plurality of operation modes.
4. The handheld controller of claim 1, wherein the gesture includes at least one of the following: A swipe gesture detected by the touch sensor of the handheld controller; Voice gestures detected by the microphone of the handheld controller; or The handheld controller detects a multi-button gesture as a combination of pressing multiple buttons on the handheld controller.
5. The handheld controller of claim 1, wherein the computer-executable instructions, when executed by the processor, further cause the handheld controller to: The second user input received by the handheld controller is determined to include a second gesture; The second gesture is determined to be a second mode selection gesture associated with a second operation mode among the multiple operation modes, wherein, The second operating mode is associated with the first wireless protocol; as well as Based at least in part on determining that the second gesture is a second mode selection gesture associated with the second operating mode, the second controller input data is transmitted using the radio to a second device configured to use the first wireless protocol.
6. The handheld controller according to claim 1, wherein: The operating modes include Bluetooth Low Energy (BLE) mode; and The second wireless protocol includes the BLE protocol.
7. The handheld controller of claim 1, wherein the handheld controller is a game controller configured to control one aspect of a video game.
8. A method for sending controller input data to a device via a handheld controller, the method comprising: The handheld controller determines that the user input received by the handheld controller includes gestures; The handheld controller determines the gesture as a mode selection gesture associated with an operating mode among multiple operating modes of the handheld controller, wherein the operating mode is associated with a second wireless protocol; and At least in part, this is based on determining that the gesture is a mode selection gesture associated with the operation mode: The handheld controller switches from using a first wireless protocol to using a second wireless protocol, wherein the first wireless protocol is different from the second wireless protocol; and The handheld controller transmits controller input data to a device configured to use the second wireless protocol via its radio.
9. The method according to claim 8, further comprising: The handheld controller determines that the gesture is a device-specific gesture; The device is selected by the handheld controller at least in part based on the association between the device and a specific gesture of the device; as well as The handheld controller establishes a wireless connection with the device before sending the controller input data to the device.
10. The method of claim 8, further comprising: The handheld controller determines that the gesture is not a device-specific gesture; The device is selected by the handheld controller at least in part based on the fact that the handheld controller was last connected to the device before receiving the user input; as well as The handheld controller establishes a wireless connection with the device before sending the controller input data to the device.
11. The method according to claim 8, wherein: Determining that the gesture is the mode selection gesture associated with the operation mode includes determining that the gesture is a first mode selection gesture among a plurality of mode selection gestures; and The plurality of mode selection gestures include at least a first mode selection gesture and a second mode selection gesture associated with a second operation mode among the plurality of operation modes.
12. The method of claim 8, wherein the gesture comprises at least one of the following: A swipe gesture detected by the touch sensor of the handheld controller; Voice gestures detected by the microphone of the handheld controller; or The handheld controller detects a multi-button gesture as a combination of pressing multiple buttons on the handheld controller.
13. The method of claim 8, further comprising: The handheld controller determines that the second user input received by the handheld controller includes a second gesture; The handheld controller determines that the second gesture is a second mode selection gesture associated with a second operation mode among the multiple operation modes, wherein the second operation mode is associated with the first wireless protocol; as well as Based at least in part on determining that the second gesture is a second mode selection gesture associated with the second operating mode, the handheld controller sends second controller input data to a second device configured to use the first wireless protocol using the radio.
14. The method according to claim 8, wherein: The operating modes include Bluetooth Low Energy (BLE) mode; and The second wireless protocol includes the BLE protocol.
15. A method for sending controller input data to a device via a handheld controller, the method comprising: The handheld controller determines that the user input received by the handheld controller includes gestures; The handheld controller determines that the gesture is a paired gesture associated with one of multiple operating modes of the handheld controller, wherein the operating mode is associated with a second wireless protocol; and At least in part, this is based on determining that the gesture is a paired gesture associated with the operating mode: The handheld controller switches from using a first wireless protocol to using a second wireless protocol, wherein the first wireless protocol is different from the second wireless protocol; and The handheld controller transmits controller input data to a device configured to use the second wireless protocol via its radio.
16. The method of claim 15, wherein: Determining that the gesture is a paired gesture associated with the operation mode includes determining that the gesture is a first paired gesture among a plurality of paired gestures; and The plurality of paired gestures includes at least a first paired gesture and a second paired gesture associated with a second operating mode in the plurality of operating modes.
17. The method of claim 15, further comprising: The device is identified by the handheld controller at least in part based on the fact that the device has been placed in a pairing mode associated with the second wireless protocol. The handheld controller establishes a wireless connection with the device before sending the controller input data to the device.
18. The method of claim 15, further comprising: The handheld controller determines that the second user input received by the handheld controller includes a second gesture; The handheld controller determines that the second gesture is a second paired gesture associated with a second operating mode among the multiple operating modes, wherein the second operating mode is associated with the first wireless protocol; as well as Based at least in part on determining that the second gesture is a second paired gesture associated with the second operating mode, the handheld controller sends second controller input data to a second device configured to use the first wireless protocol using the radio.
19. The method of claim 15, wherein the handheld controller is a game controller configured to control one aspect of a video game.
20. The method of claim 15, wherein: The operating modes include Bluetooth Low Energy (BLE) mode; and The second wireless protocol includes the BLE protocol.
Citation Information
Patent Citations
System and method for providing complex haptic stimulation during input of control gestures, and relating to control of virtual equipment
CN102473034A
Gesture controlled game screen navigation
US20090305785A1
