Smart user input event routing for cloud-connected game accessories

Abstract

A method for managing user input events from a cloud-connected accessory device performed by an accessory service involves identifying a user associated with the accessory device remotely connected to the service. The method also identifies a group of clients associated with the user, including first and second clients, each of which maintains a connection with the service. User input events are routed based on event type and routing policy, determining whether to direct the event to an active client or a cloud gaming service. A routing destination is calculated for each event, and the event is sent accordingly. Specific routing scenarios include directing events to an active client or a cloud gaming service based on event type and client state, ensuring efficient and effective routing.

Description

Technical Field

[0001] This disclosure relates to systems, methods, and apparatus for routing user input events received from directly connected cloud accessory devices connected to a cloud gaming service. Background Technology

[0002] Cloud gaming is a technology that allows users to stream and play video games on various client devices, such as browsers, personal computers, mobile devices, and smart TVs, without the need to install or download the games. The game is hosted and executed on remote servers in a data center, and video and audio output is streamed to the user's client device via the internet. User input, such as controller, keyboard, mouse, or touch commands, is sent from the user's client device to the server to control the game.

[0003] Traditionally, accessories used for cloud streaming game input (such as game controllers, keyboards, and mice) have been connected to client devices via local connections (such as short-range wireless technologies like Bluetooth or short-range wired technologies like Universal Serial Bus). This means that the client device relays user input events to the cloud gaming service.

[0004] Recently, Direct Connect to the Cloud (DTC) accessories have been able to directly connect to and communicate with cloud gaming services (e.g., using wireless LAN protocols such as Wi-Fi). DTC accessories eliminate the need to manually connect input accessories separately to client devices and can reduce input latency (e.g., by reducing the number of network hops between the accessory and the cloud gaming service). The term DTC accessory refers to handheld, portable, or wearable user input devices, such as game controllers, keyboards, mice, or head-mounted displays (HMDs), that can send user input event data related to user input events performed by the user using the DTC accessory to the cloud gaming service, bypassing the associated game display device. This associated game display device is the device the user uses to view the game they are playing, provided by the cloud gaming service. The user operates the DTC accessory to perform user input, influencing the game's progress.

[0005] The subject matter claimed herein is not limited to embodiments that address any shortcomings or operate solely in the environment described above. Rather, this background is provided merely to illustrate an example technical field in which some of the embodiments described herein may be practiced. Summary of the Invention

[0006] In some aspects, the technology described herein relates to methods, systems, and computer program products for routing multiple user input events received from a Direct Connected Cloud (DTC) accessory device. The method includes: identifying a user associated with the DTC accessory device, which is remotely connected to an accessory service via a network connection; identifying a set of clients associated with the user, each client in the set having a connection to the accessory service, the client set including a first client and a second client; and routing the multiple inputs received from the DTC accessory device, including, for each specific user input event: identifying a user input event type for that specific user input event; identifying a routing policy for that user input event type, the routing policy indicating that user input events of that user input event type should be routed to active clients in the client set. The client is still routed to the cloud gaming service; based on the routing strategy, the routing destination for the specific user input event is calculated, the routing destination including the client with an active state or the cloud gaming service; and the specific user input event is routed to the routing destination, wherein routing the plurality of user input events received from the DTC accessory device includes at least: routing the first user input event to the first client based on the first user input event having a first user input event type and the first client having an active state; routing the second user input event to the cloud gaming service based on the second user input event having a second user input event type; and routing the third user input event to the second client based on the second user input event having the first user input event type and the active state having been transferred from the first client to the second client.

[0007] This invention provides a simplified overview of some concepts, which will be further described in the detailed embodiments below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to define the scope of the claimed subject matter.

[0008] BRIEF DESCRIPTION OF DRAWINGS

[0009] To illustrate how the advantages of the systems and methods described herein are obtained, the embodiments briefly described above are described in more detail with reference to specific examples illustrated in the accompanying drawings. These drawings depict only typical embodiments of the systems and methods described herein and should not be considered as limiting their scope. The systems and methods are described and explained with additional specificity and detail using the accompanying drawings, in which: Figure 1A –1D illustration shows an example of a computer architecture that facilitates the routing of user input events generated by Direct Connected Cloud (DTC) accessory devices used with cloud gaming services; Figure 2A The illustration shows an example of a user interacting with a server-generated client-side user interface (UI) of a game session; Figure 2B The illustration shows an example of a user interacting with a client-generated UI on a client device; Figure 3 The illustration shows a sample UI that includes optional options for client devices to transition to an active state; Figure 4 The illustration shows an example of chat UI overlaid on game UI; and Figure 5 The diagram illustrates a flowchart of an example method implemented in the accessory service for routing user input events received from DTC accessory devices. Detailed Implementation

[0010] One of the challenges of cloud gaming is providing a seamless and consistent user experience across different devices and platforms, especially when users want to switch from one client device to another during a cloud gaming session. In one example scenario, a user might want to start playing a streaming game in a browser on a personal computer (PC) and then continue playing the same game on a smart TV in another room without losing game progress or connection to the server. For instance, with accessory devices connected via a traditional bluetouch or universal serial bus, if a user wants to play a cloud streaming game on their mobile device, they must manually connect their accessory to that device. Then, if the user changes their mind and wants to play the cloud streaming game on a smart TV, they need to manually connect their accessory to that smart TV, thus interrupting the gaming session. While newer Direct Connect to the Cloud (DTC) accessory devices eliminate the need to manually connect accessories to various client devices, once a DTC accessory device is connected to the cloud gaming service, the service completely consumes the user input events generated by that DTC accessory device. In these cases, the client device acts as a viewer of the streaming game, and all user interfaces (UI) interacted with by the DTC user input events are generated by the cloud gaming service. Furthermore, when using DTC accessory devices, there is no mechanism for transferring cloud gaming sessions from one client device to another. User input events are data about user input, which can be timestamps, information about the type of user input event (e.g., mouse click, joystick movement, button press), the duration of the user input event, an identifier of the accessory device from which the user performed the user input event, or other characteristics of the user input.

[0011] At least some of the embodiments described herein implement a DTC environment that includes an accessory service that intelligently routes user input events generated from a given DTC accessory device to a cloud gaming service or a client device based on context. This enables the client device to provide functionality beyond a streaming game viewer, such as interacting with a client-generated UI using the DTC accessory device, even if the user input events generated by the DTC accessory device were initially sent to the cloud service to which the DTC accessory device is connected. For example, during a game streaming session, these embodiments enable DTC accessory device user input events to be intelligently routed to a cloud gaming service (e.g., for interacting with a streaming game UI) or to a client device (e.g., for interacting with a client-generated UI rendered on top of, instead of, the game UI).

[0012] Therefore, the embodiments described herein enable a single DTC accessory device to seamlessly interact with both cloud-rendered and locally generated UIs. It is worth noting that the ability to seamlessly switch user input event routing between client-rendered UIs and game streaming sessions is particularly relevant for features traditionally rendered by client devices rather than cloud services, such as title selection, party features, and watching game trailers.

[0013] Furthermore, in at least some embodiments, the DTC environment described herein intelligently routes individual user input events in a stream of user input events generated by DTC accessory devices among multiple client devices simultaneously connected and associated with a given user. In these embodiments, the accessory service identifies one client device as active and one or more accessory client devices as idle. Then, when routing user input events generated by the DTC accessory device to a client device, the accessory service routes these user input events to the active client device. The accessory service continuously tracks changes in the activity or idle state of these client devices. In one example, the accessory service sends polling messages to these client devices to determine their state. In another example, the accessory service receives messages from these client devices reporting their state. In yet another example, the accessory service receives messages from these client devices that implicitly indicate an active client device (e.g., the client device that recently sent the message initiating a game session is implicitly considered the active client device). Therefore, the accessory service can route DTC accessory device user input events to one client device when it is active, and then route them to another client device later when it becomes active. This provides a seamless user experience, allowing users to switch between client devices without interrupting their cloud gaming sessions.

[0014] It is worth noting that while the embodiments are primarily described in the context of routing user input event streams, the embodiments described herein can be applied to any data stream communicated by a DTC accessory device. For example, a DTC accessory device may include a microphone and / or a speaker. In this example, the accessory service can seamlessly route audio generated by the DTC accessory device, streaming games, chat services, etc. For instance, the accessory service can route audio generated by both streaming games and chat services to the DTC accessory device for mixing at that DTC accessory device and then playback by a speaker. Furthermore, depending on the context (e.g., UI features the user has already interacted with), the accessory service can route audio captured by the microphone of the DTC accessory device to the streaming game (e.g., for using the game's own chat functionality to talk to other players) and / or to a party chat service or event client device (e.g., for using party chat to talk to other players).

[0015] Figure 1A –1D illustrates example 100 (including examples 100a–100d) of a computer architecture that facilitates the routing of user input events generated by DTC accessory devices used with cloud gaming services. Initially, in example 100a, Figure 1A The diagram illustrates a computer architecture that includes a DTC accessory device 101 and one or more client devices (e.g., client devices 102a to 102n, collectively referred to as client devices 102), each connected to a network 106 (e.g., a wide area network, such as the Internet). In some examples, the DTC accessory device 101 is a game controller, headset, keyboard, mouse, etc. In some examples, client devices 102 include game consoles, web browsers (e.g., web applications), PCs (e.g., laptops, desktops), mobile devices, smart TVs, set-top boxes, etc.

[0016] Typically, DTC accessory devices and client devices are each associated with a corresponding user, and each DTC accessory device and at least one client device are associated with the same user. For the sake of brevity, the DTC accessory device 101 and client device 102 depicted in Examples 100a–100d are both described as being associated with the same user.

[0017] Typically, a user operates a DTC accessory device 101 to interact with a game session 116 at a cloud gaming service 103. The interface of the game session 116 is generated at the cloud gaming service 103 and streamed from the cloud gaming service 103 to at least one of the client devices 102. The client device then displays the streamed game session on a display. Therefore, the DTC accessory device 101 is illustrated to include inputs 107 (e.g., buttons, switches, joysticks, touch-sensitive controls, etc.) that generate user input events when the user interacts with them. However, in some examples, the DTC accessory device 101 may include additional hardware, such as audio hardware 108 (e.g., microphone, speaker), video hardware 109 (e.g., camera, screen), etc. Furthermore, each client device 102 is illustrated to include a corresponding display 110 (e.g., displays 110a at client device 102a to 110n at client device 102n) that display the streamed feed of the game session 116. It is worth noting that, in Example 100, each client device 102 has a local client UI 111 (e.g., client UI 111a at client device 102a to client UI 111n at client device 102n) for interacting with the cloud gaming service 103 and input devices 113 (e.g., input device 113a at client device 102a to input device 113n at client device 102n). Client devices 102 may include other hardware and / or software components, such as chat clients 112 for interacting with the chat service 105 (e.g., chat client 112a at client device 102a to chat client 112n at client device 102n), both of which are described below.

[0018] In the embodiments described herein, DTC accessory device 101 is a DTC device configured to communicate user input events generated from user interaction with input 107 to cloud gaming service 103 via network 106. In one example, DTC accessory device 101 utilizes a Wi-Fi network to communicate these user input events to cloud gaming service 103 via the Internet (network 106).

[0019] As described above, in the current DTC environment, the cloud gaming service completely consumes the user input events of the DTC accessory device, making the client device act as a viewer of the streaming game, and all UI that can be interacted with by the DTC is generated by the cloud gaming service. This means that the current DTC environment cannot allow the DTC accessory device to interact with the client device's local UI. Instead, local UI interaction is performed using a separate input device associated with the client device. Analogous to the environment of Example 100, under the current DTC model, the DTC accessory device 101 will only be able to interact with the game session 116, and any interaction with, for example, the client UI 111a will require the use of input device 113a.

[0020] However, unlike these current DTC environments, Example 100 includes an accessory service 104, which includes a DTC routing component 117 that receives user input events (based on input 107) from the DTC accessory device 101 and intelligently routes these events to either the cloud gaming service 103 (e.g., for interacting with the game session 116) or one of the client devices 102 (e.g., for interacting with the client UI 111 on that device). Therefore, the accessory service 104 provides the DTC accessory device 101 with the additional ability to interact with the local client UI, which simplifies and streamlines the user experience, as the user can now interact with both the game session 116 and the client UI 111 using a single device without the need for a separate input device 113.

[0021] The dashed boxes surrounding cloud gaming service 103 and accessory service 104 indicate that, in some embodiments, they are part of the same overall cloud service. In embodiments, establishing a connection to cloud gaming service 103 implies a connection to accessory service 104, and vice versa. As shown, in embodiments, cloud gaming service 103 and accessory service 104 may share access to database 114.

[0022] In an embodiment, after establishing a connection between a DTC accessory device (e.g., DTC accessory device 101) and cloud gaming service 103 and / or accessory service 104, accessory service 104 identifies the user associated with that DTC accessory device. For example, Example 100a illustrates a database 114 that includes associations 115. In an embodiment, association 115 associates various DTC accessory devices with various users, such as by associating accessory identifiers (e.g., serial numbers, globally unique identifiers) with user identifiers (e.g., usernames, globally unique identifiers).

[0023] In some embodiments, the attachment service 104 also identifies associated client device connections based on the identified user. For example, the attachment service 104 determines that client device 102a is connected to cloud gaming service 103 and / or attachment service 104. In some embodiments, the attachment service 104 identifies multiple associated client device connections and the active or standby states of these client device connections. For example, the attachment service 104 determines that both client device 102a and client device 102n are connected to cloud gaming service 103 and / or attachment service 104. In this example, the attachment service 104 may determine that client device 102a is active and may determine that client device 102n is in standby mode.

[0024] In this embodiment, only one client device is active at any given time, but multiple client devices may be in a standby state. In this embodiment, the attachment service 104 applies one or more strategies to determine which client device is active and which(s) are in a standby state. In one example, the first client device to establish a connection is active, and any subsequent client devices establishing a connection are in a standby state. In another example, the client device that most recently initiated the game session acquires the active state (e.g., by acquiring the active state from another client device that has transitioned to a standby state). In yet another example, a client device may explicitly request to acquire the active state (e.g., by acquiring the active state from another client device that has transitioned to a standby state).

[0025] In Example 100, cloud gaming service 103 and accessory service 104 are each illustrated as comprising multiple instances, for example, in different geographical locations. In this embodiment, after accessory service 104 has identified the user associated with DTC accessory device 101 and the associated client device 102 with which an established connection has been established, accessory service 104 routes all connections to a single instance of accessory service 104, such as an instance geographically close to the cloud gaming service 103 to which the user is connected and / or an instance geographically close to the client device 102. In this embodiment, this ensures favorable latency characteristics between client device 102 and accessory service 104.

[0026] In this embodiment, after the accessory service 104 has identified the user associated with the DTC accessory device 101 and the associated established client device 102, the accessory service 104 uses the DTC routing component 117 to intelligently route user input events based on the context of the game session 116 and the context of the client device 102. The DTC routing component 117 can apply various routing strategies, but in general, these strategies operate in a user-intuitive manner to seamlessly route user input events. One goal of these strategies is to allow users to seamlessly and dynamically switch between different client devices of the client device 102 without losing the game session 116. Another goal of these strategies is to allow users to interact with both the game session 116 UI and the client UI 111 using the DTC accessory device 101. Figure 1B –1D, 2A, 2B, 3, and 4 describe example scenarios.

[0027] In an embodiment, the DTC routing component 117 maintains a routing policy that specifies the expected destination for each type of user input event (e.g., different types of button presses). This routing policy indicates whether a given type of user input event will be routed to one of the active client devices 102 or to the cloud gaming service 103. In an embodiment, and as will be described below, the routing policy can route a given user input event type differently based on the UI context, such as which device (cloud gaming service 103 or one of the client devices 102) is rendering an active (e.g., foreground) UI.

[0028] exist Figure 1B In Example 100b, client device 102a is illustrated with a solid line to indicate that accessory service 104 has assigned it to an active state, and client device 102n is illustrated with a dashed line to indicate that accessory service 104 has assigned it to a standby state. Furthermore, the thick arrow connecting DTC accessory device 101 and accessory service 104 indicates that DTC accessory device 101 sends a stream of user input events to accessory service 104 based on user interaction with input 107. Correspondingly, two thin arrows extend from accessory service 104, one connecting accessory service 104 to cloud gaming service 103, and the other connecting accessory service 104 to client device 102a. In this embodiment, these two arrows indicate that DTC routing component 117 routes various user input events received from DTC accessory device 101 to cloud gaming service 103 or client device 102a according to the context.

[0029] Figure 2A Example 200a is illustrated, in which a user interacts with a game UI 201 corresponding to game session 116 on a display 110a of client device 102a. In this context, DTC routing component 117 routes various user input events generated by the user's interaction with input 107 to cloud gaming service 103 to interact with game UI 201. For example, DTC routing component 117 routes user input event data corresponding to directional buttons, selection buttons, etc., on DTC accessory device 101 to cloud gaming service 103 so that the user can navigate the menus of game UI 201, for example, to customize game characters. In this context, DTC routing component 117 may simultaneously route other input event data (such as user input event data corresponding to the user pressing a system menu button on DTC accessory device 101 (e.g., the XBOX button on an XBOX game controller)) to client device 102a to interact with or trigger client UI 111a.

[0030] Figure 2BExample 200b is illustrated, in which a user now interacts with a client UI 202 corresponding to client UI 111a on a display 110a of client device 102a. In one example, client UI 111a is initiated by the user interacting with the system menu button of DTC accessory device 101. In example 200b, client UI 202 overlays game UI 201, although client UI 202 can completely replace game UI 201. In this context, DTC routing component 117 routes various user input events to client device 102a to interact with client UI 111a. For example, DTC routing component 117 routes user input events corresponding to user interactions with directional buttons, selection buttons, etc., of input 107 to client device 102a so that the user can navigate menus generated by client UI 111a (e.g., game selection control 203, settings control 204, profile control 205). In this context, DTC routing component 117 can simultaneously route other user input events (such as user input events corresponding to the user pressing the pause button, voice chat button, etc. of input 107) to cloud gaming service 103 to interact with game UI 201.

[0031] exist Figure 1C In Example 100c, client device 102n is now illustrated with a solid line to indicate that accessory service 104 has now assigned it to an active state, and client device 102a is illustrated with a dashed line to indicate that accessory service 104 has now assigned it to a standby state. Furthermore, a thin arrow extending from accessory service 104 now connects accessory service 104 to client device 102n, rather than client device 102a. Therefore, in Example 100c, DTC routing component 117 routes various user input events received from DTC accessory device 101 to cloud gaming service 103 or client device 102n based on the context. In some embodiments, client device 102n becomes an active client based on a user initiating a game session 116 at client device 102n (e.g., based on interacting with client UI 111n using device 113n).

[0032] In other embodiments, client device 102n becomes an active client based on an explicit request from client device 102n. For example, Figure 3Example 300 is illustrated, where client device 102n displays a cloud gaming menu while in standby mode. Client UI 111n displays Connection UI 301, indicating the presence of DTC accessory device 101 and providing an optional option to obtain an active status (Connect My Controller 302). Based on selecting this option using device 113n, client device 102n transitions to an active state, while client device 102a transitions to standby mode. Therefore, this optional option (e.g., Connect My Controller 302) allows the user to "pull" DTC accessory device input to a given client. This can be useful when DTC routing component 117 selects a strategy that might not suit the user's preferences at a particular moment.

[0033] As described above, the routing embodiments described herein can be applied to any data stream communicating with DTC accessory device 101, such as audio (audio hardware 108) and video (video hardware 109). In some examples, accessory service 104 seamlessly routes audio and / or video between DTC accessory devices, streaming games, chat services, etc. Figure 1D Example 100d is illustrated, in which a chat service 105 is present, a DTC accessory device 101 includes audio hardware 108, and a client device 102 includes a chat client 112. The thick dashed line connecting the DTC accessory device 101 and the accessory service 104 indicates that the DTC accessory device 101 communicates with the accessory service 104 (e.g., sends and / or receives) audio data. Furthermore, the thin dashed line connecting the accessory service 104 to the cloud gaming service 103, the chat service 105, and the client device 102n indicates that the game session 116 can route the audio in various ways. For example, the DTC routing component 117 can route audio generated by the game session 116 (e.g., game audio) and / or audio generated by the chat service 105 (e.g., incoming voice chat audio) to the DTC accessory device 101 for output at the speaker of the DTC accessory device 101 (in some examples, this can mix audio from multiple sources). Additionally or alternatively, DTC routing component 117 may route audio generated by DTC accessory device 101 (e.g., audio captured by a microphone) to game session 116 (e.g., for in-game chat) and / or to chat service 105 (e.g., for outgoing voice chat audio). In an embodiment, DTC routing component 117 does not communicate audio with chat service 105, but rather with chat client 112 (e.g., chat client 112n), which in turn communicates with chat service 105.

[0034] In this embodiment, the DTC routing component 117 considers the chat client when routing user input events. For example, Figure 4Example 400 is illustrated, showing the routing of user input events to a chat UI. In Example 400, a game UI 401 is presented on a corresponding display 110 (e.g., display 110n), and a client UI 402 (e.g., generated by client UI 111n) and a chat UI 406 (e.g., generated by chat client 112n). Here, the DTC routing component 117 can appropriately route user input events received from the DTC accessory device 101 to these UIs based on context (such as which UI is being actively presented on the client device). For example, depending on the context and the specific input used, the DTC routing component 117 can route user input events to the client device 102n to interact with the client UI 402 (e.g., with the game selection control 403, settings control 404, profile control 405, etc.) and / or with the chat UI 406 (e.g., with the join control 408, chat control 409, talk control 410, etc.).

[0035] Now combine Figure 5 Describe the embodiments, Figure 5 A flowchart illustrating an example method 500 for routing user input events received from a DTC accessory device, implemented at an accessory service, is shown. In an embodiment, instructions for implementing method 500 are encoded as computer-executable instructions (e.g., DTC routing component 117) stored on a computer storage medium, which can be executed by a processor to cause a computer system (e.g., accessory service 104) to perform method 500.

[0036] The following discussion now concerns methods and method actions. Although method actions are discussed in a specific order or shown in flowcharts as occurring in a specific order, no order is required unless explicitly stated or required, because one action depends on another action being completed before that action is performed.

[0037] Reference Figure 5 In one embodiment, method 500 includes action 501 identifying a user associated with a connected DTC accessory device. In some embodiments, action 501 includes identifying a user associated with a DTC accessory device that is remotely connected to the accessory service via a network connection. For example, DTC accessory device 101 establishes a connection with cloud gaming service 103 and / or accessory service 104. As discussed above, DTC accessory device 101 may be a game controller, headset, keyboard, mouse, etc. Based on the established connection, accessory service 104 uses association 115 to determine the user associated with DTC accessory device 101.

[0038] Method 500 further includes an action 502 of identifying client devices associated with the DTC accessory device. In some embodiments, action 502 includes identifying a set of clients associated with the user, each client in the set having established a connection with the accessory service. For example, based on the established connections, accessory service 104 uses association 115 to determine one or more client devices 102 associated with the user. As discussed above, DTC accessory device 101 may be a game console, web browser, PC, mobile device, smart TV, set-top box, etc. For example, in the context of example 100b, accessory service 104 identifies client device 102a as an active client device and client device 102b as a standby client device. In some embodiments of method 500, the client set includes a first client and a second client, and identifying the set of clients associated with the user includes determining that the first client is active and the second client is in standby mode.

[0039] Method 500 further includes an action 503 of routing data received from the DTC accessory device. In some embodiments, the data received from the DTC accessory device is a stream of user input events (e.g., based on user interaction with input 107), and action 503 includes routing a plurality of user input events received from the DTC accessory device. In an embodiment, action 503 includes, for each specific user input event, identifying a user input event type for that specific user input event and identifying a routing policy for that user input event type. In an embodiment, each user input event type corresponds to a different type of user interaction with the input hardware at the DTC accessory device. In an embodiment, the routing policy indicates whether the user input event of that user input event type should be routed to an active client in the client set or to a cloud gaming service.

[0040] In these embodiments, action 503 further includes calculating a routing destination for each specific user input event based on the routing policy. The routing destination includes i) the client with the active state or ii) the cloud gaming service. In these embodiments, action 503 further includes routing each specific user input event to the routing destination.

[0041] As shown in the figure, in this embodiment, action 503 includes one or more of action 504, which routes first data to the client device, and action 505, which routes second data to the cloud gaming service. Figure 5 In this process, actions 504 and 505 do not need to be ordered. Therefore, these actions can be executed differently, either serially (in any order) or in parallel.

[0042] In some embodiments, action 504 includes routing a first user input event received from the DTC accessory device to the first client based on the first user input event having a first user input event type and the first client having an active state, while action 505 includes routing a second user input event received from the DTC accessory device to the cloud gaming service based on a second user input event having a second user input event type. In an embodiment, the first client uses the first user input event as navigation input for a first UI generated for the first client, and the cloud gaming service uses the second user input event as input for a game streamed by the cloud gaming service.

[0043] For example, in one embodiment, the routing policy indicates that when the UI generated by the client overrides the UI generated by the cloud gaming service, a user input event of the first input event type will be routed to the client with an active state; otherwise, the user input event of the first input event type will be routed to the cloud gaming service. In these embodiments, method 500 may include receiving from the first client an indication that the UI generated by the first client overrides the UI generated by the cloud gaming service.

[0044] For example, in Example 100b, DTC routing component 117 routes some user input events to client device 102a (action 504) based on client device 102a being active, and also routes some user input events to cloud gaming service 103 (action 505). For example, as combined with Figure 2A The DTC routing component 117 can route user input events corresponding to directional buttons, selection buttons, etc., to the cloud gaming service 103 (e.g., so that the user can navigate the menu of the game UI 201), and can route other user input events to the client device 102a (e.g., for interacting with or triggering the client UI 111a). The technical effect of this routing is to enable the user to seamlessly interact with both the game UI 201 and the client UI 111a using a single input device (i.e., the DTC accessory device 101).

[0045] Method 500 also includes an action 506 that identifies a context change. In some embodiments, action 506 determines a change in the UI context or the client device context. Therefore, as shown, action 506 includes one or more of an action 507 that identifies a client context change and an action 508 that identifies a cloud context change. Figure 5In this context, actions 507 and 508 do not need to be ordered. Therefore, these actions can be executed differently, either sequentially (in any order) or in parallel. Furthermore, it is not required that both actions 507 and 508 occur when a context change is identified. Examples of context changes include a transition of a client device between active and standby states (e.g., action 507), a client UI becoming the active UI (e.g., action 507), and a game session becoming the active UI (e.g., action 508).

[0046] Method 500 also includes action 509, which modifies the routing policy for user input events. For example, based on a context change detected in action 506, DTC routing component 117 changes its routing policy. Method 500 shows that after action 509, the method returns to action 503 to implement the new routing policy. For example, some embodiments update the routing policy based on at least one of a change in which client is active or which game the cloud gaming service is streaming.

[0047] As an example of a client device transitioning between active and standby states, in example 100c, client device 102n is now assigned an active state, and client device 102a is now assigned a standby state. Therefore, DTC routing component 117 changes its routing policy to route user input events to client device 102n instead of client device 102a. Thus, some embodiments of method 500 include identifying a transition of the first client to a standby state and a transition of the second client to an active state, and routing a first set of user input events received from the DTC accessory device to the second client instead of the first client based on this transition. In embodiments, the active state transitions from the first client to the second client based on detecting a request received from the second client to make it active (e.g., example 300) or by the second client initiating a game streaming session. In embodiments, after this transition, action 504 includes routing a third user input event to the second client based on the second user input event having the same type as the first user input event and the active state having transitioned from the first client to the second client. In one embodiment, the second client uses the third user input event as navigation input for the second UI generated by the second client.

[0048] As an example of UI transition, Figure 2A and Figure 2BIt is shown that a transition can occur between the game UI 201 and the client UI 202, and user input events can be routed accordingly. Therefore, in some embodiments of method 500, in a first context (e.g., example 200b), a specific user input event is identified for interaction with the UI at the client in the client set, and this specific user input event is routed from the DTC accessory device to the first client as part of the first set of user input events. Then, in some embodiments of method 500, in a second context (e.g., example 200a), the specific user input event is identified for interaction with the cloud gaming service, and this specific user input event is routed from the DTC accessory device to the cloud gaming service as part of the second set of user input events. In embodiments, the first context and the second context are based on one or more of a first UI context within the game UI presented by the cloud gaming service or a second UI context within the client UI presented by the first client.

[0049] Returning to action 503, in some embodiments, the data received from the DTC accessory device is audio. In these embodiments, action 503 includes routing the audio received from the DTC accessory device. In these embodiments, method 500 also includes identifying a chat session associated with the user and identifying the user's intent to interact with the chat session. Then, based on the user's intent, method 500 includes routing a first audio stream received from the DTC accessory device to the chat session and routing a second audio stream received from the chat session to the DTC accessory device. In these embodiments, routing a first set of user input events received from the DTC accessory device to the first client includes routing the user input events to a chat client executed at the first client. Referring to action 506, in an embodiment, method 500 includes later identifying the user's intent to interact with game audio and, based on the user's intent, routing the first audio stream received from the DTC accessory device to the cloud gaming service and routing a third audio stream received from the cloud gaming service to the DTC accessory device.

[0050] As described above, in the embodiments, the attachment service 104 routes all connections to a single instance of the attachment service 104, such as an instance geographically close to the client device 102. Therefore, in some embodiments of method 500, the attachment service is a first attachment service among a plurality of attachment services, each of which executes a different computer system, and the method includes connecting the DTC attachment device to the first attachment service rather than another attachment service among the plurality of attachment services based on the geographical location of the DTC attachment device relative to the computer system executing the first attachment service.

[0051] Therefore, the embodiments described herein enable a single DTC accessory device to seamlessly interact with both cloud-rendered and locally generated UIs. The ability to seamlessly switch user input event routing between client-rendered UIs and game streaming sessions is particularly relevant for features traditionally rendered by client devices rather than cloud services, such as title selection, party features, and watching game trailers. Furthermore, the DTC environment described herein intelligently routes the DTC accessory device user input event stream among multiple simultaneously connected client devices associated with a given user. This provides a seamless user experience, allowing users to switch between client devices without interrupting their cloud gaming sessions. These embodiments can be applied to any data stream communicated by DTC accessory devices, such as audio and video.

[0052] As a replacement or supplement to the other examples described in this article, some examples include any combination of the following: Clause 1. A method for routing multiple user input events received from a DTC accessory device, executed at an accessory service, the method comprising: identifying a user associated with a DTC accessory device, the DTC accessory device being remotely connected to the accessory service via a network connection; identifying a set of clients associated with the user, each client in the client set having a connection to the accessory service, the client set including a first client and a second client; and routing the multiple inputs received from the DTC accessory device, including for each specific user input event: identifying a user input event type for the specific user input event; identifying a routing policy for the user input event type, the routing policy indicating whether the user input event of the user input event type should be routed to an active client in the client set or routed to... The cloud gaming service; based on the routing policy, calculating the routing destination of the specific user input event, the routing destination including the client with an active state or the cloud gaming service; and routing the specific user input event to the routing destination, wherein routing the plurality of user input events received from the DTC accessory device includes at least: routing the first user input event to the first client based on the first user input event having a first user input event type and the first client having an active state; routing the second user input event to the cloud gaming service based on the second user input event having a second user input event type; and routing the third user input event to the second client based on the second user input event having the first user input event type and the active state having been transferred from the first client to the second client.

[0053] Clause 2. The method as described in Clause 1, wherein the first client uses the first user input event as navigation input for a first UI generated for the first client, the cloud gaming service uses the second user input event as input for a game streamed by the cloud gaming service, and the second client uses the third user input event as navigation input for a second UI generated for the second client.

[0054] Clause 3. The method as described in any one of Clauses 1 to 2, wherein the routing strategy indicates that: a user input event of the first input event type will be routed to the client with an active state when the UI generated by the client overrides the UI generated by the cloud gaming service, otherwise it will be routed to the cloud gaming service.

[0055] Clause 4. The method as described in Clause 3, wherein the method further includes receiving from the client an instruction that the UI generated by the client overrides the UI generated by the cloud gaming service.

[0056] Clause 5. The method as described in any one of Clauses 1 to 4, wherein each user input event type corresponds to a different type of user interaction with the input hardware at the DTC accessory device.

[0057] Clause 6. The method as described in any one of Clauses 1 to 5, wherein the method further comprises: updating the routing policy based on at least one of which client has an active state or which game the cloud gaming service is streaming.

[0058] Clause 7. The method of any one of Clauses 1 to 6, wherein the method further comprises transferring the active state from the first client to the second client based on detecting at least one of the following: a request received from the second client to make the second client active; or initiation of a game streaming session by the second client.

[0059] Clause 8. The method of any one of Clauses 1 to 7, wherein the method further comprises: identifying a chat session associated with the user; identifying a first user intent to interact with the chat session; and based on the first user intent: routing a first audio stream received from the DTC accessory device to the chat session; and routing a second audio stream received from the chat session to the DTC accessory device.

[0060] Clause 9. The method as described in Clause 8, wherein routing the first user input event to the first client includes routing the first user input event to a chat client executed at the first client.

[0061] Clause 10. The method of any one of Clauses 8 to 9, wherein the method further comprises: identifying a second user intent to interact with game audio; and based on the second user intent: routing the first audio stream received from the DTC accessory device to the cloud gaming service; and routing a third audio stream received from the cloud gaming service to the DTC accessory device.

[0062] Clause 11. The method of any one of Clauses 1 to 10, wherein the ancillary service is a first ancillary service of a plurality of ancillary services, each of the plurality of ancillary services executing a different computer system, and the method further comprises connecting the DTC ancillary device to the first ancillary service rather than another ancillary service of the plurality of ancillary services based on the geographical location of the DTC ancillary device relative to the computer system executing the first ancillary service.

[0063] Clause 12. The method of any one of Clauses 1 to 11, wherein the DTC accessory device is selected from the group consisting of a game controller, a headset, a keyboard, and a mouse.

[0064] Clause 13. The method as described in any one of Clauses 1 to 12, wherein each client in the client set is selected from the group consisting of game consoles, web browsers, personal computers, mobile devices, smart TVs, and set-top boxes.

[0065] Clause 14. A system comprising a processor and a storage medium storing instructions which, when executed by the processor, cause the system to perform any one of Clauses 1 to 13.

[0066] Clause 15. A machine-readable medium carrying machine-readable instructions that, when executed by a processor of a machine, cause the machine to perform any one of Clauses 1 to 13.

[0067] Embodiments of this disclosure include or utilize dedicated or general-purpose computer systems (e.g., Appendix Service 104) that include computer hardware, such as, for example, processors and system memory, as discussed in more detail below. Embodiments within the scope of this disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media can be any medium accessible by a general-purpose or dedicated computer system. Computer-readable media storing computer-executable instructions and / or data structures are computer storage media (e.g., storage media). Computer-readable media carrying computer-executable instructions and / or data structures are transmission media. Therefore, embodiments of this disclosure may include at least two distinct types of computer-readable media: computer storage media and transmission media.

[0068] Computer storage media are physical storage media that store computer-executable instructions and / or data structures. Physical storage media include computer hardware such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), solid-state drives (SSDs), flash memory, phase-change memory (PCM), optical disc storage, magnetic disk storage, or other magnetic storage devices, or any other hardware storage devices that store program code in the form of computer-executable instructions or data structures, which can be accessed and executed by general-purpose or special-purpose computer systems to perform the disclosed functions.

[0069] Transmission media include networks and / or data links that carry program code in the form of computer-executable instructions or data structures, accessible to general-purpose or special-purpose computer systems. A "network" is defined as a data link that enables the transmission of electronic data between computer systems and other electronic devices. When information is transmitted or provided to a computer system via a network or another communication connection (hardwired, wireless, or a combination thereof), the computer system may consider that connection as a transmission medium. The scope of computer-readable media includes combinations thereof.

[0070] Upon arrival at various computer system components, program code in the form of computer-executable instructions or data structures can be automatically transferred from the transmission medium to the computer storage medium (or vice versa). For example, computer-executable instructions or data structures received via a network or data link can be buffered in RAM within a network interface module and ultimately transferred to the computer system RAM and / or a less volatile computer storage medium at the computer system level. Therefore, computer storage media can be included in computer system components that also utilize the transmission medium.

[0071] Computer-executable instructions include, for example, instructions and data, which, when executed at a processor system, cause a general-purpose computer system, a special-purpose computer system, or a special-purpose processing device to perform one or more functions. In embodiments, computer-executable instructions include binary files, intermediate format instructions (e.g., assembly language), or source code. In embodiments, the processor system includes one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more neural processing units (NPUs), etc.

[0072] In some embodiments, the disclosed systems and methods are practiced in a networked computing environment with many types of computer system configurations, including personal computers, desktop computers, laptops, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframes, mobile phones, PDAs, tablets, pagers, routers, switches, etc. In some embodiments, the disclosed systems and methods are practiced in a distributed system environment, wherein different computer systems linked via a network (e.g., via a hardwired data link, a wireless data link, or a combination of hardwired and wireless data links) both perform tasks. Therefore, in a distributed system environment, computer systems can include multiple constituent computer systems. Program modules can reside in local and remote memory storage devices within the distributed system environment.

[0073] In some embodiments, the disclosed systems and methods are practiced in a cloud computing environment. In some embodiments, the cloud computing environment is distributed, but this is not required. When distributed, the cloud computing environment may be distributed within an organization and / or have components owned across multiple organizations. In this description and the following claims, "cloud computing" is a model for enabling on-demand network access to a shared pool of configurable computing resources, such as networks, servers, storage, applications, and services. Cloud computing models can consist of various characteristics, such as on-demand self-service, broad network access, resource pooling, rapid elasticity, metered services, etc. Cloud computing models can also appear in the form of various service models, such as Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), etc. Cloud computing models can also be deployed using different deployment models, such as private cloud, community cloud, public cloud, hybrid cloud, etc.

[0074] Some embodiments, such as cloud computing environments, include systems having one or more hosts capable of running one or more virtual machines (VMs). During operation, the VMs emulate the operating computing system, supporting an operating system (OS) and possibly one or more other applications. In some embodiments, each host includes a hypervisor that uses physical resources abstracted from the VM view to emulate virtual resources for the VM. The hypervisor also provides appropriate isolation between VMs. Thus, from the perspective of any given VM, the hypervisor provides the illusion that the VM is interfaced with physical resources, even if the VM only interfaces with the appearance of physical resources (e.g., virtual resources). Examples of physical resources include processing power, memory, disk space, network bandwidth, media drives, etc.

[0075] Although the subject matter has been described using language specific to structural features and / or methodological actions, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above or the order of such actions. Rather, the features and actions described above are disclosed as exemplary forms of implementing the claims.

[0076] This disclosure may be embodied in other specific forms without departing from its essential characteristics. The described embodiments are illustrative only and not restrictive. All modifications falling within the meaning and scope of the equivalents of the claims are covered by the scope of the claims.

[0077] When an element is introduced in an appended claim, the articles “a,” “an,” “the,” and “the” are intended to indicate the presence of one or more of that element. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that additional elements may exist besides those listed. Unless otherwise stated, the terms “set,” “superset,” and “subset” are intended to exclude an empty set; therefore, a “set” is defined as a non-empty set, a “superset” as a non-empty superset, and a “subset” as a non-empty subset. Unless otherwise stated, the term “subset” excludes the entirety of its superset (i.e., the superset contains at least one item not included in the subset). Unless otherwise stated, a “superset” may include at least one additional element, and a “subset” may exclude at least one element.

Claims

1. A computer-implemented method (500) for routing multiple user input events received from a directly connected cloud-based DTC accessory device, executed at an accessory service (104), the method comprising: The identifier (501) is associated with a user of a DTC accessory device (101), which is remotely connected to the accessory service via a network connection (106); The identifier (502) is associated with a client set (102) of the user, each client in the client set having a connection with the attachment service, the client set including a first client (102a) and a second client (102n). as well as Routing (503) of multiple inputs received from the DTC accessory device, including for each specific user input event: The user input event type that identifies the specific user input event; Identify a routing policy (117) for the user input event type, the routing policy indicating whether the user input event of the user input event type should be routed to an active client in the client set or to the cloud gaming service; Based on the routing strategy, the routing destination of the specific user input event is calculated, and the routing destination includes the client with an active state or the cloud gaming service; as well as The specific user input event is routed to the routing destination. The routing of the plurality of user input events received from the DTC accessory device includes at least the following: The first user input event is routed (504) to the first client based on the fact that the first user input event has a first user input event type and the first client has an active state; Based on the second user input event having a second user input event type, the second user input event is routed (506) to the cloud gaming service; and The third user input event is routed (504) to the second client based on the fact that the second user input event has the first user input event type and the activity state has been transferred (507) from the first client to the second client.

2. The method as described in claim 1, characterized in that, The first client uses the first user input event as navigation input for the first user interface (UI) generated by the first client. The cloud gaming service uses the second user input event as input for the game streamed by the cloud gaming service, and The second client uses the third user input event as navigation input for the second UI generated by the second client.

3. The method according to any one of claims 1 to 2, characterized in that, The routing policy indicates that: a user input event of the first input event type will be routed to the client with an active state when the user interface (UI) generated by the client overwrites the UI generated by the cloud gaming service; otherwise, it will be routed to the cloud gaming service.

4. The method as described in claim 3, characterized in that, The method further includes receiving from the client an instruction that the UI generated by the client overrides the UI generated by the cloud gaming service.

5. The method according to any one of claims 1 to 4, characterized in that, Each user input event type corresponds to a different type of user interaction with the input hardware at the DTC accessory device.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes updating (509) the routing policy based on at least one of which client is active or which game the cloud gaming service is streaming.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes transferring the activity state from the first client to the second client based on detecting at least one of the following: A request received from the second client to make the second client active; or The second client initiates the game streaming session.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: Identify the chat session associated with the user; Identify the first user intent to interact with the chat session; and Based on the first user intent: The first audio stream received from the DTC accessory device is routed to the chat session; and The second audio stream received from the chat session is routed to the DTC accessory device.

9. The method as described in claim 8, characterized in that, Routing the first user input event to the first client includes routing the first user input event to a chat client that is executed at the first client.

10. The method according to any one of claims 8 to 9, characterized in that, The method further includes: Identify the second user intent to interact with the game audio; and Based on the second user intent: The first audio stream received from the DTC accessory device is routed to the cloud gaming service; and The third audio stream received from the cloud gaming service will be routed to the DTC accessory device.

11. The method according to any one of claims 1 to 10, characterized in that, The attachment service is the first attachment service among a plurality of attachment services, each of which executes on a different computer system, and The method further includes connecting the DTC accessory device to the first accessory service, rather than another accessory service among the plurality of accessory services, based on the geographical location of the DTC accessory device relative to the computer system performing the first accessory service.

12. The method according to any one of claims 1 to 11, characterized in that, The DTC accessory device is selected from the group consisting of a game controller, a headset, a keyboard, and a mouse.

13. The method according to any one of claims 1 to 12, characterized in that, Each client in the client set is selected from a group consisting of game consoles, web browsers, personal computers, mobile devices, smart TVs, and set-top boxes.

14. A system (104) comprising a processor and a storage medium storing instructions which, when executed by the processor, cause the system to perform the method of any one of claims 1 to 13.

15. A machine-readable medium carrying machine-readable instructions, which, when executed by a processor of a machine, cause the machine to perform the method of any one of claims 1 to 13.

Images