Intelligent user input event routing for cloud-connected gaming accessory

Abstract

A method executed by an accessories service for managing user input events from a direct-to-cloud accessory device involves identifying the user linked to the accessory device connected remotely to the service. The method further identifies a group of clients associated with the user, including a first and second client, each maintaining a connection with the service. User input events are routed based on event type and routing policies, determining whether to direct events to an active client or a cloud gaming service. The routing destination is calculated for each event, and the event is sent accordingly. Specific routing scenarios include directing events to active clients or the cloud gaming service based on event type and client states, ensuring efficient and effective event routing.

Description

INTELLIGENT USER INPUT EVENT ROUTING FOR CLOUD-CONNECTED GAMING ACCESSORYTECHNICAL FIELD

[0001] The present disclosure relates to systems, methods, and devices that route user input events from a direct-to-cloud accessory device connected to a cloud gaming service.BACKGROUND

[0002] Cloud gaming is a technology that allows users to stream and play video games on various client devices— such as browsers, personal computers, mobile devices, smart televisions, etc.— without the need to install or download the games. The games are hosted and executed on remote servers in data centers, and the video and audio output is streamed to the user's client device over the Internet. The user's inputs, such as controller, keyboard, mouse, or touch commands, are sent from the user's client device to the server to control the game.

[0003] Conventionally, accessories (e.g., game controllers, keyboards, mice) that are used for input to a cloud-streamed game have been connected to a client device via a local connection, such as short-range wireless technology standard such as BLUETOOTH or a short-range wired technology standard such as UNIVERSAL SERIAL BUS. This means that the client device relays user input events to the cloud gaming service.

[0004] More recently, direct-to-cloud (DTC) accessories have become capable of connecting to and communicating with cloud gaming services directly (e.g., using a wireless local area networking protocol such as WI-FI). DTC accessories remove the need to manually connect input accessories to client devices individually and can reduce input latency (e.g., by shortening the number of network hops between the accessory and the cloud gaming services). The term DTC accessory is used to refer to a hand-held, portable, or wearable user input device such as a game controller, keyboard, mouse, or head-mounted display (HMD), which is capable of sending user input event data about user input events performed by a user using the DTC accessory, to a cloud gaming service bypassing an associated game display device. The associated game display device is a device used by the userto view a game played by the user where the game is provided by the cloud gaming service. The user operates the DTC accessory to make user input that influences the game's progress.

[0005] The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described supra. Instead, this background is only provided to illustrate one example technology area where some embodiments described herein may be practiced.SUMMARY

[0006] In some aspects, the techniques described herein relate to methods, systems, and computer program products for routing a plurality of user input events received from a direct- to-cloud (DTC) accessory device, the method including: identifying a user associated with a DTC accessory device that is remotely connected to the accessories service via a network connection; identifying a set of clients associated with the user, each client in the set of clients having an established connection with the accessories service, the set of clients including a first client and a second client; and routing a plurality of inputs received from the DTC accessory device, including, for each particular user input event: identifying a user input event type for the particular user input event; identifying a routing policy for the user input event type, the routing policy indicating whether user input events of the user input event type are to be routed to a client in the set of clients having an active state or to a cloud gaming service; based on the routing policy, calculating a routing destination for the particular user input event, the routing destination including either the client having the active state or the cloud gaming service; and routing the particular user input event to the routing destination, wherein routing the plurality of user input events received from the DTC accessory device includes at least: routing a first user input event to the first client based on the first user input event having a first user input event type and the first client having the active state; routing a second user input event device to the cloud gaming service based on the second user input event having a second user input event type; and routing a third user input event to the second client based on the second user input event having the first user input event type and the active state having transitioned from the first client to the second client.

[0007] This Summary introduces a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.BRIEF DESCRIPTION OF THE DRAWINGS

[0008] To describe how the advantages of the systems and methods described herein can be obtained, a more particular description of the embodiments briefly described supra is rendered by reference to specific embodiments thereof, which are illustrated in the appended drawings. These drawings depict only typical embodiments of the systems and methods described herein and are not, therefore, to be considered to be limiting in their scope. Systems and methods are described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0009] Figures 1A-1D illustrate examples of a computer architecture that facilitates routing user input events generated by a direct-to-cloud (DTC) accessory device used with a cloud gaming service;

[0010] Figure 2A illustrates an example in which a user interacts with a server-generated client user interface (Ul) of a gaming session;

[0011] Figure 2B illustrates an example in which a user interacts with a client-generated Ul on a client device;

[0012] Figure 3 illustrates an example Ul that includes a selectable option for a client device to transition to an active state;

[0013] Figure 4 illustrates an example in which a chat Ul is overlayed over a game Ul; and

[0014] Figure 5 illustrates a flow chart of an example of a method implemented at an accessories service for routing user input events received from a DTC accessory device.DETAILED DESCRIPTION

[0015] One of the challenges of cloud gaming is providing a seamless and consistent user experience across different devices and platforms, especially when the user wants to switch from one client device to another during a cloud gaming session. In an example scenario, the user may want to start playing a streamed game on a browser on a personal computer (PC) and then continue playing the same game on a smart television (TV) in another room without losing the game progress or the connection to the server. For example, with a conventional BLUETOOTH or UNIVERSAL SERIAL BUS-connected accessory device, if a user wants to play a cloud-streamed game on their mobile device, they must manually connect their accessory to that device. Then, if the user changes their mind and wants to play the cloud-streamed game on a smart TV, they need to connect their accessory to the smart TV manually, disrupting thegaming session. While newer direct-to-cloud (DTC) accessory devices eliminate the need to manually connect an accessory to various client devices, once a DTC accessory device has connected to a cloud gaming service, the cloud gaming service entirely consumes user input events generated by the DTC accessory device. In these situations, a client device acts as a viewer of the streamed game, with all user interface (Ul) with which DTC user input events interact being generated by the cloud gaming service. Furthermore, there is no mechanism for transitioning a cloud gaming session from one client device to another when using DTC accessory devices. A user input event is data about a user input, where the data may be a time stamp, information about a type of the user input event (e.g., mouse click, joystick movement, button press), a duration of the user input event, an identifier of an accessory device where a user made the user input event, or other characteristic of a user input.

[0016] At least some embodiments described herein enable DTC environments that include an accessories service that intelligently routes user input events generated by a given DTC accessory device to either a cloud gaming service or a client device based on context. This enables the client device to provide functionality beyond being a mere streamed game viewer, such as by presenting a client-generated Ul that users can interact with using a DTC accessory device, even though user input events generated by the DTC accessory device are initially sent to a cloud service to which the DTC accessory device is connected. For example, during a game streaming session, these embodiments enable DTC accessory device user input events to be intelligently routed to either the cloud gaming service (e.g., for interacting with a streamed game Ul) or to a client device (e.g., for interacting with a client-generated Ul, which is rendered over the game Ul, in place of the over the game Ul, etc.).

[0017] Thus, the embodiments described herein enable a single DTC accessory device to interact seamlessly with both cloud-generated and locally-generated Uls. Notably, the ability to seamlessly switch the routing of user input events between a client-rendered Ul and a game-streaming session is particularly relevant for features, such as title selection, party features, watching game trailers, etc., that have conventionally been rendered by a client device rather than a cloud service.

[0018] Furthermore, in at least some embodiments, the DTC environments described herein intelligently route individual user input events in a stream of user input events generated by a DTC accessory device among a plurality of concurrently connected client devices that are associated with a given user. In these embodiments, the accessories service identifies a clientdevice as having an active state and one or more additional client devices as having a standby state. Then, when routing user input events generated by the DTC accessory device to a client device, the accessories service routes those user input events to the active client device. The accessories service continuously tracks the active or standby states of these client devices as those states change. In an example the accessories service sends polling messages to the client devices to determine their state. In another example, the accessories service receives messages from the client devices reporting their status. In another example, the accessories service receives messages from the client devices that implicitly indicate an active client device (e.g., the most recent client device to send a message initiating a game session is implicitly the active client device). Thus, the accessories service can route DTC accessory device user input events to one client device when it is active and later route DTC accessory device user input events to another client device when it becomes active. This provides a seamless user experience, enabling a user to switch between client devices without interruption to their cloud gaming session.

[0019] Notably, while embodiments are described primarily in the context of routing a stream of user input events, the embodiments described herein can apply to any data stream communicated by a DTC accessory device. For example, a DTC accessory device may include a microphone and / or speaker. In this example, an accessories service can seamlessly route audio generated by a DTC accessory device, a streamed game, a chat service, etc. For instance, the accessories service may route both audio generated by the streamed game and the chat service to the DTC accessory device for mixing at the DTC accessory device before being played by the speaker. Additionally, depending on context (e.g., a Ul function the user has interacted with), the accessories service can route audio captured by the DTC accessory device's microphone to the streamed game (e.g., for speaking with other players using the game's own chat features) and / or to a party chat service or an active client device (e.g., for speaking with other players using party chat).

[0020] Figures 1A-1D illustrate an example 100 (including examples lOOa-lOOd) of a computer architecture that facilitates routing user input events generated by a DTC accessory device used with a cloud gaming service. Initially, in example 100a, Figure 1A illustrates a computer architecture that includes a DTC accessory device 101 and one or more client devices (e.g., client device 102a to client device 102n, collectively, client devices 102), each connected to a network 106 (e.g., a wide-area network, such as the Internet). In someexamples, DTC accessory device 101 is a game controller, a headset, a keyboard, a mouse, and the like. In some examples, client devices 102 include a gaming console, a web browser (e.g., a web application), a PC (e.g., laptop, desktop), a mobile device, a smart TV, a set-top box, and the like.

[0021] In general, DTC accessory devices and client devices are each associated with a corresponding user, and each DTC accessory device is associated with the same user as at least one client device. For simplicity in description, the DTC accessory device 101 and the client devices 102 depicted in examples lOOa-lOOd are all described as being associated with the same user.

[0022] In general, a user operates the DTC accessory device 101 to interact with a gaming session 116 at a cloud gaming service 103, with the interface of the gaming session 116 being generated at the cloud gaming service 103 and streamed from the cloud gaming service 103 to at least one of the client devices 102. That client device, in turn, displays that streamed gaming session on a display. Thus, DTC accessory device 101 is illustrated as including inputs 107 (e.g., buttons, switches, control sticks, touch-sensitive controls, and the like) that generate user input events when interacted with by a user. However, in some examples, DTC accessory device 101 may include additional hardware, such as audio hardware 108 (e.g., microphone, speaker), video hardware 109 (e.g., camera, screen), and the like. Additionally, each of client devices 102 is illustrated as including a corresponding display 110 (e.g., display 110a at client device 102a to display llOn at client device 102n) that displays a streamed feed of gaming session 116. Notably, in example 100, each of the client devices 102 a client Ul 111 local to the client device (e.g., client Ul Illa at client device 102a to client Ul llln at client device 102n) for interacting with cloud gaming service 103 and input device 113 (e.g., input device 113a at client device 102a to input device 113n at client device 102n). Client devices102 may include other hardware and / or software components, such as a chat client 112 (e.g., chat client 112a at client device 102a to chat client 112n at client device 102n) for interacting with a chat service 105, both described later.

[0023] In the embodiments described herein, the DTC accessory device 101 is a DTC device configured for communicating user input events resulting from a user's interaction with inputs 107 to cloud gaming service 103 via network 106. In one example, DTC accessory device 101 utilizes a WI-FI network to communicate those user input events to cloud gaming service103 over the Internet (network 106).

[0024] As mentioned, in current DTC environments, a cloud gaming service entirely consumes the DTC accessory device's user input events, such that a client device acts as a viewer of the streamed game, with all Ul that can be interacted with by a DTC being generated by the cloud gaming service. This means that current DTC environments cannot enable a DTC accessory device to interact with the local Ul of a client device. Instead, local Ul interactions are performed using a separate input device associated with that client device. Drawing an analogy to the environment of example 100, under current DTC models, DTC accessory device 101 would only be able to interact with gaming session 116, and any interactions with, e.g., client Ul Illa would need to be carried out using input device 113a.

[0025] Unlike these current DTC environments, however, example 100 includes an accessories service 104, which includes a DTC routing component 117 that receives user input events (based on inputs 107) from the DTC accessory device 101 and intelligently routes those events to either cloud gaming service 103 (e.g., for interacting with gaming session 116) or to one of the client devices 102 (e.g., for interacting with a client Ul 111 at that device). Thus, the accessories service 104 provides the DTC accessory device 101 the additional capability of interacting with local client Uis, which simplifies and streamlines the user experience, because a user can now use a single device to interact with gaming session 116 as well as client Ul 111, rather than needing also to use input device 113.

[0026] A broken box surrounding cloud gaming service 103 and accessories service 104 indicates that, in some embodiments, they are part of the same overall cloud service. In embodiments, establishing a connection with cloud gaming service 103 implies a connection with accessories service 104, and vice versa. As shown, in embodiments, cloud gaming service 103 and accessories service 104 may have shared access to a database 114.

[0027] In embodiments, upon establishing a connection between a DTC accessory device (e.g., DTC accessory device 101) and cloud gaming service 103 and / or accessories service 104, accessories service 104 identifies a user associated with that DTC accessory device. For example, example 100a shows database 114 that includes associations 115. In embodiments, associations 115 associate various DTC accessories devices with various users, such as by associating accessory identifiers (e.g., serial numbers, globally unique identifiers) with user identifiers (e.g., user names, globally unique identifiers).

[0028] In embodiments, the accessories service 104 also identifies an associated client device connection based on the identified user. For example, the accessories service 104 determinesthat client device 102a is connected with cloud gaming service 103 and / or accessories service 104. In some embodiments, the accessories service 104 identifies a plurality of associated client device connections, along with an active or standby state for those client device connections. For example, the accessories service 104 determines that client device 102a and client device 102n are each connected with cloud gaming service 103 and / or accessories service 104. In this example, the accessories service 104 may determine that client device 102a has an active state and may determine that client device 102n has a standby state.

[0029] In embodiments, only a single client device has the active state at any given time, but there can be a plurality of client devices having the standby state. In embodiments, the accessories service 104 applies one or more policies for determining which client device has the active state and which client device(s) have the standby state. In one example, the first client device to establish a connection has the active state, and any subsequent client device to establish a connection has the standby state. In another example, the most recent client device to initiate a gaming session obtains the active state (e.g., taking the active state from another client device, which transitions to the standby state). In another example, a client device can expressly request to obtain the active state (e.g., taking the active state from another client device, which transitions to the standby state).

[0030] In example 100, cloud gaming service 103 and accessories service 104 are each illustrated as including a plurality of instances, e.g., in different geographies. In embodiments, afterthe accessories service 104 has identified the user associated with DTC accessory device 101, along with the client devices 102 associated therewith that have established connections, the accessories service 104 routes all connections to a single instance of the accessories service 104, such as an instance that is geographically near an instance of the cloud gaming service 103 to which the user is connected and / or an instance that is geographically near the client devices 102. In embodiments, this ensures favorable latency characteristics between client devices 102 and accessories service 104.

[0031] In embodiments, after the accessories service 104 has identified the user associated with DTC accessory device 101, along with the client devices 102 associated therewith that have established connections, the accessories service 104 uses the DTC routing component 117 to intelligently route user input events based on the context of gaming session 116, and based on the context of client devices 102. The DTC routing component 117 can apply various routing policies, but in general, those policies operate to seamlessly route user input eventsin a manner that would be intuitive to the user of DTC accessory device 101. One goal of these policies is to allow the user to seamlessly and dynamically switch between different client devices of client devices 102 without losing gaming session 116. Another goal of these policies is to allow the user to use DTC accessory device 101 to interact with both a gaming session 116 Ul and client III 111. Example scenarios are now described in connection with Figures IBID, 2A-2B, 3, and 4.

[0032] In embodiments, DTC routing component 117 maintains a routing policy that specifies a desired destination for each user input event type (e.g., different types of button presses). The routing policy indicates when user input events of a given type are to be routed to one of the client devices 102 with an active state or to cloud gaming service 103. In embodiments, and as will be described later, routing policy may route a given user input event type differently depending on Ul context, such as which device (e.g., cloud gaming service 103 or one of the client devices 102) is rendering an active (e.g., foreground) Ul.

[0033] In Figu re IB, example 100b illustrates client device 102a in solid lines to indicate accessories service 104 has assigned it the active state and illustrates client device 102n in broken lines to indicate accessories service 104 has assigned it the standby state. Additionally, a heavy arrow that connects DTC accessory device 101 with accessories service 104 indicates that DTC accessory device 101 sends a stream of user input events to accessories service 104, based on a user's interaction with inputs 107. In turn, there are two thin arrows extending from accessories service 104, one connecting the accessories service 104 with the cloud gaming service 103, and another connecting the accessories service 104 with the client device 102a. In embodiments, these two arrows indicate that DTC routing component 117 routes various user input events received from DTC accessory device 101 to either cloud gaming service 103 or client device 102a, depending on context.

[0034] Figure 2A illustrates an example 200a where a user interacts with a game Ul 201, corresponding to gaming session 116, at display 110a of client device 102a. In this context, DTC routing component 117 routes various user input events generated by a user's interaction with inputs 107 to cloud gaming service 103 for interacting with game Ul 201. For example, DTC routing component 117 routes user input event data corresponding to directional buttons, selection buttons, etc., of DTC accessory device 101 to cloud gaming service 103 so that a user can navigate menus of game Ul 201, e.g., for customizing a game character. In this context, DTC routing component 117 may concurrently route other inputevent data, such as user input event data corresponding to a user pressing a system menu button (e.g., the XBOX button on an XBOX game controller) of DTC accessory device 101 to client device 102a for interacting with or triggering client III Illa.

[0035] Figure 2B illustrates an example 200b where the user now interacts with a client Ul 202, corresponding to client Ul Illa, at display 110a of client device 102a. In one example, client Ul Illa is initiated by a user interacting with a system menu button of DTC accessory device 101. In example 200b, client Ul 202 is overlayed over game Ul 201, though client Ul 202 could entirely replace game Ul 201. In this context, DTC routing component 117 routes various user input events to client device 102a for interacting with client Ul Illa. For example, DTC routing component 117 routes the user input events corresponding to a user interacting with directional buttons, selection buttons, etc., of inputs 107 to client device 102a so that a user can navigate menus (e.g., game selection control 203, settings control 204, profile control 205) generated by client Ul Illa. In this context, DTC routing component 117 may concurrently route other user input events, such as user input events corresponding to a user pressing a pause button, voice chat button, etc., of inputs 107 to cloud gaming service 103 for interacting with game Ul 201.

[0036] In Figure 1C, example 100c now illustrates client device 102n in solid lines to indicate accessories service 104 has now assigned it the active state and illustrates client device 102a in broken lines to indicate accessories service 104 has now assigned it the standby state. Additionally, a thin arrow extending from accessories service 104 now connects accessories service 104 with client device 102n ratherthan client device 102a. Thus, in example 100c, DTC routing component 117 routes various user input events received from DTC accessory device 101 to either cloud gaming service 103 or client device 102n, depending on context. In some embodiments, client device 102n became the active client based on a user initiating gaming session 116 at client device 102n (e.g., based on using device 113n to interact with client Ul llln).

[0037] In other embodiments, client device 102n became the active client based on an express request from client device 102n. For example, Figure 3 illustrates an example 300 in which client device 102n displayed a cloud gaming menu while client device 102n was in the standby state. The client Ul llln displayed a connection Ul 301, indicating the presence of DTC accessory device 101 and providing a selectable option (connect my controller 302) for obtaining the active state. Based on selecting this option with device 113n, client device 102ntransitioned to the active state, and client device 102a transitioned to the standby state. This selectable option (e.g., connect my controller 302), therefore, allows a user to "pull" DTC accessory device input to a given client. This can be useful when DTC routing component 117 chooses a policy that may not fit the user's preference at a given moment.

[0038] As mentioned, the routing embodiments described herein can apply to any data stream communicated by DTC accessory device 101, such as audio (audio hardware 108) and video (video hardware 109). In some examples, the accessories service 104 seamlessly routes audio and / or video among a DTC accessory device, a streamed game, a chat service, and the like. Figure ID illustrates an example lOOd in which chat service 105 is present, in which DTC accessory device 101 includes audio hardware 108 and client devices 102 include chat client 112. A heavy dashed arrow connecting DTC accessory device 101 and accessories service 104 indicates that DTC accessory device 101 communicates (e.g., sends and / or receives) audio data with accessories service 104. Additionally, light dashed arrows connecting accessories service 104 with cloud gaming service 103, chat service 105, and client device 102n indicate that gaming session 116 can route this audio in various ways. For example, DTC routing component 117 can route audio generated by gaming session 116 (e.g., game audio) and / or by chat service 105 (e.g., incoming voice chat audio) to DTC accessory device 101 for output at a speaker of DTC accessory device 101 (which can mix audio from multiple sources, in some examples). Additionally, or alternatively, DTC routing component 117 can route audio generated by DTC accessory device 101 (e.g., audio capture by a microphone) to gaming session 116 (e.g., for an in-game chat) and / or to chat service 105 (e.g., outgoing voice chat audio). In embodiments, rather than communicating audio with chat service 105, DTC routing component 117 communicates audio with chat client 112 (e.g., chat client 112n), and chat client 112, in turn, communicates with chat service 105.

[0039] In embodiments, DTC routing component 117 considers a chat client when routing user input events. For example, Figure 4 illustrates example 400, showing the routing of user input events to a chat Ul. In example 400, a game Ul 401 is presented at corresponding display 110 (e.g., display llOn), along with a client Ul 402 (e.g., generated by client Ul llln) and a chat Ul 406 (e.g., generated by chat client 112n). Here, DTC routing component 117 can route user input events received from DTC accessory device 101 among these Uls as appropriate for the context, such as which Ul is being actively presented at a client device. For example, depending on context and the particular input being used, DTC routing component 117 mayroute user input events to client device 102n for interacting with client Ul 402 (e.g., to interact with a game selection control 403, a settings control 404, a profile control 405, and the like) and / or for interacting with chat Ul 406 (e.g., to interact with a join control 408, a chat control 409, a talk control 410, and the like).

[0040] E mbodiments are now described in connection with Figure 5, which illustrates a flow chart of an example method 500, implemented at an accessories service, for routing user input events received from a DTC accessory device. In embodiments, instructions for implementing method 500 are encoded as computer-executable instructions (e.g., DTC routing component 117) stored on a computer storage medium that are executable by a processor to cause a computer system (e.g., accessories service 104) to perform method 500.

[0041] The following discussion now refers to a method and method acts. Although the method acts are discussed in specific orders or are illustrated in a flow chart as occurring in a particular order, no order is required unless expressly stated or required because an act is dependent on another act being completed before the act is performed.

[0042] Referring to Figure 5, in embodiments, method 500 comprises act 501 of identifying a user associated with a connected DTC accessory device. In some embodiments, act 501 comprises identifying a user associated with a DTC accessory device that is remotely connected to the accessories service via a network connection. For example, DTC accessory device 101 establishes a connection with cloud gaming service 103 and / or with accessories service 104. As discussed, the DTC accessory device 101 may be a game controller, a headset, a keyboard, a mouse, and the like. Based on the established connection, the accessories service 104 uses associations 115 to determine a user to which the DTC accessory device 101 is associated.

[0043] Method 500 also comprises act 502 of identifying client devices associated with the DTC accessory device. In some embodiments, act 502 comprises identifying a set of clients associated with the user, each client in the set of clients having an established connection with the accessories service. For example, based on the established connection, the accessories service 104 uses associations 115 to determine one or more client devices 102 to which the user is associated. As discussed, the DTC accessory device 101 may be a gaming console, a web browser, a PC, a mobile device, a smart TV, a set-top box, etc. In the context of example 100b, for instance, the accessories service 104 identifies client device 102a as an active client device and identifies client device 102b as a standby client device. In someembodiments of method 500, the set of clients includes a first client and a second client, and identifying the set of clients associated with the user includes determining that the first client has an active state and that the second client has a standby state.

[0044] Method 500 also comprises act 503 of routing data received from the DTC accessory device. In some embodiments, the data received from the DTC accessory device is a stream of user input events (e.g., based on a user's interaction with inputs 107), and act 503 comprises routing a plurality of user input events received from the DTC accessory device. In embodiments, act 503 comprises, for each particular user input event, identifying a user input event type for the particular user input event and identifying a routing policy for the user input event type. In embodiments, each user input event type corresponds to a different type of user interaction with input hardware at a DTC accessory device. In embodiments, the routing policy indicates whether user input events of the user input event type are to be routed to a client in the set of clients having an active state or to a cloud gaming service.

[0045] In these embodiments, act 503 also comprises, for each particular user input event, calculating a routing destination for the particular user input event based on the routing policy. The routing destination comprises either i) the client that has the active state or ii) the cloud gaming service. In these embodiments, act 503 also comprises, for each particular user input event, routing the particular user input event to the routing destination.

[0046] As shown, in embodiments, act 503 comprises one or more of act 504 of routing first data to a client device and act 505 of routing second data to a cloud gaming service. In Figure 5, no ordering is required between act 504 and act 505. Thus, the acts may variously be performed serially (in any order) or in parallel.

[0047] In some embodiments, act 504 comprises routing a first user input even received from the DTC accessory device to the first client based on the first user input event having a first user input event type and the first client having the active state, while act 505 comprises routing a second user input event received from the DTC accessory device to a cloud gaming service based on the second user input event having a second user input event type. In embodiments, the first client uses the first user input event as navigational input to a first Ul generated by the first client, and the cloud gaming service uses the second user input event as input to a game being streamed by the cloud gaming service.

[0048] For example, in embodiments, the routing policy indicates that user input events of the first input event type are to be routed to the client having the active state when a Ulgenerated by the client overlays a Ul generated by the cloud gaming service, and are to be routed to the cloud gaming service otherwise. In these embodiments, method 500 may comprise receiving an indication from the first client indicating that the Ul generated by the first client overlays the Ul generated by the cloud gaming service.

[0049] In example 100b, for instance, DTC routing component 117 routed some user input events to client device 102a (act 504) based on it being active, while also routing some user input events to cloud gaming service 103 (act 505). As described in connection with Figure 2A, for instance, DTC routing component 117 may route the user input events corresponding to directional buttons, selection buttons, etc., to cloud gaming service 103 (e.g., so that a user can navigate menus of game Ul 201), and may route other user input events to client device 102a (e.g., for interacting with or triggering client Ul Illa). A technical effect of this routing is to enable a user to seamlessly interact with both game Ul 201 and client Ul Illa using a single input device, namely, DTC accessory device 101.

[0050] Method 500 also comprises act 506 of identifying a context change. In some embodiments, act 506 determines a change in a Ul context or a client device context. Thus, as shown, act 506 comprises one or more of act 507 of identifying a client context change and act 508 of identifying a cloud context change. In Figure 5, no ordering is required between act 507 and act 508. Thus, acts may variously be performed serially (in any order) or in parallel. Additionally, there is no requirement that both act 507 and 508 occur when identifying a context change. Examples of context changes include a transition of a client device between active and standby (e.g., act 507), a client Ul becoming an active Ul (e.g., act 507), and a gaming session becoming an active Ul (e.g., act 508).

[0051] Method 500 also comprises act 509 of modifying user input event routing policy. For example, based on the context change(s) detected in act 506, DTC routing component 117 changes its routing policy. Method 500 shows that, after act 509, the method returns to act 503 to implement the new routing policy. For example, some embodiments update the routing policy based on a change in at least one of which client has the active state or which game is being streamed by the cloud gaming service.

[0052] As an example of a transition of a client device between active and standby, in example 100c, client device 102n is now assigned the active state, and client device 102a is now assigned the standby state. Thus, DTC routing component 117 changes its routing policy to route user input events to client device 102n rather than client device 102a. Someembodiments of method 500, therefore, include identifying a transition of the first client to the standby state and of the second client to the active state and routing the first set of user input events received from the DTC accessory device to the second client rather than the first client based on the transition. In embodiments, the transitioning the active state from the first client to the second client based on detecting a request received from the second client to be made active (e.g., example 300) or initiation of a game streaming session by the second client. In embodiments, after this transition, act 504 comprises routing a third user input event to the second client based on the second user input event having the first user input event type and the active state having transitioned from the first client to the second client. In embodiments, the second client uses the third user input event as navigational input to a second III generated by the second client.

[0053] As an example of a Ul transition, Figures 2A and 2B showed that there can be transitions between game Ul 201 and client Ul 202 and that user input events can be routed accordingly. Therefore, in some embodiments of method 500, in a first context (e.g., example 200b) a particular user input event is identified for interacting with the Ul at the client in the set of clients, and the particular user input event is routed from the DTC accessory device to the first client as being part of the first set of user input events. Then, in some embodiments of method 500, in a second context (e.g., example 200a) the particular user input event is identified for interacting with the cloud gaming service, and the particular user input event is routed from the DTC accessory device to the cloud gaming service as being part of the second set of user input events. In embodiments, the first context and the second context are based on one or more of a first Ul context within a game Ul being presented by the cloud gaming service or a second Ul context within a client Ul being presented by the first client.

[0054] Returning to act 503, in some embodiments, the data received from the DTC accessory device is audio. In these embodiments, act 503 comprises routing audio received from the DTC accessory device. In these embodiments, method 500 further comprises identifying a chat session associated with the user and identifying a user's intent to interact with the chat session. Then, based on the user's intent, method 500 comprises routing a first audio stream received from the DTC accessory device to the chat session and routing a second audio stream received from the chat session to the DTC accessory device. In these embodiments, routing the first set of user input events received from the DTC accessory device to the first client comprises routing a user input event to a chat client executing at the first client. Referring toact 506, in embodiments, method 500 comprises later identifying the user's intent to interact with game audio and, based on the user's intent, routing the first audio stream received from the DTC accessory device to the cloud gaming service and routing a third audio stream received from the cloud gaming service to the DTC accessory device.

[0055] As mentioned, in embodiments, the accessories service 104 routes all connections to a single instance of the accessories service 104, such as an instance that is geographically near client devices 102. Thus, in some embodiments of method 500, the accessories service is a first accessories service of a plurality of accessories services that each executes a different computer system, and the method comprises connecting the DTC accessory device to the first accessories service rather than another of the plurality of accessories services based on a geolocation of the DTC accessory device in relation to a computer system at which the first accessories service executes.

[0056] Accordingly, the embodiments described herein enable a single DTC accessory device to interact seamlessly with cloud-generated and locally generated Uls. The ability to seamlessly switch user input event routing between a client-rendered Ul and a gamestreaming session is particularly relevant for features, such as title selection, party features, watching game trailers, etc., that have conventionally been rendered by a client device rather than a cloud service. Furthermore, the DTC environments described herein intelligently route a stream of DTC accessory device user input events among a plurality of concurrently connected client devices associated with a given user. This provides a seamless user experience, enabling a user to switch between client devices without interruption to their cloud gaming session. These embodiments can apply to any data stream communicated by a DTC accessory device, such as audio and video.

[0057] Alternatively or in addition to the other examples described herein, examples include any combination of the following:

[0058] Clause 1. A method, implemented at an accessories service, for routing a plurality of user input events received from a DTC accessory device, the method comprising: identifying a user associated with a DTC accessory device that is remotely connected to the accessories service via a network connection; identifying a set of clients associated with the user, each client in the set of clients having an established connection with the accessories service, the set of clients including a first client and a second client; and routing a plurality of inputs received from the DTC accessory device, including, for each particular user input event:identifying a user input event type for the particular user input event; identifying a routing policy for the user input event type, the routing policy indicating whether user input events of the user input event type are to be routed to a client in the set of clients having an active state or to a cloud gaming service; based on the routing policy, calculating a routing destination for the particular user input event, the routing destination comprising either the client having the active state or the cloud gaming service; and routing the particular user input event to the routing destination, wherein routing the plurality of user input events received from the DTC accessory device includes at least: routing a first user input event to the first client based on the first user input event having a first user input event type and the first client having the active state; routing a second user input event device to the cloud gaming service based on the second user input event having a second user input event type; and routing a third user input event to the second client based on the second user input event having the first user input event type and the active state having transitioned from the first client to the second client.

[0059] Clause 2. The method of clause 1, wherein, the first client uses the first user input event as navigational input to a first III generated by the first client, the cloud gaming service uses the second user input event as input to a game being streamed by the cloud gaming service, and the second client uses the third user input event as navigational input to a second III generated by the second client.

[0060] Clause 3. The method of any one of clauses 1 to 2, wherein the routing policy indicates that user input events of the first input event type are to be routed to the client having the active state when a III generated by the client overlays a Ul generated by the cloud gaming service, and are to be routed to the cloud gaming service otherwise.

[0061] Clause 4. The method of clause 3, wherein the method further comprises receiving an indication from the client indicating that the Ul generated by the client overlays the Ul generated by the cloud gaming service.

[0062] Clause 5. The method of any one of clauses 1 to 4, wherein each user input event type corresponds to a different type of user interaction with input hardware at the DTC accessory device.

[0063] Clause 6. The method of any one of clauses 1 to 5, wherein the method further comprises: updating the routing policy based on a change in at least one of which client has the active state or which game is being streamed by the cloud gaming service.

[0064] Clause 7. The method of any one of clauses 1 to 6, wherein the method further comprises transitioning the active state from the first client to the second client based on detecting at least one of: a request received from the second client to be made active; or an initiation of a game streaming session by the second client.

[0065] Clause 8. The method of any one of clauses 1 to 7 , wherein the method further comprises: identifying a chat session associated with the user; identifying a first user intent to interact with the chat session; and based on the first user intent: routing a first audio stream received from the DTC accessory device to the chat session; and routing a second audio stream received from the chat session to the DTC accessory device.

[0066] Clause 9. The method of clause 8, wherein routing the first user input event to the first client comprises routing the first user input event to a chat client executing at the first client.

[0067] Clause 10. The method of any one of clauses 8 to 9, wherein the method further comprises: identifying a second user intent to interact with game audio; and based on the second user intent: routing the first audio stream received from the DTC accessory device to the cloud gaming service; and routing a third audio stream received from the cloud gaming service to the DTC accessory device.

[0068] Clause 11. The method of any one of clauses 1 to 10, wherein, the accessories service is a first accessories service of a plurality of accessories services that each executes a different computer system, and the method further comprises connecting the DTC accessory device to the first accessories service rather than another of the plurality of accessories services based on a geolocation of the DTC accessory device in relation to a computer system at which the first accessories service executes.

[0069] Clause 12. The method of any one of clauses 1 to 11, wherein the DTC accessory device is selected from a group consisting of a game controller, a headset, a keyboard, and a mouse.

[0070] Clause 13. The method of any one of clauses 1 to 12, wherein each client in the set of clients is selected from a group consisting of a gaming console, a web browser, a personal computer, a mobile device, a smart television, and a set-top box.

[0071] Clause 14. A system comprising a processor and a storage medium storing instructions which, when executed by the processor, causes the system to carry out the method of any one of clauses 1 to 13.

[0072] Clause 15. A machine-readable medium carrying machine-readable instructions which, when executed by a processor of a machine, causes the machine to carry out the method of any one of clauses 1 to 13.

[0073] Embodiments of the disclosure comprise or utilize a special-purpose or general- purpose computer system (e.g., accessories service 104) that includes computer hardware, such as, for example, a processor and system memory, as discussed in greater detail below. Embodiments within the scope of the present disclosure also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media can be any media accessible by a general- purpose or special-purpose computer system. Computer-readable media that store computer-executable instructions and / or data structures are computer storage media (e.g., storage medium). Computer-readable media that carry computer-executable instructions and / or data structures are transmission media. Thus, embodiments of the disclosure can comprise at least two distinctly different kinds of computer-readable media: computer storage media and transmission media.

[0074] Computer storage media are physical storage media that store computer-executable instructions and / or data structures. Physical storage media include computer hardware, such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), solid state drives (SSDs), flash memory, phase-change memory (PCM), optical disk storage, magnetic disk storage or other magnetic storage devices, or any other hardware storage device(s) which store program code in the form of computerexecutable instructions or data structures, which can be accessed and executed by a general- purpose or special-purpose computer system to implement the disclosed functionality.

[0075] Transmission media include a network and / or data links that carry program code in the form of computer-executable instructions or data structures that are accessible by a general-purpose or special-purpose computer system. A "network" is defined as a data link that enables the transport of electronic data between computer systems and other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination thereof) to a computer system, the computer system may view the connection as transmission media. The scope of computer-readable media includes combinations thereof.

[0076] Upon reaching various computer system components, program code in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to computer storage media (or vice versa). For example, computerexecutable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module and eventually transferred to computer system RAM and / or less volatile computer storage media at a computer system. Thus, computer storage media can be included in computer system components that also utilize transmission media.

[0077] Computer-executable instructions comprise, for example, instructions and data which when executed at a processor system, cause a general-purpose computer system, a specialpurpose computer system, or a special-purpose processing device to perform a function or group of functions. In embodiments, computer-executable instructions comprise binaries, intermediate format instructions (e.g., assembly language), or source code. In embodiments, a processor system comprises one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more neural processing units (NPUs), and the like.

[0078] In some embodiments, the disclosed systems and methods are practiced in network computing environments with many types of computer system configurations, including personal computers, desktop computers, laptop computers, message processors, hand-held devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, tablets, pagers, routers, switches, and the like. In some embodiments, the disclosed systems and methods are practiced in distributed system environments where different computer systems, which are linked through a network (e.g., by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links), both perform tasks. As such, in a distributed system environment, a computer system may include a plurality of constituent computer systems. Program modules may be located in local and remote memory storage devices in a distributed system environment.

[0079] In some embodiments, the disclosed systems and methods are practiced in a cloud computing environment. In some embodiments, cloud computing environments are distributed, although this is not required. When distributed, cloud computing environments may be distributed internally within an organization and / or have components possessed across multiple organizations. In this description and the following claims, "cloud computing"is a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). A cloud computing model can be composed of various characteristics, such as on-demand self- service, broad network access, resource pooling, rapid elasticity, measured service, and so forth. A cloud computing model may also come in the form of various service models such as Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (laaS), etc. The cloud computing model may also be deployed using different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, etc.

[0080] Some embodiments, such as a cloud computing environment, comprise a system with one or more hosts capable of running one or more virtual machines (VMs). During operation, VMs emulate an operational computing system, supporting an operating system (OS) and perhaps one or more other applications. In some embodiments, each host includes a hypervisor that emulates virtual resources for the VMs using physical resources that are abstracted from the view of the VMs. The hypervisor also provides proper isolation between the VMs. Thus, from the perspective of any given VM, the hypervisor provides the illusion that the VM is interfacing with a physical resource, even though the VM only interfaces with the appearance (e.g., a virtual resource) of a physical resource. Examples of physical resources include processing capacity, memory, disk space, network bandwidth, media drives, and so forth.

[0081] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described supra or the order of the acts described supra. Rather, the described features and acts are disclosed as example forms of implementing the claims.

[0082] The present disclosure may be embodied in other specific forms without departing from its essential characteristics. The described embodiments are only illustrative and not restrictive. All changes that come within the meaning and range of equivalency of the claims are to be embraced within their scope.

[0083] When introducing elements in the appended claims, the articles "a," "an," "the," and "said" are intended to mean there are one or more of the elements. The terms "comprising," "including," and "having" are intended to be inclusive and mean that there may be additional elements other than the listed elements. Unless otherwise specified, the terms "set,""superset," and "subset" are intended to exclude an empty set, and thus "set" is defined as a non-empty set, "superset" is defined as a non-empty superset, and "subset" is defined as a non-empty subset. Unless otherwise specified, the term "subset" excludes the entirety of its superset (i.e., the superset contains at least one item not included in the subset). Unless otherwise specified, a "superset" can include at least one additional element, and a "subset" can exclude at least one element.

Claims

CLAIMSWhat is claimed:

1. A computer-implemented method (500), implemented at an accessories service (104), for routing a plurality of user input events received from a direct-to-cloud, DTC, accessory device, the method comprising: identifying (501) a user associated with a DTC accessory device (101) that is remotely connected to the accessories service via a network connection (106); identifying (502) a set of clients associated with the user (102), each client in the set of clients having an established connection with the accessories service, the set of clients including a first client (102a) and a second client (102n); and routing (503) a plurality of inputs received from the DTC accessory device, including, for each particular user input event: identifying a user input event type for the particular user input event; identifying a routing policy (117) for the user input event type, the routing policy indicating whether user input events of the user input event type are to be routed to a client in the set of clients having an active state or to a cloud gaming service; based on the routing policy, calculating a routing destination for the particular user input event, the routing destination comprising either the client having the active state or the cloud gaming service; and routing the particular user input event to the routing destination, wherein routing the plurality of user input events received from the DTC accessory device includes at least: routing (504) a first user input event to the first client based on the first user input event having a first user input event type and the first client having the active state; routing (506) a second user input event to the cloud gaming service based on the second user input event having a second user input event type; and routing (504) a third user input event to the second client based on the second user input event having the first user input event type and the active state having transitioned (507) from the first client to the second client.

2. The method of claim 1, wherein, the first client uses the first user input event as navigational input to a first user interface, Ul, generated by the first client, the cloud gaming service uses the second user input event as input to a game being streamed by the cloud gaming service, and the second client uses the third user input event as navigational input to a second Ul generated by the second client.

3. The method of any one of claims 1 to 2, wherein the routing policy indicates that user input events of the first input event type are to be routed to the client having the active state when a user interface, Ul, generated by the client overlays a Ul generated by the cloud gaming service, and are to be routed to the cloud gaming service otherwise.

4. The method of claim 3, wherein the method further comprises receiving an indication from the client indicating that the Ul generated by the client overlays the Ul generated by the cloud gaming service.

5. The method of any one of claims 1 to 4, wherein each user input event type corresponds to a different type of user interaction with input hardware at the DTC accessory device.

6. The method of any one of claims 1 to 5, wherein the method further comprises: updating (509) the routing policy based on a change in at least one of which client has the active state or which game is being streamed by the cloud gaming service.

7. The method of any one of claims 1 to 6, wherein the method further comprises transitioning the active state from the first client to the second client based on detecting at least one of: a request received from the second client to be made active; or an initiation of a game streaming session by the second client.

8. The method of any one of claims 1 to 7, wherein the method further comprises: identifying a chat session associated with the user; identifying a first user intent to interact with the chat session; and based on the first user intent: routing a first audio stream received from the DTC accessory device to the chat session; androuting a second audio stream received from the chat session to the DTC accessory device.

9. The method of claim 8, wherein routing the first user input event to the first client comprises routing the first user input event to a chat client executing at the first client.

10. The method of any one of claims 8 to 9, wherein the method further comprises: identifying a second user intent to interact with game audio; and based on the second user intent: routing the first audio stream received from the DTC accessory device to the cloud gaming service; and routing a third audio stream received from the cloud gaming service to the DTC accessory device.

11. The method of any one of claims 1 to 10, wherein, the accessories service is a first accessories service of a plurality of accessories services that each executes a different computer system, and the method further comprises connecting the DTC accessory device to the first accessories service rather than another of the plurality of accessories services based on a geolocation of the DTC accessory device in relation to a computer system at which the first accessories service executes.

12. The method of any one of claims 1 to 11, wherein the DTC accessory device is selected from a group consisting of a game controller, a headset, a keyboard, and a mouse.

13. The method of any one of claims 1 to 12, wherein each client in the set of clients is selected from a group consisting of a gaming console, a web browser, a personal computer, a mobile device, a smart television, and a set-top box.

14. A system (104) comprising a processor and a storage medium storing instructions which, when executed by the processor, causes the system to carry out the method of any one of claims 1 to 13.

15. A machine-readable medium carrying machine-readable instructions which, when executed by a processor of a machine, causes the machine to carry out the method of any one of claims 1 to 13.

Images