System and device for controlling network applications

Abstract

This disclosure describes, in part, techniques for controlling network applications. For example, a remote system may transmit data representing the state of the application to a display device, such as a television, over a network. The remote system may then receive input data from a control device over the network. The input data may represent one or more inputs received by the control device. Using the input data, the remote system may update the state of the application. The remote system may then transmit data representing the updated state of the application to the display device. In some instances, the remote system may further transmit audio data representing a sound corresponding to the updated state of the application to the control device. The control device may synchronize the output of the sound with the display device displaying the updated state of the application.

Description

[Technical field]

[0001] Related Applications This application claims priority to U.S. Patent Application No. 16 / 202,959, entitled "SYSTEMS AND DEVICES FOR CONTROLLING NETWORK APPLICATIONS," filed November 28, 2018, which claims priority to U.S. Provisional Patent Application No. 62 / 715,146, entitled "Game Controllers," filed August 6, 2018, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] As games have become more popular, companies have created electronic devices, such as video game consoles, that allow users to play various types of games. For example, a video game console connected to a television may cause the television to display content related to the state of the game. While the television is displaying the content, the video game console may receive input directly from a video game controller held by the user. The video game console may then process the input to update the state of the game. Based on the updates, the video game console may cause the television to display content related to the updated state of the game. If the user is playing a multiplayer online game, the video game console may further send data representing the updates to a remote server. [Brief description of the drawings]

[0003] The detailed description is now described with reference to the accompanying drawings, in which the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. Use of the same reference number in different figures indicates similar or identical items or features.

[0004] [Figure 1] FIG. 1 illustrates a schematic diagram of an example system for controlling network applications. [Diagram 2]1 illustrates a block diagram of an example architecture of a control device configured to communicate with a remote system. [Diagram 3] 1 illustrates a block diagram of an example architecture of a remote system configured to communicate with a control device and a display device. [Figure 4] 1 illustrates a block diagram of an example of synchronizing data sent to a control device with data sent to a display device. [Diagram 5] 1 illustrates a schematic diagram of an exemplary system for communicating with multiple control and display devices in a network. [Figure 6A] 1 illustrates an exemplary process for controlling a network application. [Figure 6B] 1 illustrates an exemplary process for controlling a network application. [Figure 7] 1 illustrates a first exemplary process of a remote system communicating with a control device and a display device to control a network application. [Figure 8] 1 illustrates a second exemplary process of a remote system communicating with a control device and a display device to control a network application. [Figure 9] 1 illustrates an example process for a control device communicating with a remote system to control a network application. [Figure 10] 1 illustrates a block diagram of an example architecture of a voice processing system that receives audio data and / or additional data from a control device and implements techniques for determining a user's intent based on the audio data and / or additional data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] The present disclosure describes, in part, systems and techniques for controlling network applications. For example, a user may control an application running on a remote system using a control device that communicates with the remote system via a network connection. For example, the control device may send data to the remote system, where the data represents at least the input received by the control device. The remote system may then use the data received from the control device to update the current state of the application. In addition, the remote system may send video data to a display device via the network connection, where the video data represents the current state of the application. Thus, and in some instances, an additional electronic device, such as a video game console, may not be needed to run the application and / or process data from the control device. In such instances, this may eliminate the need for an additional electronic device.

[0006] More specifically, a user may first configure a controlling device. The controlling device may include, but is not limited to, a video game controller, a keyboard, a mouse, a tablet, a cell phone, a computer, a camera, a microphone, and / or any other type of device capable of receiving input from a user and / or generating data. To configure the controlling device, a user may input configuration settings into the controlling device and / or into another device. If the configuration settings are entered into another device, the other device may then transmit data representing the configuration settings to the controlling device. The configuration settings may include, but are not limited to, identifiers associated with network devices (e.g., a service set identifier (SSID) associated with a router), passwords, etc. The controlling device may then store the data representing the configuration settings. Additionally, the controlling device may connect to a network using the configuration settings to communicate with a remote system.

[0007] In some cases, the user may further configure the control device with the remote system. For example, the remote system may receive data representing an identifier of the control device from the control device and / or another device. As described herein, the identifier of the device may include, but is not limited to, an Internet Protocol (IP) address, a Media Access Control (MAC) address, a numeric identifier, an alphabetic identifier, a mixed numeric and alphabetic identifier, and / or any other type of identifier that may be used to identify a device. The remote system may then store the data representing the identifier of the control device in a user profile associated with the user. In some cases, if the remote system does not already store a user profile, the remote system may first generate the user profile before storing the data representing the identifier in the user profile.

[0008] In some instances, a user may further configure one or more display devices with the remote system. A display device may include, but is not limited to, a television, a projector, a monitor, a tablet, a mobile phone, a peripheral device that connects to another display device, and / or any other type of device that can display content (e.g., image(s), video, etc.) and / or cause another device to display content. To configure a display device, the remote system may receive data representing an identifier of the display device from the display device and / or another device. The remote system may then store the data representing the identifier of the display device in a user profile.

[0009] In some cases, the remote system may further store data representing one or more applications that are accessible by the control device over the network. The applications may include, but are not limited to, gaming applications, word processing applications, educational applications, multimedia applications (e.g., movies, music, etc.), and / or any other type of application that can be accessed over the network. The remote system may associate one or more of the applications with a user profile. For example, the remote system may receive data indicating that a user has acquired (e.g., purchased, leased, etc.) an application from the remote system. In some cases, the remote system may then store data indicating that the user has acquired by the application in the user profile.

[0010] To access the applications, the control device may connect to the network using configuration settings. The control device may then transmit data representing at least an identifier associated with the control device to the remote system. The remote system may receive the data and use a user profile to determine that the data is received from the control device. The remote system may further determine that the control device is associated with at least a display device using the user profile. In some instances, the remote system may then transmit data (e.g., video data, audio data, etc.) representing one or more applications acquired by the user to the display device. In some instances, the remote system may transmit data (e.g., video data, audio data, etc.) representing one or more additional applications that the user can access to the display device. In either example, the display device may receive the data and display content identifying the one or more applications. For example, the content may include a respective identifier (e.g., name) of each of the one or more applications that the user can access.

[0011] The remote system may then receive data (e.g., input data) from the controlling device representing a selection of the application. Based at least in part on the data, the remote system may begin executing data (e.g., computer code) representing the application. The remote system may then begin receiving data representing the input received by the controlling device. In some cases, the remote system receives the data every time the controlling device receives an input. In some cases, the remote system receives the data at a given time interval (e.g., every millisecond, every 10 milliseconds, every second, etc.). In either case, the remote system uses the data to update a state of the application based on the input(s) received by the controlling device. The remote system may then transmit the data representing the state of the application to the display device.

[0012] For example, the remote system may send data (e.g., video data, audio data, etc.) representing a first state of the application to the display device. For example, if the application includes a game application, the first state of the application may include an object (e.g., a character) located at a first location in the game environment (e.g., a forest). The display device may receive the data from the remote system. Using the data, the display device may display an image(s) representing the first state of the application. For example, again, if the application includes a game application, the display device may display content representing the object located at the first location in the game environment. In some instances, the display device may further output a sound represented by the audio data. The user may then use the control device to provide input to the application via the remote system.

[0013] For example, the control device may receive input, such as a selection of a button, joystick, key, touch-sensitive pad, sensor, etc. associated with the control device. The control device may then transmit data representing the input to the remote system. Using the data, the remote system may update a first state of the application to a second state of the application. For example, again, if the application includes a game application, the input may be associated with advancing an object a given amount in the game environment. Thus, the remote system may update the first state of the application to a second state of the application by advancing the object in the environment a given amount. The remote system may then transmit data (e.g., video data, audio data, etc.) representing the second state of the application to the display device. The display device may display an image(s) representing the second state of the game. For example, the display device may display content representing an object located at a second position in the game environment. Additionally, in some instances, the display device may further output a sound represented by the audio data. The remote system may then continue to perform a similar process to update the state of the application on the display device as the remote system continues to receive data from the control device.

[0014] In some cases, the remote system may further transmit data (e.g., video data, audio data, etc.) to the controlling device, where the data is associated with a state of the application being displayed by the display device. For example, the remote system may transmit to the controlling device audio data representing a sound corresponding to the current state of the application being displayed by the display device. The controlling device may receive the audio data from the remote system and, in response, output a sound. In some cases, when transmitting the audio data to the controlling device, the remote system, the controlling device, and / or another device may synchronize the output of the audio data with the current state of the application.

[0015] For the first example, the remote system may generate first timestamp data representing a first timestamp associated with the data transmitted to the display device and / or second timestamp data representing a second timestamp associated with the audio data transmitted to the control device. The remote system may then transmit the first timestamp data to the display device and the second timestamp data to the control device. In some cases, the remote system transmits the first timestamp data to the display device as part of a first data packet, the first data packet further including the video data. In other cases, the remote system transmits the first timestamp data to the display device separately from the video data. Additionally, in some cases, the remote system transmits the second timestamp data to the control device as part of a second data packet, the second data packet further including the audio data. In other cases, the remote system transmits the second timestamp data to the control device separately from the audio data.

[0016] The second timestamp data may be used by the control device to synchronize the output of the audio data with the display of the current state of the application by the display device. For example, the first timestamp data may include a first instruction for displaying the current state of the application at a first time, and the second timestamp data may include a second instruction for outputting a sound at a second time, the first time being similar to the second time (e.g., the same with a threshold period to that time, etc.). Thus, the control device may store the audio data and / or the second timestamp data in a buffer. The control device may then determine when the current time includes the second time represented by the second timestamp data. When the current time includes the second time, the control device may output the sound represented by the audio data.

[0017] For a second example, the remote system may send the audio data, the video data, the first timestamp data, and the second timestamp data to a separate electronic device, such as a dongle. The electronic device may then use the first timestamp data and the second timestamp data to determine when to send the video data to the display device and the audio data to the control device, respectively. For example, at a first time, the electronic device may send the video data to the display device. Based at least in part on receiving the video data, the display device may display a current state of an application. In addition, at a second time, the electronic device may send the audio data to the control device. Based at least in part on receiving the audio data, the control device may output a sound.

[0018] In some instances, the remote system may receive, from the control device, audio data representing user speech from a user. The remote system may then process the audio data and perform one or more functions based on the processing. For a first example, if the user speech corresponds to an action to be performed by an application, the remote system may update a state of the application based on the user speech. For example, if the user speech includes one or more words such as "advance an object," the remote system may update a state of the application by advancing an object within a game environment. For a second example, if the user speech corresponds to a request to send a message to a contact of the user, the remote system may send the message to the contact. For example, if the application includes a game application, the message may include a request to join a multiplayer game with the user.

[0019] For a third example, if the user voice corresponds to a request to select an application, the remote system may initiate selecting the application and / or transmitting data representing the application to the display device. For example, if the user voice includes one or more words such as "play a first game application," the remote system may initiate selecting the first game application and / or transmitting data representing the first game application to the display device. These are just some examples of using user voice to perform one or more functions, although in other examples, the remote system may perform additional functions based on audio data received from the control device.

[0020] In some instances, such as when the application includes a gaming application, the remote system may implement similar processes and / or techniques as a video game console, computing device, tablet, and / or other electronic device to process input received by the control device. However, unlike a conventional video game console, computing device, table, and / or other electronic device running a gaming application, the remote system may be remote from the control device and / or the display device. For example, the control device and / or the display device may be located in the user's environment, such as the user's home, apartment, office, etc. The remote system may be remote from the user's environment and communicate with each of the control device and the display device over a network.

[0021] In the above examples, the remote system may be located remotely from the control device and the display device. For example, the control device and the display device may be located in an environment such as a user's home, apartment, business, etc. The remote system may then be located remotely from the environment, such as a cloud-based system. In other cases, the remote system may be located in the same environment as the control device and / or the display device. For example, the remote system may include devices located within the environment.

[0022] In some instances, by having the control device communicate directly with the remote system, the processes and techniques described herein may reduce latency when using applications. For example, with respect to online gaming, instead of using a video game console to process input from the control device and then sending data representing updates to the game to the remote system, the control device sends the input data directly to the remote system (i.e., the input data does not have to be sent to the video game console), which then processes the input. This may reduce latency caused by the input data being sent first to the video game console and / or latency caused by the video game console processing the input data.

[0023] FIG. 1 is a schematic diagram of an exemplary system 100 for controlling network applications. The system 100 may include, for example, a remote system 102, a control device 104, and a display device 106. In the example of FIG. 1, the control device 104 may communicate with the remote system 102 over a network 108, such as by using a first communication channel. For example, the control device 104 may transmit data to the remote system 102 (represented by 112), which may transmit data to the control device 104 (represented by 114). In addition, the display device 106 may communicate with the remote system 102 over the network 108, such as by using a second communication channel. For example, the display device 106 may transmit data to the remote system 102 (represented by 116), which may transmit data to the display device 106 (represented by 118). By transmitting and receiving data with the control device 104 and the display device 106, the remote system 102 enables a user 110 to control an application displayed by the display device 106 using the control device 104 via the remote system 102.

[0024] For example, the user 110 may initially configure the control device 104. The control device 104 may include, but is not limited to, a video game controller, a keyboard, a mouse, a tablet, a cell phone, a computer, a camera, a microphone, and / or any other type of device capable of receiving input from the user 110 and / or generating data. To configure the control device 104, the user 110 may enter configuration settings into the control device 104 and / or into another device. As the configuration settings are entered into another device, the other device may then transmit data representing the configuration settings to the control device 104. The configuration settings may include, but are not limited to, an identifier associated with a network device (e.g., an SSID associated with a router (not shown)), a password, etc. The control device 104 may then store the data representing the configuration settings. Additionally, the control device 104 may connect to the network 108 using the configuration settings to communicate with the remote system 102.

[0025] In some instances, the user 110 may further configure the control device 104 with the remote system 102. For example, the remote system 102 may receive identifier data 120 representing an identifier of the control device 104 from the control device 104 and / or another device. The identifier of the control device 104 may include, but is not limited to, an IP address, a MAC address, a numeric identifier, an alphabetic identifier, a mixed numeric and alphabetic identifier, and / or any other type of identifier that may be used to identify the control device 104. The configuration component 122 may then be configured to store the identifier data 120 representing the identifier of the control device 104 associated with a user profile, where the user profile is represented by the user profile data 124.

[0026] In some instances, the user 110 may further configure one or more display devices with the remote system 102, such as the display device 106. The display device 106 may include, but is not limited to, a television, a projector, a monitor, a tablet, a mobile phone, a peripheral device that connects to another display device, and / or any other type of device that can display content (e.g., image(s), video, etc.) and / or cause another device to display content. To configure the display device 106, the remote system 102 may receive identifier data 120 representing an identifier of the control device 106 from the control device 106 and / or another device. The identifier of the display device 106 may include, but is not limited to, an IP address, a MAC address, a numeric identifier, an alphabetic identifier, a mixed numeric and alphabetic identifier, and / or any other type of identifier that may be used to identify the display device 106. The configuration component 122 may then be configured to store the identifier data 120 representing the identifier of the display device 106 associated with the user profile.

[0027] In some cases, the remote system 102 may further store application data 126 representing one or more applications that are accessible by the control device 104 over the network 108. The applications may include, but are not limited to, gaming applications, word processing applications, educational applications, multimedia applications (e.g., movies, music, etc.), and / or any other type of application that can be accessed over the network 108. In some cases, the remote system 102 may associate one or more of the applications with a user profile. For example, the remote system 102 may receive data indicating that the user 110 has acquired (e.g., purchased, leased, etc.) an application from the remote system 102. In some cases, the remote system 102 may then store application data 128 indicating that the user 110 has acquired by the application. The remote system 102 may then enable the user 110 to access the application using the control device 104.

[0028] To access the application, the control device 104 may connect to the network 108, such as by using configuration settings. The control device 104 may then transmit to the remote system 102 identifier data 120 representing at least an identifier associated with the control device 104. The remote system 102 may receive the identifier data 120 and use the user profile data 124 (e.g., the stored identifier data 120) to determine that the identifier data 120 is received from the control device 104. For example, the remote system 102 may match the identifier represented by the identifier data 120 received from the control device 104 with the identifier represented by the stored identifier data 120. Based at least in part on the match, the remote system 102 may determine that the identifier data 120 was transmitted from the control device 104.

[0029] The remote system 102 may further determine, using the user profile data 124, that the control device 104 is associated with at least the display device 106. For example, based at least in part on determining that the identifier data 120 was transmitted from the control device 104, the remote system 102 may analyze the user profile data 124 associated with the user 110. The remote system 102 may then determine that the user profile data 124 includes and / or is associated with identifier data 120 that represents an identifier of the display device 106. Based at least in part on the determination, the remote system 102 may determine that the control device 104 is associated with the display device 106.

[0030] In some cases, remote system 102 may transmit data (e.g., video data, audio data, etc.) representing one or more additional applications that are accessible to user 110 to display device 106. In some cases, the one or more applications may include one or more applications obtained by user 110 (e.g., one or more applications represented by application data 128). Additionally or alternatively, in some cases, the one or more applications may include one or more applications that are free to the user. In either example, display device 106 may receive the data from remote system 102. Display device 106 may then display an image(s) representing one or more applications available to user 110. For example, the image(s) may include one or more identifiers (e.g., one or more names) of the one or more applications available to user 110.

[0031] The remote system 102 may then receive input data 130 from the control device 104 representing the input(s) received by the control device 104. The remote system 102 may analyze the input data 130 to identify the application selected by the user 110. For example, the input(s) may correspond to movement(s) of an object, such as a cursor, across an image(s) representing one or more applications. The input(s) may further correspond to a selection of one of the applications (e.g., selection of a control on the control device 104 when the object is located over the selected application). The remote system 102 may then determine that the user 110 has selected an application based at least in part on the selection.

[0032] The remote system 102 may then transmit data representing the first state of the application to the display device 106. The data may include first video data 132 representing an image(s) of the first state of the application, first audio data 134 representing a sound corresponding to the first state of the application, and / or first timestamp data 136 representing a time for displaying the first state of the application and / or a time for outputting a sound. In some cases, the remote system 102 transmits a first data packet including the first video data 132, the first audio data 134, and / or the first timestamp data 136. In other cases, the remote system 102 may transmit the first video data, the first audio data, and / or the first timestamp data 136 separately to the display device 106.

[0033] The display device 106 may receive data (and / or the first data packet) from the remote system 102. Using the first video data 132, the display device 106 may display an image(s) representing a first state of the application. For example, if the application includes a game application, the display device 106 may display content representing a first state of the game. In the example of FIG. 1, the first state of the game may include an object (e.g., a character) located at a first location within the game environment (e.g., a forest). In some instances, the display device 106 may further output a sound represented by the first audio data 134. In some instances, the display device 106 displays the image(s) and / or outputs a sound according to a time represented by the first timestamp data 136.

[0034] For example, the display device 106 may determine when the current time includes a time represented by the first timestamp data 136 for displaying the first video data 132. Based at least in part on the determination, the display device 106 may begin displaying an image(s) representing a first state of an application. In addition, the display device 106 may determine when the current time includes a time represented by the first timestamp data 136 for outputting a sound. Based at least in part on the determination, the display device 106 may begin outputting a sound represented by the first audio data 134.

[0035] In some cases, the remote system 102 may additionally transmit data representing a first state of the application to the control device 104. The data may include second audio data 134 representing a sound corresponding to the first state of the application, and / or second timestamp data 136 representing a time for outputting the sound. In some cases, the remote system 102 transmits a second data packet including the second audio data 134 and / or the second timestamp data 136. In other cases, the remote system 102 may transmit the second audio data 134 that separately forms the second timestamp data 136. The control device 104 may then output the sound represented by the second audio data 134. In some cases, the control device 104 may output the sound according to the time represented by the second timestamp data 136. In some cases, the first timestamp data 136 and / or the second timestamp data 136 synchronize the output of the sound by the display device 106 with the display of the image(s) by the display device 106.

[0036] For example, the synchronization component 138 may be configured to at least synchronize the display of an image(s) by the display device 106 with the output of a sound by the control device 104. To perform the synchronization, the synchronization component 138 may generate first timestamp data 136 for displaying the image(s) and / or second timestamp data 136 for outputting a sound. For example, the first timestamp data 136 may represent an instruction to display the image(s) at a first time. Additionally, the second timestamp data 136 may represent an instruction to output a sound at a second time. In some cases, the first time may be similar to the second time. In some cases, the first time may be within a threshold period to the second time. The threshold period may include, but is not limited to, 1 millisecond, 10 milliseconds, 1 second, etc.

[0037] The user 110 may then provide input to the application using the control device 104. For example, the control device 104 may receive input, such as a selection of a button, joystick, key, touch-sensitive pad, sensor, etc. associated with the control device 104. The control device 104 may then transmit input data 130 representing the input to the remote system 102. Using the input data 130, the processing component 140 may update a first state of the application to a second state of the application. For example, using the above example where the application includes a game application, the input may be associated with advancing an object by a given amount within the game environment. Thus, the processing component 140 may analyze the input data 130 to determine that the input includes advancing an object by a given amount. The processing component 140 may then update the first state of the application to a second state of the application by advancing the object by the given amount within the environment.

[0038] The remote system 102 may then transmit data representing the second state of the application to the display device 106. The data may include third video data 132 representing an image(s) of the second state of the application, third audio data 134 representing a sound corresponding to the second state of the application, and / or third timestamp data 136 representing a time for displaying the second state of the application and / or a time for outputting the sound. In some instances, the remote system 102 transmits a third data packet including the third video data 132, the third audio data 134, and / or the third timestamp data 136. In other instances, the remote system 102 may transmit the third video data, the third audio data, and / or the third timestamp data 136 separately to the display device 106.

[0039] The display device 106 may receive data (and / or a third data packet) from the remote system 102. Using the third video data 132, the display device 106 may display an image(s) representing a second state of the application. For example, again using an example where the application includes a gaming application, the display device 106 may display an object located at a second location within the gaming environment. In some instances, the display device 106 may further output a sound represented by the third audio data 134. In some instances, the display device 106 displays the image(s) and / or outputs a sound according to the time(s) represented by the third timestamp data 136.

[0040] In some cases, the remote system 102 may additionally transmit data representing a second state of the application to the control device 104. The data may include fourth audio data 134 representing a sound corresponding to the second state of the application, and / or fourth timestamp data 136 representing a time for outputting the sound. In some cases, the remote system 102 transmits a fourth data packet including the fourth audio data 134 and / or the fourth timestamp data 136. In other cases, the remote system 102 may transmit the fourth audio data separately from the fourth timestamp data 136. In either case, the control device 104 may then output the sound represented by the fourth audio data 134. In some cases, the control device 104 may output the sound according to the time represented by the fourth timestamp data 136. In some cases, the third timestamp data 136 and / or the fourth timestamp data 136 synchronize the output of the sound by the display device 106 with the display of the image(s) by the display device 106.

[0041] In some cases, the remote system 102 may continue to receive the input data 130 from the control device 104. The processing component 140 may then continue to process the input data 130 to update the state of the application. Based at least in part on the updates, the remote system 102 may continue to send data representing the current state of the application (e.g., video data 132, audio data 134, timestamp data 136, etc.) to the display device 106. In some cases, the remote system 102 sends the data to the display device 106 as a data packet. The remote system 102 may further send the data representing the current state of the application (e.g., audio data 134, timestamp data 136, etc.) to the control device. In some cases, the remote system 102 sends the data to the control device 104 as part of a data packet.

[0042] 1, the remote system 102 may include a speech processing system 142 including an automatic speech recognition (ASR) component 144 and a natural language understanding (NLU) component 146. In some cases, the input data 130 received from the control device 104 may include audio data representing user speech from the user 110. For example, the control device 104 may include at least one microphone that captures the user speech from the user 110. The control device 104 may then generate audio data representing the user speech and transmit the audio data (e.g., the input data 130) to the remote system 102. In cases where the input data 130 includes audio data, the ASR component 144 may analyze the audio data to generate text data representing the user speech.

[0043] The NLU component 146 may then use the text data to perform semantic interpretation of the text. That is, the NLU component 146 determines the meaning behind the text based on individual words and then implements that meaning. The NLU component 146 interprets the text string to derive the intent or desired action from the user 110, as well as the associated piece of information in the processing component 140 to determine how to update the current state of the application. For example, if the user speech processed using the ASR component 144 includes the text "move the object forward," the NLU component 146 may determine that the user 110 intends to move the object forward in the environment. The processing component 140 may then use that intent to update the state of the application, similar to the process described above.

[0044] In an example, some or each of the components and / or systems of the remote system 102 may include their own processor(s), network interface(s), and / or memory. Thus, by way of example, the application system 148 may include and / or be associated with a processor(s), network interface(s), and / or memory. In addition, the voice processing system 142 may include and / or be associated with a different processor(s), network interface(s), and / or memory, or one or more of these components may utilize some or all of the same processor(s), network interface(s), and / or memory utilized by the application system 148. These components are described in detail herein. In addition, the operations and / or functions associated with and / or described with respect to the components of the remote system 102 may be implemented utilizing cloud-based computing resources. For example, a web-based system such as an Elastic Compute Cloud system or a similar system may be utilized to generate and / or present a virtual computing environment for implementation of some or all of the functions described herein. Additionally or alternatively, one or more systems may be utilized that may be configured to perform operations without provisioning and / or managing servers, such as a Lambda system or similar system.

[0045] 2 illustrates a block diagram of an example architecture of a control device 104 configured to communicate with a remote system 102. As shown, the control device 104 includes one or more processors 202, one or more network interfaces 204, one or more speakers 206, one or more microphones 208, one or more input devices 210, one or more output devices 212, one or more power components 214, one or more lights 216, one or more peripheral inputs 218, and memory 220. In some cases, the control device 104 may include one or more additional components not illustrated in the example of FIG. 2. In some cases, the control device 104 may not include one or more speaker(s) 206, microphone(s) 208, input device(s) 210, output device(s) 212, lights 216, or peripheral input(s) 218.

[0046] Although the control device 104 is illustrated as having one or more integrated speaker(s) 206, in other examples, the control device 104 may not include the speaker(s) 206. For example, the control device 104 may generate an audio output signal that drives an external loudspeaker, such as headphones connected to the control device 104 via the peripheral input 218. As another example, the control device 104 may drive or control an external loudspeaker through a wireless data connection, such as a Bluetooth connection. In other situations, the control device 104 may be used in conjunction with a loudspeaker device that receives audio data and other instructions from the remote system 102 rather than from the control device 104.

[0047] The microphone(s) 208 may include a sensor (e.g., a transducer) configured to receive sound. The microphone(s) 208 may generate an input signal for an audio input (e.g., sound). For example, the microphone(s) 208 may determine a digital input signal for a user's speech. In some cases, the microphone(s) 208 are implemented in an array. The array may be arranged in a geometric pattern, such as a linear geometric form, a circular geometric form, or any other configuration. For example, for a given point, an array of four sensors may be arranged in a circular pattern in 90 degree increments (e.g., 0, 90, 180, 2112) to receive sound from four directions. The microphone(s) 208 may be in a planar configuration or may be positioned apart in a non-planar three-dimensional area. In some cases, the microphone(s) 208 may include an array of spatially distinct sensors in data communication. For example, an array of networked sensors may be included. The microphone(s) 208 may include an omnidirectional microphone, a directional microphone (eg, a shotgun microphone), and the like.

[0048] In some instances, the microphone(s) 208 and speaker(s) 206 facilitate interaction, such as a dialogue, with a user. The microphone(s) 208 generate audio data representing sounds from the environment of the control device 104, e.g., vocal utterances by a user. The audio data generated by the microphone(s) 208 may include or be used to generate directional audio signals, each of which emphasizes audio from a different radial direction relative to the microphone(s) 208.

[0049] The input device(s) 210 may include button(s), key(s), joystick(s), touch-sensitive pad(s), trigger(s), sensor(s) that detect movement of the control device 104 (e.g., accelerometer(s), magnetometer(s), etc.), and / or any other type of device capable of receiving input from a user. The output device(s) 212 may include devices that provide feedback to the user. For example, the output device(s) 210 may include haptic driver(s) that actuate actuator(s).

[0050] The power component(s) 214 may be configured to provide power to the control device 104. For a first example, the power component(s) 214 may include one or more batteries. For a second example, the power component(s) 214 may include components connected to an external AC (alternating current) power source, such as a household AC power source (alternatively referred to herein as "AC mains" or "wall power").

[0051] The light(s) 216 may include visual indicator(s) (e.g., light emitting diodes (LEDs)) that emit light when activated by the control device 104. In some cases, the visual indicator(s) may include a light ring disposed on a portion of the control device 104. In other cases, the visual indicator(s) may be disposed on various portions on the control device 104. The peripheral input(s) 218 ​​may include one or more components for connecting peripheral devices to the control device 104. For example, the peripheral input 218 may include a jack for connecting headphones to the control device 104.

[0052] The control device 104 may include a memory 220. The memory 220 may be used to store any number of software components executable by the processor(s) 202. The software components stored in the memory 220 may include an operating system 222 configured to manage hardware and services within and coupled to the control device 104. In addition, the executable components stored by the memory 220 may include an audio processing component(s) 224 configured to generate audio data using the microphone 208. The audio processing component(s) 224 may include functionality for processing microphone audio data generated by the microphone(s) 208 and / or output audio data provided to the speaker(s) 206. As an example, the audio processing component(s) 224 may include an acoustic echo cancellation or suppression component for reducing acoustic echoes generated by acoustic coupling between the microphone(s) 208 and the speaker(s) 206. The audio processing component 224 may also include a noise reduction component to reduce noise in the received audio data, such as components of the microphone audio data other than the user's voice.

[0053] In addition, the audio processing component(s) 224 may include a wake word detection component that monitors the audio signal to detect user speech of the system of trigger expressions. Wake word detection may be implemented using keyword spotting techniques, as one example.

[0054] 2, memory 220 may include input data 130 generated by control device 104, which transmits input data 130 to at least remote system 102. In addition, memory 220 may include video data 132, audio data 134, and / or timestamp data 136 that control device 104 receives from remote system 102. In addition, memory 220 may store identifier data 120 that control device 104 may transmit to remote system 102.

[0055] 2, the memory 220 may store configuration data 226. The control device 104 may use the configuration data 226 to establish a connection with a network, such as the network 108. For example, the configuration data 226 may represent an SSID and a password.

[0056] 2, the electronic device 104 may be connected to peripheral device(s) 226. The peripheral device(s) 226 may include, but are not limited to, headphones, speaker(s), display(s), input device(s) (e.g., keyboard, mouse, button(s), etc.). In some cases, the electronic device 104 includes a wired connection with the peripheral device(s) 226. For example, the peripheral device(s) 226 may connect to the peripheral input(s) 218. In some cases, the electronic device 104 includes a wireless connection with the peripheral device(s) 226. For example, the electronic device 104 may communicate with the peripheral device(s) 226 over a network.

[0057] 2 illustrates the peripheral device(s) 226 as being separate from the electronic device 104, in other cases, the electronic device 104 may include the peripheral device(s) 226. For example, the electronic device 104 may include headphones.

[0058] 3 illustrates a block diagram of an example architecture of a remote system 102 configured to communicate with a control device 104 and a display device 106. As shown, the application system 148 may include a processor(s) 302, a network interface(s) 304, and a memory 306. In addition, the audio processing system 142 may include a processor(s) 308, a network interface(s) 310, and a memory 312.

[0059] As used herein, a processor, such as processor(s) 202, processor(s) 302, and / or processor(s) 308, may include multiple processors and / or processors with multiple cores. Additionally, a processor may include one or more cores of different types. For example, a processor may include an application processor unit, a graphics processing unit, etc. In one instance, a processor may include a microphone controller and / or a microphone processor. Processor(s) 202, processor(s) 302, and / or processor(s) 308 may include a graphics processing unit (GPU), a microphone processor, a digital signal processor, or other processing units or components known in the art. Alternatively, or in addition, the functions described herein may be implemented, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), etc. Additionally, each of processor(s) 202, processor(s) 302, and / or processor(s) 308 may have its own local memory that may also store program components, program data, and / or one or more operating systems.

[0060] Memory 220, memory 306, and / or memory 312 may include volatile and non-volatile memory, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program components, or other data. Memory 220, memory 306, and / or memory 312 may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage, RAID storage systems, or any other medium that may be used to store the desired information and that may be accessed by a computing device. Memory 220, memory 306, and / or memory 312 may be implemented as a computer-readable storage medium ("CRSM"), which may be any available physical medium accessible by processor(s) processor(s) 202, processor(s) 302, and / or processor(s) 308 to execute instructions stored in memory 220, memory 306, and / or memory 312. In one basic case, the CRSM may include random access memory ("RAM") and flash memory. In other cases, the CRSM may include, but is not limited to, read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), or any other tangible medium that may be used to store desired information and that may be accessed by the processor(s).

[0061] Further, the functional components may be stored in respective memories, or the same functions may alternatively be implemented in hardware, firmware, an application specific integrated circuit, a field programmable gate array, or a system on a chip (SoC). In addition, although not illustrated, each respective memory discussed herein, such as memory 220, memory 306, and / or memory 312, may include at least one operating system (OS) component configured to manage hardware resource devices, such as network interface(s), I / O devices of the respective device, and provide various services to applications or components executing on the processor. Such OS components may implement a variant of the FreeBSD operating system published by the FreeBSD Project, other UNIX or UNIX-like variants, a variant of the Linux operating system published by Linus Torvalds, the FireOS operating system of Amazon.com Inc. of Seattle, Washington, USA, the Windows operating systems of Microsoft Corporation of Redmond, Washington, USA, LynxOS published by Lynx Software Technologies, Inc. of San Jose, California, Embedded Operating System (Enea OSE) published by ENEA AB of Sweden, and the like.

[0062] Network interface(s) 204, network interface(s) 304, and / or network interface(s) 312 may enable messages between remote system 102, control device 104, display device 106, and / or one or more other remote systems, as well as other networked devices. Network interface(s) 204, network interface(s) 304, and / or network interface(s) 310 may include one or more network interface controllers (NICs) or other types of transceiver devices for sending and receiving messages over network 108.

[0063] For example, each of network interface(s) 204, network interface(s) 304, and / or network interface(s) 310 may include a personal area network (PAN) component for enabling messages over one or more short-range wireless message channels. For example, the PAN component may enable messages conforming to at least one standard of IEEE 802.15.4 (ZigBee), IEEE 802.15.1 (Bluetooth), IEEE 802.11 (WiFi), or any other PAN message protocol. Additionally, each of network interface(s) 204, network interface(s) 304, and / or network interface(s) 310 may include a wide area network component for enabling messages over a wide area network (WAN).

[0064] 4 illustrates a block diagram of an example of synchronizing data transmitted to the control device 104 with data transmitted to the display device 106. In the example of FIG. 4, the control device 104 may establish a network connection with a network device 402, such as an access point. In addition, the display device 106 may be connected to an electronic device 404, which includes a network interface(s) for establishing a network connection with the network device 402. However, in other examples, the electronic device 404 may be included within the display device 106. Still, in other examples, the display device 106 may include a network interface(s) for establishing a network connection with the network device 402 (e.g., the electronic device 404 would not be included in the example of FIG. 4).

[0065] The remote system 102 may transmit at least the audio data 134 and the first timestamp data 136 to the control device 104. In some instances, the remote system 102 transmits the audio data 134 and the first timestamp data 136 as part of a first data packet. The audio data 134 may represent a sound corresponding to a state of an application. For example, if the application is a game application, the audio data 134 may represent a sound corresponding to a state of the game. The first timestamp data 136 may represent a first instruction to output the sound represented by the audio data 134 at a first time.

[0066] Additionally, the remote system 102 may transmit at least the video data 132 and / or the second timestamp data 136 to the control device 104. In some cases, the remote system 102 transmits the video data 132 and the second timestamp data 136 as part of a second data packet. The video data 132 may represent an image(s) corresponding to a state of an application. The second timestamp data 136 may represent a second instruction to display the image(s) represented by the video data 132 at a second time. In some cases, the second time may be similar to the first time (e.g., including the first time). In other cases, the second time may be within a threshold period to the first time.

[0067] In the example of FIG. 4, the control device 104 may store the audio data 134 and the first timestamp data 136 in the memory 220. The control device 104 may then determine when the current time includes the first time represented by the first timestamp data 136. Based at least in part on the determination, the control device 104 may cause a sound represented by the audio data 134 to be output. In some instances, to cause the sound to be output, the control device 104 may output the sound using the speaker(s) 206. Additionally or alternatively, in some instances, to cause the audio to be output, the control device 104 may transmit the audio data 134 to the peripheral device(s) 406, which include speaker(s) for outputting the audio. For example, the control device 104 may be connected to headphones (e.g., the peripheral device 406). In such an example, the control device 104 may transmit the audio data 134 to the headphones, which may output the audio.

[0068] In addition, the display device 106 may display the image(s) represented by the video data 132. In some cases, such as when the remote system 102 does not transmit the second timestamp data 136 to the display device 106, the display device 106 may begin displaying the image(s) when the display device 106 is receiving the video data 132. In other cases, such as when the remote system 102 transmits the second timestamp data 136, the display device 106 may store the video data 132 in memory. The display device 106 may then determine when the current time includes the second time represented by the second timestamp data 136. Based at least in part on the determination, the display device 106 may begin displaying the image(s).

[0069] 4, the control device 104, the display device 106, the network device 402, the electronic device 404, and the peripheral device(s) 406 may be located within an environment, such as a user's home. In addition, the remote system 102 may be located remotely from the environment. However, in other examples, one or more of the display device 106, the network device 402, the electronic device 404, and the peripheral device(s) 406 may be located remotely from the environment.

[0070] FIG. 5 illustrates a schematic diagram of an exemplary system 500 for communicating with multiple control devices and display devices in a network. The system 500 may include, for example, a remote system 102, a control device 104, a display device 106, one or more additional control devices 502, and one or more additional display devices 504. In the example of FIG. 5, the remote system 102 may receive input data 130 from the control device 104 over the network 108. The remote system 102 may then use the input data 130 to update a local state of an application associated with the control device 104. In addition, the remote system 102 may transmit video data 132, audio data 134, and / or timestamp data 136 representing a local state of an application associated with the control device 104 over the network 108 to the display device 106. Furthermore, the remote system 102 may transmit data representing a current local state of an application associated with the control device 104 to the one or more display devices 504 over the network 108.

[0071] The remote system 102 may also perform a similar process for each of the other one or more other control devices 502. For example, the remote system 102 may receive input data 130 from the control device 502 over the network 108. The remote system 102 may then use the input data 130 to update a local state of an application associated with the control device 502. In addition, the remote system 102 may transmit video data 132, audio data 134, and / or timestamp data 136 representing the local state of the application associated with the control device 502 over the network 108 to the display device 504 (associated with the control device 502). Further, the remote system 102 may transmit data representing the current local state of the application associated with the control device 502 to the display device 106 and / or one or more other display devices 504 over the network 108.

[0072] In some cases, such as when the application includes a game application, the remote system 102 may store an official version of the game's environment. The remote system 102 may then use input data 130 from the control device to update the positions of objects (e.g., characters, vehicles, buildings, etc.) in the environment. Thus, data representing the local state of an application associated with a control device, such as the control device 104, may represent the position(s) of the object(s) controlled by the control device. The remote system 102 transmits this data to the display device 106 and / or one or more display devices 504 so that the display device 106 and / or one or more display devices 504 may update the position(s) of the object(s).

[0073] For example, a first local state of an application associated with the control device 104 may differ from a second local state of an application associated with the control device 502. For example, the first local state of an application associated with the control device 104, such as during a multiplayer network game, may represent a first position of a first object in a game environment, and the control device 104 is configured to control the first object. Thus, the display device 106 may display an image(s) representing the first local state of the application. In addition, a second local state of an application associated with the control device 502 may represent a second position of a second object in the game environment, and the control device 502 is configured to control the second object. Thus, the display device 504 (associated with the control device 502) may be displaying an image(s) representing the second local state of the application.

[0074] In such instances, when the remote system 102 updates the second local state of the application associated with the control device 502, such as by changing the position of the second object in the game environment, the remote system 102 may further update the first local state of the application to include the updated position of the second object. Thus, the first local state of the application may represent both the first position of the first object and the updated position of the second object.

[0075] While the above examples include a network application having multiple users, in other cases, the control device 104 may be associated with both the display device 106 and one or more of the display devices 504. For example, when the remote system 102 receives input data from the control device 104 and uses the input data to update the state of the application, the remote system 102 may transmit data (e.g., video data, audio data, timestamp data, etc.) to the display device 106 and the second display device 504 over the network 104. In some cases, the data transmitted to the display device 106 may be the same as the data transmitted to the second display device 504. For example, the display device 106 and the second display device 504 may display similar content (e.g., image(s) representing the state of the application). In addition, the display device 106 and the second display device 504 may receive similar timestamp data that causes the display device 106 and the second display device 504 to synchronize the display of content.

[0076] In some instances, the data sent to the display device 106 may differ from the data sent to the second display device 504. For example, the display device 106 and the second display device 504 may operate as a "split screen" where the display device 106 displays a first portion of the content and the second display device 504 displays a second portion of the content. In addition, the display device 106 and the second display device 504 may receive similar timestamp data that causes the display device 106 and the second display device 504 to synchronize the display of the content.

[0077] In some instances, the control device 104 and one or more other control devices 502 may be associated with the display device 106. For example, a first user may use the control device 104 to control an application, while a second user uses the second control device 502 to control the application. For example, the remote system 102 may receive data (e.g., input data) from each of the control device 104 and the second control device 502 over the network 108. The remote system 102 may then update the state of the application using the data received from each of the control device 104 and the second control device 106. In addition, the remote system 102 may transmit data (e.g., video data, audio data, timestamp data, etc.) representing the current state of the application to the display device 106 over the network 108.

[0078] In some cases, and in each of the above examples, control device 104 and one or more of control device 502 may be located in the same environment, and / or display device 106 and / or one or more of display device 504 may be located in the same environment. In some cases, and in each of the above examples, control device 104 and one or more of control device 502 may be located in separate environments, and / or display device 106 and / or one or more of display device 504 may be located in separate environments.

[0079] 5, one or more of the control devices 502 and / or one or more of the display devices 504 may be similar to the control device 104 and / or the display device 106, respectively. In other examples, one or more of the control devices 502 and / or one or more of the display devices 504 may be different from the control device 104 and / or the display device 106, respectively. For example, one or more of the control devices 502 may correspond to a video game console that processes input from a user. After processing the input, the video game console may transmit data to the remote system 102, where the data represents a change in the local state of an application.

[0080] 6A-9 illustrate various processes for controlling network applications. The processes described herein are illustrated as a collection of blocks in a logical flow diagram representing a sequence of operations, some or all of which may be implemented in hardware, software, or a combination thereof. In the context of software, the blocks may represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, programs the processors to perform the recited operations. In general, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular data type. The order in which the blocks are described should not be construed as limiting unless otherwise stated. Any number of the described blocks may be combined in any order and / or in parallel to implement a process, or alternative processes, and not all blocks need to be executed.

[0081] 6A, 6B illustrate an exemplary process for controlling a network application. At 602, the process 600 may include storing first data indicative of a first identifier associated with a first electronic device and a second identifier associated with a second electronic device. For example, the remote system 102 may store first data representing a first identifier associated with a first electronic device (e.g., the control device 104) and a second identifier associated with a second electronic device (e.g., the display device 106). In some cases, the first data may include user profile data including and / or associated with the first identifier data representing the first identifier and the second identifier data representing the second identifier. In some cases, the remote system 102 receives the first data during a configuration process of the first electronic device and / or the second electronic device.

[0082] At 604, process 600 may include storing second data representative of the application. For example, the remote system 102 may store the second data representative of the application. As described herein, the application may include, but is not limited to, a gaming application, a word processing application, an educational application, a multimedia application (e.g., movies, music, etc.), and / or any other type of application that can be accessed over a network.

[0083] At 606, process 600 may include receiving third data from the first electronic device, the third data representing the first identifier. For example, the remote system 102 may receive the third data (e.g., identifier data) from the first electronic device, the third data representing the first identifier. In some instances, the remote system 102 receives the third data based at least in part on the first electronic device connecting to the network.

[0084] At 608, process 600 may include determining, using the first data, that third data is received from the first electronic device. For example, the remote system 102 may use the first data to determine that third data is received from the first electronic device. In some instances, the remote system 102 may make the determination based at least in part on matching the first identifier represented by the first data with the first identifier represented by the third data.

[0085] At 610, the process 600 may include using the first data to determine that the second electronic device is associated with the first electronic device. For example, the remote system 102 may use the first data to determine that the second electronic device is associated with the first electronic device. In some cases, such as when the first data includes user profile data, the remote system 102 may determine that the user profile data includes and / or is associated with both first identifier data representing the first identifier and second identifier data representing a second identifier of the second electronic device. Based at least in part on the determination, the remote system 102 may determine that the second electronic device is associated with the first electronic device.

[0086] At 612, process 600 may include transmitting fourth data representing the first state of the application to the second electronic device. For example, remote system 102 may transmit the fourth data representing the first state of the application to the second electronic device. The fourth data may include a data packet, the data packet including video data representing an image(s) corresponding to the first state of the application, audio data representing a sound corresponding to the first state of the application, and / or timestamp data representing an instruction to display the first state of the application at a particular time and / or an instruction to output a sound at a particular time. In some cases, such as when the application includes a game application, the first state of the application may include an object (e.g., a character) located at a first location within the game environment.

[0087] At 614, process 600 may include receiving fifth data from the first electronic device representative of an input received by the first electronic device. For example, the remote system 102 may receive fifth data (e.g., input data) from the first electronic device representative of an input received by the first electronic device. In some cases, the input may be received by the first electronic device using button(s), key(s), joystick(s), touch-sensitive pad(s), trigger(s), sensor(s), etc. In some cases, the input may include audio data representative of sound captured by microphone(s) of the first electronic device.

[0088] At 616, the process 600 may include updating the first state of the application to a second state of the application based at least in part on the fifth data. For example, the remote system 102 may use the fifth data to update the first state of the application to a second state of the application. For example, again, if the application includes a gaming application, the input may correspond to a movement of an object. Thus, the remote system 102 may update the first state of the application to the second state of the application by moving the object from a first position in the gaming environment to a second position in the gaming environment.

[0089] At 618, process 600 may include transmitting sixth data representing a second state of the application to the second electronic device. For example, the remote system 102 may transmit the sixth data representing the second state of the application to the second electronic device. The sixth data may include a data packet including video data representing an image(s) corresponding to the second state of the application and / or audio data representing a sound corresponding to the second state of the application.

[0090] At 620, process 600 may include transmitting to the second electronic device seventh data representing a first timestamp associated with displaying the second state of the application. For example, remote system 102 may transmit to the second electronic device seventh data representing a first timestamp associated with displaying the second state of the application. In some cases, the seventh data may include timestamp data representing instructions to display the second state of the application at a particular time and / or instructions to output a sound at a particular time. In some cases, the seventh data is included in the same data packet that also includes the sixth data.

[0091] At 622, process 600 may include transmitting eighth data representing a sound corresponding to the second state of the application to the first electronic device. For example, remote system 102 may transmit the eighth data representing the sound corresponding to the second state of the application to the first electronic device. The sixth data may include a data packet including at least audio data representing the sound corresponding to the second state of the application.

[0092] At 624, the process 600 may include transmitting ninth data to the first electronic device, the ninth data representing a second timestamp associated with the output of the sound. For example, the remote system 102 may transmit to the first electronic device, the ninth data representing a second timestamp associated with the output of the sound. In some cases, the ninth data may include timestamp data representing an instruction to output the sound at a particular time. For example, the ninth data may synchronize the output of the sound by the first electronic device with the display of a second state of the application by the second electronic device. In some cases, the ninth data is included in the same data packet that also includes the audio data (e.g., included in the eighth data).

[0093] FIG. 7 illustrates a first exemplary process 700 of a remote system communicating with a control device and a display device to control a network application. At 702, the process 700 may include determining, by the remote system, that a first electronic device is associated with a second electronic device. For example, the remote system 102 may determine that a first electronic device (e.g., the control device 104) is associated with a second electronic device (e.g., the display device 106). In some instances, the remote system 102 may make the determination using user profile data. For example, the user profile data may include and / or be associated with first identifier data associated with the first electronic device and second identifier data associated with the second electronic device.

[0094] At 704, process 700 may include transmitting, by the remote system to the second electronic device, first data representing a first state of the application. For example, the remote system 102 may transmit the first data representing the first state of the application to the second electronic device. The first data may include a data packet, the data packet including video data representing an image(s) corresponding to the first state of the application, audio data representing a sound corresponding to the first state of the application, and / or timestamp data representing an instruction to display the first state of the application at a particular time and / or an instruction to output a sound at a particular time.

[0095] At 706, process 700 may include receiving, by the remote system, from the first electronic device, second data representative of the input received by the first electronic device. For example, the remote system 102 may receive, from the first electronic device, second data (e.g., input data) representative of the input received by the first electronic device. In some cases, the input may be received by the first electronic device using button(s), key(s), joystick(s), touch-sensitive pad(s), trigger(s), sensor(s), etc. In some cases, the second data may include audio data representative of sound captured by microphone(s) of the first electronic device.

[0096] At 708, the process 700 may include determining, by the remote system, a second state of the application based at least in part on the first state of the application and the second data. For example, the remote system 102 may determine the second state of the application based at least in part on the first state of the application and the second data. In some cases, such as when the application includes a gaming application, the first state of the application may include an object located at a first location within the gaming environment. In such cases, the remote system 102 may update the first location of the object to a second location based at least in part on the input. The remote system 102 may then determine the second state of the application as including the object located at the second location.

[0097] At 710, process 700 may include transmitting, by the remote system to the second electronic device, third data representing the second state of the application. For example, the remote system 102 may transmit the third data representing the second state of the application to the second electronic device. The third data may include a data packet including video data representing an image(s) corresponding to the second state of the application, audio data representing a sound corresponding to the second state of the application, and / or timestamp data representing an instruction to display the second state of the application at a particular time and / or an instruction to output a sound at a particular time.

[0098] 8 illustrates a second example process 800 of a remote system communicating with a control device and a display device to control a network application. At 802, the process 800 may include storing, by the remote system, first data representative of the application. For example, the remote system 102 may store the first data representative of the application (e.g., application data). As described herein, the application may include, but is not limited to, a gaming application, a word processing application, an educational application, a multimedia application (e.g., movies, music, etc.), and / or any other type of application that can be accessed over a network.

[0099] At 804, the process may include receiving, by the remote system, second data from the first electronic device, the second data representing the selection of the application. For example, the remote system 102 may receive, from the first electronic device (e.g., the control device 104), the second data representing the selection of the application.

[0100] At 806, process 800 may include transmitting, by the remote system to the second electronic device, first data representing the first state of the application. For example, the remote system 102 may transmit the first data representing the first state of the application to the second electronic device. The third data may include a data packet including video data representing an image(s) corresponding to the first state of the application, audio data representing a sound corresponding to the first state of the application, and / or timestamp data representing an instruction to display the first state of the application at a particular time and / or an instruction to output a sound at a particular time.

[0101] At 808, process 800 may include receiving, by the remote system, from the first electronic device, fourth data representative of the input received by the first electronic device. For example, the remote system 102 may receive, from the first electronic device, fourth data (e.g., input data) representative of the input received by the first electronic device. In some cases, the input may be received by the first electronic device using button(s), key(s), joystick(s), touch-sensitive pad(s), trigger(s), sensor(s), etc. In some cases, the second data may include audio data representative of sound captured by microphone(s) of the first electronic device.

[0102] At 810, the process 800 may include determining, by the remote system, a second state of the application based at least in part on the first state of the application and the fourth data. For example, the remote system 102 may determine a second state of the application based at least in part on the first state of the application and the fourth data. In some cases, such as when the application includes a gaming application, the first state of the application may include an object located at a first location within the gaming environment. In such cases, the remote system 102 may update the first location of the object to a second location based at least in part on the input. The remote system 102 may then determine a second state of the application as including an object located at the second location.

[0103] At 812, process 800 may include transmitting, by the remote system to the second electronic device, fifth data representing the second state of the application. For example, the remote system 102 may transmit the fifth data representing the second state of the application to the second electronic device. The fifth data may include a data packet including video data representing an image(s) corresponding to the second state of the application, audio data representing a sound corresponding to the second state of the application, and / or timestamp data representing an instruction to display the second state of the application at a particular time and / or an instruction to output a sound at a particular time.

[0104] 9 illustrates an example process 900 of a control device communicating with a remote system to control a network application. At 902, the process 900 may include storing first data representing settings associated with a network. For example, the control device 104 may store the first data (e.g., configuration data) representing settings associated with the network. In some instances, the first data may include an SSID associated with a router and a password.

[0105] At 904, the process 900 may include establishing a connection to a network using the first data. For example, the control device 104 may establish a connection with the network using the first data.

[0106] At 906, the process 900 may include receiving second data from the remote system representing a sound associated with the first state of the application. For example, the control device 104 may receive second data (e.g., audio data) representing a sound associated with the first state of the application from the remote system 102. In some cases, the control device 104 may further receive timestamp data representing a time for outputting the sound. The control device 104 may then cause the sound to be output. In some cases, the control device 104 outputs the sound using a speaker(s). In some cases, the control device 104 transmits the second data to a peripheral device for outputting the sound.

[0107] At 908, the process 900 may include receiving input using at least one input device. For example, the control device 104 may receive input using the at least one input device. In some cases, the at least one input device may include button(s), key(s), joystick(s), touch-sensitive pad(s), trigger(s), sensor(s), etc. In some cases, the at least one input device may include microphone(s).

[0108] At 910, the process 900 may include transmitting third data representative of the input to the remote system. For example, the control device 104 may transmit third data representative of the input (e.g., input data) to the remote system 102.

[0109] At 912, the process 900 may include receiving fourth data from the remote system representing a sound associated with the second state of the application. For example, the control device 104 may receive fourth data (e.g., audio data) representing a sound associated with the second state of the application from the remote system 102. In some cases, the control device 104 may further receive timestamp data representing a time for outputting the sound. The control device 104 may then cause the sound to be output. In some cases, the control device 104 outputs the sound using a speaker(s). In some cases, the control device 104 transmits the second data to a peripheral device for outputting the sound.

[0110] 10 illustrates a block diagram of an example architecture of a voice processing system that receives audio data and / or additional data from a voice-enabled device, such as the control device 104, and implements techniques for determining how to respond to a user's voice command expressed in the audio data. The block diagram of FIG. 10 illustrates an example architecture of a voice processing system (e.g., the remote system 102) that receives audio data 1002 and additional data 1004 (e.g., metadata such as timestamps, SRN values, etc.) from the control device 104, and implements techniques for determining the intent of the audio data 1002.

[0111] FIG. 10 includes a conceptual diagram of how a voice command 1006 may be processed, enabling the system to capture and execute a command spoken by a user 1008, such as a spoken command that may follow a wake word. The various components illustrated may be located on the same or different physical devices. Communication between the various components illustrated in FIG. 10 may occur directly or through a network. An audio capture component, such as microphone(s) 208 of the control device 104, captures audio corresponding to the voice command 1006. The control device 104 then processes the voice command 1006, or audio data corresponding to the voice command 1006, using a wake word detection component 1010 to determine whether a keyword (such as a wake word) is detected within the voice command 1006. Following detection of the wake word, the control device 104 transmits audio data 1002 corresponding to the voice command 1006 to a computing device of the remote system 102 that includes the ASR component 144. The audio data 1002 may be output from an acoustic front end (AFE) 1014 located on the control device 104 before transmission, or the audio data 1002 may be in a different form for processing by a remote AFE 1014, such as an AFE 1014 located with the ASR component 144.

[0112] In various examples, the remote system 102 may include one or more processors 308 for powering components, devices, and operations of the remote system 102, and one or more network interfaces 310, such as a wireless or Wi-Fi network communication interface, an Ethernet communication interface, a cellular network communication interface, a Bluetooth communication interface, etc., for communication by the remote system 102 over various types of networks (e.g., the network(s) 108), including wide area networks, local area networks, private networks, public networks, etc. In the case of a wireless communication interface, such an interface may include a wireless transceiver and associated control circuitry and logic for implementing an appropriate communication protocol.

[0113] The remote system 102 may include a voice processing component 1018, which includes various components for processing the voice commands 1006, such as an automatic speech recognition (ASR) component 144 and a natural language understanding (NLU) component 146. The various components illustrated may be located on the same or different physical devices.

[0114] In some examples, audio data (e.g., audio data 1002) may be received by the remote system 102 for voice processing (either for the purpose of enabling voice communication and / or for the purpose of executing the command in the voice) to interpret the included voice command 1006. The audio data 1002 may include data corresponding to a wake word, or a portion of the audio data corresponding to the wake word may be removed by the control device 104 before transmission. Upon receipt by the remote system 102, the ASR component 144 may convert the audio data to text. The ASR component 144 transcribes the audio data into text data representing the spoken words contained within the audio data. The text data may then be used by other components for various purposes, such as reconciliation, executing system commands, entering data, etc. The voice command 1006 in the audio data is input to a processor configured to perform ASR, which then interprets the utterance based on a similarity between the utterance and a pre-established ASR language model stored in an ASR model knowledge base (ASR model storage 1020). For example, an ASR process may compare input audio data to models of sounds (e.g., sub-word units or phonemes) and sequences of sounds to identify words that match sequences of sounds spoken in an utterance of the audio data.

[0115] Different ways (i.e., different hypotheses) in which the voice command 1006 may be interpreted may each be assigned an ASR probability or ASR confidence score that represents the likelihood that a particular set of words matches the words spoken in the utterance. The ASR confidence score may be based on several factors, including, for example, the similarity of the sounds in the utterance to a model of the language sounds (e.g., the acoustic model 1022 stored in the ASR model storage 1020) and the likelihood that a particular word matching the sounds will be included in a sentence in a particular location (e.g., using a language or grammar model). Thus, each potential text interpretation of the spoken utterance (hypothesis) is associated with an ASR confidence score. Based on the factors considered and the assigned ASR confidence score, the ASR component 144 outputs the text that is most likely to be recognized in the audio data. The ASR component 144 may also output multiple ASR hypotheses in the form of a lattice or N-best list, with each hypothesis corresponding to an ASR confidence score or other score (such as a probability score).

[0116] The one or more devices performing the ASR processing may include an acoustic front end (AFE) 1014 and a speech recognition engine 1024. The acoustic front end (AFE) 1014 converts audio data from a microphone into data for processing by the speech recognition engine 1024. The speech recognition engine 1024 compares the speech recognition data to an acoustic model 1022, a language model 1026, and other data models and information to recognize the speech conveyed in the audio data. The AFE 1014 reduces noise in the audio data and divides the digitized audio data into frames representing time intervals, for which the AFE 1014 determines some values, called features, that represent the quality of the audio data, along with a set of those values, called a feature vector, that represent the features / quality of the audio data in a frame. Many different features may be determined, as known in the art, with each feature representing some quality of the audio that may be useful for ASR processing. Several approaches may be used by the AFE 1014 to process the audio data, such as Mel-Frequency Cepstral Coefficients (MFCC), Perceptual Linear Prediction (PLP) techniques, neural network feature vector techniques, linear discriminant analysis, semi-associative covariance matrices, or other approaches known to those skilled in the art.

[0117] The speech recognition engine 1024 may process the output from the AFE 1014 with reference to information stored in the voice / model storage. Alternatively, data after front-end processing (such as feature vectors) may be received by the device performing the ASR processing from another source other than the internal AFE 1014. For example, the control device 104 may process audio data into feature vectors (e.g., using the AFE 1014 on the device) and transmit that information over a network to a server for ASR processing. The feature vectors may arrive at the remote system 102 encoded, in which case they may be decoded prior to processing by the processor executing the speech recognition engine 1024.

[0118] The speech recognition engine 1024 attempts to match the received feature vectors to known language phonemes and words in stored acoustic models 1022 and language models 1026. The speech recognition engine 1024 calculates a recognition score for the feature vectors based on the acoustic and linguistic information. The acoustic information is used to calculate an acoustic score that represents the likelihood that the intended sounds represented by the feature vectors match the language phonemes. The linguistic information is used to adjust the acoustic score by considering which sounds and / or words are used in context with each other, thereby improving the likelihood that the ASR process will output a grammatically meaningful speech result. The particular model used may be a general model or may be a model corresponding to a particular domain, such as music, banking, etc.

[0119] The speech recognition engine 1024 may use a number of techniques to match feature vectors against phonemes to determine the probability that the feature vector may match a phoneme, for example, using a hidden Markov model (HMM). A received sound may be represented as a path between states of the HMM, and multiple paths may represent multiple possible text matches for the same sound.

[0120] Following ASR processing, the ASR results may be transmitted by the speech recognition engine 1024 to other processing components, which may be local to the device performing the ASR and / or distributed across a network(s). For example, the ASR results in the form of a single text representation of the speech, an N-best list containing multiple hypotheses and their respective scores, a lattice, etc., may be transmitted to the remote system 102 for natural language understanding (NLU) processing, such as conversion of the text into commands for execution, either by the control device 104, the remote system 102, or another device (such as application system 148).

[0121] The NLU component 146 (e.g., a server) may include various components, including potentially dedicated processor(s), memory, storage, and the like. As shown in FIG. 10, the NLU component 146 may include a recognizer 1028 that includes a named entity recognition (NER) component 1030 that is used to identify portions of the query text that correspond to named entities that may be recognizable by the system. A downstream process called named entity resolution links text portions to specific entities known to the system. To perform named entity resolution, the system may utilize gazetteer information 1032 stored in an entity library storage 1034. The gazetteer information may be used for entity resolution, for example, to query ASR results with different entities (such as contact names). Gazetteers may be linked to users (e.g., a particular gazetteer may be associated with a particular user's game collection), to a particular domain (such as games), or may be organized in a variety of other manners.

[0122] In general, the NLU component 146 receives text input (such as text input determined by the ASR component 144) and attempts to perform a semantic interpretation of the text. That is, the NLU component 146 determines the meaning behind the text based on individual words and then implements that meaning. The NLU component 146 interprets text strings to derive an intent or desired action from the user and relevant pieces of information within the text that enable a device (e.g., the control device 104, the application system 148, etc.) to complete that action. For example, if a spoken utterance is processed using the ASR component 144 and outputs the text "advance the character," the NLU component 146 may determine that the user 1008 intended to advance a character in a gaming application. For a second example, if a spoken utterance is processed using the ASR component 144 and outputs the text "start the gaming application," the NLU component 146 may determine that the user 1008 intended for the application system 148 to start a gaming application.

[0123] The NLU component 146 may process several text inputs related to the same utterance. For example, if the ASR component 144 outputs N text segments (as part of an N-best list), the NLU component 146 may process all N outputs to obtain an NLU result.

[0124] To properly perform NLU processing of speech input, the NLU component 146 may be configured to determine the "domain" of the utterance to determine and narrow down which services offered by an endpoint device (e.g., remote system 102, control device 104, application system 148, etc.) may be relevant. For example, an endpoint device may offer services related to interaction with a phone service, a contact list service, a calendar / scheduling service, a gaming service, an application service, etc. Words in a single text query may involve more than one service, and some services may be functionally linked (e.g., both a phone service and a gaming service may utilize data from a contact list).

[0125] The named entity recognition (NER) component 1030 receives a query in the form of an ASR result and attempts to identify relevant grammatical and lexical information that can be used to interpret the meaning. To do so, the NLU component 146 may begin by identifying potential domains that may be relevant to the received query. The NLU storage 1036 includes a database of device domains 1052 that identify domains associated with a particular device. For example, the control device 104 may be associated with domains for gaming, phone, calendaring, contact list, and device-specific communications, but not video. In addition, the device domains 1052 may include database entries for particular services on particular devices, indexed by either a device ID, a user ID, or a household ID, or some other indicator.

[0126] In NLU processing, a domain may represent a distinct set of activities with a common theme, such as "gaming," "phone," "calendaring," etc. Thus, each domain may be associated with a particular recognizer 1028, a language model and / or grammar database 1040, a particular set of domain intents / actions 1038, and a particular personalized domain lexicon 1042. Each lexicon 1032 may include domain-indexed lexical information associated with a particular user and / or device. A user's contact list lexical information may include names of contacts. This personalized information improves entity resolution, since every user's contact list is likely different.

[0127] As described above, in conventional NLU processing, a query may be processed applying rules, models, and information applicable to each identified domain. For example, if a query potentially involves both communication and, for example, music, the query may be NLU processed using the grammar model and vocabulary information for communication, and processed using the grammar model and vocabulary information for music, in substantially parallel fashion. The query-based responses generated by each model set are scored, and the overall highest ranked result from all applied domains is typically selected as the correct result.

[0128] An intent classification (IC) component 1044 analyzes the query to determine one or more intents for each identified domain, where the intent corresponds to an action to be performed in response to the query. Each domain is associated with a database of words linked to intents (e.g., domain intent 1038). For example, a gaming domain intent database 1038 may link words and phrases such as "go forward," "go backward," and "jump" to an "action" intent. Meanwhile, a voice message domain intent database may link words and phrases such as "send message," "send voice message," and "send below." The IC component 1044 identifies potential intents for each identified domain by comparing words in the query to words and phrases in the domain intent database 1038. In some instances, intent determination by the IC component 1044 is performed using a set of rules or templates that are processed against the input text to identify matching intents.

[0129] To generate a specific interpreted response, the NER 1030 applies the grammar model and lexical information associated with the respective domain to actually recognize mention of one or more entities in the text of the query. In this manner, the NER component 1030 identifies "slots" (i.e., specific words in the query text) that may be required for subsequent command processing. Depending on the complexity of the NER component 1030, the NER component 1030 may also label each slot by type (noun, place, city, artist name, song name, etc.) with various levels of specificity. Each grammar model 1040 includes names of entities (i.e., nouns) commonly found in speech for a particular domain (i.e., general terms), while the lexical information from the gazetteer 1032 is individualized to the user(s) and / or device. For example, a grammar model associated with a gaming domain may include a database of words commonly used when people are discussing games and / or playing gaming applications.

[0130] The intents identified by the IC component 1044 are linked to domain-specific grammar frameworks (included in 1040) that have "slots" or "fields" to be filled. Each slot / field corresponds to a portion of the query text that the system believes corresponds to an entity. To make the resolution more flexible, these frameworks are typically structured based on associating slots with grammar tags, rather than structured as sentences. For example, if "send a message" is an identified intent, one or more grammar (1040) frameworks may correspond to sentence structures such as "send a message to {contact}", "send a voice message to {contact}", "send the following to {contact}", etc.

[0131] For example, the NER component 1030 may parse the query to identify words as subjects, objects, verbs, prepositions, etc. based on grammar rules and / or models before recognizing named entities. The identified verbs may be used by the IC component 1044 to identify the intent, which is then used by the NER component 1030 to identify the framework. Meanwhile, the framework for the intent "send a message" may specify a list of applicable slots / fields to serve the identified "contact" and any object modifiers, etc. (e.g., specify the recipient's device). The NER component 1030 then searches the corresponding fields in the domain-specific and personalized lexicon(s) and attempts to match words and phrases in the query tagged as grammatical objects or object modifiers with those identified in the database(s).

[0132] This process involves semantic tagging, which is the labeling of words or word combinations according to their type / semantic meaning. Parsing can be performed using heuristic grammar rules, or NER models can be built using techniques such as hidden Markov models, maximum entropy models, log-linear models, conditional random fields (CRFs), etc.

[0133] The intent-linked framework is then used to determine what database fields should be searched to determine the meaning of these phrases, such as searching the user's gazetteer for similarity to the framework slots. If the gazetteer search does not resolve the slot / field using the gazetteer information, the NER component 1030 may search a database of common words associated with the domain (in the knowledge base 1046). Thus, for example, if the query was "send a message to Joe," after failing to determine the last name of "Joe," the NER component 1030 may search the domain lexicon for the word "Joe." In alternative examples, common words may be checked before the gazetteer information, or both may be tried, potentially resulting in two different results.

[0134] Output data from the NLU component 146 (which may include tagged text, commands, etc.) may then be sent to a command processing component 1048. The destination command processing component 1048 may be determined based on the NLU output. For example, if the NLU output includes a command to send a message, the command processing component 1048 may be a message sending application, such as one located on the control device 104, on the application system 148, or in a message sending appliance, configured to execute the message sending command. If the NLU output includes an input to a gaming application, such as moving a character in a given direction, the destination command processor may include the application system 148 configured to execute the command.

[0135] The NLU operations of the existing system may take the form of a multi-domain architecture. Each domain (which may include a set of intents and entity slots that define larger concepts such as gaming, messaging, and components such as trained models used to perform various NLU operations such as NER, IC, etc.) may be built separately and made available to the NLU component 146 during runtime operations (such as text output from the ASR component 1014) where the NLU operations are performed on text. Each domain may have components specifically configured to perform various steps of the NLU operations.

[0136] For example, in an NLU system, the system may include a multi-domain architecture composed of multiple domains for intents / commands that can be executed by the system (or by other devices connected to the system), such as gaming, messaging, and information. The system may include multiple domain recognizers, each of which may include its own recognizer 1028. Each recognizer may include various NLU components, such as a NER component 1030, an IC component 1044, and other components, such as an entity resolver, or other components.

[0137] The command processing component 1048 and / or the NLU component 146 may determine the domain based on the intent and, based on this determination, route the request corresponding to the audio data to an appropriate domain speechlet, such as the illustrated domain speechlet 1050. The domain speechlet 1050 may include any type of device or group of devices (e.g., hardware devices, virtual devices or partitions, servers, etc.) and may receive the text data and / or intent associated with the audio data 1002 and determine how to respond to the request. For example, the intent of the command "make the character walk forward" may be routed to a gaming domain speechlet 1050 that controls a device such as the application system 148.

[0138] Various types of domain speechlets 1050 can be used to determine which device 1002 to use to respond to the voice command 1006, as well as the appropriate response and potential additional content (e.g., audio data). For example, domain speechlets 1050 include third party skill domain speechlets 1050 that can handle intents associated with gaming, productivity, etc., music domain speechlets 1050 that can handle intents associated with music play requests (e.g., Amazon Music, Pandora, Spotify, iHeart, etc.), video play requests (e.g., Amazon The speechlets 1050 may include a video domain speechlet 1050 that may handle intents associated with video, audio, video content, video streaming, video streaming services ...

[0139] Although the above invention has been described with respect to specific examples, it should be understood that the scope of the invention is not limited to these specific examples. Since other modifications and variations adapted to suit specific operating requirements and environments will be apparent to those skilled in the art, the present invention is not to be considered limited to the examples selected for the purpose of disclosure, but extends to all variations and variations that do not depart from the true concept and scope of the invention.

[0140] Although the present application describes embodiments having particular structural features and / or methodological acts, it should be understood that the claims are not necessarily limited to the particular features or acts described. Rather, the particular features and acts are merely illustrative of some embodiments that fall within the scope of the claims.

[0141] Example clauses A: A system located remotely from an environment including a handheld controller and a display device, comprising: at least one database storing profile data indicative of a first identifier associated with the handheld controller and a second identifier associated with the display device, and game data representative of a game; one or more processors; and one or more computer readable media storing instructions, which when executed by the one or more processors, cause the one or more processors to: receive first data representative of the first identifier from the handheld controller over a network; determine using the profile data that the first data is received from the handheld controller; determine using the profile data that the handheld controller is associated with the display device; and transmit first video data over the network to the display device, wherein the first video data indicates a first state of the game. receiving, over a network from the handheld controller, input data representing an input received by the handheld controller, updating a first state of the game to a second state of the game based at least in part on the input data; transmitting, over the network, second video data representing the second state of the game to a display device; transmitting, over the network, first timestamp data to the display device, the first timestamp data including first instructions for displaying the second state of the game at a first time; transmitting, over the network, audio data to the handheld controller, audio data representing a sound corresponding to the second state of the game; and transmitting, over the network, second timestamp data to the handheld controller, the second timestamp data including second instructions for outputting the sound at a second time, the second time being within a threshold period to the first time.

[0142] B: The system described in paragraph A, wherein the one or more computer-readable media store further instructions that, when executed by the one or more processors, cause the one or more processors to perform further operations including: transmitting, via the network to the display device, additional audio data representing a sound corresponding to a second state of the game; and transmitting, via the network to the display device, third timestamp data, the third timestamp data including third instructions for outputting the sound at a third time, the third time being similar to the first time.

[0143] C: The system of either paragraph A or paragraph B, wherein the first state of the game includes an object located at a first position within the game environment, and updating the first state of the game to a second state of the game includes determining a second position of the object within the game environment using the input data, and updating the position of the object within the game environment from the first position within the game environment to the second position within the game environment.

[0144] D: A method comprising: determining, by a remote system, that an electronic device is associated with a display device; transmitting, by the remote system, first video data representing a first state of a game to the display device; receiving, by the remote system, input data from the electronic device representing an input received by the electronic device; determining, by the remote system, a second state of the game based at least in part on the first state of the game and the input data; and transmitting, by the remote system, second video data representing the second state of the game to the display device.

[0145] E: The method of paragraph E, further including: transmitting, by the remote system to the electronic device, audio data representing a sound associated with a second state of the game; and transmitting, by the remote system to the electronic device, data representing a timestamp, the timestamp being associated with synchronizing output of the sound by the electronic device with display of the second video data by the display device.

[0146] F: The method of either paragraph D or E, further including: establishing, by the remote system, a first network connection with the electronic device, wherein receiving input data is established via the first network connection; and establishing, by the remote system, a second network connection with the display device, wherein transmitting second video data is established via the second network connection.

[0147] G: The method of any one of paragraphs D-F, wherein determining a second state of the game based at least in part on the first state of the game and the input data includes: determining, by the remote system, that the first state of the game includes the object being located at a first location within the game environment; determining, by the remote system, based at least in part on the first location and the input data, that a second location of the object is located at a second location within the environment; and determining, by the remote system, that the second state of the game includes the object being located at a second location within the environment.

[0148] H: The method of any one of paragraphs D-G, further including: transmitting, by the remote system to the display device, third video data representing one or more games, the one or more games including a game; receiving, by the remote system, additional input data from the electronic device representing additional input received by the electronic device; and initiating, by the remote system, the game based at least in part on the additional input data.

[0149] I: The method of any one of paragraphs D-H, wherein the input data includes audio data representing a user voice, and the method further includes determining, by the remote system, one or more words represented by the user voice, and determining, by the remote system, a second state of the game based at least in part on the first state of the game and the input data, includes determining, by the remote system, a second state of the game based at least in part on the first state of the game and the one or more words.

[0150] J: The method of any one of paragraphs D-I, further including storing, in a database by the remote system, first data representing a first identifier associated with the electronic device and a second identifier associated with the display device, receiving, by the remote system, second data representing the first identifier from the electronic device, and determining, by the remote system, using the first data that the second data was sent from the electronic device, wherein determining that the display device is associated with the electronic device is based at least in part on the first data.

[0151] K: The method of any one of paragraphs D-J, further including: receiving, by the remote system, first data representing a first identifier associated with the electronic device; receiving, by the remote system, second data representing a second identifier associated with the display device; and generating, by the remote system, third data representing a user profile, the third data indicating at least the first identifier associated with the electronic device and the second identifier associated with the display device.

[0152] L: The method of any one of paragraphs D-K, including determining, by the remote system, that the electronic device is associated with an additional display device, transmitting, by the remote system, first video data representing a first state of the game to the additional display device, and transmitting, by the remote system, second video data representing a second state of the game to the additional display device.

[0153] M: The method of any one of paragraphs D-L, further including: transmitting, by the remote system to the display device, first data representing a first timestamp, the first data representing the first timestamp associated with a second state of the game; and transmitting, by the remote system to the additional display device, second data representing a second timestamp, the second timestamp being associated with synchronizing display of the second state of the game by the additional display device with display of the second state of the game by the display device.

[0154] N: The method of any one of paragraphs D-M, further including receiving, by the remote system, additional input data from the additional electronic device representing additional input received by the additional electronic device, determining, by the remote system, a third state of the game based at least in part on the second state of the game and the additional input data, and transmitting, by the remote system to the display device, third video data representing the third state of the game.

[0155] O: A system comprising: one or more processors; and one or more computer-readable media having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations including: storing data associated with a game; receiving first input data from the electronic device over a network representing a selection of the game; transmitting first video data representing a first state of the game to a display device over the network; receiving second input data from the electronic device over the network representing an input received by the electronic device; determining a second state of the game based at least in part on the first state of the game and the second input data; and transmitting second video data representing the second state of the game to the display device over the network.

[0156] P: The system described in paragraph O, wherein the one or more computer-readable media store further instructions that, when executed by the one or more processors, cause the one or more processors to perform further operations including sending, to the electronic device, audio data representing a sound associated with a second state of the game, determining a time at which the electronic device outputs the sound, and sending, to the electronic device, additional data representing instructions for outputting the sound at the time.

[0157] Q: The system of either paragraph O or paragraph P, wherein the one or more computer-readable media store further instructions which, when executed by the one or more processors, cause the one or more processors to perform further operations including establishing a first network connection with the electronic device, wherein the receiving of second input data is established via the first network connection, and establishing a second network connection with the display device, wherein the transmitting of second video data is established via the second network connection.

[0158] R: The system of any one of paragraphs O-Q, wherein determining a second state of the game based at least in part on the first state of the game and the second input data includes: determining that the first state of the game includes an object being located at a first location within the game environment; determining a second location of the object based at least in part on the first location and the second input data; and determining that the second state of the game includes the object being located at a second location within the environment.

[0159] S: The system of any one of paragraphs O-R, wherein the one or more computer-readable media store further instructions that, when executed by the one or more processors, cause the one or more processors to perform further operations, including transmitting, to a display device, third video data representing the one or more games, the one or more games including the games.

[0160] T: The system of any one of paragraphs O-S, wherein the data is first data, and the one or more computer-readable media stores further instructions that, when executed by the one or more processors, cause the one or more processors to perform further operations including storing second data representing an association between the electronic device and the display device; receiving third data from the electronic device representing an identifier associated with the electronic device; and determining using the second data that the electronic device is associated with the display device, wherein at least one of the transmission of the first video data or the transmission of the second video data is based at least in part on determining that the electronic device is associated with the display device.

Claims

1. 1. A system located remotely from an environment including a handheld controller and a display device, comprising: at least one database storing profile data indicative of a first identifier associated with the handheld controller and a second identifier associated with the display device, and game data representative of a game; one or more processors; one or more computer readable media storing instructions; the instructions, when executed by one or more processors, cause the one or more processors to: receiving first data representative of the first identifier from the handheld controller over a network; determining, using the profile data, that the first data is received from the handheld controller; determining that the handheld controller is associated with the display device based on determining, when it is determined that the first data has been received from the handheld controller, that the second identifier included in the profile data corresponding to the first data is associated with the first identifier; transmitting first video data representing a first state of the game to the display device over the network; receiving input data from the handheld controller over the network, the input data representing inputs received by the handheld controller; updating the first state of the game to a second state of the game based at least in part on the input data; transmitting second video data representative of the second state of the game to the display device over the network; transmitting, via the network to the display device, first timestamp data including first instructions to cause the display device to display the second state of the game at a first time; transmitting audio data representing a sound corresponding to the second state of the game to the handheld controller over the network; transmitting second time stamp data over the network to the handheld controller, the second time stamp data including second instructions to cause the handheld controller to output the sound at a second time within a threshold period of time until the first time; A system that performs an operation including:

2. The one or more computer-readable media storing further instructions that, when executed by the one or more processors, cause the one or more processors to: transmitting additional audio data representing the sound corresponding to the second state of the game to the display device over the network; transmitting, via the network to the display device, third timestamp data including third instructions to cause the display device to output the sound at a third time within a threshold period of time until the first time; The system of claim 1 , further comprising:

3. 1. A method comprising: determining, by a remote system, that the electronic device is associated with a display device based on determining, upon determining that first data representing a first identifier has been received from an electronic device, that a second identifier included in identifier data corresponding to the first data is associated with the first identifier, the identifier data including data representing the first identifier associated with the electronic device and data representing the second identifier associated with the display device; transmitting, by the remote system to the display device, first video data representative of a first state of a game; receiving, by the remote system, input data from the electronic device representative of input received by the electronic device; determining, by the remote system, a second state of the game based at least in part on the first state of the game and the input data; transmitting, by the remote system to the display device, second video data representative of the second state of the game, data representative of a time to display the second state, and data representative of a time to output a sound corresponding to the second state; A method comprising:

4. transmitting, by the remote system to the electronic device, audio data representative of a sound associated with the second state of the game; transmitting, by the remote system to the electronic device, data representing a timestamp associated with synchronizing the display of the second video data by the display device and the output of the sound by the electronic device; The method of claim 3 further comprising:

5. establishing, by the remote system, a first network connection with the electronic device; establishing, by the remote system, a second network connection with the display device; 4. The method of claim 3, further comprising: wherein the receiving the input data comprises receiving the input data over the first network connection; and wherein the transmitting the second video data comprises transmitting the second video data over the second network connection.

6. determining the second state of the game based at least in part on the first state of the game and the input data; determining, by the remote system, that the first state of the game includes an object located at a first position within a game environment; determining, by the remote system, a second position of the object based at least in part on the first position and the input data; determining, by the remote system, that the second state of the game includes the object being located at the second position within the game environment; The method of claim 3 further comprising:

7. transmitting, by the remote system, to the display device, third video data representative of one or more games including the game; receiving, by the remote system, additional input data from the electronic device representative of additional input received by the electronic device; initiating, by the remote system, the game based at least in part on the additional input data; and The method of claim 3 further comprising:

8. The input data includes audio data representing a user's voice, the method comprising: determining, by the remote system, one or more words represented by the user speech; 4. The method of claim 3, wherein the determining the second state of the game based at least in part on the first state of the game and the input data comprises determining, by the remote system, the second state of the game based at least in part on the first state of the game and the one or more words.

9. storing said identifier data in a database by said remote system; The method of claim 3 further comprising:

10. receiving, by the remote system, first data representative of a first identifier associated with the electronic device; receiving, by the remote system, second data representative of a second identifier associated with the display device; generating, by the remote system, third data representative of a user profile indicative of at least the first identifier associated with the electronic device and the second identifier associated with the display device; The method of claim 3 further comprising:

11. 1. A system comprising: one or more processors; one or more computer readable media storing instructions; the instructions, when executed by the one or more processors, cause the one or more processors to: determining that the electronic device is associated with a display device based on determining, upon determining that first data representing a first identifier has been received from an electronic device, that a second identifier included in identifier data corresponding to the first data is associated with the first identifier, the identifier data including data representing the first identifier associated with the electronic device and data representing the second identifier associated with the display device; storing data associated with the game; receiving first input data from the electronic device over a network representing a selection of the game; transmitting first video data representing a first state of the game over the network to the display device; receiving second input data from the electronic device over the network representative of an input received by the electronic device; determining a second state of the game based at least in part on the first state of the game and the second input data; transmitting, via the network to the display device, second video data representing the second state of the game, data representing a time for displaying the second state, and data representing a time for outputting a sound corresponding to the second state; A system that performs an operation including:

12. The one or more computer-readable media storing further instructions that, when executed by the one or more processors, cause the one or more processors to: transmitting to the electronic device audio data representing a sound associated with the second state of the game; determining a time when the electronic device outputs the sound; sending additional data to said electronic device representing instructions to output said sound at said time; The system of claim 11 , further comprising:

13. The one or more computer-readable media storing further instructions that, when executed by the one or more processors, cause the one or more processors to: Establishing a first network connection with the electronic device; establishing a second network connection with the display device; 12. The system of claim 11, further comprising: wherein the receiving the second input data comprises receiving the second input data over the first network connection; and wherein the transmitting the second video data comprises transmitting the second video data over the second network connection.

14. determining the second state of the game based at least in part on the first state of the game and the second input data; determining that the first state of the game includes an object located at a first position within a game environment; determining a second position of the object based at least in part on the first position and the second input data; determining that the second state of the game includes the object being located at the second position within the game environment; The system of claim 11 , comprising:

15. The one or more computer-readable media storing further instructions that, when executed by the one or more processors, cause the one or more processors to: storing said identifier data; The system of claim 11 , further comprising:

Images