Systems and handheld controller for establishing direct communication between server system and video game controller

Abstract

To provide systems and methods for establishing direct communication between a server system and a video game controller.SOLUTION: Systems and methods include a video game controller and a computing device. An application is executed by a server system when a session is established by the computing device with the server system. Once the application is executed, the video game controller is used to send an identifier to the server system. The server system verifies the identifier to pair the session with the video game controller. When the video game controller is paired with the session, the video game controller can be used to change a state of a virtual scene that is displayed on the computing device or on a display screen.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] The present disclosure relates to a system and method for establishing direct communication between a server system and a video game controller. [Background technology]

[0002] Typically, games are accessed for play through a gaming console. The gaming console is connected to a game controller. The gamer connects the gaming console to a television. The game or game updates are downloaded from a server to the gaming console, and the gamer plays the game using the game controller.

[0003] It is in this context that the embodiments described in this disclosure arise. Summary of the Invention

[0004] The embodiments described in this disclosure provide a system and method for establishing direct communication between a server system and a video game controller.

[0005] In some embodiments, the systems and methods described herein provide a way for a video game controller to be used with any display device coupled to a computer network. A gaming console need not be used to access an application running on a server system. For example, a video game controller can be used with a computing device such as a mobile phone, tablet, or television. A gaming console need not be used. An execution session of the application is accessed using the computing device. Upon accessing the session, an identifier associated with the video game controller is sent to the server system. The server system verifies the identifier. Upon verifying the identifier, the server system determines to pair the video game controller with the session. Once the video game controller is paired with the session, a user can use the video game controller to change the state of a virtual scene displayed on the computing device.

[0006] Some advantages of the systems and methods described herein for establishing direct communication between a server system and a video game controller include eliminating the need for a game console between the video game controller and the server system. A user accesses a session by providing their login information. Once a session between a computing device and the server system is established, an identifier associated with the video game controller is sent from the video game controller to the server system. Once the identifier is verified by the server system, the server system pairs the video game controller with the session, allowing the user to modify the state of a virtual scene displayed on the computing device or another display device. Thus, a game console is not required to use the video game controller. The video game controller can be used regardless of whether the game console is used to access a virtual scene generated by execution of an application stored on the server. Eliminating the use of a game console reduces the number of hops between the video game controller and the server system. Each hop is used to receive data, analyze the data to determine a destination address, and transmit the data to the destination address. Eliminating the game console reduces the hops associated with the game console. Reducing the number of hops between the video game controller and the server system reduces latency between the video game controller and the server system, speeding up game play.

[0007] Other aspects of the disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, which illustrate, by way of example, the principles of the embodiments described in the present disclosure.

[0008] The various embodiments described in this disclosure will be best understood by referring to the following description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0009] [Figure 1A]FIG. 1 illustrates an embodiment of a system for illustrating pairing of a controller with a session. [Figure 1B] FIG. 10 is a diagram of an embodiment of a controller to illustrate wireless communication between the controller and a router. [Figure 1C] FIG. 1 illustrates an embodiment of a system for illustrating use of cellular connection channels by a controller. [Figure 1D] 1C to illustrate the use of a broadband communication integrated circuit (IC) chip to transfer identification information from the controller of FIG. 1C to a server system. [Figure 1E] FIG. 1 is a diagram of an embodiment of a server system to illustrate pairing of a session with a controller. [Figure 1F-1] FIG. 10 illustrates an embodiment of a pairing request notification displayed on a display screen of a computing device. [Figure 1F-2] 10A and 10B are diagrams of an embodiment of a controller illustrating the display of a pairing request notification on a display screen of the controller. [Figure 1G] FIG. 1 is a diagram of an embodiment of a controller illustrating the controller's haptic feedback device, audio device, and display device. [Figure 2] FIG. 1 illustrates an embodiment of a system for illustrating the mirroring effect of computing devices. [Figure 3] FIG. 1D is a diagram of an embodiment of a system to illustrate execution of a discovery program to discover the device ID of the controller of FIG. 1E. [Figure 4A] FIG. 1 is a diagram of an embodiment of a controller to illustrate the use of a calibration processor within the controller. [Figure 4B] FIG. 1 is a diagram of an embodiment of a controller to illustrate the use of a calibration processor within the controller. [Figure 5] FIG. 1 is a diagram of an embodiment of a server system to illustrate calibration of sensor values ​​by a calibration processor of the server system. [Figure 6] 1 shows a perspective view of an embodiment of a video game controller for interfacing with an interactive program. DETAILED DESCRIPTION OF THE INVENTION

[0010] SUMMARY OF THE INVENTION A system and method for establishing direct communication between a server system and a video game controller is described herein.

[0011] It should be noted that the various embodiments described in this disclosure may be practiced without some or all of these specific details. In other instances, well-known process operations have not been described in detail in order to not unnecessarily obscure the various embodiments described in this disclosure.

[0012] FIG. 1A is a diagram of an embodiment of a system 100 illustrating pairing of a controller 102 with a session 106. Examples of controllers 102 include handheld controllers, which may be a Sony® DualShock® controller, a gun controller, a PlayStation® Move® controller, a stick controller, a mobile phone, a mobile device, a tablet, a video game controller, a joystick, a glove controller, or a steering wheel controller. Examples of sessions 106 include a game session, a video session, a virtual image session, an augmented reality image session, a virtual reality image session, etc. Illustratively, session 106 is an instance of execution of application 1, which may be a game execution application that enables the playing of a game application or a video conferencing application that enables images of a real-world environment to be transferred between two clients over a computer network 122. Examples of real-world environments include a room or confined area, an enclosed environment, or an environment surrounded by walls or cubicles. Session 106 begins when user A logs into user account 1 and ends when user A logs out of user account 1. For example, an instance of execution of session 106 begins when user A logs into his / her user account 1 and ends for user account 1 when user A logs out of user account 1. Logging out by user A disables game play by user A, and logging in by user A into user account 1 is performed to enable game play. User A logs out of user account 1 by selecting a button on controller 102 or another controller described herein.

[0013] The system 100 includes a controller 102, a computing device 114, a computer network 122, a server system 104, a router 118, and a modem 120. By way of example, the router 118 and the modem 120 are located locally relative to the controller 102. For example, the router 118 and the modem 120 are located in the same real-world environment in which the controller 102 is located.

[0014] The computing device 114 is a machine for performing calculations. Examples of the computing device 114 include a smart television, a tablet, a smartphone, a head-mounted display (HMD), an electronic computer, an information processing system, a desktop computer, and a laptop computer. Another example of the computing device 114 includes a combination of a display device and a gaming console. The display device is coupled to the gaming console. The computer network 122 is a group of computing hardware devices linked together via a communication channel to facilitate communication and resource sharing between users. The computer network 122 can be a wide area network or a local area network, or a combination of a wide area network and a local area network. The Internet is an example of a wide area network, and an intranet is an example of a local area network.

[0015] The server system 104 includes one or more servers that run the application 1 and other applications. For example, each server includes a server operating system (OS) designed to run on the server. Each server is a dedicated computer that processes requests from clients, such as computing devices or controllers or display devices, as described herein. The dedicated computer includes a processor and a memory device. The processor is coupled to the memory device. As used herein, terms such as processor, application specific integrated circuit (ASIC), programmable logic device (PLD), and central processing unit (CPU), server, and microprocessor are used interchangeably herein. Examples of memory devices include read-only memory and random access memory. Illustratively, the memory device may be a flash memory, a hard disk drive, or a redundant array of independent disks (RAID).

[0016] Modem 120 is a device that performs modulation and demodulation of data to allow clients to connect to computer network 122. As an example, modem 120 applies a network communication protocol, such as Transmission Control Protocol / Internet Protocol (TCP / IP), to extract data from transmission packets. Modem 120 applies a network communication protocol to embed data into transmission packets. An example of modem 120 is a broadband modem that allows clients to access the Internet via cable or digital subscriber line (DSL). Modem 120 routes data signals received from router 118 to server system 104 via computer network 122. Additionally, modem 120 routes data signals received from server system 104 to router 118 via computer network 122. Modem 120 is coupled at one end to a cable or DLS, which is coupled to computer network 122. In one embodiment, the terms data and information are used interchangeably herein.

[0017] The router 118 directs a data signal received from the server system 104 via the computer network 112 to either the computing device 114 or the controller 102 based on a destination address received by the router 118 along with the data signal. For example, the router 118 determines that the data signal has a destination address of the computing device 114 and directs the data signal to the computing device 114 to which the destination address is assigned. Examples of destination addresses are an Internet Protocol (IP) address, a Media Access Layer (MAC) address, or a combination thereof. The destination address of the computing device 114 is different from the destination address of the controller 102. For example, at least one alphanumeric character in the destination address of the controller 102 is different from at least one alphanumeric character in the destination address of the computing device 114. The router 118 determines that another data signal has a destination address of the controller 102 and directs the other data signal to the controller 102. An example of a router 118 is a broadband router. The computing device 114 has a display screen 116, for example a liquid crystal display (LCD) screen, a light emitting diode (LED) display screen, or a plasma display screen, which is the display portion of a monitor or display device.

[0018] The server system 104 stores a plurality of user accounts 1 to N. Each user account is assigned to a user. For example, user account 1 is assigned to user A, and another user account 2 is assigned to another user B.

[0019] The controller 102 is coupled to the router 118 via a connection channel CC1. An example of the connection channel CC1 is a wireless connection such as a wireless local area network (LAN), a Wi-Fi connection, or a Bluetooth® connection. Wi-Fi is a wireless network technology that uses radio waves or electromagnetic waves to provide a wireless high-speed connection to the computer network 122. Similarly, the computing device 114 is connected to the router 118 via a connection channel CC2, which is also a wireless connection. The connection channel CC1 or CC2 can be a radio frequency and has a specific data rate measured in bits per second and a bandwidth measured in Hertz. Examples of the connection channels CC1 and CC2 are wireless connection channels that apply wireless communication protocols such as Wi-Fi® or Bluetooth®. The connection channel CC2 of the computing device 114 with the router 118 is different from the connection channel CC1 of the controller 102 with the router 118. For example, the wireless connection between the computing device 114 and the router 118 has a different data transfer rate between the computing device 114 and the router 118 compared to the data transfer rate between the controller 102 and the router 118.

[0020] Router 118 is coupled to modem 120 via Ethernet cable EC. Modem 120 is coupled to computer network 122 via connection C1, and server system 104 is coupled to computer network 122 via another connection C2. Connection C1 includes a group of cables or digital subscriber lines, and connection C2 includes a gateway device and a group of cables or digital subscriber lines. The combination of connection channel CC1, router 118, cable EC, modem 120, and connection C1 is sometimes referred to herein as connection channel 111.

[0021] User A powers on computing device 114 and accesses user account 1. For example, user A accesses a login website from server system 104 via computer network 122. User A uses an input device, such as a keyboard, pad, or mouse, on computing device 114 to communicate his or her login information, which may include a username and password assigned to user A. The login information is transmitted from computing device 114 to router 118 via connection channel CC2. Router 118 transmits the login information to modem 120 via cable EC. Modem 120 generates one or more transport packets embedding the login information and transmits the transport packets to server system 104 via connection C1, computer network 122, and connection C2.

[0022] The server of server system 104 determines that the login information is authentic, provides access to user account 1, and establishes session 106. To establish session 106, server system 104 executes an instance of application 1 and sends one or more transport packets containing information related to session 106 to computing device 11 via connection C2, computer network 122, connection C1, modem 120, cable EC, router 118, and communication channel CC2. Examples of information related to session 106 include information identifying application 1, such as the name of application 1 to be granted access when providing access to user account 1, the title of application 1, or virtual image data identifying application 1, audio data identifying application 1, or a combination of two or more thereof. Explained, information related to session 106 includes image data for displaying an introductory image of a video game on computing device 114 and audio data synchronized with the display of the introductory image. As another example, information related to session 106 includes a uniform resource locator (URL) of a game website accessed to display features of the video game and play the video game. Another example of information related to session 105 includes a notification that user A will provide their biometric information via controller 102 by using the biometric information scanner BIS of controller 102. Explaining, information related to session 106 includes a notification that user A will press their finger against the fingerprint reader of controller 102, or that user A will speak into the microphone of controller 102, or that user A will place their eye in the biometric eye scanner of controller 102 to be scanned. Examples of biometric information scanners include a fingerprint reader, a microphone, and a biometric eye scanner. Another example of information related to session 105 includes information identifying application 1 and a notification that user A will provide their biometric information via controller 102. The biometric information is unique to user A and distinguishes user A from other users.Yet another example of information related to the session 106 includes information identifying application 1 and instructions sent to the controller 102 to provide its device identification (ID). Examples of device IDs include the serial number of the controller 102, the MAC address of the controller 102, and combinations thereof. The device ID of the controller 102 is unique to the controller 102 and distinguishes the controller 102 from other controllers, which may be identical in structure and function to the controller 102. Yet another example of information related to the session 106 includes information identifying application 1 and instructions sent to the controller 102 to provide its network identification (ID). An example of a network ID includes the network address of the router 118. The network ID stored in the controller 102 is unique to the router 118 and distinguishes the router 118 from other routers.

[0023] The one or more forwarded packets containing information associated with session 106 and transmitted from server system 104 also include the destination address of router 118 and one or more additional destination addresses, such as the destination address of computing device 114 and the destination address of controller 102. Upon receiving the forwarded packets containing information related to session 106, modem 120 applies a network communication protocol to depacketize the forwarded packets and identify the destination address of router 118, the destination address of computing device 114, the destination address of controller 102, and information related to session 106. Note that the destination address of controller 102 is pre-stored in a memory device of server system 104. For example, when user A directly purchases controller 102, user A provides his or her user account information to the entity selling controller 102 to user A. The entity makes a selection via an input device of the client device to register controller 102 with user account 1. Upon receiving the signal generated based on the client device selection, one or more processors of the server system 104 described herein register the controller 102 with user account 1 to register the device ID of the controller 102 with user account 1. Explained, the one or more processors of the server system 104 store a link between user account 1 and the device ID of the controller 102 in a mapping database, as described below. As another example, when user A purchases the controller 102 online, user A provides his or her user account information to a website, such as a retailer's website or a manufacturer's website, accessed by user A for the purchase. The user account information is transmitted from the website to the server system 104 for the one or more processors to register the controller 102 with user account 1.Examples of user account information include an email address used by accessing User A's User Account 1, a username assigned to User Account 1, an address associated with User Account 1, or a telephone number associated with User Account 1, or any combination thereof. Modem 120 transmits the destination address of computing device 114, the destination address of controller 102, and information related to session 106 to router 118 via cable EC.

[0024] Upon receiving the destination address of computing device 114 and the information related to session 106, router 118 determines that the information related to session 106 should be sent to computing device 114 and sends the information related to session 106 to computing device 114 over connection channel CC2. The combination of connection channel CC2, router 118, cable EC, modem 120, connection C1, computer network 122, and connection C2 is referred to herein as communication channel 108 and is indicated using an "x".

[0025] Similarly, upon receiving the destination address of controller 102 and the information related to session 106, router 118 determines that the information related to session 106 should be sent to controller 102 and sends the information related to session 106 to controller 102 over connection channel CC1. The combination of connection channel CC1, router 118, cable EC, modem 120, connection C1, computer network 122, and connection channel C2 is referred to herein as communication channel 110, and is indicated using an "o".

[0026] Upon receiving the information related to session 106, computing device 114 displays the information on display screen 116, outputs audio data of the information related to session 106 through one or more speakers of computing device 114, or both displays the information and outputs audio data. Upon reading the information related to session 106 displayed on display screen 116 or listening to the audio data, user A provides his / her biometric ID, such as voice or fingerprint, or retinal information to biometric information scanner BIS. For example, user A says "Hey, controller" to transmit his / her voice to biometric information scanner BIS.

[0027] If the information associated with the session 106 includes instructions for the controller 102 to provide its device ID, the controller 102 accesses the device ID from a memory device of the controller 102. Also, if the information associated with the session 106 includes instructions for the controller 102 to provide a network ID, the controller 102 accesses the network ID from a memory device of the controller 102. A combination of two or more of a biometric ID or a device ID or a network ID is referred to herein as identification information.

[0028] The identification information and instructions for providing the identification information to the destination address of the server system 104 are transmitted from the controller 102 to the router 118 via communication channel CC1. As an example, the URL of a gaming website is stored in firmware in a read-only memory or another memory device of the controller 102. The URL includes the destination address of the server system 104. As another example, the computing device 114 presents the URL of the server system 104 to the controller 102 via a wireless connection 115. The computing device 114 or another device, such as a gaming console, is bypassed or not used in transmitting the identification information. For example, the identification information is not transmitted from the controller 102 to the computing device 114 or the gaming console. As another example, the identification information is not addressed by the controller 102 transmitted to the computing device 114 or the gaming console. The identification information is addressed by the controller 102 transmitted to the server system 104. The computing device 114 is coupled to the controller 102 via a Bluetooth® connection, or a wireless connection 115, such as a Bluetooth® connection or another short-range connection.

[0029] The router 118 receives the identification information and the command via communication channel CC1, identifies the destination address of the server system 104 in the command, and routes the identification information via cable EC to the modem 120. The modem 120 receives the identification information and the command, identifies the destination address of the server system 104 in the command, applies a network communication protocol to the identification information to generate a transport packet, and transmits the transport packet to the server system 104 via connection C1, computer network 122, and connection C2. The server system 104 receives the identification information and, upon verifying the identification information, pairs the controller 102 with the session 106.

[0030] In response to pairing the controller 102 with the session 106, the server system 104 generates and transmits a pairing notification. For example, the pairing notification may include an image frame to be displayed on the computing device 114, an audio frame to be output as sound by the computing device 114, or a combination thereof. The image frame may include information for displaying the pairing notification, such as the color, intensity, or texture of the pairing notification. As another example, the pairing notification may include instructions for a rendering program executed by a graphics processing unit (GPU) of the computing device 114, instructions for an audio processor of the computing device 114 to process audio data, or both. The server system 104 transmits the pairing notification to the computing device 114 via connection C2, computer network 122, connection C1, modem 120, cable EC, router 108, and connection channel CC2. The modem 120 receives one or more transfer packets from the server system 104 via connection C2, computer network 122, and connection C1, the transfer packets including pairing information and a destination address of the computing device 114. Modem 120 applies a network communication protocol to the forwarded packets to extract the destination address of computing device 114 and the pairing notification, and transmits the pairing notification and destination address to router 118 via cable EC. From the destination address of computing device 114, router 118 determines that the pairing notification should be sent to computing device 114, and transmits the pairing notification to computing device 114 via connection channel CC2. For example, router 118 identifies from the IP address of computing device 114 that the pairing notification should be sent to computing device 114.Upon receiving the pairing notification including an image frame or an audio frame, or a combination thereof, the pairing information is output as one or more images on the display screen 116 of the computing device 114, as sound via one or more speakers of the computing device 114, or as a combination thereof. Upon receiving the pairing information including instructions for a rendering program, the GPU of the computing device 114 executes the rendering program to display the pairing notification on the display screen 116. Additionally, if the pairing notification includes instructions for an audio processor, the audio processor and one or more speakers of the computing device 114 execute the instructions to process the audio data and output the audio data as sound. Similarly, if the pairing notification includes both instructions, both the GPU and the audio processor output the pairing notification as images and sound in synchronization with each other.

[0031] Once the session 106 is coupled, associated, or otherwise paired with the controller 102, user A can use the controller 102 to interact with the session 106 and establish direct communication between the server system 104 and the controller 102. For example, the controller 102 may be used by user A to play a game generated by executing application 1. Illustratively, the controller 102 may be used by user A to provide input data, such as controller movement data, controller button presses, etc., to the server system 104 via the communication channel 110 to change the state of a virtual scene displayed on the computing device 114.

[0032] Before session 106 is paired with controller 102, user 102 cannot use controller 102 to interact with session 106 of execution of application 1 to change the state of the virtual scene displayed on computing device 114. For example, user A selects or moves an input device on controller 102. The selection or movement of the input device generates input information that is not processed by server system 102 or permitted by server system 102 to change the state of the virtual scene. Input information transferred from controller 102 to server system 102 via communication channel 110 to change the state of the virtual scene generated by execution of application 1 is not processed by server system 104 to change the state of the virtual scene until controller 102 is paired with session 106.

[0033] In one embodiment, in a multiplayer game, such as a multiplayer race car game or soccer game, a logout by user A from user account 1 does not affect other instances of session 106 being executed by other users who also play the game. The other users' instances of session 106 continue until they log out of their corresponding user accounts.

[0034] In an embodiment, user A physically transfers the controller 102 described herein or another controller to another user. If the other user uses user account 1 assigned to user A, the instance of session 106 associated with user account 1 continues. However, if the other user logs out of user account 1 and then logs into their own user account 2, another instance of session 106 starts for user account 2.

[0035] In one embodiment, the session 106 ends when the computing device 114 on which the virtual scene is being displayed is disconnected from the server system 104, or when the computing device 114 is powered off, or when the communication signal between the computing device 114 and the server system 104 is weak, e.g., has a power amount below a threshold, or a communication device of the computer network 122 is not functioning or malfunctioning, or a communication device of the computing device 114 is not functioning or malfunctioning.

[0036] In an embodiment, both the router 118 and the modem 120 are integrated into a single hardware device.

[0037] In one embodiment, connection channel CC2 is a wired connection channel, such as a coaxial cable. Similarly, connection channel CC1 is a wired connection.

[0038] In an embodiment, functions described herein as being performed by server system 104 are performed by one or more processors of server system 104. Similarly, in an embodiment, functions described herein as being performed by modem 120 are performed by one or more processors of modem 120. Also, in an embodiment, functions described herein as being performed by router 118 are performed by one or more processors of router 118. In an embodiment, functions described herein as being performed by controller 102 are performed by one or more processors of controller 102.

[0039] In an embodiment, the pairing notification described herein is not sent from the server system 104 to the computing device 114 .

[0040] In one embodiment, the computing device 114 is not coupled to the controller 102 via a wireless connection 115 .

[0041] In one embodiment, the controller 102 sends a request to the computing device 114 to obtain a destination address of the server system 104 before sending the identification information to the server system 104. For example, the computing device 114 requests that a wireless connection 115 occur between the computing device 114 and the controller 102. Once the wireless connection 115 is established, the controller 102 sends a request to the server system 104 via the wireless connection 115 to obtain the destination address. Upon receiving the request, the computing device 114 presents the destination address of the server system 104 to the controller 102 via the wireless connection 115.

[0042] In an embodiment, session 106 does not end when another user uses controller 102. The other user receives controller 102 from user A after session 106 has been initiated and while it is active. The other user presents their login information to server system 104 via computing device 114 to log in to their user account and start another instance of session 106 for the other user's user account. Session 106 remains active and does not end when the other user switches from user account 1 to their own account.

[0043] 1B is a diagram of an embodiment of controller 102 to illustrate wireless communication between controller 102 and router 118. Controller 102 includes a motion sensor system 152, a wireless communication integrated circuit (IC) 156, an input device 150, a computer network ID chip 151, a wireless communication IC 157, a biometric identification sensor BIS, a processor 194, and a device ID chip 158. Processor 194 is coupled to motion sensor system 152, wireless communication integrated circuit (IC) 156, input device 150, computer network ID chip 151, wireless communication IC 157, biometric identification sensor BIS, and device ID chip 158. Input device 150 is coupled to motion sensor system 152. An example of motion sensor system 152 includes one or more gyroscopes, one or more accelerometers, and one or more magnetometers to facilitate determining the position and orientation of controller 102 and to facilitate determining changes in position and orientation. Examples of input device 150 include buttons, switches, touchscreens, styluses, joysticks, microphones, gun triggers, etc. Examples of wireless communication IC 157 include a Bluetooth® or Wi-Fi® device that enables communication between controller 102 and computing device 114 (FIG. 1A) over wireless connection 115 (FIG. 1A). Explained, a Bluetooth® device includes a processor that facilitates data transfer over short distances using short-wavelength ultra-high frequency (UHF) radio waves in the Industrial, Scientific, and Medical (ISM) band of 2.4 to 2.485 GHz between the controller and another device, such as a computing device, creating a personal area network (PAN).

[0044] The device ID chip 158 is a memory chip that stores a device ID such as the MAC address or unique identification number of the controller 102. For example, the device ID chip 158 is unique to the controller 102 and stores a series of alphanumeric characters that distinguish the controller 102 from other controllers. Examples of the wireless communication IC 156 include Wi-Fi (registered trademark) devices that enable Wi-Fi (registered trademark) communication between the controller 102 and the router 118 via the connection channel CC1 (FIG. 1A). To explain, the wireless communication IC 156 is a network interface card (NIC) that couples the controller to the computer network 122 via the connection channel CC1 and the router 118 as described herein.

[0045] The computer network ID chip 151 is another memory chip that stores a network ID such as the Internet protocol (IP) address of the router 118 in FIG. 1A. The router 118 is identified within the computer network 122 using the network ID stored in the computer network ID chip 151. For example, the computer network ID chip 151 is unique to the router 118 and stores a series of alphanumeric characters that distinguish the router 118 from other routers within the computer network 122.

[0046] The device ID of the controller 102 and the other corresponding device IDs of other controllers are pre-registered with the server system 104 before the session 106 (FIG. 1A) starts. For example, the device ID unique to the controller 102 is wired to the controller 102, and the server system 104 stores the device ID. The device ID of the controller 102 and the additional device IDs of other controllers are stored in the device ID database of the server system 104. The device ID database is stored in one or more memory devices of the server system 104.

[0047] Additionally, the network ID identifying the router 118 is pre-registered with the server system 104 before the controller 102 transmits identifying information to the server system 104 to pair the controller 102 with the session 106. For example, during establishment of the session 106, the server system 104 transmits an instruction to the computing device 114 over the communication channel 108 to present the network ID. Upon receiving the instruction, the computing device 114 accesses the network ID stored in a memory device of the computing device 114 and transmits the network ID to the server system 104 over the communication channel 108 (FIG. 1A). The server system 104 receives the network ID and stores the network ID in a network ID database of the server system 104. The network ID database is stored in one or more memory devices of the server system 104. Additionally, the server system 104 associates, such as by establishing a one-to-one communication or link, between the network ID and the user account 1 after the session 106 is established.

[0048] Furthermore, a biometric ID identifying user A is pre-registered with the server system 104 before the controller 102 transmits the identifying information to the server system 104 to pair the controller 102 with the session 106. For example, the computing device 114 has a biometric scanner. After or while the session 106 is established, the server system 104 transmits an instruction to the computing device 114 via the communication channel 108 to request a biometric ID from user A. When the computing device 114 receives the instruction, the computing device 114 displays a message or outputs a sound to request a biometric ID from user A. When the computing device 114 receives the biometric ID from user A via the biometric scanner of the computing device 114, the computing device 114 transmits the biometric ID to the server system 104 via the communication channel 108. The server system 104 receives the biometric ID and stores the biometric ID in a biometric ID database of the server system 104. The biometric ID database is stored in one or more memory devices of the server system 104. The server system 104 also associates, such as by establishing a one-to-one communication or link between the biometric ID and the user account 1 .

[0049] After the session 106 is established between the computing device 114 and the server system 104, user A presents his / her biometric ID to the biometric identification scanner BIS of the controller 102 to pair the controller 102 with the session 106. For example, after the session 106 begins, user A presses his / her finger against the fingerprint reader of the controller 102, or user A speaks into the microphone of the controller 102, or user 102 sees a notification to place his / her eye in the biometric eye scanner of the controller 102 for scanning, and user A provides his / her biometric information to the biometric information scanner BIS of the controller 102. Upon receiving the biometric ID, the biometric identification scanner BIS of the controller 102 generates a biometric identification signal and transmits the biometric identification signal to the processor 194. The processor 194 receives the biometric identification signal and generates instructions for transmitting the biometric identification signal to the server system 104. For example, the instructions include a destination address of the server system 104. The processor 194 transmits the command and the biometric identification signal to the wireless communication IC 156. The wireless communication IC 156 applies a wireless communication protocol to the biometric identification signal and the command and transmits the biometric identification signal and the command to the router 118 via connection channel CC1 (FIG. 1A). The router 118 determines from the destination address in the command that the biometric identification signal should be transmitted to the server system 104 and transmits the command and the biometric identification signal to the modem 120. The modem 120 applies a network communication protocol to the biometric identification signal to generate one or more transport packets and transmits the transport packets to the server system 104 via connection C1, computer network 122, and connection C2. The transport packet with the biometric identification signal is an example of a connection request.

[0050] Alternatively, or in addition to transmitting the biometric identification signal after the session 106 is established between the computing device 114 and the server system 104, the processor 194 accesses the device ID from the device ID chip 158 and generates instructions to transmit the device ID to the server system 104, thereby pairing the controller 102 with the session 102. The processor 194 transmits the instructions and the device ID to the wireless communication IC 156. The wireless communication IC 156 applies a wireless communication protocol to the device ID and the instructions and transmits the device ID and the instructions to the router 118 over connection channel CC1 (FIG. 1A). The router 118 determines from the instructions that the device ID should be transmitted to the server system 104 and transmits the instructions and the device ID to the modem 120. The modem 120 generates one or more transport packets with the device ID and transmits the transport packets to the server system 104 via connection C1, computer network 122, and connection C2. The transport packets with the device ID are another example of a connection request.

[0051] Alternatively, or in addition to transmitting a biometric identification signal after initiating the session 106, or alternatively, or in addition to transmitting a device ID after initiating the session 106, the processor 194 accesses a network ID from the computer network ID chip 151 and generates instructions to transmit the network ID to the server system 104 to pair the controller 102 with the session 106. The processor 194 transmits the instructions and the device ID to the wireless communication IC 156. The wireless communication IC 156 applies a wireless communication protocol to the network ID and the instructions and transmits the network ID and the instructions to the router 118 over connection channel CC1. The router 118 determines from the instructions that the network ID should be transmitted to the server system 104 and transmits the instructions and the network ID to the modem 120. The modem 120 generates one or more transport packets from the network ID and transmits the transport packets to the server system 104 over connection C1, computer network 122, and connection C2. The transport packets with the network ID are another example of a connection request.

[0052] Note that the connection request is not sent through the gaming console or computing device 114. The gaming console or computing device 114 cannot be used to forward the connection request from the controller 102 to the server system 104.

[0053] Once the device ID, biometric ID, network ID, or a combination of two or more thereof is used by the server system 104 to pair the controller 102 with the session 106, the controller 102 may be used by user A to generate and provide input to change the state of a virtual scene displayed when application 1 is running. For example, once the controller 102 is paired with the server system 104, user A can use the controller 102 to change the state of the virtual scene. User A uses the controller 102 by selecting or moving the input device 150 or by moving the controller 102 to provide a selection. The input device 150 generates an input signal upon receiving the selection or movement of the input device 150, and / or the motion sensor system 152 generates an input signal upon movement of the controller 102. The input signal generated by the motion sensor 152 includes data for determining the position or orientation of the controller 102 and / or data for determining the position and orientation of the input device 150. Examples of data for calculating the position or orientation of the controller 102 include the acceleration of the controller 102, the angular velocity of the controller 102, and the orientation of the controller 102. The position and orientation of the controller 102 and the position and orientation of the input device 150 are measured with reference to a reference coordinate system within the controller 102. Examples of data for calculating the position and orientation of the input device 150 include the acceleration of the input device 150, the angular velocity of the input device 150, and the orientation of the input device 150. The processor 194 receives one or more input signals.

[0054] The processor 194 receives one or more input signals and outputs input information. For example, the input information includes data for calculating the position or orientation, or a combination thereof, of the controller 102, or the position or orientation, or a combination thereof, of the input device 150 or a selection made by user A via the input device 150. Explained, the input information includes which of multiple buttons of the input device 150 is selected by user A, or which of multiple joysticks of the input device 150 is moved in which direction. In addition to the input information, the processor 194 also generates instructions for transmitting the input information to a destination address of the server system 104.

[0055] The processor 194 transmits the input information to the server system 104 via the communication channel 110. For example, the processor 194 transmits the input information and instructions to the wireless communication IC 156. The wireless communication IC 156 applies a wireless communication protocol to the input information and instructions and transmits the input information and instructions to the router 118 via connection channel CC1 (FIG. 1A). The router 118 determines from the instructions that the input information should be transmitted to the server system 104 and transmits the instructions and the input information to the modem 120. The modem 120 generates one or more transport packets from the input information and transmits the transport packets to the server system 104 via connection C1, computer network 122, and connection C2.

[0056] In one embodiment, wireless communication IC 156 and wireless communication IC 157 are integrated into a single integrated circuit chip. In an embodiment, wireless communication IC 156 is integrated into one integrated circuit chip and wireless communication IC 157 is integrated into another integrated circuit chip.

[0057] FIG. 1C is a diagram of an embodiment of a system 159 to illustrate the use of communication channels 155, including cellular connection channel 151 and cellular connection channel 153. Communication channel 155 is shown as "o" in FIG. 1C. Each of cellular connection channels 151 and 152 is a wireless connection channel, such as a radio frequency connection channel. Communication channel 155 includes cellular network 154 or a mobile network, which includes multiple cell towers, such as tower TW. Examples of cellular network 154 include a mobile broadband network, a fourth-generation (4G) mobile network, a 4G long-term evolution (4G LTE), and a fifth-generation (5G) mobile network. Cellular network 154 transfers data using a cellular communication protocol, such as a mobile broadband protocol, a 4G protocol, a 4G LTE protocol, or a 5G protocol. System 159 includes a controller 160 that is similar in structure and function to controller 102 (FIG. 1B), except that controller 160 couples to computer network 122 via cellular network 154 instead of via router 118. For example, a destination address of server system 104 is stored in firmware of controller 160 such that controller 160 can transmit identifying information to server system 104. As another example, controller 160 is coupled to computing device 114 via wireless connection 115 (FIG. 1A) to send a request for the destination address of server system 114. Upon receiving the request, computing device 114 transmits the destination address to controller 160 via wired connection 115.

[0058] Each cell tower includes a transceiver for forwarding data received from controller 160 to computer network 122 and for forwarding data received from computer network 122 to controller 160. The combination of cellular connection channel 151, the plurality of towers, cellular connection channel 153, computer network 122, and connection C2 is communication channel 155.

[0059] 1A , except that controller 160 is employed in system 159. Controller 160 is similar in structure and function to controller 102, except that controller 160 communicates with server system 104 via computer network 122 and one or more cell towers. Instead of communicating with computer network 122 via connection channel CC1, router 118, cable EC, modem 120, and connection C1, controller 160 communicates with computer network 122 via cellular connection channel 151, tower TW, and cellular connection channel 152.

[0060] When a session 106 (FIG. 1A) for application 1 is established or initiated, a controller 160, such as a subscriber identity module (SIM) card in the controller 160, transmits the identification information to the server system 104 to apply a cellular communication protocol to the identification information and instructions to generate one or more transfer units for transfer to the server system 104. The controller 160 transmits the transfer unit containing the identification information and instructions to the tower TW via the cellular connection channel 151. The computing device 114 or another device, such as a gaming console, is bypassed when transmitting the identification information. For example, the identification information is not transmitted from the controller 160 to the computing device 114 or the gaming console. As another example, the identification information is not addressed by the controller 160 to be transmitted to the computing device 114 or the gaming console. The identification information is addressed by the controller 160 to be transmitted to the server system 104.

[0061] A transceiver in tower TW receives the transfer unit having the identification information and instructions, and the transceiver transfers the transfer unit to computer network 122. A gateway, such as a modem, in computer network 122 demodulates the transfer unit by applying a cellular communication protocol to extract the instructions and identification information from the transfer unit, applies a network communication protocol to the identification information to generate a transfer packet, and transmits the transfer packet having the identification information to server system 104 via connection C2. Transfer of input information from controller 160 to server system 102 via communication channel 155 for changing the state of the virtual scene generated by execution of application 1 is disabled or not enabled by controller 160 until controller 160 is paired with session 106.

[0062] 1D is a diagram of an embodiment of controller 160 to illustrate the use of broadband communication IC 162 to transfer identification information from controller 160 to server system 104 (FIG. 1C). Controller 160 includes input device 150, motion sensor system 152, wireless communication IC 157, biometric identification scanner BIS, processor 194, broadband communication IC 162, and device ID chip 158. An example of broadband communication IC 162 is a SIM card. Processor 194 is coupled to input device 150, motion sensor system 152, wireless communication IC 157, biometric identification scanner BIS, broadband communication IC 162, and device ID chip 158.

[0063] 1A operate in the same manner as described above with respect to controller 102, except that the components operate as described above with respect to controller 160 instead of controller 102. For example, motion sensor system 152 measures data for calculating the position and orientation of controller 160 instead of controller 102.

[0064] The device ID of the controller 160 is pre-registered with the server system 104 before the session 106 (FIG. 1C) begins. For example, a device ID unique to the controller 160 is embedded in the controller 160, and the server system 104 stores the device ID. The device ID of the controller 160 is stored in a device ID database of the server system 104.

[0065] After session 106 is established between computing device 114 (FIG. 1C) and server system 104 (FIG. 1C), user A presents his / her biometric ID to biometric identification scanner BIS of controller 160 to pair controller 160 with session 106. For example, after session 106 begins, user A presses his / her finger against the fingerprint reader of controller 102, or user A speaks into the microphone of controller 160, or user 102 sees a notification on computing device 114 that user A has placed his / her eye in the biometric eye scanner of controller 160 for scanning, and user A presents his / her biometric information to biometric information scanner BIS of controller 160. Upon receiving the biometric ID, biometric identification scanner BIS of controller 160 generates a biometric identification signal and transmits the biometric identification signal to processor 194. Processor 194 receives the biometric identification signal and generates instructions for transmitting the biometric identification signal to server system 104. For example, the instructions include a destination address of server system 104. Processor 194 transmits the instructions and the biometric identification signal to broadband communication IC 162. Broadband communication IC 162 applies a cellular communication protocol to the biometric identification signal and the instructions to generate one or more transmission units and transmits the transmission units over cellular connection channel 151 (FIG. 1C) to tower TW, which forwards the transmission units to computer network 122. Computer network 122 applies a cellular communication protocol to the transmission units to extract the biometric ID and instructions from the transmission units, and further applies a network communication protocol to the biometric ID and instructions to generate one or more transmission packets and transmits the transmission packets to server system 104 over connection C2.

[0066] Alternatively, or in addition to transmitting a biometric identification signal after initiation of session 106, processor 194 of controller 160 accesses the device ID of controller 160 from device ID chip 158 and generates instructions to transmit the device ID to server system 104 to pair controller 160 with session 106. Processor 194 transmits the instructions and device ID to broadband communication IC 162. Broadband communication IC 162 applies a cellular communication protocol to the device ID and instructions, generates one or more transfer units, transmits the transfer units over cellular connection channel 151 (FIG. 1C) to tower TW, which forwards the transfer units to computer network 122. Computer network 122 applies a cellular communication protocol to the transfer units to extract the device ID and instructions from the transfer units, and further applies a network communication protocol to the device ID and instructions to generate one or more transfer packets and transmits the transfer packets to server system 104 over connection C2.

[0067] Similarly, when the controller 160 pairs with the session 106 and establishes direct communication between the controller 160 and the server system 104, the processor 194 transmits input information to the server system 104 via the communication channel 155. For example, the processor 194 transmits the input information and instructions to the wireless communication IC 156 to transmit the input information. The wireless communication IC 156 applies a wireless communication protocol to the input information and instructions to generate one or more transfer units and transmits the transfer units having the input information and instructions to the tower TW via the cellular connection channel 151. The tower TW determines from the instructions that the transfer unit having the instructions and the input information is to be transmitted to the server system 104 and transmits the transfer units to the computer network 122 via another cellular tower. The computer network 122 applies a cellular communication protocol to the transfer units to extract the input information and instructions from the transfer units, and further applies a network communication protocol to the input information and instructions to generate multiple transfer packets and transmits the transfer packets to the server system 104 via the connection C2.

[0068] In one embodiment, wireless communication IC 162 and wireless communication IC 157 are integrated into a single integrated circuit chip. In an embodiment, broadband communication IC 162 is integrated into one integrated circuit chip and wireless communication IC 157 is integrated into another integrated circuit chip.

[0069] 1B or 1D , the processor 194 of the controller determines whether to perform a handover from using the access channel 111 ( FIG. 1A ) to using the cellular network 154 ( FIG. 1C ), or vice versa. For example, the processor 194 determines that the cellular network 154 has a better quality of service (QoS) than the access channel 111 or that there is less latency in transferring data over the cellular network 154 compared to the access channel 111. Explained, the processor 194 determines that the cellular network 154 has a lower amount of data loss or a lower ping time compared to the amount of data loss for data transmitted over the access channel 111. If the processor 194 determines that the cellular network 154 has better QoS, the processor 194 determines to use the broadband communication IC 162 instead of the wireless communication IC 156 to transfer data, such as identification information, described herein from the controller to the server system 104. To illustrate, when user A uses the controller in his / her car, cellular network 154 allows for better QoS and lower latency compared to when user A uses the controller in his / her home. When user A uses the controller in his / her home, connection channel 111 allows for better QoS and lower latency compared to cellular network 154.

[0070] It should be noted that in an embodiment, processor 194 determines the amount of data loss by transmitting a predetermined number of transfer units for provision to server system 104 via communication channel 155 (FIG. 1C) and requesting one or more processors of server system 104, as described herein, for a count of the transfer units received by server system 104. Similarly, processor 194 determines the amount of lost transfer packets by transmitting a predetermined number of transfer units for provision to server system 104 via communication channel 110 (FIG. 1A) and requesting one or more processors of server system 104, as described herein, for a count of the transfer packets received by server system 104 via communication channel 110. Processor 194 compares the amount of lost transfer units to the amount of lost transfer packets to determine the amount of lesser data loss. Processor 194 also determines the latency by requesting one or more processors, as described herein, of server system 104 to transmit a predetermined number of transfer units for presentation to server system 104 over communication channel 155 and transmit a count of the transfer units back to processor 194 over communication channel 155. The latency is determined by determining the time for sending and receiving the transfer units over communication channel 155. Similarly, processor 194 determines the latency by requesting one or more processors, as described herein, of server system 104 to transmit a predetermined number of transfer packets for presentation to server system 104 over communication channel 110 and transmit a count of the transfer packets back to processor 194 over communication channel 110. The latency is determined by determining the time for sending and receiving the transfer packets over communication channel 110. Both latency times are compared by processor 194 to determine the shorter latency time.

[0071] In an embodiment, one controller includes both the wireless communication IC 156 and the broadband communication IC 162 of FIG. 1B in addition to the remaining components shown in FIG. 1B or 1D. The controller's processor 194 determines the power cost associated with using the cellular network 154 compared to using the connection channel 111. For example, the processor 194 queries the controller's battery after a predetermined period of time during which the wireless communication IC 156 (FIG. 1B) is used and the broadband communication IC 162 (FIG. 1D) is not used. The battery is coupled to the processor 194 and provides power to all components of the controller described herein. The processor 194 also queries the battery after a predetermined period of time equal to the length of time during which the broadband communication IC 162 is used and the wireless communication IC 156 (FIG. 1B) is not used. The processor 194 determines whether more battery power is used during the predetermined period of time during which the wireless communication IC 156 is used and the broadband communication IC 162 is not used, or during the predetermined period of time during which the broadband communication IC 162 is used and the wireless communication IC 156 is not used. If processor 194 determines that the battery is used more during a predetermined period when wireless communication IC 156 is used and broadband communication IC 162 is not used, processor 194 determines that the power cost associated with using connection channel 111 is higher than that associated with using cellular network 154. On the other hand, if processor 194 determines that the battery is used more during a predetermined period when broadband communication IC 162 is used and wireless communication IC 156 is not used, processor 194 determines that the power cost associated with using cellular network 154 is higher than that associated with using connection channel 111. Processor 194 uses either wireless communication IC 156 or broadband communication IC 162, which has the lower power cost.

[0072] 1B or 1D , the processor 194 presents a message to user A, either via the controller's display screen or in the form of audio data output as a voice from one or more speakers on the controller, indicating whether the cellular network 154 or the connection channel 111 should be used. User A may decide to use either the cellular network 154 or the connection channel 111 to obtain better QoS, which may positively impact their play time. Upon receiving a selection from user A via the input device 150 indicating that the cellular network 154 will be used, the processor 194 determines to use the broadband communication IC 162 to transfer data described herein to the server system 104. On the other hand, upon receiving a selection from user A via the input device 150 indicating that the connection channel 111 will be used, the processor 194 determines to use the wireless communication IC 156 to transfer data described herein to the server system 104.

[0073] In this embodiment, the controller includes both the wireless communication IC 156 and the broadband communication IC 162 of FIG. 1B in addition to the remaining components shown in FIG. 1B or 1D. In this embodiment, there is no handover as described above. Rather, the processor 194 determines to connect to the server system 104 using both the wireless communication IC 156 and the broadband communication IC 162. The processor 194 transmits the same data, such as identification information, as described herein to the server system 104 via the wireless communication IC 156 and the broadband communication IC 162. The server system 104 processes the same data initially received from either the wireless communication IC 156 or the broadband communication IC 162. For example, the communication interface of the server system 104 receives one or more forwarded packets having a packet ID or timestamp from the wireless communication IC 156, receives one or more forwarded packets having the same packet ID or timestamp from the broadband communication IC 162, and processes the initially received one or more forwarded packets.

[0074] Figure 1E is a diagram of an embodiment of server system 104 to illustrate pairing of session 106 (Figure 1A) with controller 164. Controller 102 of Figure 1A and controller 160 of Figure 1C are examples of controller 164. Server system 104 includes authentication processor 176, session link processor 178, a device ID database, a network ID database, a user account database, a mapping database, a session ID database, a biometric ID database, a pairing notification processor 180, communication interface 170, session ID processor 183, and execution server ES1.

[0075] Communications interface 170 is coupled to session ID processor 183, pairing notification processor 180, authentication processor 176, and execution server ES1. Pairing notification processor 180 is coupled to authentication processor 176 and session link processor 178. Authentication processor 176 is coupled to session link processor 178 and to a mapping database stored in one or more memory devices of server system 104. The mapping database is coupled to a device ID database, a user account database, a biometric ID database, and a network ID database. Session link processor 178 is coupled to the session ID database. Session ID processor 183 is coupled to execution server ES1, the session ID database, and session link processor 178.

[0076] Server system 104 is coupled to controller 164 via communication network 177. Examples of communication network 177 include communication channel 110 of Figure 1A and communication channel 155 of Figure 1C. Additional examples of communication network 177 include communication channel 108 of Figures 1A and 1C. Illustratively, communication network 177 includes communication channels 110 and 108 or communication channels 155 and 108.

[0077] The server system 104 is coupled to a display device 166, such as an LED, LCD, or plasma display device. The display device 166 may be an input device, such as a touchscreen display, that receives selections from an administrative user. An example of the display device 166 is the computing device 114 of FIGS. 1A and 1C.

[0078] The execution server ES1 has an input channel 184, such as a computer port, coupled to the communication interface 170 via the session ID processor 183 to receive data from the communication interface 170. The execution server ES1 further has an execution engine 186 that executes an application 1, such as a video conferencing application or a game, examples of which include video games, virtual reality games, and augmented reality games. Examples of engines as described herein include a processor and computer software modules. The application is executed to change the state of a virtual scene displayed on the computing device 114 or another display screen described herein. Examples of the state of the virtual scene include the position of a virtual object in the virtual scene displayed to the client, the orientation of the virtual object, the color of the object, the hue of the virtual object, the texture of the virtual object, the position of a background in the virtual scene, the orientation of the background, the color of the background, the hue of the background, the texture of the background, or a combination of two or more thereof. One or more video frames, one or more audio frames, or a combination thereof, are generated by the execution engine ES1, where the one or more frames include a state or a change to a state. The generated one or more frames may be referred to herein as a video output. Examples of video frames include I-frames, P-frames, and B-frames.

[0079] Communications interface 170 may be a network interface, such as a network interface controller or network interface card, that applies a network communications protocol to forward packets received from controller 164 over communications network 177, extracts data from the forwarded packets, and applies the network communications protocol to the data to generate one or more forwarded packets that are forwarded over communications network 177 to controller 164. An example of communications interface 170 includes a processor coupled to the network interface.

[0080] Examples of encoder 188 include a processor or computer software module that performs compression of frames. For example, encoder 188 may receive frames from execution engine 186 and perform intra-frame or inter-frame compression by applying a compression protocol such as H.264 or another frame compression standard.

[0081] In operation 1, display device 166 transmits one or more forwarded packets having login information to communication interface 170 over communication network 177. Communication interface 170 applies a network communication protocol to the forwarded packets to extract the login information from the forwarded packets. A processor in communication interface 170 identifies the information extracted from the forwarded packets as login information and, upon identification, provides the login information to session ID processor 183. For example, the processor in communication interface 170 determines that a structure, such as a series of alphanumeric characters or a username structure or a password structure, of the information received from the forwarded packets matches the structure of login information and determines that the information is login information.

[0082] The session ID processor 183 determines whether the login information is authentic, and if so, establishes a session 106 for the execution of application 1. If the session ID processor 183 determines that the login information is authentic, it grants access to user account 1. An example of session 1 is a temporary, interactive exchange or communication of data or information between the display device 166 and the server system 102. The session 106 ends when user A logs out of user account 1 or when the connection between the server system 102 and the display device 166 is lost. The loss of the connection between the server system 102 and the display device 166 may be due to a malfunction of the communication channel 110 in FIG. 1A or the communication channel 155 in FIG. 1C. On the other hand, the session ID processor 183 does not establish the session 106 when it determines that the login information is not authentic.

[0083] The session ID processor 183 assigns a session ID 1, such as an alphanumeric character, to the session 106, links the session ID to user account 1 assigned to user A, and stores session ID 1 in a session ID database. Session ID 1 is linked to user account 1 by establishing a one-to-one relationship between session ID 1 and user account 1 for the session, and the link is stored in a mapping database by the session ID processor 183.

[0084] The input channel 184 receives a determination from the session ID processor 183 that the login information is authentic and provides the determination to the game execution engine 186. Upon receiving a determination that the login information is authentic, the game execution engine 186 generates one or more frames of information related to the session 106, such as an audio frame, a video frame, or a combination thereof. Specifically, the game execution engine 186 generates a frame including the name of application 1, the title of application 1, or a virtual image of application 1. The frame is encoded by the encoder 188 and sent to the communication interface 170 in operation 2, which outputs the encoded frame. The communication interface 170 applies a network communication protocol to the encoded frame to generate a stream having one or more transport packets and transmits the stream to the display device 166 via the communication network 177. Examples of streams sent from the communication interface 170 include a video stream, an audio stream, or a combination thereof.

[0085] After receiving the information related to the session, in operation 3, the controller 164 transmits the identification information, such as biometric ID1 or device ID1 or network ID1 or a combination of two or more thereof, to the communication interface 170 via the communication network 177 to pair the controller 164 with the session 106. The communication interface 170 applies a network communication protocol to the one or more forwarded packets containing the identification information to extract the identification information from the forwarded packets and transmits the identification information to the authentication processor 176.

[0086] The processor of communication interface 170 identifies the information extracted from the forwarded packet as identification information and, upon identification, provides the identification information to authentication processor 176. For example, the processor of communication interface 170 determines that the structure of the information received from the forwarded packet matches that of the identification information, such as a series of phonemes or a fingerprint or an IP address or a device ID, and determines that the information is identification information.

[0087] Upon receiving the biometric ID1 from the communication interface 170, the authentication processor 176 searches for the pre-registered biometric ID1 from the biometric ID database and processes the received biometric ID1 to determine whether the received biometric ID1 is authentic. For example, the authentication processor 176 determines whether there is a match between the received biometric ID1 and the pre-registered biometric ID1. The pre-registered biometric ID1 is pre-registered to and linked to user account 1 assigned to user A by being stored in the biometric ID database. If a match is determined to occur, the authentication processor 176 determines that the received biometric ID1 is authentic in order to verify the received biometric ID1. On the other hand, if a match is determined not to occur, the authentication processor 176 determines that the received biometric ID1 is not authentic. Similarly, upon receiving the device ID1 from the communication interface 170, the authentication processor 176 searches for the pre-registered device ID1 from the device ID database and processes the received device ID1 to determine whether the received device ID1 is authentic. Similarly, upon receiving network ID1 from communication interface 170, authentication processor 176 searches for pre-registered network ID1 from the network ID database and processes the received network ID1 to determine whether the received network ID1 is authentic.

[0088] It should be noted that the mapping database pre-stores an association, such as a one-to-one correspondence, link, or mapping, between the pre-registered biometric ID 1 and the user account 1. For example, before the biometric ID 1 is received from the controller 164 to pair the controller 164 with the session 106, the session ID processor 183 receives the biometric ID 1 from the computing device 114 via the communication channel 108 of FIG. 1A or 1C and stores the pre-registered biometric ID 1 in the biometric ID database. Furthermore, the session ID processor 183 establishes an association between the pre-registered biometric ID 1 and the user account 1, stores the association in the mapping database, and stores the pre-registered biometric ID 1 in the biometric ID database. In a similar manner, before network ID1 and device ID1 are received from controller 164 to pair controller 164 with session 106, session ID processor 183 stores an association between pre-registered device ID1 and / or user account 1 in a mapping database, and an association between pre-registered network ID1 and user account 1 in the mapping database, stores pre-registered network ID1 in the network ID database, and stores pre-registered device ID in the device ID1 database. User account 1 and other user accounts 2 through N are stored in the user account database.

[0089] The authentication processor 176 provides a determination that the received biometric ID1 is authentic to the session link processor 178. The session link processor 178 establishes an association between the session ID1 and the received biometric ID1 and stores the association in a mapping database to pair the session 116 with the controller 164. For example, the session link processor 178 generates a one-to-one link or mapping or correspondence between the session ID1 of the session 106 and the received biometric ID1. Similarly, the session link processor 178 establishes an association between the session ID1 and the received device ID1 and stores the association in the mapping database. For example, the session link processor 178 generates a one-to-one link or mapping or correspondence between the session ID1 of the session 106 and the received device ID1. Similarly, the session link processor 178 also establishes an association between the session ID1 and the received network ID1 and stores the association in the mapping database. For example, the session link processor 178 generates a one-to-one link or mapping or correspondence between the session ID1 of the session 106 and the received network ID1.

[0090] Upon establishing an association between the identification information received from controller 164 and session 106, in operation 4, session link processor 178 provides an association determination to pairing notification processor 180. In response to receiving the association determination, pairing notification processor 180 generates a pairing notification and transmits the pairing notification to communication interface 170. Pairing notification processor 180 also generates instructions for transmitting the pairing notification to display device 166, or to controller 164, or to both display device 166 and controller 164. Communication interface 170 receives instructions from pairing notification processor 180, applies a network communication protocol to the pairing notification and a destination address of display device 166, or to the destination address of controller 164, or to both destination addresses, generates one or more forwarded packets, and transmits the forwarded packets via communication network 177 to display device 166 or controller 164, or to both controller 164 and display device 166.

[0091] If authentication processor 176 determines that received biometric ID1, received device ID1, or received network ID1, or a combination of two or more thereof, is not authentic, authentication processor 176 notifies pairing notification processor 180 of the determination. Upon receiving the determination, pairing notification processor 180 generates an unpairing notification and instructions to send the unpairing notification to display device 166 or controller 164, or both controller 164 and display device 166. Upon receiving the instructions and the unpairing notification, communication interface 170 applies a network communication protocol to a destination address of display device 166, or a destination address of controller 164, or both destination addresses to generate one or more forwarded packets, and transmits the forwarded packets via communication network 177 to display device 166 or controller 164, or both controller 164 and display device 166.

[0092] The communication interface 170 transmits a forwarded packet including a pairing-related notification, such as an unpairing notification or a pairing notification, to the display device 166 via the connection C2 (FIG. 1A), the computer network 122 (FIG. 1A), the connection C1 (FIG. 1A), the modem 120 (FIG. 1A), the cable EC (FIG. 1A), the router 108 (FIG. 1A), and the connection channel CC2 (FIG. 1A). The modem 120 of FIGS. 1A and 1C receives the forwarded packet including the pairing-related notification and the destination address of the display device 166. The modem 120 applies a network communication protocol to the forwarded packet to extract the destination address of the display device 166 and the pairing-related notification, and transmits the pairing-related notification and the destination address to the router 118 of FIGS. 1A and 1C. The router 118 determines from the destination address of the display device 166 that the pairing-related notification should be transmitted to the display device 166 and transmits the pairing-related notification to the display device 166. For example, router 118 identifies, from the IP address of display device 166, that the pairing-related notification is to be sent to display device 166. Upon receiving the pairing-related notification, the GPU of display device 166 displays the pairing-related notification on the display screen of display device 166. If the pairing-related notification is audio data, the audio processor of display device 166 outputs the pairing-related notification as sound. Similarly, if the pairing-related notification includes both image and sound data, both the GPU and audio processor of display device 166 output the pairing-related notification as image and sound in synchronization with each other.

[0093] In one embodiment, the communication interface 170 transmits a forwarded packet including the pairing-related notification to the controller 102 of FIG. 1A via connection C2, computer network 122, connection C1, modem 120, cable EC, router 108, and connection channel CC1. The modem 120 of FIGS. 1A and 1C receives the forwarded packet including the pairing-related notification and a destination address of the display device 166. The modem 120 applies a network communication protocol to the forwarded packet to extract the destination address of the controller 102 and the pairing-related notification, and transmits the pairing-related notification and the destination address to the router 118 of FIGS. 1A and 1C. The router 118 determines from the destination address of the controller 102 that the pairing-related notification should be transmitted to the controller 102 via connection channel CC1 and transmits the pairing-related notification to the controller 102. For example, the router 118 identifies from the IP address of the controller 102 that the pairing-related notification should be transmitted to the controller 102. Upon receiving the pairing-related notification, the GPU of the controller 102 displays the pairing-related notification on a display screen of a display device of the controller 102. If the pairing-related notification is audio data, an audio processor of the display device of the controller 102 outputs the pairing-related notification as sound. Similarly, if the pairing-related notification includes both image and sound data, both the GPU and the audio processor of the controller 102 output the pairing-related notification as image and sound in synchronization with each other.

[0094] In an embodiment, communication interface 170 transmits a forwarding packet including a pairing-related notification to controller 160 of FIG. 1C via connection C2, computer network 122, connection C1, and cellular network 154 (FIG. 1C). Computer network 122 of FIG. 1C receives the forwarding packet including the pairing-related notification and a destination address of controller 160. Computer network 122 applies a network communication protocol to the forwarding packet to extract the destination address of controller 160 and the pairing-related notification, and further applies a cellular communication protocol to the destination address and the pairing-related notification to generate one or more forwarding units and transmits the forwarding units to tower TW of cellular network 154 via cellular connection channel 153. Tower TW forwards the forwarding units to controller 160 of FIG. 1C via cellular connection channel 151 according to the destination address of controller 160. Controller 160 applies the cellular communication protocol to obtain the pairing-related notification. Upon receiving the pairing-related notification, the GPU of the controller 160 displays the pairing-related notification on a display screen of the display device of the controller 160. If the pairing-related notification is audio data, the audio processor of the display device of the controller 160 outputs the pairing-related notification as sound. Similarly, if the pairing-related notification includes both image and sound data, both the GPU and the audio processor of the controller 160 output the pairing-related notification as image and sound in synchronization with each other.

[0095] In operation 5, when controller 164 is paired, e.g., linked, with session 106, user A can use controller 164 to generate one or more transfer packets having input information and session ID 1 to change the state of the virtual scene. For example, user A's selection of a button on controller 164 can fire a virtual weapon or cause a virtual user to jump in a video game. As another example, user A's selection of a joystick on controller 164 can change several areas on the display screen of display device 166 where live streams showing images of different users are displayed. Explaining, instead of two live streams showing two users from two different locations on the display screen of display device 166, only the live stream of one of the two users is received by display device 166 for display on the display screen.

[0096] Communications interface 170 receives a forwarded packet containing the input information and session ID 1 from controller 164 via communications network 177 and applies a network communications protocol to the forwarded packet to extract the input information and session ID 1. The processor of communications interface 170 determines that the information in the forwarded packet has a structure that matches the structure of the input information, such as a series of button presses or a series of joystick movements, and, upon such determination, sends the input information to session ID processor 183. Session ID processor 183 provides the input information to execution engine 186 via input channel 184. Session ID processor 183 may authenticate session ID 1 before sending the input information to execution engine 186.

[0097] The execution engine 186 interacts with the controller 164 to determine changes to the state of the virtual scene according to the input information and generates multiple frames, such as audio frames, video frames, or a combination thereof, having the state of the virtual scene. For example, the execution engine 186 determines changes to the position, orientation, intensity, color, shape, or a combination thereof of the virtual object based on the input information to generate the frames. Examples of frames include I-frames, P-frames, and B-frames. The execution engine 186 provides the frames to the encoder 188, which compresses the frames and outputs encoded frames. For example, the frames are compressed using the H.264 standard. Furthermore, the execution engine 186 generates instructions to transmit the frames to the display device 166.

[0098] Encoder 180 provides the encoded frames to communication interface 170. Communication interface 170 applies a network communication protocol to the encoded frames and instructions to generate one or more transport packets and transmits the transport packets to modem 120 of Figures 1A and 1C via connection C2 of Figures 1A and 1C, computer network 122, and connection C1 of Figures 1A and 1C. Modem 120 applies a network communication protocol to the transport packets to extract the instructions and encoded frames from the transport packets and transmits the instructions and encoded frames to router 118 of Figures 1A and 1C via Ethernet connection EC. Router 118 determines from the destination address in the instructions that the encoded frames should be transmitted to display device 166 and transmits the encoded frames to display device 166 via connection channel CC2 of Figures 1A and 1C. The display device 166 decodes the encoded frames and outputs the frames, and further displays an image of the virtual scene having the virtual object and the virtual background on the display screen of the display device 166, or outputs audio data through a speaker of the display device 166, or a combination thereof. For example, the display device 166 displays a virtual object that has changed its position, orientation, color, intensity, or texture. As another example, the display device 166 outputs a sound generated by the virtual object. When the display device 166 displays an image or outputs a sound related to the image according to the encoded frame received from the server system 104 in response to the input information received from the controller 164, the server system 104 or the application 1 and the controller 164 interact with each other.

[0099] It should be noted that in an embodiment, each processor of the server system 104 is replaced by a software module, such as a portion of a computer software program, which is executed by one or more processors of the server system 104. Each software module may include one or more routines. In an embodiment, the functionality of some processors of the server system 104 is implemented as modules, while the remaining processors of the server system 104 are hardware components, such as integrated circuits.

[0100] In an embodiment, the functions described herein as being performed by authentication processor 176, session ID processor 183, processor of communication interface 170, pairing notification processor 180, encoder 188, and execution engine 186 are instead performed by any other number of processors or servers of server system 104.

[0101] In one embodiment, user A decides whether to pair controller 164 with session 106. Instead of sending a determination that biometric ID1, receiving device ID1, or receiving network ID1, or a combination of two or more thereof, is authentic to session link processor 178, authentication processor 176 sends a determination regarding authentication to pairing notification server 180. Upon receiving the determination from authentication processor 176, pairing notification processor 180 generates a pairing request notification and instructions for sending the pairing request notification to display device 166. The pairing request notification includes a request to pair controller 164 with session 106. Pairing notification processor 180 sends the pairing request notification and instructions to communication interface 170.

[0102] In this embodiment, upon receiving the command and pairing request notification, communication interface 170 applies a network communication protocol to the destination address of display device 166 and the pairing request notification to generate one or more forwarded packets and transmits the forwarded packets to display device 166 via communication channel 108 of FIGS. 1A and 1C.

[0103] Continuing with the embodiment, communication interface 170 transmits a forwarded packet to display device 166 via connection C2, computer network 122, connection C1, modem 120, cable EC, router 108, and connection channel CC2. Modem 120 of FIGS. 1A and 1C receives the forwarded packet including a pairing request notification and a destination address of display device 166. Modem 120 applies a network communication protocol to the forwarded packet to extract the destination address of display device 166 and the pairing request notification, and transmits the pairing request notification and the destination address to router 118 of FIGS. 1A and 1C. Router 118 determines from the destination address of display device 166 that a pairing request notification should be sent to display device 166 and transmits the pairing request notification to display device 166. For example, router 118 identifies from the IP address of display device 166 that the pairing request notification should be sent to display device 166. Upon receiving the pairing request notification, the GPU of the display device 166 displays the pairing request notification on the display screen of the display device 166. If the pairing request notification is audio data, the audio processor of the display device 166 outputs the pairing request notification as sound. Similarly, if the pairing request notification includes both image and sound data, both the GPU and the audio processor of the display device 166 output the pairing request notification as image and sound in synchronization with each other.

[0104] Further, in an embodiment, upon hearing, viewing, or both viewing and hearing the pairing request notification, user A makes a selection on controller 164 to indicate whether or not to pair controller 164 with session 106. For example, user A selects a first button on controller 164 to provide an affirmative response indicating that user A wishes to pair with session 106. User A selects a second button on controller 164 to provide a negative response indicating that user A does not wish to pair with session 106.

[0105] In an embodiment, controller 164 generates input data indicating a selection made by user A regarding pairing. Wireless communication IC 157 (FIG. 1B or 1D) of controller 164 transmits the input data to display device 166 via wireless connection 115 between controller 164 and display device 166. A GPU of display device 166 receives the input data and displays a positive or negative response on display device 166.

[0106] In an embodiment, controller 164 also transmits input data to server system 104 via communication channel 110 of Figure 1A or communication channel 155 of Figure 1C. For example, the input data is transmitted from controller 164 to router 108 of Figure 1A via connection channel CC1 of Figure 1A, and router 108 routes the input data to modem 120 of Figure 1A via cable EC. Modem 120 generates one or more transport packets by applying a network communication protocol to the input data and transmits the transport packets to server system 104 via connection C1 of Figure 1A, computer network 122 of Figure 1A, and connection C2 of Figure 1A.

[0107] Further, in an embodiment, communication interface 170 receives a forwarded packet having input data and applies a network communication protocol to extract the input data from the forwarded packet. The processor of communication interface 170 identifies information in the forwarded packet as having the structure of the input data and, upon identification, transmits the input data to session link processor 178. Session link processor 178 determines whether the response in the input signal is positive or negative. In response to receiving input data including a positive response, session link processor 178 pairs controller 164 with session 106. On the other hand, in response to receiving input data including a negative response, session link processor 178 does not pair controller 164 with session 106. In this manner, user A determines whether to pair controller 164 with session 106.

[0108] In one embodiment, controller 164 includes components of both controllers 102 and 160. For example, processor 194 is coupled to both broadband communication IC 162 of FIG. 1D and wireless communication IC 156 of FIG. 1B. In this example, processor 194 of controller 164 transmits input data to server system 104 via wireless communication IC 156 and communication channel 110 of FIG. 1A and via broadband communication IC 162 and communication channel 155 of FIG. 1C. In an embodiment, a pairing request notification is transmitted from server system 104 to controller 164 via communication network 177 for display on a display screen of controller 164 or for output via one or more speakers of controller 164.

[0109] Note that in one embodiment, the identification information is different from the login information. For example, the login information is a series of alphanumeric characters, such as a user A or a password, that is provided or entered by the user A into the server system 104 via an input device of the controller. The identification information is not a series of alphanumeric characters that is provided or entered by the user A into the server system 104 via an input device of the controller.

[0110] In an embodiment, user A cannot use controller 102 to change the state of the virtual scene until controller 102 is paired with server system 104. For example, user A uses controller 102 to provide a selection to generate further input information for changing the state of the virtual scene by selecting or moving input device 150, and one or more transfer packets having the input information are generated and transmitted over communication network 177 to communication interface 170. Communication interface 170 applies a network communication protocol to extract the input information for changing the state of the virtual scene and provides the input information to session ID processor 183. The processor of communication interface 170 determines that the information in the transfer packets has a structure that matches the structure of input information for changing the virtual scene, such as a series of button presses or a series of joystick movements, and, upon such determination, transmits the input information to session ID processor 183.

[0111] In this embodiment, upon receiving the input information, the session ID processor 183 sends a request to the session link processor 178 to determine whether the controller 164 receiving the input information is paired with the session 106. Upon receiving a determination from the session link processor 178 that the controller 164 is not paired with the session 106, the session ID processor 183 does not provide the input information to the execution engine 186 via the input channel 184, and the execution engine 186 cannot process the input information to change the state of the virtual scene.

[0112] 1B and broadband communication IC 162 of FIG. 1B in addition to the remaining components shown in FIG. 1B or 1D. In this embodiment, communication interface 170 determines whether to perform a handover from using access channel 111 (FIG. 1B) to using cellular network 154 (FIG. 1C), or vice versa. For example, the processor of communication interface 170 determines that cellular network 154 has better QoS than access channel 111 or has lower latency in transferring data over cellular network 154 compared to access channel 111. Explained, the processor of communication interface 170 determines that cellular network 154 has a lower amount of packet loss or a shorter ping time compared to access channel 111. If the cellular network 154 is determined to have better QoS, the processor 194 of the communication interface 170 decides to use the broadband communication IC 162 instead of the wireless communication IC 156 to transfer data described herein, such as pairing information or pairing request notifications, from the server system 104 to the controller.

[0113] In this embodiment, the controller includes both the wireless communication IC 156 and the broadband communication IC 162 of FIG. 1B in addition to the remaining components shown in FIG. 1B or 1D. In this embodiment, there is no handover as described above. Rather, the processor of the communication interface 170 determines to connect to the controller using both the wireless communication IC 156 and the broadband communication IC 162. The processor of the communication interface 170 transmits the same data described herein, such as pairing information or pairing request notifications, to the controller via the wireless communication IC 156 and the broadband communication IC 162. The processor 194 processes data initially received from the server system 104 by either the wireless communication IC 156 or the broadband communication IC 162. For example, the wireless communication IC 156 receives one or more forwarded packets with a packet ID or timestamp from the server system 104 and the broadband communication IC 162, and receives one or more forwarded packets with the same packet ID or timestamp from the server system 104. When the wireless communication IC 156 first receives one or more transfer packets compared to receiving a transfer unit by the broadband communication IC 162, the wireless communication IC 156 extracts information from the one or more transfer packets and sends the information to the processor 194 for processing.

[0114] In an embodiment, computing device 114 (FIG. 1A) may be activated from sleep mode when a selection of one or more input devices 150 (FIG. 1B) is made by user A at controller 164. Server system 104 wakes computing device 114 when user A toggles between selecting the PS® button on controller 164 or selecting the fingerprint pad on controller 164. In sleep mode, computing device 114 operates in a low-power state in which it can receive messages from server system 104.

[0115] FIG. 1F-1 is a diagram of an embodiment of a pairing request notification 190 displayed on the display screen 116 of the computing device 114. The pairing request notification 190 is an example of a pairing request notification generated by the pairing notification processor 180 of FIG. 1E. The pairing request notification 190 includes information indicating that user A's biometric information has been authenticated. Additionally, the pairing request notification 190 includes a question regarding whether user A wishes to pair with the controller 164 of FIG. 1E. User A selects one or more buttons and one or more joysticks on the controller 164 to indicate whether user A wishes to pair the session 106 with the controller 164. The controller 164 generates input data indicating the selection made by user A.

[0116] FIG. 1F-2 is a diagram of an embodiment of controller 192 to illustrate the display of pairing request notification 190 on display screen 195 of controller 192. Examples of display screens described herein include a liquid crystal display screen, a light emitting diode screen, or a plasma display screen. Controller 192 is an example of controller 102 of FIG. 1A, controller 160 of FIG. 1C, and controller 164 of FIG. 1E. Controller 192 has display screen 195 on which pairing request notification 190 is displayed. Controller 192 includes a GPU coupled to processor 194 of FIG. 1B and 1D, which is coupled to display screen 195.

[0117] 1E generates a pairing request notification 190 and instructions for transmitting the pairing request notification 190 to a controller 192. The pairing notification processor 180 transmits the pairing request notification and instructions to the interface 170 of FIG.

[0118] Upon receiving the instructions and pairing request notification 190, communication interface 170 applies a network communication protocol to the destination address of controller 192 and pairing request notification 190 to generate one or more forwarded packets and transmits the forwarded packets to controller 192 over communication channel 110 of FIG. 1A. Referring to FIG. 1A, communication interface 170 transmits the forwarded packets to controller 192 over connection C2, computer network 122, connection C1, modem 120, cable EC, router 108, and connection channel CC1. Modem 120 receives the forwarded packets including the pairing request notification 190 and the destination address of controller 192. Modem 120 applies a network communication protocol to the forwarded packets to extract the destination address of controller 192 and pairing request notification, and transmits the pairing request notification and destination address to router 118. The router 118 determines from the destination address of the controller 192 that a pairing request notification 190 should be sent to the controller 192 and sends the pairing request notification to the controller 192 via the connection channel CC1. For example, the router 118 identifies from the IP address of the controller 192 that the pairing request notification 190 should be sent to the controller 192. Upon receiving the pairing request notification 190, the GPU of the controller 192 displays the pairing request notification 190 on the display screen 195. If the pairing request notification is audio data, the audio processor and one or more speakers of the controller 192 output the pairing request notification as sound. Similarly, if the pairing request notification includes both image and sound data, both the GPU and the audio processor of the controller 192 output the pairing request notification as image and sound in synchronization with each other.

[0119] 1C , upon receiving pairing request notification 190 and an instruction to transmit pairing request notification 190 to controller 160 from pairing notification processor 180 of server system 104, communication interface 170 applies a network communication protocol to the destination address of controller 160 and pairing request notification 190 to generate one or more forwarding packets and transmits the forwarding packets to controller 160 over communication channel 155 of FIG. 1C . Referring to FIG. 1C , communication interface 170 transmits the forwarding packets to controller 160 over connection C2, computer network 122, cellular connection channel 153, tower TW, and cellular connection channel 151. The gateway of computer network 122 applies the network communication protocol to extract the destination address and pairing request notification 190 from the forwarding packets, applies a cellular communication protocol to the destination address and pairing request notification 190 to generate one or more forwarding units, and transmits the forwarding units to tower TW over cellular communication channel 155. The transceiver of cell tower TW transfers the transfer unit to controller 160 via cellular communication channel 151 according to the destination address of controller 160. Broadband communication IC 162 (FIG. 1D) of controller 160 applies a cellular communication protocol to the transfer unit to extract pairing request notification 190 and provides pairing request notification 190 to the GPU of controller 160 for displaying pairing request notification 190 on a display screen of controller 160. If pairing request notification 190 should be output as sound, broadband communication IC 162 provides pairing request notification 190 to an audio processor of controller 160 to output pairing request notification 190 as sound. If the pairing request notification 190 includes both image data and audio data, the broadband communication IC 162 provides the image data of the pairing request notification 190 to the GPU of the controller 160, and also provides the audio data of the pairing request notification 190 to the audio data of the controller 160, and synchronizes the output of the audio data as sound with the display of the video data.

[0120] 1G is a diagram of an embodiment of controller 192 illustrating haptic feedback device 196, audio device 101, and display device 107 of controller 192. Controller 192 has the components of controller 102 of FIG. 1B and has additional components such as haptic feedback device 196, audio device 101, and display device 107.

[0121] The display device 107, the haptic feedback device 196, and the audio device 101 are coupled to the processor 194. The display device 107 includes a processor 109, such as a GPU, and a display screen 195. The display screen 195 is coupled to the processor 109.

[0122] The audio device 101 includes an audio processor 105 and a speaker 103. The speaker 103 is coupled to the audio processor 105. The haptic feedback device 196 includes a processor 198C, a driver 198B, and a mechanical component 198A. The processor 198C is coupled to the driver 198B, which is coupled to the mechanical component 198A. An example of a driver includes one or more transistors coupled to each other. An example of the mechanical component 198A includes a tactile sensor such as a motor or a transducer.

[0123] The processor 194 receives notifications, such as pairing-related notifications or pairing request notifications 190, from the wireless communication IC 156 and provides the notifications to the processor 109. The processor 109 applies rendering operations to the notifications to display the video data on the display screen 195.

[0124] Additionally, processor 194 receives audio data for notifications, such as pairing-related notifications or pairing request notifications 190, from wireless communication IC 156 and provides the audio data to audio processor 105. The audio data for the notification may contain the same information contained in the image or images of the notification or may contain additional information to be output as sound along with the image or images of the notification. Audio processor 105 processes the audio data, such as filtering, amplifying, or converting from digital to analog format, and provides the processed audio data to speaker 103. Speaker 103 outputs the processed audio data as sound.

[0125] Processor 198C also receives haptic feedback data output along with notifications such as pairing-related notification or pairing request notification 190 or generated during execution of application 1 ( FIG. 1A ) from wireless communication IC 156, and provides the haptic feedback data to processor 198C. Upon receiving the haptic feedback data, processor 198C transmits a signal to driver 198B, which outputs a current upon receiving the signal. Mechanical component 198A vibrates in accordance with the current, providing haptic or tactile feedback to user A holding controller 192.

[0126] 2 is a diagram of an embodiment of a system 200 for illustrating the mirroring effect of computing device 114. System 200 includes the components of system 100 of FIG. 1A or system 159 of FIG. 1D, except that system 200 includes a display device 202. An example of display device 202 is a television or smart television or monitor. Display device 202 has a display screen 204, such as a light-emitting diode screen, a liquid crystal display screen, or a plasma display screen.

[0127] Computing device 114 communicates with display device 202 via a wireless or wired connection 206. For example, dongle 252 plugs into a High-Definition Multimedia Interface (HDMI®) port on display device 202, allowing computing device 114 to wirelessly stream media, such as video and audio frames, from computing device 114 to display device 202. Video frames of media are displayed on display screen 204, and audio frames of media are output through one or more speakers of display device 202. An example of wired connection 206 includes an HDMI® cable.

[0128] In one embodiment, in addition to outputting media to display device 202, media is output by computing device 114. For example, video frames of the media are displayed on display screens 116 and 204, and audio frames are output via one or more speakers of computing device 114 and one or more speakers of display device 202.

[0129] When session 106 is established between computing device 114 and server system 104, and before session 106 is paired with controller 164, information related to session 106, such as an image of the title of the game, is initially displayed on display screen 116 of computing device 114. A processor of computing device 114 streams or transmits media having information related to session 106 to display device 202 over wired or wireless connection 206. A GPU of display device 202 displays an image of the information related to session 106 on display screen 204, and one or more speakers of display device 202 output sound having information related to session 106 or sound having other information synchronized with the information related to session 106.

[0130] Additionally, when a notification, such as a pairing-related notification or a pairing request notification, is displayed on computing device 114, the notification is streamed or transmitted from a processor of computing device 114 to display device 202 via wired or wireless connection 206. A GPU of display device 202 displays the notification on display screen 204. Additionally, an audio processor of display device 202 outputs the sound of the notification through one or more speakers of display device 202.

[0131] 3 is a diagram of an embodiment of a system 300 illustrating execution of a discovery program 306 to discover the device ID of the controller 164. The computing device 114 includes a processor 302, a memory device 304, a wireless communication IC 308, and a wireless communication IC 310. The wireless communication IC 310 applies a wireless communication protocol to one or more communication packets received from the computing device 114 over connection channel CC2 to extract data from the communication packets and apply it to the data to be transferred from the processor 302 to the computing device 114 over connection channel CC2 to generate one or more communication packets. An example of the wireless communication IC 308 includes a Bluetooth® device that enables communication between the controller 164 and the computing device 114 (FIG. 1A). The processor 302 is coupled to the memory device 304 and the wireless communication IC 308. The discovery program 306 is stored in the memory device 304. The processor 302 is coupled to the memory device 304, the wireless communication IC 310, and the wireless communication IC 308.

[0132] When a session 106 of execution of application 1 is established between the computing device 114 and the server system 104, and before the session 106 is paired with the controller 164, the processor 302 executes a discovery program 306 to request the device ID of the controller 164 from the controller 164. The processor 194 (FIG. 1B or 1D) of the controller 164 retrieves the device ID from the device ID chip 158 and provides the device ID to the wireless communication IC 157 (FIG. 1B or 1D). The wireless communication IC 157 applies a wireless communication protocol, such as the Bluetooth® protocol, to the device ID to generate one or more forwarded datagrams and transmits the forwarded datagrams to the wireless communication IC 308 of the computing device 114 over the wireless connection 115.

[0133] The wireless communication IC 308 applies a wireless communication protocol to one or more forwarded datagrams received from the controller 164 to extract a device ID from the forwarded datagrams and provide the device ID to the processor 302. The processor 302 generates instructions to transmit the device ID to the server system 104. The processor 302 transmits the instructions and the device ID to the wireless communication IC 310, which applies a wireless communication protocol to the device ID and instructions to generate one or more communication packets and transmits the communication packets to the router 118 via connection channel CC2.

[0134] Router 118 applies a wireless communication protocol to the communication packets to obtain a device ID and instructions, and directs the device ID and instructions to modem 120. Modem 120 applies a network communication protocol to the device ID and instructions to generate one or more transport packets and transmits the transport packets to server system 104 via connection C1, computer network 122, and connection C2.

[0135] 4A is a diagram of an embodiment of a controller 408 to illustrate the use of the calibration processor 404 within the controller 408. The controller 408 is an example of the controller 102 of FIG. 1A, or the controller 164 of FIG. 1E, or the controller 192 of FIG. 1F-2. Illustratively, the controller 408 includes the components of the controller 102. The controller 408 includes the motion sensor system 152, the calibration processor 404, the processor 194, and the wireless communication IC 156. The calibration processor 404 is coupled to the motion sensor system 152 and the processor 194.

[0136] The motion sensor system 152 measures sensor values, such as data for calculating the position and orientation of the controller 408, or data for calculating the position and orientation of the input device 150 of the controller 408, or a combination thereof, and provides the sensor values ​​to the calibration processor 404. For example, the motion sensor system 152 measures the acceleration of the controller 408 and the orientation of the controller 408 relative to a reference coordinate system of the controller 408. As another example, the motion sensor system 152 measures the acceleration of the input device 150 and the orientation of the input device 150 relative to a reference coordinate system of the controller 408.

[0137] The calibration processor 404 receives sensor values ​​from the motion sensor system 152 and calibrates the values ​​of one or more sensors. For example, the calibration processor 404 scales the sensor values, such as by multiplying, adding, or subtracting a factor, to output calibrated sensor values, which are example input information described above with reference to FIG. 1B . Examples of factors include real numbers. For example, when the controller 408 moves from position P1 to position P2, which is two inches apart, the calibration processor 404 scales the distance between positions P1 and P2 to be less than two inches or greater than two inches apart. As another example, when the input device 150 moves from position PO1 to position PO2, which is one centimeter apart, the calibration processor 404 scales the distance between positions PO1 and PO2 to be less than one centimeter or greater than one centimeter.

[0138] The calibrated sensor values ​​are provided by the calibration processor 404 to the processor 194, which generates instructions to transmit the calibrated sensor values ​​to the server system 104. The processor 194 transmits the instructions and the calibrated sensor values ​​to the wireless communication IC 156. The wireless communication IC 156 generates one or more transmission units incorporating the instructions and the calibrated sensor values ​​by applying a network communication protocol and transmits the transmission units to the router 118. The router 118 applies the wireless communication protocol to obtain the instructions and the calibrated sensor values ​​and transmits the instructions and the calibrated sensor values ​​to the modem 120 (FIG. 1A). The modem 120 applies the network communication protocol to the calibrated sensor values ​​to generate one or more transmission packets and transmits the transmission packets to the server system 104 (FIG. 1A) via the communication channel 108 (FIG. 1A). Calibrating the sensor values ​​in the controller 408 reduces the workload of the server system 104 of FIG. 1A.

[0139] 4A is a diagram of an embodiment of a controller 450 to illustrate the use of a calibration process 404 within the controller 450. The controller 450 is an example of the controller 160 of FIG. 1C, or the controller 164 of FIG. 1E, or the controller 192 of FIG. 1F-2. The controller 450 includes the motion sensor system 152, the calibration processor 404, the processor 194, and the broadband communication IC 162.

[0140] The calibrated sensor values ​​are provided by calibration processor 404 to processor 194, which generates instructions to transmit the calibrated sensor values ​​to server system 104. Processor 194 transmits the instructions and the calibrated sensor values ​​to broadband communication IC 162. Broadband communication IC 162 generates one or more transfer units incorporating the calibrated sensor values ​​and the instructions by applying a cellular communication protocol and transmits the one or more transfer units to server system 104 (FIG. 1C) via cellular network 154 (FIG. 1C). Calibrating the sensor values ​​in controller 450 reduces the workload of server system 104 of FIG. 1C.

[0141] 5 is a diagram of an embodiment of a server system 500 to illustrate calibration of sensor values ​​by a calibration processor 404 of the server system 500. Instead of calibration being performed by the controller, calibration is performed by the server system 500. The server system 500 is an example of the server system 104 of FIG. 1A or 1C. The server system 500 includes the components of the server system 104. The server system 500 further includes a calibration processor 404 coupled to the input channel 184 and the communication interface 170.

[0142] The communication interface 170 of the server system 500 receives one or more forwarded packets with input information further including sensor values ​​and applies a network communication protocol to the forwarded packets to extract the sensor values. The processor of the communication interface 170 determines that the information in the forwarded packets matches the structure of the sensor values ​​and sends the sensor values ​​to the calibration processor 404. The calibration processor 404 receives the sensor values, calibrates the sensor values, and outputs the calibrated sensor values. The calibration processor 404 provides the calibrated sensor values ​​to the execution engine 186 via the input channel 184. The execution engine 186 applies the calibrated sensor values ​​during execution of the application 1 to modify the state of the virtual scene and generate one or more frames for encoding.

[0143] FIG. 6 shows a perspective view of a video game controller 600 for interfacing with an interactive program, such as application 1, according to an embodiment described in this disclosure. Video game controller 600 is an example of controller 102 of FIG. 1A, controller 160 of FIG. 1C, or controller 164 of FIG. 1E. Video game controller 600 includes a main body 602 and extensions 604A and 604B extending from main body 602. Extensions 604A and 604B are configured to be held by user A's left and right hands, respectively, and thus function as handles that allow user A to securely grip video game controller 600. Included on the top surface of main body 602 are various input devices, such as buttons 606A, 606B, 606C, and 606D, joysticks 608A and 608B, and multiple directional pads 610A, 610B, 610C, and 610D. Also shown is the top of a three-dimensional (3D) control bar 612 that extends from top to bottom through the body 602 of the video game controller 600. A speaker 614 is provided for playing sounds that provide feedback to user A.

[0144] Additionally, video game controller 600 includes a touch panel 616 defined on the top surface of main body 602 facing user A's head when extensions 604A and 604B are held by user A in his left and right hands, respectively. Touch panel 616 is oriented substantially horizontally and positioned between groups A and B, such that user A holding extensions 604A and 604B can easily use touch panel 616 with the thumb of either hand. Group A includes buttons 606A, 606B, 606C, and 606D, and group B includes directional pads 610A, 610B, 610C, and 610D. Touch panel 616 utilizes touch-sensitive technology (e.g., resistive, capacitive, etc.) to detect touch gestures made by user A. In the illustrated embodiment, the touch panel 616 provides a tactile sensation due to its shape, and user A easily determines the approximate vertical position of his or her thumb on the touch panel 616 based solely on sensation.

[0145] Video game controller 600 includes a fingerprint pad 618 that receives the touch of one or more fingers of user A and outputs the biometric ID described above. Fingerprint pad 618 is an example of the fingerprint reader described above. Fingerprint pad 618 is located below touch panel 616, between joysticks 608A and 608B.

[0146] In one embodiment, fingerprint pad 618 is located anywhere else on controller 600. For example, fingerprint pad 618 is integrated into touch panel 616. As another example, the fingerprint pad is integrated into the surface of extension 604A or the surface of extension 604B, or into both surfaces. As yet another example, touch panel 616 is part of a display screen, such as a liquid crystal display, a light-emitting diode display, or a plasma display, and fingerprint pad 618 is integrated into the display screen. As yet another example, the fingerprint pad is integrated into the top surface of either joystick 608A or joystick 608B, or multiple fingerprint pads are integrated on the surfaces of joysticks 608A and 608B. As yet another example, fingerprint pad 618 is located on the back side of controller 600.

[0147] In one embodiment, the video game controller 600 includes one or more microphones for capturing sounds from the real-world environment. By way of example, the microphones are arranged as a microphone array. The arrangement is a linear array of microphones. When three or more microphones are included in the microphone array, the location of a sound source relative to the microphone array can be determined based on an analysis of audio data captured by the microphone array. More specifically, a sound source can be identified relative to the microphone array based on the relative timing of its sound captured by each microphone in the microphone array. In combination with the position and orientation of the video game controller 600 (e.g., as determined based on the motion sensor system 152 of FIGS. 1B and 1D and the tracking methods defined elsewhere herein), the microphone array can in turn determine the location of a sound source within the interactive environment. Furthermore, the captured sounds can be processed to filter out sounds that do not emanate from a specific region of the real-world environment. User A's sounds are captured by the microphone, and the biometric information described above is output.

[0148] The embodiments described in this disclosure may be implemented in a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. The embodiments described in this disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0149] With the foregoing embodiments in mind, it should be understood that the embodiments described herein may employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the various embodiments described herein are useful machine operations. Some embodiments described herein also relate to devices or apparatus for performing these operations. An apparatus may be specially constructed for the required purposes, or the apparatus may be a computer selectively activated or configured by a computer program stored in the computer. In particular, various machines may be used with computer programs written in accordance with the teachings herein, or it may be more convenient to construct a more specialized apparatus to perform the required operations.

[0150] Various embodiments described in this disclosure may also be embodied as computer-readable code on a non-transitory computer-readable medium. The non-transitory computer-readable medium is any data storage device that can store data, such as RAM, ROM, flash memory, or a disk, which can then be read by a computer system. Examples of computer-readable media include hard drives, network-attached storage (NAS), ROM, RAM, compact disc ROM (CD-ROM), CD-R, CD-RW, magnetic tape, and other optical and non-optical data storage devices. The non-transitory computer-readable medium may include computer-readable tangible media distributed across network-coupled computer systems so that the computer-readable code is stored and executed in a distributed fashion.

[0151] Although the operations of the method have been described in a particular order, it should be understood that other housekeeping operations may be performed between operations, or operations may be arranged to occur at slightly different times, or operations may be distributed throughout the system to allow operations to occur at various intervals relative to processing, and operations may be performed in a different order, so long as the processing of the overlay operation is performed in the desired manner.

[0152] Furthermore, it should be noted that in embodiments, one or more features from any of the above embodiments may be combined with one or more features of any other embodiment without departing from the scope of the various embodiments described in this disclosure.

[0153] Although the foregoing embodiments have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered as illustrative rather than restrictive, and the various embodiments described in this disclosure are not to be limited to the details provided herein, but may be modified within the scope of the appended claims and their equivalents.

Claims

1. 1. A method for associating a handheld controller (HHC) with a session, comprising: receiving, by a computing device, first instructions for providing, from a server system, information associated with the session and information for identifying the HHC, the session being established between the computing device and the server system; providing the first instructions to the computing device to provide to the HHC information associated with the session and information identifying the HHC; Including, providing causes the HHC to provide information identifying the HHC to the server system and associate the session with the HHC; The method, wherein the session is a gaming session, a video session, a virtual imaging session, an augmented reality imaging session, or a virtual reality imaging session.

2. obtaining login information from the computing device; determining whether the login information is authentic, wherein the session is established when an application is executed by the server system, and the application is executed when the login information is determined to be authentic; The method of claim 1 further comprising:

3. The method of claim 2 , wherein the information associated with the session includes information for identifying the application and a destination address of the server system.

4. generating second instructions for providing information identifying the HHC to the destination address of the server system, wherein providing the information identifying the HHC includes: applying a communications protocol to the second instructions and information identifying the HHC; After applying the communication protocol, transmitting the second command and the information for identifying the HHC via a router to a modem for transferring the information for identifying the HHC to the server system via a computer network; The method of claim 3, comprising:

5. generating, by a processor of the HHC after receiving the destination address of the server system from the computing device, second instructions for transmitting information identifying the HHC to the server system; In response to receiving the second instruction, the HHC applies a communications protocol to information identifying the HHC to generate one or more transfer units; further comprising The method of claim 3 , wherein providing information to identify the HHC includes transmitting the one or more transfer units to the server system over a network.

6. 3. The method of claim 2, wherein the HHC is configured to enable the HHC to influence images displayed by the computing device after the HHC associates with the session, the images being displayed by the computing device based on execution of the application.

7. The method of claim 1 , wherein the information for identifying the HHC includes a device identification of the HHC.

8. The method of claim 1 , wherein providing the information associated with the session and the first instruction from the computing device to the HHC is performed using a wireless communication protocol.

9. The method of claim 1 , wherein the computing device comprises at least one of a gaming console and a display device. The method of claim 1.

10. The method of claim 1 , wherein the HHC comprises a mobile device.

11. The method of claim 1 , wherein the HHC comprises a video game controller, a cellular phone, or a tablet.

12. 1. A system for associating a handheld controller (HHC) with a session, comprising: a computing device coupled to the HHC, the computing device configured to receive, from a server system, a first instruction to provide information associated with the session and information for identifying the HHC, the session being established between the computing device and the server system; Equipped with the computing device is configured to provide the first instructions to the HHC to provide information associated with the session and information identifying the HHC; the providing includes having the HHC provide information identifying the HHC to the server system and associating the session with the HHC; The system, wherein the session includes at least one of a gaming session, a video session, a virtual image session, an augmented reality image session, and a virtual reality image session.

13. 13. The system of claim 12, wherein the computing device is configured to provide login information to the server system, the session is established between the computing device and the server system when the login information is authenticated, and the computing device is configured to display an application after the login information is authenticated.

14. The system of claim 13 , wherein the information associated with the session includes information identifying the application and a destination address of the server system.

15. The HHC is a processor; a communication circuit coupled to the processor; Including, the processor is configured to generate second instructions for providing information identifying the HHC to the destination address of the server system; To provide information identifying the HHC, the communication circuitry applying a communications protocol to the second instructions and information identifying the HHC; 15. The system of claim 14, further configured to, after applying the communication protocol, transmit the second instruction and the information for identifying the HHC via a router to a modem for transferring the information for identifying the HHC to the server system via a computer network.

16. The HHC is a processor; a communication circuit coupled to the processor; Including, after the destination address of the server system is received, the processor is configured to generate second instructions for transmitting information identifying the HHC to the server system and transmit the second instructions to the communication circuit; When the second instruction is received, the communication circuitry is configured to apply a communication protocol to information identifying the HHC to generate one or more transfer units; 15. The system of claim 14, wherein the communication circuitry is configured to transmit the one or more transfer units to the server system via a cellular network and a computer network to provide information identifying the HHC.

17. 14. The system of claim 13, wherein the HHC is configured to affect an image displayed by the computing device after the HHC associates the session, the image being generated based on execution of the application.

18. The system of claim 12 , wherein the HHC comprises a mobile device.

19. The system of claim 12 , wherein the HHC comprises a video game controller, a cellular phone, or a tablet.

20. a handheld controller (HHC) associated with the session, a first communication circuit configured to receive a first instruction to provide information associated with the session and information for identifying the HHC, the information associated with the session being provided by a computing device, and the session being established between the computing device and a server system; and a second communication circuit; a processor coupled to the first communication circuit and the second communication circuit, the processor configured to receive information associated with the session and the first instruction from the first communication circuit, and configured to access information for identifying the HHC and provide the information for identifying the HHC to the second communication circuit when the first instruction is received by the processor; Equipped with the second communication circuitry is configured to provide information identifying the HHC to the server system to associate the session with the HHC; The HHC, wherein the session includes at least one of a gaming session, a video session, a virtual image session, an augmented reality image session, and a virtual reality image session.

21. 21. The HHC of claim 20, wherein the information associated with the session includes information for identifying an application and a destination address of the server system.

22. The HHC of claim 21, wherein the processor is configured to generate second instructions for providing information for identifying the HHC to the destination address of the server system, and the first communication circuit is configured to apply a communication protocol to the second instructions and the information for identifying the HHC to transmit the information for identifying the HHC to the server system, and after applying the communication protocol, transmit the second instructions and the information for identifying the HHC via a router to a modem for forwarding the information for identifying the HHC to the server system via a computer network.

23. Upon receiving the destination address of the server system from the computing device, the processor is configured to generate second instructions for transmitting information identifying the HHC to the server system, and transmit the second instructions and the information identifying the HHC to the second communications circuit; In response to receiving the second instruction, the second communications circuitry is configured to apply a communications protocol to information identifying the HHC to generate one or more transfer units; 22. The HHC of claim 21, wherein the second communication circuit is configured to transmit the one or more transfer units to the server system via a network to transmit information identifying the HHC to the server system.

24. 21. The HHC of claim 20, wherein the HHC comprises a mobile device.

25. 21. The HHC of claim 20, wherein the HHC comprises a video game controller, a cellular phone, or a tablet.

Images