Systems and methods for reducing hops associated with head mounted systems

Abstract

This invention describes systems and methods for reducing hops associated with a head-mounted display. The head-mounted display includes communication circuitry for receiving and transmitting interactive media associated with a game program via a network. The interactive media is processed by a game cloud system and streamed directly to the head-mounted display. The head-mounted display further includes user input circuitry for generating input from actions received from a user, the input including positioning and movement detected by the user input circuitry. The head-mounted display includes game processing circuitry for decoding the interactive media received from the network. The game processing circuitry drives a portion of interactivity associated with the game program. This portion of interactivity is generated based on the input.

Description

[0001] This divisional application is a divisional application with application date of June 6, 2014, application number 201410250174.3, and invention name “System and method for reducing hops associated with head-mounted systems”. Technical Field

[0002] The present disclosure relates to systems and methods for reducing hops associated with head mounted systems. Technical Background

[0003] Video game consoles are often used to play games. For example, users often purchase a controller and a console to play games. The console can be connected to the Internet. The user turns on the console and operates the controller to play the game through the console and the Internet. However, the many hops (e.g., many network devices) through which the game data passes in order to facilitate the game play may slow down the speed of playing the game. For example, the user may have to wait for graphics to be downloaded from the Internet to the console before playing certain stages of the game.

[0004] It is against this background that the embodiments described in this disclosure arise. SUMMARY OF THE INVENTION

[0005] Embodiments described in the present disclosure provide systems and methods for reducing the number of hops associated with a head mounted system.

[0006] In summary, in various embodiments, the systems and methods allow for game play in which media data is streamed from a network to a head mounted display by bypassing a router between the head mounted display and the network. Not using a router reduces the number of hops between the head mounted display and a game cloud that executes a game program to allow a user to play the game.

[0007] In some embodiments, a head mounted display for playing games through a game cloud system is described. The head mounted display includes a communication circuit for receiving and transmitting interactive media associated with a game program through a network. The interactive media is processed by the game cloud system and streamed directly to the communication circuit of the head mounted display. The head mounted display further includes a user input circuit for receiving actions from a user to generate input, the input including positioning and movement detected by the user input circuit. The head mounted display includes a game processing circuit for decoding the interactive media received from the network. The game processing circuit drives a portion of interactivity associated with the game program. The portion of interactivity is generated based on the input.

[0008] In some embodiments, a method for playing a game through a game cloud system is described. The method includes directly streaming interactive media associated with a game program through a network. The interactive media is processed by the game cloud system. The direct streaming operation is performed by a head-mounted display. The method further includes: receiving input, the input including location and motion detected by the head-mounted display; decoding the interactive media received from the network; and driving a portion of interactivity associated with the game program. The portion of interactivity is generated based on the input.

[0009] In various embodiments, a non-transitory computer-readable medium is described, on which computer-executable instructions for playing a game through a game cloud system are stored. When the computer-executable instructions are executed by a processor of a computer, interactive media associated with a game program is directly streamed over a network. The interactive media is processed by the game cloud system. In addition, when the computer-executable instructions are executed by the processor, input is received from the head-mounted display. The input includes positioning and movement detected by the head-mounted display. When the computer-executable instructions are executed by the processor, the interactive media received from the network is decoded and a portion of the interactivity associated with the game program is driven. The portion of the interactivity is generated based on the input.

[0010] Other aspects of the description will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the embodiments described in the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The various embodiments described in this disclosure may be best understood by referring to the following description taken in conjunction with the accompanying drawings, in which:

[0012] Figure 1A is a diagram of a system for reducing the number of hops between a gaming cloud and a head mounted display (HMD) or handheld controller (HHC) according to one embodiment described in the present disclosure.

[0013] Figure 1B is a diagram of a system for transmitting data between an HMD or HHC and a gaming cloud through a network and a router according to one embodiment described in the present disclosure.

[0014] Figure 1C is a diagram of a system for communicating media data using a console and for communicating input data and / or media data with or without a router, according to one embodiment described in the present disclosure.

[0015] Figure 2 is a diagram of a system according to one embodiment described in the present disclosure, the system being used to illustrate generating input data based on the user's head movements and / or hand movements.

[0016] Figure 3 is a diagram of an HMD according to one embodiment described in the present disclosure, the HMD being Figure 1A An example of an HMD.

[0017] Figure 4A is a diagram of an image according to an embodiment described in the present disclosure, the image being displayed in Figure 3 on the display of the HMD in order to access the wireless network.

[0018] Figure 4B is a diagram of an image displayed on a display screen of a computer to access a wireless network according to one embodiment described in the present disclosure.

[0019] Figure 5A is a diagram of a game image according to one embodiment described in the present disclosure, the image being displayed in Figure 3 On a display screen of an HMD, wherein a peripheral area of ​​the display screen of the HMD has a low resolution and a central area has a high resolution.

[0020] Figure 5B is a diagram of a game image according to one embodiment described in the present disclosure, the image being displayed in Figure 3 On a display screen of an HMD, a top area and a bottom area of ​​the display screen of the HMD have a low resolution and an area between the top area and the bottom area has a high resolution.

[0021] Figure 5C is a diagram of a game image according to one embodiment described in the present disclosure, the image being displayed in Figure 3 On a display screen of an HMD, a right area and a left area of ​​the display screen of the HMD have a low resolution and an area between the right area and the left area has a high resolution.

[0022] Figure 6 is a diagram illustrating various types of HHCs according to one embodiment described in the present disclosure.

[0023] Fig. 7A is a diagram of an embodiment of an HMD according to one embodiment described in the present disclosure, the HMD being Figure 3 An example of an HMD.

[0024] Figure 7Bis an isometric view of an embodiment of an HMD according to one embodiment described in the present disclosure, the HMD being Figure 3 An example of an HMD.

[0025] Figure 8 is a block diagram of a console according to one embodiment described in the present disclosure, the console being compatible with HHC and Figure 3 HMD docking.

[0026] Fig. 9 is a block diagram of an embodiment of a gaming system according to one embodiment described in the present disclosure. DETAILED DESCRIPTION

[0027] The present invention describes systems and methods for reducing the number of hops associated with a head-mounted system. For example, network devices such as routers, repeaters, hubs, computers, consoles, etc. add hops to data transmitted through the network devices. One or more network devices may be located between a head-mounted display (HMD) and a network and / or between a handheld controller (HHC) and a network. The addition of hops may be due to: a network device receiving data from another device; a network device buffering data; analyzing data; and a network device resending data to another device. The system and method reduce the number of network devices between the HMD and the network and / or between the HHC and the network. The reduction in the number of network devices can reduce delays, such as lag time, time to buffer data, time to receive data, time to analyze data, time to resend data, etc. When delays are reduced, high-density graphics received from a game cloud over a network are displayed using the HMD and / or HMC with minimal delay. It should be noted that the various embodiments described in the present disclosure can be practiced without some or all of these specific details. In other cases, well-known processing operations are not described in detail to avoid unnecessary confusion of the various embodiments described in the present disclosure.

[0028] In some embodiments, the system includes a computer, an HHC, and an HMD. In various embodiments, the computer can be a special-purpose computer, a game console, a mobile phone, a tablet computer, or other such devices that execute one or more parts of the interactive game program presented on the display. In these embodiments, any remaining part of the interactive game program is executed in a game cloud system (e.g., one or more virtual machines (VMs)). In some embodiments, the game cloud system includes many servers, and the servers execute the game program to generate a game environment on a game device, an HMD, a television, a computer screen, etc. For example, a virtual machine monitor (hypervisor) is executed on physical resources (e.g., processors, storage devices, servers, etc.) to execute many operating systems and many computer software applications to generate data, which is further used to generate a game environment on the HMD. Examples of game consoles include Sony Computer Entertainment, Inc., a ... TM The interactive game program can be a multi-user game program played by multiple users or a single-user game program played by a user and a computer.

[0029] In some embodiments, all portions of the interactive game program are executed on a game cloud system or on a computer.

[0030] In various embodiments, an interactive game program is executed to generate an immersive environment in the HMD. For example, the immersive environment is created by the processor and the user immerses himself in the environment. The immersive environment makes the user feel like he / she is part of the virtual world created by the processor. The immersive environment mentally separates the user from the real world and brings the user into the virtual world to provide the user with a gaming experience. In some embodiments, the processor generates the immersive environment, and the immersive environment is displayed in the HMD to establish the following feeling in the user: the virtual world of the immersive environment is real. In some embodiments, the media data or other media data as described herein includes data used to generate the immersive environment.

[0031] An HMD is a device worn on a user's head or as part of a helmet that has small display optics, such as lenses, glass, etc., located in front of one or each eye of the user. In some embodiments, although two optics are used for both eyes, one scene, such as a virtual scene, an augmented virtual reality scene, etc., is displayed on the display screen of the HMD. Both eyes see one scene.

[0032] In some embodiments, the HMD is capable of receiving and presenting video output from the computer.In various embodiments, the HHC and / or HMD communicates wirelessly with the computer, as such wireless communication provides greater freedom of movement for the HHC and / or HMD than a wired connection.

[0033] The HHC may include any of a variety of features for providing input to an interactive gaming program, such as buttons, joysticks, directional pads, triggers, hand gestures, touch pads, touch screens, or other types of input mechanisms, etc. An example of an HHC is the Sony Dualshock 4 controller manufactured by Sony Computer Entertainment, Inc. Other examples of HHCs include controllers manufactured by other entities and having any model and any version number.

[0034] Additionally, the HHC may be a motion controller that allows a user to interface with an interactive gaming program and provide input to the interactive gaming program by moving the motion controller. An example of a motion controller is the Sony Computer Entertainment TM PlayStation Move manufactured by the company TM Controller.

[0035] Similarly, the HMD may include a user input circuit that allows the user to dock with the interactive game program and provide input to the interactive game program by moving the HMD. Various techniques can be used to detect the positioning and movement of the motion controller and / or HMD. For example, the motion controller and / or the user input circuit may include various types of inertial sensor circuits, such as accelerometers, gyroscopes, and magnetometers. In some embodiments, the accelerometer is a 6-axis low-latency accelerometer. In some embodiments, the motion controller and / or the HMD may include one or more fixed reference objects, such as light emitting diodes (LEDs), colored dots, reflectors, markers, retroreflective materials, predefined shapes, predefined colors, predefined objects, barcodes, labels, quick response (QR) codes, etc., and the image of the fixed reference object is captured by one or more digital cameras. In some embodiments, the digital camera includes a video camera, which further includes a single charge coupled device (CCD), an LED indicator, and a hardware-based real-time data compression and encoding device so that the compressed video data can be transmitted in an appropriate format (such as based on the moving picture experts group (MPEG) standard within the image). The position and movement of the motion controller and / or may then be determined by analyzing images captured by one or more digital cameras.

[0036] Figure 1A1 is a diagram of an embodiment of a system 100 for reducing the number of hops between a game cloud 102 and an HMD 104 or HHC 106. In some embodiments, the game cloud 102 is referred to herein as a game cloud system. In various embodiments, a user 108 places an HMD 104 on his head so that the lenses of the HMD 104 are located in front of his / her eyes, in a manner similar to how the user 108 puts on a helmet. In some embodiments, the HMD 104 is put on like glasses (e.g., prescription glasses, goggles, etc.). In many embodiments, the HMD 104 covers both eyes of the user 108. The user 106 holds the HHC 106 in his / her hand.

[0037] In various embodiments, the hands of the user 106 are used instead of the HHC 106 to provide gestures to the HMD 104, such as hand gestures, finger gestures, etc. For example, a digital camera within the HMD 104 captures an image of the gesture, and a processor within the HMD 104 analyzes the gesture to determine whether a game displayed by the HMD 104 is affected by the gesture.

[0038] As used herein, a processor may be a microprocessor, a programmable logic device (PLD), an application specific integrated circuit (ASIC), or a combination thereof.

[0039] The system 100 includes a network 110, which may be a local area network (LAN), a wide area network (WAN), or a combination thereof. Examples of the network 110 include the Internet, an intranet, or a combination thereof. In some embodiments, the network 110 uses the Transmission Control Protocol (TCP) / Internet Protocol (IP) or the User Datagram Protocol / IP (UDP / IP) to communicate media data between the game cloud 102 and the HMD 104 or HHC 106 through the network 110. In various embodiments, the network 110 uses the Ethernet protocol, the TCP / IP protocol, or both protocols to communicate media data between the game cloud 102 and the HMD 104 or HHC 106 through the network 110.

[0040] The game cloud 102 includes an encoder / decoder (codec) 112 and a stream buffer 114. The stream buffer 114 stores a stream of media data 116 generated when executing the game program 117. The media data 116 includes virtual environment data, virtual game object data, a combination thereof, etc. In various embodiments, the virtual environment data is presented to generate a virtual environment of the game, and the virtual game object data is presented to generate one or more virtual game objects, such as virtual game characters, virtual points, virtual prizes, game interfaces, etc. In some embodiments, the virtual environment of the game includes virtual game objects. Examples of virtual environments include virtual geographic areas, such as virtual cities, virtual roads, virtual lakes, virtual oceans, etc. The game program 117 is an example of an interactive game program and is executed by one or more servers of the game cloud 102. Examples of codecs 112 include compressors / decompressors. For example, codecs 112 encode / decode media data 116. Examples of compression include lossy compression, lossless compression, etc. In some embodiments, the codec 112 performs packetization and depacketization, and performs encoding and decoding. In these embodiments, the codec 112 includes a stream buffer 114.

[0041] The user 108 uses the HMD 104 to access an operating system (OS) executed by a processor of the HMD 104. For example, the user 108 turns on the HMD 104 through a button, and the processor of the HMD 104 executes the OS.

[0042] In some embodiments, the OS allows the HMD 104 to access the network 110. For example, when the user 108 selects a network access application (e.g., a network access icon, a network access symbol, etc.) executed by the processor of the HMD 104 on the OS, the network access application provides a list of networks to the microcontroller of the HMD 104 for display to the user 108. The user 108 uses a user input circuit to select one of the networks to access the network 110. For example, the user 108 performs one or more head movements, such as head tilt, blinking, gazing, staring, nodding, shaking the head, etc., and the user input circuit detects the head movement to generate an input for selecting one of the networks. As another embodiment, the user 108 selects a button on the HHC 106 to select one of the networks, and the communication circuit of the HHC 106 communicates this selection from the HHC 106 to the HMD 104. Examples of communication circuits include transceivers, transmit / receive circuits, network interface controllers, or combinations thereof, etc. The network access application also requests a password (eg, a security key, a numeric password, etc.) from the user 108 to access the network 110. After verifying that the password is valid, the network access application allows the user 108 to access the network 110.

[0043] Once the network 110 is accessed, the OS allows the HMD 104 to access the game program 117. For example, when the user 108 selects a game access application (e.g., a game access icon, a game access symbol, etc.) executed by the processor of the HMD 104 on the OS, the game access application requests access to the game program 117 through the network 110. For example, the user 108 performs one or more head movements, and the user input circuit detects the head movement to generate an input for selecting the game access application. As another embodiment, the user 108 selects a button on the HHC 106 to select the game access application, and the communication circuit of the HHC 106 communicates this selection from the HHC 106 to the HMD 104.

[0044] After accessing the game program 117 through the network 110, the microcontroller of the HMD 104 displays the game on the display screen of the HMD 104. In some embodiments, the display screen of the HMD 104 is a high-performance screen to reduce blur when the HMD 104 moves quickly. The user 108 performs one or more of the head movements and each head movement triggers the user input circuit to generate input, which can be used to play the game. In some embodiments, the user 108 uses his / her hand to perform one or more buttons of the HHC 106, and each hand movement (e.g., pressing a finger, moving a finger, rotating a finger, moving a finger up, moving a finger down, moving a finger to the right, moving a finger to the left, etc.) triggers the HHC 106 to generate input, which can be used to play the game.

[0045] In some embodiments, the game access application requests a username and / or password from the user 108 to access the game program 117. After receiving verification from the game cloud 102 that the username and / or password is valid, the game access application allows the user 108 to access the game program 117. In some embodiments, the user 108 provides the username and / or password by performing one or more of the hand movements and / or performing one or more of the head movements.

[0046] In various embodiments, instead of accessing the game access application, the user 108 requests to access a web page after accessing the network 110, and the web page allows the user 108 to access the game program 117. For example, the user 108 selects the web browser application through the user input circuit by performing one or more of the hand movements, or selects the web browser application through the HHC 106 by performing one or more of the head movements to access the web page. After accessing the web page, the user 108 plays the game displayed on the web page. When the game program 117 is executed on the game cloud 102, the game is displayed.

[0047] In some embodiments, the user name and / or password of the user 108 is verified before accessing the web page to play the game displayed when the game program 117 is executed on the game cloud 102. The manner of verifying the username and / or password is similar to the above-mentioned manner when the user 108 accesses the game through the game access application.

[0048] When accessing the game program 117, the codec 112 encodes the digital data stream of the media data 116, such as compressing it, so as to send the digital data stream of the encoded media data to the HMD 104 through the network 110. In some embodiments, the digital data stream of the encoded media data is in the form of data packets for transmission through the network 110.

[0049] The HMD 104 receives a digital data stream of encoded media data from the codec 112 through the network 110, and processes (e.g., decodes, unpacks, etc.) and presents the digital data stream to display the game on the display screen of the HMD 104. When the game is displayed on the display screen of the HMD 104, one or more inputs are received from the HMD 104 and / or the HHC 106. For example, the user 108 performs one or more of the head movements after wearing the HMD 104. As another embodiment, the user 108 performs one or more of the hand movements, such as pressing a button, moving a joystick, hand gestures, finger gestures, combinations thereof, etc., and when performing one or more hand movements, the HHC 106 generates input data, and the communication circuit of the HHC 106 converts the input data into an input signal to be sent to the communication circuit of the HMD 104. The communication circuitry of the HMD 104 converts the input signal from analog form to digital form to generate input data, and the HMD 104 packetizes and encodes the input data and sends it to the codec 112 through the network 110 .

[0050] In some embodiments, the game program 117 maps input data generated based on one or more head movements with input data generated based on one or more hand movements to determine whether to change the state of a game displayed on the HMD 104. For example, when an input indicating a head tilt and an input indicating a button press on the HHC 106 are received over the network 110, the game program 117 determines that the state of the game is to be changed. Otherwise, the game program 117 determines not to change the state of the game.

[0051] The input data of the input generated based on one or more head movements and / or one or more hand movements is packetized and encoded by the HMD 104 and sent as a stream of encoded input data to the codec 112 through the network 110. For example, when the input data is not sent through a router, the input data is sent directly from the HMD 104 to the game cloud 102 through the network 110. The router can be placed between the HMD 104 and the network 110 to route data (e.g., transfer, send, etc.) between the HMD 104 and the network 110 to an addressed device, etc.

[0052] The codec 112 decodes (e.g., decompresses, etc.) the stream of encoded input data received from the HMD 104 via the network 110, and buffers the decoded input data in the stream buffer 114 for unpacking and sending to the game program 117. One or more servers of the game cloud 102 unpack the stream of decoded input data and send the input data to the game program 117. In some embodiments, the game cloud 102 first performs unpacking and then performs decoding.

[0053] Based on the input data, the game program 117 generates other media data, and one or more servers of the game cloud 102 packetize the other media data to generate a stream of other media data. The stream of other media data is stored in the stream buffer 114, encoded by the codec 112, and sent to the HMD 104 through the network 110 as a stream of encoded other media data. In some embodiments, before the other media data is sent to the HMD 104 through the network 110, the other media data is first encoded and then packetized. The HMD 104 receives the stream of encoded other media data, unpacks the stream, and decodes the encoded other media data to provide the other media data to the microcontroller of the HMD 104.

[0054] In some embodiments, the gaming environment includes media data 116, other media data, or a combination thereof.

[0055] The microcontroller of the HMD 104 changes the display of the game executed by the game program 117 based on other media data. For example, when the user 108 nods, in the virtual world of the game generated by executing the game program 117, the avatar of the user 108 also nods. As another embodiment, when the user 108 shakes his / her head to indicate refusal, in the virtual world of the game generated by executing the game program 117, the avatar of the user 108 also shakes its head to indicate refusal. As another embodiment, when the user 108 looks at the virtual game character displayed on the display screen of the HMD 104, the virtual game character looks back at the user 108. When the user 108 looks away from the virtual game character, the virtual game character looks away from the user 108. In addition, in this embodiment, the external digital camera of the HMD 104 captures an image of the eye or head position of the user 108. Based on the gaze of the eye or head position, the game program 117 changes the position of the eye of the virtual game character. It should be noted that the input data of the input generated based on one or more head movements and / or one or more hand movements changes the state of the game. In some embodiments, the display of a game is referred to herein as part of the interactivity associated with the game program 117.

[0056] In various embodiments, instead of communicating input data generated based on one or more hand motions from the HHC 106 to the HMD 104, the input data is communicated directly from the HHC 106 to the codec 112 through the network 110. For example, when the input data is not communicated through a router (not shown) between the HHC 106 and the network 110, the input data is communicated directly from the HHC 106 to the codec 112 through the network 110. The HHC 106 communicates input data generated based on one or more hand motions from the HMD 104 in a manner similar to the manner in which the HMD 104 communicates input data. For example, the input data generated based on one or more hand motions from the HMD 104 is encoded and packetized by the HHC 106 and sent to the codec 112 through the network 110 as a stream of encoded input data.

[0057] It should be noted that the absence of a router (not shown) between the HMD 104 and the network 110 reduces the number of hops between the network 110 and the HMD 104. In embodiments without a router, the codec 112 of the gaming cloud 102 streams interactive media (e.g., media data 116, other media data, etc.) directly to the wireless access card (WAC) of the HMD 104 over the network 110. Additionally, in these embodiments, the WAC streams interactive media (e.g., input data, etc.) directly to the codec 112 of the gaming cloud 102 over the network 110.

[0058] The reduction in the number of hops saves time associated with routers. For example, a router receives data from another network device, buffers the data, analyzes the data, and resends the data to another network device. When data is transmitted between the HMD 104 and the game cloud 102 through the network 110 by bypassing the router, the time of receiving data, buffering data, analyzing data, and resending data is saved. Similarly, when data is transmitted between the HHC 106 and the game cloud 102 through the network 110 by bypassing the router, the time of receiving data, buffering data, analyzing data, and resending data is saved.

[0059] Similarly, when a computer (eg, console, mobile device, etc.) is not used between the HMD 104 or HHC 104 and the network 110, time associated with the receipt, storage, analysis, and retransmission of data is saved.

[0060] Figure 1B 1 is a diagram of an embodiment of a system 150 for transmitting data between an HMD 104 or HHC 106 and a gaming cloud 102 via a network 110 and a router 152. The system 150 is similar to the system 100 ( Figure 1A ), except that system 150 includes a router 152 between HMD 104 and network 110. Router 152 is also located between HHC 106 and network 110.

[0061] The HMD 104 is coupled to the router 152 via a wireless connection (e.g., a Bluetooth connection, a Wi-Fi connection, etc.). In addition, the HHC 106 is coupled to the router 152 via a wireless connection (e.g., a Bluetooth connection, a Wi-Fi connection, etc.). In some embodiments, the router 152 is coupled to the network 110 via a wired connection.

[0062] System 150 operates in a manner similar to system 100 ( Figure 1A ), except that the stream of encoded data is sent from the HMD 104 or HHC 106 to the router 152. The router 152 routes (e.g., passes, etc.) the stream of encoded data to a path in the network 110 to facilitate sending the stream to the codec 112. The router 152 routes the stream of encoded data to the codec 112 using the IP address and / or media access layer (MAC) address of the codec 112. In some embodiments, the router 152 determines the network path of the network 110 based on one or more network traffic factors (e.g., the amount of packet flow on the network path, congestion on the network path, etc.).

[0063] The router 152 receives the stream of encoded data from the game cloud 102 via the network 110 and routes the stream of encoded data to the HMD 104. For example, the router 152 routes the stream of encoded data received from the game cloud 102 via the network 110 to the HMD 104 based on the IP address and / or MAC address of the HMD 104.

[0064] Figure 1C is a diagram of an embodiment of a system 170 for communicating media data using a computer 172 and for communicating input data and / or media data with or without a router 152 .

[0065] In some embodiments, the list of wireless networks is presented on a display screen coupled to or within the computer 172 instead of on the HMD 104. For example, when the computer 172 is a mobile phone, the computer 172 includes a display screen for displaying the list of wireless networks. As another example, when the computer 172 is coupled to a television display screen, the list of wireless networks is displayed on the display screen. In these embodiments, the list of wireless networks is accessed when the processor 176 of the computer 172 executes a wireless access application stored in a storage device of the computer 172. When the user 108 accesses the wireless access application through the HMD 104 or HHC 106 by performing one or more head movements and / or one or more hand movements, the processor 176 executes the wireless access application. The communication circuit of the HMD 104 or HHC 106 sends input data generated based on one or more head movements and / or one or more hand movements to the computer 172. When the processor 176 of the computer 172 receives the input data, the wireless access application is executed to generate a list of wireless networks.

[0066] The computer 172 includes a network interface controller (NIC) 174, which requests a portion of the game program 117 from the game cloud 102. Examples of NICs include network interface cards and network adapters. The portion of the game program 117 is encoded by the codec 112 and streamed to the NIC 174 of the computer 172 via the network 110. The processor 176 of the computer 172 executes the portion of the game program 117 to generate media data, which is sent from the communication circuit 178 (e.g., transceiver, transmit / receive circuit, network interface controller, etc.) to the HMD 104 for display on the display screen of the HMD 104. The communication circuit of the HMD 104 receives the media data from the computer 172 and sends the media data to the microcontroller of the HMD 104 for display on the display screen of the HMD 104.

[0067] In addition, the communication circuit 178 of the computer 172 receives input data generated based on one or more head movements and / or one or more hand movements from the HMD 104 and / or the HHC 106 and sends the input data to the processor 176. In response to the input data, the processor 176 executes the portion of the game program 117 stored in the computer 172 to generate media data, which is sent from the communication circuit 178 to the communication circuit of the HMD 104. The media data is sent to the communication circuit of the HMD 104 to change the virtual game objects and / or virtual environment of the game displayed by executing the game program 117. When the virtual game objects and / or virtual environment change, the game state of the game displayed by executing the game program 117 changes.

[0068] In some embodiments, the NIC 174 of the computer 172 sends the game state to the game cloud 102 through the router 152 and the network 110 so as to inform one or more servers of the game cloud 102 of the game state.

[0069] In various embodiments, media data (e.g., media data 116, other media data, etc.) is sent from the codec 112 to the HMD 104 through the network 110 and the router 152 until a portion of the game program 117 is downloaded from the game cloud 102 to the computer 172. For example, the user 108 initially uses a game access application to access the game program 117. During access to the portion of the game program 117, the media data 116 is sent from the codec 112 to the HMD 104 through the network 110 and the router 152 for display on the display screen of the HMD 104. During the time when the media data (e.g., media data 116, other media data, etc.) from the game cloud 102 is accessed for display on the HMD 104, the NIC 174 of the computer 172 downloads a portion of the game program 117 from the game cloud 102 through the network 110 and the router 152.

[0070] In some embodiments, when the console 172 accesses a portion of the game program 117, the media data (e.g., media data 116, other media data, etc.) is sent from the codec 112 to the HMD 104 via the network 110 for display on the display screen of the HMD 104. In these embodiments, the router 152 is bypassed to reduce the number of hops. After bypassing the router 152, during the time when the media data (e.g., media data 116, other media data, etc.) from the game cloud 102 is accessed for display on the HMD 104, the NIC 174 of the computer 172 downloads a portion of the game program 117 from the game cloud 102 via the network 110 and the router 152.

[0071] In many embodiments, a portion of the input data generated based on one or more head movements and / or one or more hand movements is sent from the HMD 104 to the codec 112 of the game cloud 102 via the router 152 and the network 110, and the remainder of the input data is sent from the communication circuit of the HMD 104 to the communication circuit 178 of the computer 172.

[0072] In various embodiments, a portion of the input data generated based on one or more hand movements is sent from the communication circuit of the HHC 106 to the codec 112 of the game cloud 102 via the router 152 and the network 110, and the remainder of the input data is sent from the communication circuit of the HHC 106 to the communication circuit 178 of the computer 172.

[0073] In some embodiments, a portion of the input data generated based on one or more head movements and / or one or more hand movements is sent from the HMD 104 to the codec 112 of the gaming cloud 102 via the network 110, and the remainder of the input data is sent from the communication circuit of the HMD 104 to the communication circuit 178 of the computer 172. In these embodiments, the router 152 is bypassed.

[0074] In some embodiments, a portion of the input data generated based on one or more hand movements is sent from the communication circuit of the HHC 106 to the codec 112 of the game cloud 102 via the network 110, and the remainder of the input data is sent from the communication circuit of the HHC 106 to the communication circuit 178 of the computer 172. In these embodiments, the router 152 is bypassed.

[0075] In various embodiments, media data (e.g., media data 116, other media data, etc.) generated by executing the game program 117 is sent from the codec 112 of the game cloud 102 to the HMD 104 through the network 110 and the router 152 to present the media data, thereby displaying the game on the display screen of the HMD 104, and media data generated by executing a portion of the game program 117 by the processor 176 of the computer 172 is sent from the communication circuit 178 of the computer 172 to the HMD 104 to display the game on the display screen. In these embodiments, the game cloud 102 and the computer 172 have synchronized game states. For example, the codec 112 sends the game state generated by executing the game program 117 to the NIC 174 of the computer 172 through the network 110 and the router 152 to inform the computer 172 of the game state. As another embodiment, the NIC 174 of the computer 172 sends the game state generated by executing the portion of the game program 117 on the computer 172 to the encoding and decoding 112 of the game cloud 102 through the router 152 and the network 110 so as to inform one or more game cloud servers of the game state.

[0076] In some embodiments, media data (e.g., media data 116, other media data, etc.) generated by executing the game program 117 is sent from the codec 112 of the game cloud 102 to the HMD 104 via the network 110 to present the media data, thereby displaying the game on the display screen of the HMD 104, and media data generated by the processor 176 of the computer 172 executing a portion of the game program 117 is sent from the communication circuit 178 of the computer 172 to the HMD 104 to display the game on the display screen. In these embodiments, when the media data is sent from the codec 112 to the HMD 104, the router 152 is bypassed. In these embodiments, the game cloud 102 and the computer 172 have a synchronized game state as described above.

[0077] In some embodiments, the media data (e.g., media data 116, other media data, etc.) generated by executing the game program 117 and sent from the codec 112 of the game cloud 102 through the network 110 and the router 152 to the HMD 104 for presentation on the display screen of the HMD 104 has a greater number of graphics than the media data generated by executing a portion of the game program 117 by the processor 176 of the computer 172.

[0078] In some embodiments, the media data (e.g., media data 116, other media data, etc.) generated by executing the game program 117 and sent from the codec 112 of the game cloud 102 to the HMD 104 over the network 110 for presentation of the media data on the display screen of the HMD 104 has a greater number of graphics than the media data generated by executing a portion of the game program 117 by the processor 176 of the computer 172. In these embodiments, the router 152 is bypassed when the media data is sent from the codec 112 of the game cloud 102 to the HMD 104 over the network 110.

[0079] In various embodiments, using HMD 104 ( Figures 1A to 1C ) to display two-dimensional or three-dimensional images.

[0080] Figure 2 2 is a diagram of an embodiment of a system 200 for illustrating the generation of input data based on one or more head movements and / or one or more hand movements. A user 108 is wearing an HMD 202 and holding an HHC 204. The HMD 202 is a HMD 104 ( Figures 1A to 1C ) and HHC 204 is an example of HHC 106 ( Figures 1A to 1C ) instance.

[0081] The HMD 202 includes one or more marks, such as a mark M1, a mark M2, and a mark M3, etc. Similarly, the HHC 204 includes one or more marks, such as a mark M4, a mark M5, and a mark M6, etc. Each mark may be a light emitting diode, a light source, a color, a reflective material, etc.

[0082] The video camera 206 captures an image of the HMD 202. When the head of the user 108 is tilted or moved, the position and location of the markers M1 to M3 change in the xyz coordinate system. The video camera 206 captures the image of the markers M1 to M3 and sends the image to the computer 172. The image of the markers M1 to M3 is an example of input data. The positioning of the HMD 202 in the three-dimensional space (X, Y, Z) can be determined by the processor 176 ( Figure 1C ) is determined based on the positioning of the markers M1 to M3 in the image. In addition, the inertial motion (e.g., yaw, pitch, and roll, etc.) of the HMD 202 is determined by the processor 176 ( Figure 1C ) is determined based on the movement of markers M1 to M3.

[0083] In various embodiments, the video camera 206 is movable. For example, the video camera 206 is attached to a robotic device, such as a multicopter, a robotic arm, a robot, a robotic vehicle, a robotic car, a quadcopter, etc. For example, the video camera 206 is attached with respect to the robotic device (e.g., below, above, to one side of the robotic device, etc.) so as to capture images of the HMD 202 and / or HHC 204. The HMD 202 moves as the head of the user 108 moves. In some embodiments, a digital camera is used instead of the video camera 206.

[0084] In some embodiments, the video camera 206 captures an image of the HHC 204. When the hand of the user 108 moves, the positioning and location of the markers M4 to M6 change in the coordinate system. The video camera 206 captures the image of the markers M4 to M6 and sends the image to the computer 172. The image of the markers M4 to M6 is an example of input data. The positioning of the HHC 204 in three-dimensional space (X, Y, Z) can be determined by the processor 176 of the computer 172 based on the positioning of the markers M4 to M6 in the image. In addition, the inertial motion (e.g., yaw, tilt, and roll, etc.) of the HHC 204 is determined by the processor 176 of the computer 172 based on the movement of the markers M4 to M6.

[0085] In some embodiments, the hand of the user 108 is marked with the markers M4 to M6 instead of the HHC 204 .

[0086] Figure 3 is a diagram of an embodiment of an HMD 250, which is an HMD 104 ( Figures 1A to 1C ). HMD 250 includes a video audio separator 254, a video decoder 255, a WAC 258, a stream buffer 259, one or more speakers 260, a user input circuit 262, a display screen 266, a microcontroller 268, an audio buffer 272, an external digital camera 274, an audio codec 276, an internal digital camera 278, a video buffer 280, a video audio synchronizer 282, a microphone 284, and a controller / computer communication circuit 287. The external digital camera 274 faces the real world environment of the user 108, and the internal digital camera 278 faces the user 108, such as the eyes, head, etc. of the user 108.

[0087] In many embodiments, speaker 260 is an audio circuit. In various embodiments, audio codec 276, audio buffer 272 and / or speaker 260 are audio circuits. In some embodiments, the combination of video decoder 255 and microcontroller 268 is referred to herein as a game processing circuit. In various embodiments, microcontroller 268 is a display circuit. Examples of display screens include LED screens, liquid crystal display (LCD) screens, liquid crystal on silicon screens, organic LED (OLED) screens, plasma screens, etc. Examples of external digital video cameras 274 include Playstation 4, manufactured by Sony Computer Entertainment, Inc.

[0088] The microcontroller 268 stores a rendering program 286 and an operating system 288. The rendering program 286 and the operating system 288 are stored in a storage device of the microcontroller 286 and executed by a microprocessor of the microcontroller 268. An example of the microcontroller 268 includes a low-cost microcontroller that includes a driver, such as an LCD driver, that drives (e.g., generates a signal to provide to, generates a signal to activate, etc.) an element of the display screen 266 (e.g., LCD) to display a game on the display screen 266. Another example of the microcontroller 268 includes a graphics processing unit (GPU) and a storage device.

[0089] In some embodiments, the storage device of the microcontroller 268 is not flash memory and random access memory (RAM). For example, the storage device of the microcontroller is a buffer. In various embodiments, the storage device of the microcontroller 268 is flash memory or RAM. Examples of user input circuit 262 include gyroscopes, magnetometers, and accelerometers. Examples of WAC 258 include NIC. In some embodiments, WAC 258 is referred to as communication circuit in this article.

[0090] From the network 110 or the router 152 ( Figure 1B to Figure 1C , Figure 2 ) receives the stream of the encoded media data into the stream buffer 259. Examples of the encoded media data received by the stream buffer 259 include the encoded media data 116 ( Figures 1A to 1C , Figure 2 ) and other encoded media data, etc. It should be noted that when the router 152 is coupled to the computer 172 ( Figure 2 ), the media data received from the computer 172 is stored in a buffer (not shown) of the HMD 250 rather than in the streaming buffer 259.

[0091] The WAC 258 accesses the stream of the encoded media data from the stream buffer 259 and depacketizes the stream. In addition, the WAC 258 also includes a decoder for decoding the encoded media data.

[0092] In computer 172 ( Figure 1C ) through router 152( Figure 1C ) in an embodiment of receiving a stream of encoded media data, the NIC 174 ( Figure 1C ) depacketizes and decodes the stream of encoded media data to produce decoded data, which is sent by router 152 to a buffer (not shown) of HMD 250 for storage.

[0093] The video audio separator 254 accesses the decoded data from the WAC 258 or a buffer (not shown) of the HMD 250. The video audio separator 254 separates the audio data from the video data within the decoded data.

[0094] The video audio separator 254 sends the audio data to the audio buffer 272 and sends the video data to the video buffer 280. The video audio synchronizer 282 synchronizes the video data stored in the video buffer 280 with the audio data stored in the audio buffer 272. For example, the video audio synchronizer 282 synchronizes the video data and the audio data using the play times of the video data and the audio data.

[0095] The audio codec 276 converts the synchronized audio data from a digital format to an analog format to generate an audio signal, and the speaker 260 plays the audio signal to generate sound. The video decoder 255 decodes the synchronized video data, such as from a digital form to an analog form, etc., to generate a video signal. The microcontroller 268 executes a rendering program 286 based on the analog video signal generated by the video decoder 255 to display the game on the display screen 266. In some embodiments, the game displayed on the display screen 266 is displayed in synchronization with the playback of the audio signal.

[0096] In addition, user 108 ( Figures 1A to 1C , Figure 2 ) speaks into microphone 284, which converts sound (e.g., voice) signals into electrical signals (e.g., audio signals). Audio codec 276 converts the audio signals from analog format to digital format to generate audio data, which is stored in audio buffer 276. The audio data stored in audio buffer 276 is an example of input data generated based on the voice of user 108. WAC 258 accesses the audio data from audio buffer 276 for transmission over network 110 ( Figures 1A to 1C , Figure 2)Send to the game cloud 102( Figures 1A to 1C , Figure 2 ) of the codec 112 ( Figures 1A to 1C ). For example, WAC 258 packets and encodes audio data accessed from audio buffer 276 for transmission to codec 112 via network 110.

[0097] In some embodiments, WAC 258 accesses audio data from audio buffer 276 for transmission via router 152 ( Figure 1B to Figure 1C , Figure 2 ) and Network 110( Figures 1A to 1C , Figure 2 ) is sent to the codec 112 of the game cloud 102 ( Figures 1A to 1C ). For example, WAC 258 packetizes and encodes audio data accessed from audio buffer 276 for transmission to codec 112 via router 152 and network 110.

[0098] Internal digital camera 278 captures user 108 ( Figures 1A to 1C , Figure 2 ) to generate image data, which is an example of input data generated based on one or more head movements. Similarly, the external digital camera 278 captures one or more images of the user's 108 hands and / or one or more images of markers located on the HMD 250 or on the user's 108 hands to generate image data. Image data generated based on markers located on the user's 108 hands or based on the movement of the user's 108 hands is an example of input data generated based on one or more hand movements. The image data captured by the digital cameras 274 and 278 are stored in the video buffer 280.

[0099] In some embodiments, the image data captured by the digital cameras 274 and 278 is stored in a buffer of the HMD 250 that is not the video buffer 280. In various embodiments, the image data captured by the digital cameras 274 and 278 is decoded by the video decoder 255 and sent to the microcontroller 268 to display the image on the display screen 266.

[0100] WAC 258 accesses image data captured by digital cameras 274 and 278 from video buffer 280 for transmission over network 110 ( Figures 1A to 1C , Figure 2 )Send to the game cloud 102( Figures 1A to 1C , Figure 2 ) of the codec 112 ( Figures 1A to 1C). For example, WAC 258 packets and encodes image data accessed from video buffer 280 for transmission to codec 112 via network 110.

[0101] In some embodiments, WAC 258 accesses video data from video buffer 280 for transmission via router 152 ( Figure 1B to Figure 1C , Figure 2 ) and Network 110( Figures 1A to 1C , Figure 2 ) is sent to the codec 112 of the game cloud 102 ( Figures 1A to 1C ). For example, WAC 258 packets and encodes video data accessed from video buffer 280 for transmission to codec 112 via router 152 and network 110.

[0102] The controller / computer communication circuit 287 receives media data from the computer 172 for storage in a buffer (not shown) of the HMD 250. In addition, the controller / computer communication circuit 287 receives media data from the HHC 106 ( Figures 1A to 1C , Figure 2 ) receives an input signal, converts the input signal from analog form to digital form to generate input data, and WAC 258 accesses the input data to be transmitted through network 110 ( Figures 1A to 1C , Figure 2 )Send to the game cloud 102( Figures 1A to 1C , Figure 2 ) of the codec 112 ( Figures 1A to 1C ). For example, WAC258 packets and encodes input data accessed from controller / computer communication circuit 287 for transmission to codec 112 via network 110.

[0103] In some embodiments, WAC 258 accesses input data from controller / computer communication circuit 287 for communication via router 152 ( Figure 1B to Figure 1C , Figure 2 ) and Network 110( Figures 1A to 1C , Figure 2 ) is sent to the codec 112 of the game cloud 102 ( Figures 1A to 1C ). For example, WAC 258 packets and encodes input data accessed from controller / computer communication circuit 287 for transmission to codec 112 via router 152 and network 110.

[0104] It should be noted that two separate communication circuits may be used in place of the controller / computer communication circuit 287, one for communicating with the computer 172 ( Figure 1B ) to communicate (e.g., receive, send, etc.) data, and another for communicating with the HHC 106 ( Figures 1A to 1C, Figure 2 ) to convey data.

[0105] In many implementations, the decoder of WAC 258 is located outside of WAC 258. In various implementations, stream buffer 259 is located within WAC 258.

[0106] In some embodiments, the HMD 250 does not include an external digital camera 274. In some embodiments, the HMD 250 includes any number of microcontrollers, any number of buffers, and / or any number of storage devices.

[0107] In various embodiments, the HMD 250 includes one or more batteries that provide power to components such as a video and audio separator 254, a wireless access card 258, a stream buffer 259, one or more speakers 260, a user input circuit 262, a display screen 266, a microcontroller 268, an audio buffer 272, an external digital camera 274, an audio codec 276, an internal digital camera 278, a video buffer 280, a video and audio synchronizer 282, a microphone 284, and a controller / computer communication circuit 287. The one or more batteries are charged with a charger (not shown) that plugs into an AC outlet.

[0108] In many embodiments, input data and / or media data is referred to herein as interactive media.

[0109] In some embodiments, HMD 250 includes a communication circuit (not shown) to facilitate peer-to-peer multi-channel communication between local users through pairing. For example, HMD 250 includes a transceiver that modulates the sound signal received from microphone 284 and sends the modulated signal to the transceiver of another HMD (not shown) through a channel. The transceiver of the other HMD demodulates the signal to provide it to the speaker of the other HMD to facilitate communication between users.

[0110] In various embodiments, the transceiver of HMD 250 uses different channels to communicate with different other HMDs. For example, the channel through which the modulated signal is sent to a first other HMD is different from the channel through which the modulated signal is sent to a second other HMD.

[0111] In some embodiments, WAC 258, user input circuit 262, microcontroller 268, and video decoder 255 are integrated into one or more separate circuit chips. For example, WAC 258, video decoder 255, and microcontroller 268 are integrated into one circuit chip, and user input circuit 262 is integrated into another circuit chip. As another example, each of WAC 258, user input circuit 262, microcontroller 268, and video decoder 255 is integrated into a separate circuit chip.

[0112] In some embodiments, a video camera is used instead of a digital camera. For example, a video camera is used instead of an external digital camera 274 ( Figure 3 ), and another video camera is used to replace the internal digital camera 278 ( Figure 3 ).

[0113] In some embodiments, the HMD 250 does not have an external digital camera 274 and / or does not have an internal digital camera 278. In these embodiments, the positioning and movement of the HMD 250 is performed by one or more servers of the game cloud 102 and / or by the computer 172 ( Figure 1C ) of the processor 176 ( Figure 1C ) is determined based on input data generated by user input circuit 262 and transmitted through network 110 ( Figures 1A to 1C , Figure 2 ) or through router 152( Figure 1B to Figure 1C , Figure 2 ) and network 110 or sent through router 152, computer 172 and network 110.

[0114] In some embodiments, when the internal digital camera 278 cannot see the eyes or head of the user 108 and / or when the video camera 206 ( Figure 2 ) cannot see the head of the user 108, the user input circuit 262 generates input data. In these embodiments, the positioning and movement of the HMD 250 is controlled by one or more servers of the game cloud 102 and / or by the computer 172 ( Figure 1C ) of the processor 176 ( Figure 1C ) is determined based on input data, the input data being transmitted through the network 110 ( Figures 1A to 1C , Figure 2 ) is sent to the game cloud 102 or through the router 152 ( Figure 1B to Figure 1C , Figure 2 ) and network 110 or sent through router 152, computer 172 and network 110.

[0115] In some embodiments, the HMD 250 (e.g., the WAC 258 of the HMD 250) switches between two or more modes. The first mode includes the HMD 250 communicating with the game cloud 102 ( Figures 1A to 1C , Figure 2 ) through network 110( Figures 1A to 1C , Figure 2 ) uses router 152 ( Figure 1B to Figure 1C , Figure 2 ) and computer 172( Figure 1C and Figure 2 ). The second mode includes using the router 152 when communicating data between the HMD 250 and the gaming cloud 102 via the network 110. In the second mode, the computer 172 is bypassed when communicating data. The third mode includes using the network 110 when communicating data between the HMD 250 and the gaming cloud 102 via the network 110. In the third mode, the router 152 and the computer 172 are bypassed when communicating data.

[0116] In some embodiments, user 108 uses HMD 104 and HHC 106, and the positioning and movement of HMD 104 and the positioning and movement of HHC 106 are represented in a single virtual game scene. For example, in a game (such as an immersive environment game, etc.) generated by executing game program 117 and displayed in HMD 104, HHC 106 is represented as a sword. In this embodiment, the game has a virtual game scene. When user 108 turns his / her head to the left, the left part of the game scene is shown to user 108, and when user 108 turns his / her head to the right, the right part of the game scene is shown to user 108. The display of the part of the game scene is based on the positioning and movement of HMD 104. In addition, when user 108 moves HHC 106 to the left and moves HMD 104 to the left, the sword is shown to the user to point left in the left part of the game scene. When the user 108 moves the HHC 106 to the left and the HMD 104 to the right, the right portion of the game scene is shown to the user and the sword is not displayed in the right portion. In addition, when the user 108 moves the HHC 106 to the right and the HMD 104 to the right, the sword is shown to the user pointing right in the right portion of the game scene. When the user 108 moves the HHC 106 to the right and the HMD 104 to the left, the right portion of the game scene is shown to the user 108 and the sword is not displayed in the right portion of the game scene. Displaying or not displaying the sword in a portion of the game scene is based on the positioning and movement of the HHC 106.

[0117] Figure 4A is a diagram of an embodiment of an image 300, which is displayed on the HMD 250 ( Figure 3) on the display screen 266 for accessing the wireless network. The image 300 is on the microcontroller 268 ( Figure 3 ) executes the network access application by the microcontroller 268 ( Figure 3 ). Image 300 includes a list of networks (e.g., wireless network A, wireless network B, wireless network C, etc.) to which HMD 250 can connect in order to access game program 117 from game cloud 102. User 108 selects one of wireless network A, wireless network B, wireless network C by performing one or more head movements and / or one or more hand movements to connect HMD 250 to the wireless network.

[0118] In many embodiments, image 300 includes any number of the wireless networks listed.

[0119] Figure 4B is a diagram of an embodiment of an image 310, which is displayed on a computer 172 ( Figure 2 ) on a display screen 312 for accessing a wireless network. Image 310 is displayed on a processor 176 ( Figure 2 ) is generated by the GPU of the computer 172 when executing a network access application. The image 310 includes a list of networks (e.g., wireless network A, wireless network B, wireless network C, etc.) to which the HMD 250 can connect through the router 152 and the computer 172 to access the game program 117 from the game cloud 102. The user 108 selects one of the wireless networks A, B, and C by performing one or more head movements and / or one or more hand movements to connect the HMD 250 to the wireless network through the router 152 and the computer 172.

[0120] In many embodiments, image 310 includes any number of wireless networks listed.

[0121] Figure 5A is a diagram of a game image 320 displayed on a display screen 321, which is a display screen 266 ( Figure 3 ) is an example of a microcontroller 268 ( Figure 3 ) displays the peripheral area 322 of the image 320 with a low resolution, and the microcontroller 268 displays the central area 324 with a high resolution. The high resolution is higher than the lower ratio. The peripheral area 322 is adjacent to the edges E1 to E4 of the display screen 321.

[0122] In some embodiments, edge E1 is perpendicular to edge E2 and perpendicular to edge E4. Edge E3 is perpendicular to edges E2 and E4 and parallel to edge E1. In various embodiments, edge E1 is substantially perpendicular to edge E2 and perpendicular to edge E4 (e.g., at an angle ranging from 87 to 93 degrees, etc.). Edge E3 is substantially perpendicular to edges E2 and E4 and substantially parallel to edge E1.

[0123] It should be noted that in some embodiments, the display screen 321 is any shape, such as rectangular, square, circular, oval, polygonal, curved, etc.

[0124] Based on internal digital camera 278 ( Figure 3 ) captures the gaze image, the microcontroller 268 ( Figure 3 ) divides the display screen 321 into regions 322 and 324. For example, after determining that the user 108 ( Figures 1A to 1C , Figure 2 ) looks at the central area 324, the microcontroller 268 determines that the central area 324 has a high resolution, determines that the peripheral area 322 has a low resolution, and executes the rendering program 286 to display the central area 324 with a high resolution and the peripheral area 322 with a low resolution.

[0125] Figure 5B 330 is a diagram of an embodiment of an image 330 displayed on a display screen 321. Image 330 has a top peripheral region 332 and a bottom peripheral region 334. Top peripheral region 332 is adjacent to edge E1, and bottom peripheral region 334 is adjacent to edge E3. Peripheral regions 332 and 334 have a low resolution, and middle region 336 has a high resolution. Middle region 336 is located adjacent to edges E2 and E4 and between regions 332 and 334.

[0126] Based on internal digital camera 278 ( Figure 3 ) captures the gaze image, the microcontroller 268 ( Figure 3 ) divides the display screen 321 into regions 332, 334, and 336. For example, after determining that the user 108 ( Figures 1A to 1C , Figure 2 ) looks at the middle area 336, the microcontroller 268 determines that the middle area 336 has a high resolution, determines that the peripheral areas 332 and 334 have a low resolution, and executes the rendering program 286 to display the middle area 336 with a high resolution and the peripheral areas 332 and 334 with a low resolution.

[0127] In some embodiments, the resolution of the peripheral region 332 is different from the resolution of the peripheral region 334. For example, the peripheral region 332 has a medium resolution between a low resolution and a high resolution, and the peripheral region 334 has a low resolution. As another example, the peripheral region 334 has a medium resolution and the peripheral region 332 has a low resolution.

[0128] Figure 5C 340 is a diagram of an embodiment of an image 340 displayed on a display screen 321. Image 340 has a right peripheral region 342 and a left peripheral region 344. Right peripheral region 342 is adjacent to edge E2, and left peripheral region 344 is adjacent to edge E4. Peripheral regions 342 and 344 have a low resolution, and middle region 346 has a high resolution. Middle region 346 is located adjacent to edges E1 and E3 and between regions 342 and 344.

[0129] Based on internal digital camera 278 ( Figure 3 ) captures the gaze image, the microcontroller 268 ( Figure 3 ) divides the display screen 321 into regions 342, 344, and 346. For example, after determining that the user 108 ( Figures 1A to 1C , Figure 2 ) looks at the middle area 346, the microcontroller 268 determines that the middle area 346 has a high resolution, determines that the peripheral areas 342 and 344 have a low resolution, and executes the rendering program 286 to display the middle area 346 with a high resolution and the peripheral areas 342 and 344 with a low resolution.

[0130] In some embodiments, the resolution of the peripheral region 342 is different from the resolution of the peripheral region 344. For example, the peripheral region 342 has a medium resolution between a low resolution and a high resolution, and the peripheral region 344 has a low resolution. As another example, the peripheral region 344 has a medium resolution and the peripheral region 342 has a low resolution.

[0131] In various embodiments, any other head motion (rather than gaze) may be used to determine that one or more areas of display screen 321 have high resolution and to determine that one or more areas of display screen 321 have low resolution or low and medium resolution.

[0132] In some embodiments, the peripheral area of ​​the display screen is any shape, such as curved, semicircular, polygonal, etc., and the central area or middle area of ​​the display screen is any shape, such as rectangular, circular, polygonal, elliptical, curved, etc.

[0133] Figure 6350, 352, 354, 356, 358, and 360. Each HHC 350, 352, 354, 356, 358, and 360 is a HHC 106 ( Figures 1A to 1C , Figure 2 ). As shown, HHC 350 has a boomerang shape, HHC 352 has a microphone shape, and HHC 354 has a steering wheel shape. In addition, HHC 356 is a glove that user 108 wears on his / her hands, HHC 358 is a jacket, top, shirt, etc. that user 108 wears on his / her upper body, and HHC 360 is a pair of pants that user 108 wears on his / her lower body.

[0134] Each HHC 350, 352, 354, 356, 358 and 360 includes one or more feedback mechanisms, such as sensors, mechanical compressors, mechanical weight gainers, mechanical weight lossers, mechanical decompressors, vibration mechanisms, tactile feedback mechanisms, heaters, coolers, electrical shockers, etc., which are used to play the game by executing the game program 117 ( Figures 1A to 1C , Figure 2 ) provides feedback to user 108 during the game displayed.

[0135] In various embodiments, the user 108 wears the HHC 356 on his / her hand in addition to the HMD 104, and pets the virtual dog while playing the game displayed by executing the game program 117. The haptic feedback mechanism provides haptic feedback to the user 108 so as to allow the user 108 to feel as if the user 108 is petting a real dog. When the user 108 lifts the virtual dog, the weight increaser is used to allow the user 108 to feel as if the user 108 is lifting a real dog.

[0136] In some embodiments, the user 108 wears the HHC 358 in addition to the HMD 104 and performs a virtual scuba dive in a virtual ocean. The mechanical weight increasers increase the weight applied to the user 108 so as to allow the user 108 to feel as if the user 108 is scuba diving in a real ocean and feel the weight of the water above the user 108. Similarly, as the user 108 approaches the surface of the ocean, the mechanical weight reducers reduce the weight applied to the user 108.

[0137] In some embodiments, each HHC 350, 352, 354, 356, 358, and 360 includes one or more fixed reference objects (examples of which are provided above) that are used to determine the position and movement of the HHC.

[0138] In various embodiments, the HHC 106 has any shape, such as rectangular, circular, oval, polygonal, curvilinear, spiral, etc.

[0139] Fig. 7A 370 is a diagram of an embodiment of an HMD 370. The HMD 370 is an HMD 250 ( Figure 3 ) is an example of an HMD 370. HMD 370 includes straps 372 and 374. When user 108 wears HMD 370, the straps are attached to user 108 ( Figures 1A to 1C , Figure 2 ) behind the head. In addition, the HMD 370 includes headphones 376A and 376B (eg, speakers, etc.) that emit sounds that are similar to those emitted by executing the game program 117 ( Figures 1A to 1C ) to play the game. HMD 370 includes lenses 378A and 378B, which allow user 108 to see the sounds of the game being played by executing game program 117 ( Figures 1A to 1C ) to play and is displayed on the display screen 266 ( Figure 3 ) on the headset. The groove 380 rests on the nose of the user 108 to support the HMD 370 on the nose.

[0140] In some embodiments, the user 108 puts on the HMD 370 similar to how the user 108 puts on sunglasses, glasses, or reading glasses.

[0141] Figure 7B is an isometric view of an embodiment of HMD 400, which is HMD 250 ( Figure 3 ). In some embodiments, HMD 400 is a product of research and development by Sony Computer Entertainment America LLC. HMD 400 includes a head support frame 402. HMD 400 further includes indicia, such as: LEDs 404A and 404B, infrared light emitters 406A, 406B, 406C, 406D, 406E, 406F, 406G, and 406H. Examples of infrared light emitters include LEDs. HMD 400 includes an on / off switch 408, which allows HMD 400 to be turned on or off.

[0142] The markers are used to determine the position and movement of the HMD 400. For example, as the user 108 ( Figures 1A to 1C , Figure 2 )'s head movement, external digital camera 274 ( Figure 3 ) changes the size of the image of LEDs 404A and 404B captured. Based on the size, computer 172 ( Figure 3 ) processor or game cloud 102 ( Figures 1A to 1C , Figure 2 ) determines the positioning and movement of the HMD 400 in the coordinate system.

[0143] Figure 8 is compatible with HHC 106 and HMD 104 ( Figures 1A to 1C , Figure 2 ) is a block diagram of an embodiment of a console 500 that is docked. The console 500 can be used to execute and present a portion of the game program 117. The console 500 can be compatible with the HHC 106 and the HMD 104 ( Figures 1A to 1C , Figure 2 ) and the game program 117. The console 500 has various peripheral devices that can be connected to the console 500. The console 500 has: a unit processor 528, A dynamic random access memory (XDRAM) unit 526, a reality synthesizer graphics unit 530 with a dedicated video random access memory (VRAM) unit 532, and an input / output (I / O) bridge 534. The console 500 also includes a Blu-ray player for reading from a disk 540a. A Blu-ray Disc Read Only Memory (BD-ROM) optical disc reader 540, and a removable slot-load hard disk drive (HDD) 536, the Blu A BD-ROM optical disk reader and a removable slot-load hard disk drive (HDD) can be accessed through I / O bridge 534. Optionally, console 500 also includes a memory card reader for reading Compact Flash cards, Memory Cards, and other memory cards. A memory card reader 538 for a memory card or the like may similarly be accessed via the I / O bridge 534 .

[0144] The I / O bridge 534 is also connected to a Universal Serial Bus (USB) 2.0 port 524, a Gigabit Ethernet port 522, an IEEE 802.11b / g wireless network (Wi-Fi) port 520, and a Bluetooth capable port. Wireless link port 518.

[0145] In operation, I / O bridge 534 handles all wireless data, Universal Serial Bus (USB) data, and Ethernet data, including data from one or more game controllers 502 and 503 and data from HMD 505, which is HMD 250 ( Figure 3 ). For example, when user 108( Figures 1A to 1C , Figure 2 ) in playing by executing the game program 117 ( Figure 1A), the I / O bridge 534 receives input data from the game controller 502 or 503 or from the HMD 505 via the Bluetooth link and passes the input data to the unit processor 528, which updates the current state of the game accordingly. Each game controller 502 and 503 is a HHC 106 ( Figures 1A to 1C , Figure 2 ) instance.

[0146] The wireless port, USB port and Ethernet port also provide connectivity for other peripheral devices besides game controllers 502 and 503 and HMD 505, such as: remote control 504, keyboard 506, mouse 508, such as Sony Playstation 501, and the like. Portable entertainment device 510, such as Video camera 512, microphone headset 514, and microphone 515. Thus, such peripheral devices may in principle be connected wirelessly to console 500, for example, portable entertainment device 510 may communicate via a Wi-Fi dedicated connection, while microphone headset 514 may communicate via a Bluetooth link.

[0147] The provision of these interfaces means that the console 500 is potentially also compatible with other peripheral devices such as digital video recorders (DVRs), set-top boxes, digital video cameras, portable media players, voice over IP phones, mobile phones, printers and scanners.

[0148] Additionally, a legacy memory card reader 516 may be connected to the console 500 via the USB port 524 , thereby allowing the reading of memory cards 548 of the type used by the console 500 .

[0149] The game controllers 502 and 503 and HMD 505 may be operable to communicate wirelessly with the console 500 via a Bluetooth link, or connected to a USB port, thereby also providing power for charging the batteries of the game controllers 502 and 503 and HMD 505. In some embodiments, each of the game controllers 502 and 503 and HMD 505 may also include: memory; processor; memory card reader; permanent storage such as flash memory; light emitters such as illuminated spherical portions, LEDs or infrared lights; microphones and speakers for ultrasonic communication; acoustic chambers; digital cameras; internal clocks; recognizable shapes such as spherical portions facing the game console; wireless communications using protocols such as Bluetooth, Wi-Fi, etc.

[0150] Game controller 502 is designed to be used by both hands of user 108, and game controller 503 is a single-handed controller with attachments. HMD 505 is designed to be mounted on top of the head and / or in front of the eyes of user 108. In addition to one or more analog joysticks and conventional control buttons, each game controller 502 and 503 can also accept digital position determination. Similarly, HMD 505 can accept digital position determination. Therefore, gestures and movements of user 108 of game controller 502 and 503 and HMD 505 can be converted into inputs for the game in addition to or in place of conventional button or joystick commands. Optionally, such as Playstation 3, the game controller 502 and 503 can be used to control the user 108 to play the game. TM Other wirelessly enabled peripherals for portable devices can be used as controllers. TM In the case of a portable device, additional game or control information (e.g., control instructions or number of lives) may be provided on the device's display screen. Other alternative or supplemental control devices may also be used, such as: a dance mat (not shown); a light gun (not shown); and a steering wheel and pedals (not shown) or a custom controller, such as a single or multiple large buttons for quick response quiz games (also not shown).

[0151] The remote control 504 is also operable to wirelessly communicate with the console 500 via a Bluetooth link. The remote control 504 includes a Blu-Ray TM The operation of the optical disc BD-ROM reader 540 and the controls suitable for navigating the contents of the optical disc.

[0152] In addition to conventional pre-recorded and recordable CDs and so-called Super Audio CDs, Blu-Ray TM The optical disc BD-ROM reader 540 is also operable to read CD-ROMs compatible with the console 500. In addition to conventional pre-recorded and recordable DVDs, the reader 540 is also operable to read digital video disc ROMs (DVD-ROMs) compatible with the console 500. The reader 540 is further operable to read BD-ROMs compatible with the console 500 as well as conventional pre-recorded and recordable Blu-Ray discs.

[0153] The console 500 is operable to supply audio and video generated or decoded by the reality synthesizer graphics unit 530 to a display and sound output device 542, such as a monitor or television having a display screen 544 and one or more speakers 546, via an audio connector 550 and a video connector 552. The audio connector 550 may include conventional analog and digital outputs, while the video connector 552 may variously include component video, S-video, composite video, and one or more high-definition multimedia interface (HDMI) outputs. Thus, the video output may be in a format such as PAL or NTSC, or in high definition at 520p, 1080i, or 1080p.

[0154] Audio processing (generation, decoding, etc.) is performed by the unit processor 528. The operating system of the console 500 supports 5.1 surround sound, Theater Surround (DTS), and Decoding of the CD's 7.1 surround sound.

[0155] In some embodiments, a video camera (e.g., video camera 512, etc.) includes a single charge coupled device (CCD), an LED indicator, and hardware-based real-time data compression and encoding means so that compressed video data can be transmitted in an appropriate format (e.g., based on the MPEG (Motion Picture Experts Group) standard within an image) so that it can be decoded by the console 500. The LED indicator of the video camera 512 is arranged to illuminate in response to appropriate control data from the console 500 so as to, for example, indicate unfavorable lighting conditions. Embodiments of the video camera 512 can be connected to the console 500 differently via USB, Bluetooth, or Wi-Fi communication ports. Embodiments of the video camera can include one or more associated microphones and can also transmit audio data. In embodiments of the video camera, the CCD can have a resolution suitable for high-definition video capture. In use, the images captured by the video camera can, for example, be incorporated into a game or interpreted as a game control input. In another embodiment, the video camera is an infrared camera suitable for detecting infrared light.

[0156] In various embodiments, in order to successfully communicate data with a peripheral device such as, for example, a video camera or a remote control through one of the communication ports of console 500, a piece of software, such as a device driver, is provided.

[0157] In some embodiments, the above system devices (including console devices, portable HHCs, and portable HMDs) allow the HHCs and portable HMDs to display and capture game programs 117 ( Figures 1A to 1C) interactive session. The console device starts an interactive session of the game program 117, which defines the interactivity between the user 108 and the game program 117. The system device determines that the user 108 ( Figures 1A to 1C , Figure 2 ) is operated by the console device. The console device determines the current state of the game program 117 based on the interactivity between the user 108 and the game program 117. The system device tracks the position and orientation of the HHC and / or HMD during the interactive session between the user 108 and the game program 117. The system device generates a separator video stream for the interactive session based on the current state of the game program 117 and the tracked position and orientation of the HHC and / or HMD. In some embodiments, the HHC presents the separator video stream on the display screen of the HHC. In various embodiments, the HMD presents the separator video stream on the display screen of the HMD.

[0158] Fig. 9 6 is a block diagram of a gaming system 600 according to various embodiments described in the present disclosure. In some embodiments, gaming system 600 is a cloud gaming system in which one or more resources (e.g., servers, storage devices, etc.) are used to execute VMs (e.g., operating systems, software applications, etc.) after a relationship between the resources and the VMs is established by a virtual machine monitor. Gaming system 600 is used to provide video streaming to one or more clients 610 via network 615, which is network 110 ( Figures 1A to 1C , Figure 2 ). The gaming system 600 includes a video server system 620 and an optional game server 625. The video server system 620 is used to provide a video stream to one or more clients 610 with a certain quality of service. For example, the video server system 620 may receive a video game (e.g., by executing the game program 116 ( Figures 1A to 1C ) or game commands for viewing angles within the video game, and provides updated video streams reflecting such state changes to the client 610 with minimal lag time. The video server system 620 may be used to provide video streams in a variety of alternative video formats (including formats not yet defined). In addition, the video stream may include video clips for presentation to the user 108 ( Figures 1A to 1C , Figure 2 ) video frames. Typical frame rates are 30 frames per second, 60 frames per second, and 620 frames per second. However, various embodiments described in the present disclosure include higher or lower frame rates.

[0159] Client 610 (each referred to herein as 610A, 610B, etc.) may include HMD 104 ( Figures 1A to 1C , Figure 2)、HHC 104( Figures 1A to 1C , Figure 2 ) terminal, personal computer, game console, tablet computer, phone, set-top box, wireless device, digital board, stand-alone device and / or the like. Client 610 is used to receive the encoded video stream, decode the video stream, and present the resulting video to user 108 (e.g., game player). The process of receiving the encoded video stream and / or decoding the video stream includes storing each video frame in a receiving buffer of the client. The video stream can be presented to user 108 on a display integral to client 610 or on a separate device such as a monitor or television.

[0160] Client 610 is optionally used to support more than one game player. For example, a game console can be used to support two, three, four or more simultaneous players. Each of these players can receive a separate video stream, or a single video stream can include multiple regions of the frame generated specifically for each player (e.g., generated based on each player's perspective).

[0161] The clients 610 are optionally geographically dispersed. The number of clients included in the gaming system 600 may vary widely from one or two to thousands, tens of thousands, or more. As used herein, the term "game player" is used to refer to a person who plays a game, and the term "game playing device" is used to refer to a device used to play a game (e.g., by executing the game program 116 ( Figures 1A to 1C ) to play games, etc.). In some embodiments, a game playing device may refer to multiple computing devices that cooperate to deliver a gaming experience to the user 108. For example, a game console and HMD 104 ( Figures 1A to 1C , Figure 2 ) may cooperate with the video server system 620 to communicate the game seen through the HMD 104. In one embodiment, the game console receives the video stream from the video server system 620, and the game console forwards the video stream or updates to the video stream to the HMD 104 for presentation.

[0162] Client 610 is used to receive video streams through network 615. Network 615 can be any type of communication network, including telephone network, Internet, wireless network, power line network, local area network, wide area network, private network and / or similar network. In some embodiments, video streams are communicated through standard protocols such as TCP / IP or UDP / IP. In many embodiments, video streams are communicated through proprietary standards.

[0163] A typical example of one of the clients 610 is a personal computer that includes a processor, non-volatile memory, a display, decoding logic, network communication capabilities, and one or more input devices. The decoding logic may include hardware, firmware, and / or software stored on a non-transitory computer-readable medium.

[0164] The client 610 may (but need not) further include a system for modifying the received video. For example, the client may be configured to: perform further rendering, overlay one video image on another video image, crop a video image, and / or the like. As another example, the client 610 may be configured to receive various types of video frames, such as intra-coded pictures (I frames), predicted frames (P frames), and bi-directionally predicted picture frames (B frames), and process these frames into images for displaying a game to the user 108, for example, by executing the game program 116 ( Figures 1A to 1C ) to play games, etc. In some embodiments, components of client 610 are used to perform further rendering, shading, conversion to 3-D, or similar operations on the video stream. Components of client 610 are optionally used to receive more than one audio or video stream.

[0165] The input devices of client 610 may include, for example, a one-handed game controller, a two-handed game controller, a gesture recognition system, a gaze recognition system, a speech recognition system, a keyboard, a joystick, a pointing device, a force feedback device, a motion and / or position sensing device, a mouse, a touch screen, a neural interface, a camera, input devices not yet developed, and / or the like.

[0166] The video stream and optional audio stream received by the client 610 are generated and provided by the video server system 620. The video stream includes video frames, and the audio stream includes audio frames. The video frames, including pixel information in appropriate data structures, largely constitute the images displayed to the user 108. As used herein, the term "video frame" is used to refer to a frame that primarily includes the following information, which is used to constitute (e.g., implement, etc.) the game played by the user 108 (e.g., by executing the game program 116 ( Figures 1A to 1C ) to play games, etc. Most of the teachings in this article about "video frames" can also apply to "audio frames".

[0167] In various embodiments, client 610 is used to receive input from user 108. These inputs may include inputs to change the video game (e.g., by executing game program 116 ( Figures 1A to 1C) to play the game, etc.) or otherwise affect the game play. The game commands may be received using an input device (e.g., user input circuitry 262 (FIG. 4), etc.) and / or may be automatically generated by computing instructions executed on client 610. The received game commands are communicated from client 610 to video server system 620 and / or game server 625 via network 615. For example, the game commands are communicated to game server 625 via video server system 620.

[0168] In some embodiments, separate copies of the game commands are communicated from the client 610 to the game server 625 and the video server system 620 .

[0169] The communication of game commands optionally depends on the identity of the command.Game commands are optionally communicated from Client 610A via different routes or communication channels used to provide audio or video streams to Client 610A.

[0170] The game server 625 is optionally operated by a different entity than the video server system 620. For example, the game server 625 may be operated by a game publisher. In this embodiment, the video server system 620 is optionally viewed as a client by the game server 625, and is optionally used to appear (from the perspective of the game server 625) as a client executing a game engine. Communications between the video server system 620 and the game server 625 optionally occur over the network 615. Thus, the game server 625 may be a multi-player game server that sends game state information to multiple clients, one of which is the video server system 620.

[0171] The video server system 620 can be used to simultaneously communicate with multiple instances of the game server 625. For example, the video server system 620 can be used to provide multiple different video games to different users. Each of these different video games can be supported by a different game server 625 and / or published by a different entity.

[0172] In some embodiments, a number of geographically distributed instances of the video server system 620 are used to provide game videos to a plurality of different users. Each of these instances of the video server system 620 can communicate with the same instance of the game server 625.

[0173] The communication between the video server system 620 and the game server 625 is optionally carried out through a dedicated communication channel. For example, the video server system 620 can be connected to the game server 625 through a high-bandwidth channel, and the high-bandwidth channel is dedicated to the communication between these two systems.

[0174] The video server system 620 includes at least: a video source 630, an I / O device 645, a processor 650, and a storage body 655. The video server system 620 may include a computing device or be distributed among multiple computing devices. These computing devices are optionally connected through a communication system such as a local area network.

[0175] Video source 630 is used to provide video streams, for example, streaming video or a series of video frames forming a motion picture. In some embodiments, video source 630 includes a video game engine and rendering logic.

[0176] The video game engine is used to: receive game commands from the player and maintain the video game based on the received commands (e.g., by executing the game program 116 ( Figures 1A to 1C ) to play the game, etc.). This game state includes the positioning of objects in the game environment, and the viewing angle. The game state may also include the properties, images, colors and / or textures of the objects. The game state is maintained based on game rules and game commands (e.g., move, turn, attack, set focus, interact, use and / or similar commands). Parts of the game engine are optionally placed in the game server 625. The game server 625 may maintain a copy of the game state based on game commands received from multiple players using geographically dispersed clients. In these cases, the game server 625 provides the game state to the video source 630, stores a copy of the game state in the video source 630 and performs presentation. The game server 625 may receive game commands from the client 610 via the network 615, and / or may receive game commands via the video server system 620.

[0177] The video source 630 includes rendering logic, such as hardware, firmware, and / or software stored on a non-transitory computer-readable medium such as storage 655. This rendering logic is used to create video frames of a video stream based on the game state. All or part of the rendering logic is optionally placed in the GPU. The rendering logic includes processing stages for determining the three-dimensional spatial relationship between multiple objects and / or for applying appropriate textures, etc., based on the game state and viewing angle. The rendering logic produces raw video, which is then often encoded and then communicated to the client 610. For example, it can be based on Adobe The raw video is encoded using MPEG-4, MPEG-4, MPEG-5, MPEG-6, MPEG-7, MPEG-8, MPEG-1, MPEG-2, MPEG-3, MPEG-4A, MPEG-5B, MPEG-6C, MPEG-7, MPEG-8, MPEG-10, MPEG-11, MPEG-12, MPEG-13, MPEG-14, MPEG-15, MPEG-16, MPEG-17, MPEG-18, MPEG-19, MPEG-21, MPEG-22, MPEG-23, MPEG-24, MPEG-25, MPEG-264, MPEG-27, MPEG-28, MPEG-29, MPEG-3A, MPEG-3B, MPEG-3C ...B,

[0178] A video stream is characterized by a frame size and a frame rate. Typical frame sizes include 800x600, 1280x720 (e.g., 720p), 1024x768, but any other frame size may be used. The frame rate is the number of video frames per second. A video stream may include different types of video frames. For example, the H.264 standard includes "P" frames and "I" frames. An I frame includes information for refreshing all macroblocks / pixels on a display device, while a P frame includes information for refreshing a subset of the macroblocks / pixels. The data size of a P frame is smaller than an I frame. As used herein, the term "frame size" is intended to refer to the number of pixels within a frame. The term "frame data size" is intended to refer to the number of bytes required to store the frame.

[0179] In some embodiments, video source 630 includes a video recording device such as a camera. This camera can be used to generate delayed video or live video that can be included in the video stream of the computer game. The resulting video stream optionally includes the rendered images and images recorded using a still camera or a video camera. Video source 630 may also include a storage device for storing previously recorded video to be included in the video stream. Video source 630 may also include: a memory device for detecting HMD 104, HHC 106, or user 108 ( Figures 1A to 1C , Figure 2 )'s movement or positioning; and logic for determining game state or generating video based on the detected motion and / or positioning.

[0180] Video source 630 is optionally used to provide an overlay for placing on other videos. For example, these overlays may include a command interface, a login command, a message to the game player, an image of other game players, a video feed of other game players (e.g., a webcam video). In some embodiments where client 610A includes a touch screen interface or a gaze detection interface, the overlay may include a virtual keyboard, a joystick, a touch pad, and / or the like. In one embodiment of an overlay, the player's voice is overlaid on the audio stream. Video source 630 optionally further includes one or more audio sources.

[0181] In an embodiment where a video server system 620 is used to maintain a game state based on input from more than one player, each player may have a different perspective, including positioning and viewing direction. Video source 630 is optionally used to provide a separate video stream to each player based on the player's perspective. In addition, video source 630 can be used to provide different frame sizes, frame data sizes, and / or encodings to each of clients 610. Video source 630 is optionally used to provide 3-D video.

[0182] I / O device 645 is used for video server system 620 to send and / or receive information, such as video, commands, requests for information, game status, gaze information, device movement, device location, user movement, client identity, player identity, game commands, security information, audio, and / or similar information. I / O device 645 includes communication hardware such as a network interface card or a modem. I / O device 645 is used to communicate with game server 625, network 615, and / or client 610.

[0183] Processor 650 is used to execute logic, such as software, included in the various components of video server system 620 discussed herein. For example, processor 650 can be programmed with software instructions to perform the functions of video source 630, game server 625, and / or client qualifier 660. Video server system 620 optionally includes more than one instance of processor 650. Processor 650 can also be programmed with software instructions to execute commands received by video server system 620 or coordinate the operation of various elements of game system 600 discussed herein. Processor 650 can include one or more hardware devices. Processor 650 is an electronic processor.

[0184] Storage 655 includes non-temporary analog and / or digital storage devices. For example, storage 655 may include an analog storage device for storing video frames. Storage 655 may include a non-temporary computer-readable digital storage, such as a hard drive, an optical drive, or a solid-state storage. Storage 615 (e.g., by an appropriate data structure or file system) is used to store video frames, artificial frames, video streams including video frames and artificial frames, audio frames, audio streams, and / or the like. Storage 655 is optionally distributed among multiple devices. In some embodiments, storage 655 is used to store software components of video source 630 discussed elsewhere herein. These components can be stored in a format that can be supplied at any time when needed.

[0185] The video server system 620 optionally further includes a client qualifier 660. The client qualifier 660 is used to remotely determine the capabilities of a client such as client 610A or 610B. These capabilities may include the capabilities of the client 610A itself and the capabilities of one or more communication channels between the client 610A and the video server system 620. For example, the client qualifier 660 can be used to test a communication channel through the network 615.

[0186] The client qualifier 660 can manually or automatically determine (e.g., discover) the capabilities of the client 610A. Manual determination includes: communicating with the user 108 of the client 610A, and asking the user 108 to provide capabilities. For example, in some embodiments, the client qualifier 660 is used to display images, text, and / or the like on the web browser of the client 610A. In some embodiments, the client 610A is an HMD including a web browser, such as HMD104, etc. In another embodiment, the client 610A is a game console with a web browser, which can be displayed on the HMD. The displayed object requests the user 108 to enter information about the client 610A, such as the operating system, processor, video decoder type, network connection type, display resolution, etc. The information entered by the user 108 is communicated back to the client qualifier 660.

[0187] Automatic determination can be performed, for example, by executing an agent program on client 610A and / or by sending a test video to client 610A. The agent program may include computing instructions embedded in a web page or installed as an add-on, such as a java script. The agent program is optionally provided by client qualifier 660. In various embodiments, the agent program can find: the processing power of client 610A, the decoding and display capabilities of client 610A, the latency reliability and bandwidth of the communication channel between client 610A and video server system 620, the display type of client 610A, the firewall present on client 610A, the hardware of client 610A, the software executed on client 610A, the registry entries within client 610A, and / or the like.

[0188] The client qualifier 660 includes hardware, firmware, and / or software stored on a non-transitory computer-readable medium. The client qualifier 660 is optionally placed on a computing device separate from one or more other elements of the video server system 620. For example, in some embodiments, the client qualifier 660 is used to determine the characteristics of the communication channel between the client 610 and one or more instances of the video server system 620. In these embodiments, the information discovered by the client qualifier can be used to determine which instance of the video server system 620 is most suitable for conveying streaming video to one of the clients 610.

[0189] In some embodiments, the media data includes real world environment data, virtual world environment data, or a combination thereof. Real world environment data includes data generated by one or more images of a real world environment, such as a real car, a real person, a real geographic location, a real structure, a real house, a real tree, a real object, a real creature, a real non-living thing, etc. Real world environment data is generated by a camera (e.g., a video camera, an audio camera, etc.) when the camera captures an image of the real world environment. Virtual world environment data is data generated by a computer (e.g., a processor executing the game program 117, etc.).

[0190] In many embodiments, one or more users (e.g., user 108, another user, etc.) can interact with a virtual world. As used herein, the term virtual world means a representation of a fictitious (e.g., virtual, computer-generated, etc.) environment having interaction rules simulated by one or more processors, which a real user can perceive through one or more display screens and / or can interact with through one or more user interface devices. As used herein, the term user interface device refers to a real device that a user 108 can use to send input to the virtual world or receive output from the virtual world, such as HMD 104, HHC 106 ( Figures 1A to 1C , Figure 2 ) etc. The virtual world may be composed of one or more processor modules (e.g., game cloud 102 ( Figures 1A to 1C , Figure 2 ) of one or more servers, microcontroller 268 ( Figure 3 ), processor 176 ( Figure 1C ) etc.) to simulate. One or more processor modules can be connected via network 110 ( Figures 1A to 1C , Figure 2 ) are linked together. User 108 may interact with the virtual world via a user interface device, which may communicate with the processor module and other user interface devices via network 110. Certain aspects of the virtual world may be presented to user 108 in graphical form on a graphical display screen, such as, for example, a computer monitor, a television monitor, display screen 266, a display screen of HHC 106, or the like. Certain other aspects of the virtual world may be presented to user 108 via speakers (e.g., speakers 260( Figure 3 ) etc.) in an audible form to the user, and the speaker may be associated with the graphic display.

[0191] Within a virtual world, users may be represented by avatars. Each avatar within a virtual world may be associated uniquely with a different user. The name or pseudonym of user 108 may be displayed next to the avatar so that another user may easily identify user 108. User 108's interaction with the virtual world may be represented by one or more corresponding actions of the avatar. For example, when user 108 performs a head action, the avatar performs the same head action. As another example, when user 108 performs a hand action, the avatar performs the same hand action in the virtual world. Different users may interact with each other in a virtual public space (e.g., a virtual geographic area, etc.) through their avatars. The avatar representing user 108 may have an appearance similar to that of a person, animal, or object. A avatar in human form may have the same gender as user 108 or a different gender. The avatar may be displayed on a display screen (e.g., display screen 266 ( Figure 3 )、Display screen 544( Figure 8 )、HHC 106( Figures 1A to 1C ) display screen, computer 172( Figure 1C ) on a display screen, etc.) so that the user 108 can see the virtual image and other objects in the virtual world.

[0192] In various embodiments, a display screen (e.g., display screen 266 ( Figure 3 )、Display screen 544( Figure 8 )、HHC 106( Figures 1A to 1C ) display screen, computer 172( Figure 1C ) can display the virtual world from the perspective of the virtual image without displaying the virtual image. The perspective of the virtual image in the virtual world can be considered to be the field of view of the virtual camera. As used herein, the virtual camera refers to the perspective within the virtual world, which can be used to present a two-dimensional image of a three-dimensional scene within the virtual world. Users can interact with each other through their virtual images through the chat channel associated with each lobby. Users can enter text through their user interface in order to chat with other users. The text can then appear on or next to the user's virtual image, for example, in the form of a comic book-style dialogue bubble (sometimes called a chat bubble). This chat can be facilitated by using a recorded phrase chat system (sometimes called a quick chat). In the case of a quick chat, the user 108 can select one or more chat phrases from a menu.

[0193] In some embodiments, a public space is public in the sense that the public space is not uniquely associated with any particular user or group of users, and no user or group of users can exclude another user from the public space. In contrast, a private space is associated with a particular user among a plurality of users. A private space is private in the sense that a particular user associated with the private space can restrict other users from entering the private space. The private space can have the appearance of a common private estate. In other embodiments, the user need not control the display screen (e.g., display screen 266 ( Figure 3 )、Display screen 544( Figure 8 )、HHC 106( Figures 1A to 1C ) display screen, computer 172( Figure 1C ) on a display screen of a user's computer, etc. The avatar displayed in the virtual space may be a robot (bot) controlled by a machine (e.g., a processor, etc.). The avatar robot can therefore move around in the virtual space in a manner similar to that of an avatar controlled by user 108, but the real-world user does not actually control the avatar robot. The avatar robot can roam in the space in many ways, perform actions, publish messages, assign permissions to certain messages, interact with other avatar robots or avatars controlled by real-world users, etc. Furthermore, the robot can be set to interact in a defined manner, modify the environment, publish announcements, publish messages, establish a virtual space, a virtual building, or construct virtual objects, graphical representations of objects, exchange real or virtual money, etc.

[0194] The embodiments described in this disclosure can be practiced with various computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. Some embodiments described in this disclosure can also be practiced in distributed computing environments, where tasks are performed by remote processing devices linked by wired or wireless networks.

[0195] In view of the above embodiments, it should be understood that many embodiments described in the present disclosure may adopt various computer-implemented operations, which involve data stored in a computer system. These operations are operations that require physical operations on physical quantities. Any operation described in the present disclosure is a useful machine operation, which forms part of the various embodiments described in the present disclosure. Some embodiments described in the present disclosure also relate to equipment or devices for performing these operations. The device may be specially constructed for the desired purpose, or the device may be a computer selectively activated or configured by a computer program stored in a computer. Specifically, various machines may be used together with a computer program written according to the teachings herein, or it may be more convenient to construct a more dedicated device to perform the desired operation.

[0196] The various embodiments described in the present disclosure may also be embodied as computer-readable code on a non-transitory computer-readable medium. A non-transitory computer-readable medium is any data storage device, such as RAM, ROM, flash memory, disk, etc., that can store data, which can then be read by a computer system. Examples of computer-readable media include hard drives, network attached storage (NAS), ROM, RAM, compact disks (CD-ROMs), recordable CDs (CD-Rs), rewritable CDs (RWs), magnetic tapes, and other optical and non-optical data storage devices. Computer-readable media may include computer-readable tangible media distributed on a network-coupled computer system so that computer-readable code is stored and executed in a distributed manner.

[0197] Although the method operations are described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be adjusted so that they occur at slightly different times, or the operations may be distributed in a system that allows the processing operations to occur at various time intervals associated with the processing, or the operations may be performed in a different order, so long as the processing of the superimposed operations is performed in the desired manner.

[0198] Although the foregoing embodiments have been described in some detail for the purpose of clarity of understanding, it is apparent that certain changes and modifications may be made within the scope of the appended claims. Therefore, the embodiments of the present invention should be considered illustrative rather than restrictive, and the various embodiments described in this disclosure are not limited to the details given herein, but may be modified within the scope and equivalent scope of the appended claims.

Claims

1. A video game controller comprising: an input control configured to receive a selection from a user for playing a video game, the input control configured to generate input data for the selection; as well as A communication circuit that interfaces with the input control interface, wherein the communication circuit is configured to generate a data packet with the input data, wherein the communication circuit is configured to send the data packet to a game cloud system via a router and the computer network without using a game console between the video game controller and the computer network, wherein the game cloud system is configured to process the input data to generate interactive control of a video output, the video output being encoded to produce a plurality of compressed frames, wherein the plurality of compressed frames are configured to be streamed via the computer network to a display device for displaying the video output in response to the input data.

2. The video game controller of claim 1 , wherein the input control comprises one or more buttons for the user to select to generate the input data, wherein the input control comprises a joystick that generates the input data, wherein the input data indicates selection of the input control.

3. The video game controller of claim 1 , wherein the input controls include one or more inertial sensors, wherein the inertial sensors include a magnetometer, an accelerometer, and a gyroscope that detect motion of the video game controller, wherein the input data instructs the video game controller to perform the motion.

4. The video game controller of claim 1, wherein the communication circuit is coupled to the input control to receive the input data from the input control.

5. The video game controller of claim 1, wherein the data packets comply with the Transmission Control Protocol over Internet Protocol to be sent to the gaming cloud system via the computer network, wherein the computer network is the Internet.

6. The video game controller of claim 1, wherein the router is configured to receive the data packet from the communication circuit, buffer the data packet, and send the data packet to the gaming cloud system via the computer network.

7. The video game controller of claim 1 , wherein the game console is configured to receive the input data from the video game controller, execute a game program based on the input data to output media data, and provide the media data to the display device for displaying the video output.

8. The video game controller of claim 1 , wherein the game cloud system is configured to process the input data to determine a state of the video game, wherein the state includes a position, color, and texture of a virtual game object within the video game, wherein the interactive control includes the state.

9. The video game controller of claim 1 , wherein the video output comprises a plurality of video frames, wherein the plurality of video frames are encoded according to the H.264 standard, wherein each of the plurality of compressed frames has a frame size and is a P-frame, an I-frame, or a B-frame, wherein, The stream of compressed frames is configured to have a frame rate.

10. The video game controller of claim 1, wherein the display device comprises a television screen or a computer screen or a head mounted display, wherein the head mounted display is configured to be worn on the user's head.

11. The video game controller of claim 1 , wherein the router is coupled to the communication circuitry via a wireless connection.

12. A system comprising: A video game controller comprising: an input control configured to receive a selection from a user for playing a video game, the input control configured to generate input data for the selection; and a communication circuit that interfaces with the input control, wherein the communication circuit is configured to generate a data packet having the input data, wherein the communication circuit is configured to send the data packet to a gaming cloud system via a router and the computer network without using a gaming console between the video game controller and the computer network, wherein the gaming cloud system is configured to process the input data to generate interactive controls for a video output that is encoded to produce a plurality of compressed frames, wherein the plurality of compressed frames are configured to be streamed via the computer network to a display device for displaying the video output in response to the input data; and a display device coupled to the game cloud system via the computer network, Wherein, the display device is configured to receive the plurality of compressed frames and decode the plurality of compressed frames to display the video output.

13. The system of claim 12, wherein the input control comprises one or more buttons for the user to select to generate the input data, wherein the input control comprises a joystick that generates the input data, wherein the input data indicates selection of the input control.

14. The system of claim 12, wherein the input controls include one or more inertial sensors, wherein the inertial sensors include a magnetometer, an accelerometer, and a gyroscope that detect motion of the video game controller, wherein the input data instructs the video game controller to perform the motion.

15. The system of claim 12, wherein the communication circuit is coupled to the input control to receive the input data from the input control.

16. The system of claim 12, wherein the data packets comply with the Transmission Control Protocol over Internet Protocol to be sent to the gaming cloud system via the computer network, wherein the computer network is the Internet.

17. The system of claim 12, wherein the router is configured to receive the data packet from the communication circuit, buffer the data packet, and send the data packet to the gaming cloud system via the computer network.

18. The system of claim 12, wherein the game console is configured to receive the input data from the video game controller, execute a game program based on the input data to output media data, and provide the media data to the display device for displaying the video output.

19. The system of claim 12, wherein the game cloud system is configured to process the input data to determine a state of the video game, wherein the state includes a position, color, and texture of a virtual game object within the video game, wherein the interactive control includes the state.

20. The system of claim 12, wherein the video output comprises a plurality of video frames, wherein the plurality of video frames are encoded according to the H.264 standard, wherein each of the plurality of compressed frames has a frame size and is a P frame, an I frame, or a B frame, wherein, The stream of compressed frames is configured to have a frame rate.

21. The system of claim 12, wherein the display device is a head mounted display, wherein the head mounted display comprises a decoder configured to decode the plurality of compressed frames, wherein the head mounted display comprises an external digital camera, wherein, The external digital camera is configured to capture one or more images of a real-world environment surrounding the head-mounted display for transmission to the gaming cloud system via the router and the computer network without using the gaming console.

Images