Remote play using local projector

Abstract

SOLUTION: A controller device for interactive gameplay of a video game configured to be handheld by a user includes: at least one motion sensor, the motion sensor generating motion data in response to movement of the controller device when handheld by the user during the interactive gameplay of the video game; a wireless communications device that receives gameplay video over a network from a remotely executing session of the video game; and a projector, the projector being integrated with the controller device and configured to project the gameplay video onto a projection surface in a local environment, the projection of the gameplay video being stabilized based on the motion data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] SUMMARY Embodiments of the present disclosure relate to a method and system for remote play using a local projector. [Background technology]

[0002] 2. Description of Related Art Projectors can receive a video signal and project the video onto a surface, allowing viewers to view attractive, potentially large images, and even allowing existing surfaces to be used as video display areas. Continuing advances in technology are leading to the miniaturization of projectors, including the development of small form factors such as those seen in today's state-of-the-art picoprojectors.

[0003] Video game technology continues to evolve, and advances in computing and networking technology now enable gameplay of video games running on computing devices in remote locations.

[0004] It is in this context that embodiments of the present disclosure arise. Summary of the Invention

[0005] SUMMARY Embodiments of the present disclosure relate to a method and system for remote play using a local projector.

[0006] In some embodiments, a controller device is provided for interactive gameplay of a video game, the controller device being configured to be handheld by a user; at least one motion sensor that generates motion data responsive to movement of the controller device when held by the user during interactive gameplay of the video game; a wireless communication device that receives gameplay video over a network from a remote execution session of the video game; and a projector that is integrated with the controller device and configured to project the gameplay video onto a projection surface in a local environment, wherein the projection of the gameplay video is stabilized based on the motion data.

[0007] In some implementations, the movements of the controller device are generated from interactive gameplay of a video game, and the motion data is provided as input to a remote execution session of the video game.

[0008] In some embodiments, the remote execution session is executed by a cloud gaming service.

[0009] In some embodiments, the remote execution session is executed by a remotely located gaming console.

[0010] In some implementations, the gameplay video depicts a view of the virtual environment of the video game.

[0011] In some implementations, stabilizing the gameplay video includes adjusting the direction of the projector toward the location of the projection surface.

[0012] In some implementations, stabilizing the gameplay video includes adjusting the keystone of the projector in response to lateral movement of the controller relative to the projection surface.

[0013] In some implementations, the stabilization of the gameplay video includes adjusting the zoom of the projector in response to movement of the controller towards or away from the projection surface.

[0014] In some embodiments, the at least one motion sensor includes one or more of an accelerometer, a gyroscope, or a magnetometer.

[0015] In some implementations, the controller device further includes at least one input device selected from a button, a trigger, a joystick, or a touchpad.

[0016] In some embodiments, a system for interactive gameplay of a video game is provided, the system including: a controller device configured to be handheld by a user, the controller device having at least one motion sensor that generates motion data in response to movements of the controller device held by the user during interactive gameplay of the video game; a projector configured to be removably connected to the controller device, the projector having a wireless communication device that receives gameplay video from a remote execution session of the video game over a network, the projector configured to project the gameplay video onto a projection surface in a local environment, and the projection of the gameplay video is stabilized based on the motion data.

[0017] In some implementations, the movements of the controller device are generated from interactive gameplay of a video game, and the motion data is provided as input to a remote execution session of the video game.

[0018] In some embodiments, the remote execution session is executed by a cloud gaming service.

[0019] In some embodiments, the remote execution session is executed by a remotely located gaming console.

[0020] In some implementations, the gameplay video depicts a view of the virtual environment of the video game.

[0021] In some implementations, stabilizing the gameplay video includes adjusting the direction of the projector toward the location of the projection surface.

[0022] In some implementations, stabilizing the gameplay video includes adjusting the keystone of the projector in response to lateral movement of the controller relative to the projection surface.

[0023] In some implementations, stabilizing the gameplay video includes adjusting the zoom of the projector in response to movement of the controller towards or away from the projection surface.

[0024] In some embodiments, the at least one motion sensor includes one or more of an accelerometer, a gyroscope, or a magnetometer.

[0025] In some implementations, the controller device further includes at least one input device selected from a button, a trigger, a joystick, or a touchpad.

[0026] Other aspects and advantages of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the disclosure.

[0027] The present disclosure, together with further advantages thereof, may best be understood by reference to the accompanying drawings and the following description. [Brief explanation of the drawings]

[0028] [Figure 1] 1 conceptually illustrates a system for providing remote gameplay via a projector according to an embodiment of the present disclosure.

[0029] [Figure 2A] 1 illustrates a front view of a controller with a projector attached, according to an embodiment of the present disclosure.

[0030] [Figure 2B] 1 illustrates a front view of a controller with a projector attached, according to an embodiment of the present disclosure.

[0031] [Figure 2C] 1 illustrates a front view of a controller with an integrated projector according to an embodiment of the present disclosure.

[0032] [Figure 3] 1 conceptually illustrates a system for enabling remote game play using a projector attached to or integrated into a controller, according to an embodiment of the present disclosure.

[0033] [Figure 4] 1 conceptually illustrates a user operating a controller with a projector attached or integrated therein, according to an embodiment of the present disclosure.

[0034] [Figure 5] FIG. 1 is a block diagram of a gaming system according to various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0035] Broadly speaking, embodiments of the present disclosure relate to implementations of a projector for enabling gameplay of a video game. In some implementations, the projector is a standalone device that can be enabled for remote gameplay of a video game run by a cloud video game service or by a remote computing device, such as a game console or personal computer. In other implementations, the projector is integrated with a controller device operated by a user for gameplay. In related implementations, the projector is configured as an attachment that can be connected to an existing controller device. In such implementations, the image projected by the projector is stabilized using motion data captured by the controller device, so that the position of the image on a particular projection surface is maintained in a consistent manner, allowing for easy viewing by the player.

[0036] FIG. 1 conceptually illustrates a system for providing remote gameplay via a projector, according to embodiments of the present disclosure. In the illustrated embodiment, a user 100 operates a controller 102 to provide input to a video game. The video game is executed at a location / device remote from the user's location. For example, in some embodiments, the video game is executed in the cloud by a cloud gaming service 110. In such embodiments, the video game may be executed by a cloud gaming machine, a virtualized gaming machine, or other hardware or server resource capable of executing a video game session. In some embodiments, the video game is executed by a remote computing device (e.g., a gaming console, computer, or machine) and made available through a remote gaming service. The remote gaming service facilitates remote gameplay of the video game, such as facilitating the transmission of video from the remote computing device and the transmission of input to the remote computing device.

[0037] Projector 104 is configured to receive video from a video game, e.g., a cloud / remote gaming service 110, and display / project the video onto surface 106, which in the illustrated implementation is conceptually shown as projected video 108. By way of non-limiting example, surface 106 may be a wall, a ceiling, a projection screen, or any other surface suitable for projecting an image. In some implementations, controller 102 communicates with projector 104 via a wired or wireless connection, and input from controller 102 is transmitted from controller 102 to projector 104, which passes the input to a running video game for processing and to facilitate execution of the game.

[0038] It will be appreciated that in some implementations, projector 104 can have a portable form factor for ease of transport. Additionally, projector 104 includes all necessary components to enable the projection of an image onto a surface, such as electronics for receiving and processing video signals, a light source, and optics. In some implementations, projector 104 includes a battery so that it can be used without being plugged into an electrical outlet. In some implementations, projector 104 includes wireless communication electronics to enable wireless communication over various networks / wireless protocols, such as 5G cellular, WiFi, Bluetooth, etc.

[0039] It will be appreciated that projector 104 as described enables a user to play video games remotely. For example, a user may be away from their home where their gaming console is located, but projector 104 can be used to enable the user to remotely play a video game running on their gaming console at their home. That is, the video game can be run remotely on the user's gaming console, and video can be transmitted over one or more networks to projector 104 for display on surfaces in the user's local environment. In some implementations, video transmission can be enabled through a remote gaming service. In other implementations, the video game is run by a cloud gaming service, as described above. With appropriate connectivity and a suitable projection surface for viewing, projector 104 constitutes a portable device that allows a user to enjoy gameplay of a video game anywhere.

[0040] In some implementations, the projector 104 is configured to project an anchor image onto the projection surface that is used to track the controller, i.e., the position / orientation of the controller 102 in the local environment can be determined, at least in part, based on images captured by a camera included in the controller, by identifying and tracking at least a portion of the projected anchor image in the captured images.

[0041] In some implementations, projector 104 includes a video pass-through connector, such as an HDMI input, a DVI input, or other video input, through which projector 104 can receive video from another source or device and project it onto the projection surface.

[0042] In some implementations, the projector is connected to or integrated with a controller device that provides input to a video game, and the projector projects a video of the game and can utilize motion data from the controller to stabilize the projected image of the projector.

[0043] 2A illustrates a front view of a controller with an attached projector according to an embodiment of the present disclosure. In the illustrated embodiment, the projector 206 is attached to the underside of the controller 200. The controller 200 includes various input devices, such as a joystick 202 and triggers / buttons 204. Mounting the projector 206 to the underside of the controller 200 exposes the top surface of the controller 200, allowing easy access to the joystick 202 and any other input devices located on the top surface, such as buttons or a touchpad. It will be appreciated that the projector 206 is configured to be removably connected to and communicate with the controller 200. As described herein, the projector 206 receives motion data from the controller 200 to enable stabilization of the projected image.

[0044] 2B shows a front view of a controller with a mounted projector according to an embodiment of the present disclosure. In the illustrated embodiment, the projector 208 is mounted on the top side of the controller 200. The mounting configuration of the projector 208 is such that the projector 208 is elevated and supported above the top surface of the controller 200, thereby allowing continued access to input devices on the top surface of the controller. It will be appreciated that the projector 208 is configured to be removably connected to the controller 200, to communicate with the controller 200, and to receive motion data from the controller 200 to enable stabilization of the projected image.

[0045] 2C shows a front view of a controller with an integrated projector according to an embodiment of the present disclosure. In the embodiment shown, controller 210 includes projector 212. The image from the projector is stabilized as described herein.

[0046] 3 conceptually illustrates a system for enabling remote gameplay using a projector attached to or integrated into a controller, according to an embodiment of the present disclosure. In the illustrated embodiment, projector 300 is communicatively connected to controller 320. Projector 300 can be removably connected to the controller via an attachment mechanism, or in other embodiments, can be integrated with controller 320.

[0047] The controller 320 includes one or more input devices 330, such as a joystick, buttons, a directional pad, a touchpad, or a biometric sensor, for generating video game input. The controller 320 further includes one or more motion sensors 326 for detecting the position, orientation, and movement of the controller 320. In some implementations, the motion sensors 326 may include one or more accelerometers, magnetometers, and gyroscopes. In some implementations, the motion sensors 326 are defined in the form of an inertial measurement unit (IMU). In some implementations, the controller 326 includes one or more cameras 328, which can be used for tracking, such as by using simultaneous localization and mapping (SLAM) techniques.

[0048] The controller 326 includes at least one processor 322 configured to execute controller tracking logic 324, which determines and tracks the position / location and orientation of the controller and the movement of the controller based on motion data from the aforementioned motion sensors 326 and / or image data from the camera 328. In some implementations, the controller 320 further includes one or more haptic devices 332 that can be activated by the controller to provide haptic feedback to the user, such as by generating vibrations or other haptic sensations.

[0049] Projector 300 is connected to or integrated with controller 320. Projector 300 includes a battery 316 for powering its components. Projector 300 includes a wireless communication module 314 that enables wireless communication (e.g., WiFi, 5G, etc.) with a remote game session 342 of a video game over a network 340. As described above, in some implementations, remote game session 342 is performed by a cloud gaming service, and in some implementations, remote game session 342 is performed by a user's remote gaming device, such as a gaming console or computer. Video from the remote game session is streamed to projector 300 over network 340 and displayed.

[0050] The projector includes a processor 302 that executes video logic 306 to process and enhance input video for projection. The processor 302 also executes stabilization logic 304 configured to stabilize the projected image of the projector 300 based on motion data from the controller 320 to maintain the projected image in a substantially stationary configuration (e.g., stationary position, orientation, magnification, keystone, brightness / color consistency, etc.). In some implementations, the stabilization logic 304 communicates with an orientation controller 308 to control and adjust the projection orientation / tilt (e.g., pitch, yaw, roll) to stabilize the projected image. In some implementations, the stabilization logic 304 communicates with an optical controller 312 to control and adjust the projector's optical properties, such as zoom (size) and keystone, to stabilize the projected image. In some implementations, the stabilization logic 304 communicates with an image generation controller 310 to adjust the generation of the projected image to stabilize the projected image, for example, by adjusting the brightness, color intensity, or other image properties of various regions of the projection to stabilize the image.

[0051] It will be appreciated that motion and / or image data from controller 320 can be used as input for a remote game session 342 of a video game. In such an implementation, the motion / image data can be communicated to the remote game session 342 via the projector over network 340, or by the controller itself over the network to the remote game session. In such an implementation, the motion data of controller 320 serves a dual purpose: as input to the game and as data that enables stabilization of the projected video image of projector 300.

[0052] In some implementations, image data from camera 328 is processed and used as feedback to the system to determine and verify that the projected video is properly stabilized. For example, the image data can be analyzed to identify the projected video on the projection surface, as well as to identify other anchors in the local environment, and to determine whether the projected video is maintained in a stable position on the projection surface.

[0053] In some implementations, a motion sensor 326 and / or a camera 328 are included in the projector 300, and data therefrom is processed by the stabilization logic 304 in the projector.

[0054] In some implementations, projector 300 receives motion data and / or image data generated by motion sensor 326 and camera 328, respectively, from controller 320. In some implementations, projector 300 receives such data after it is first processed in the controller by controller tracking logic. In some implementations, projector 300 receives processed movement information from the motion / image data that indicates movement detected based on the motion / image data.

[0055] In some implementations, the remote game session 342 is configured to optimize the rendering of gameplay video based on the detected position / orientation and / or movement of the controller or projector. For example, the remote game session 342 can adjust the brightness or color intensity or other graphical properties of areas of the video based on such information, thereby improving the stability of the image quality of the projected video and compensating for projector movement.

[0056] In some implementations, in response to a sudden change in projector / controller orientation, the virtual view is temporarily reoriented in a similar direction (in the virtual environment of the video game), thereby rendering an augmentation of the view in the direction the projector moved, and then, when the projector stabilizes and returns to its original orientation, the virtual view is also reoriented back to its original orientation in the virtual environment.

[0057] Therefore, in view of the above, a hybrid projector stabilization technique is provided in which stabilization is performed by both the projector 300 and the remote game session 342.

[0058] 4 conceptually illustrates a user operating a controller with an attached or integrated projector, according to an embodiment of the present disclosure. In the illustrated embodiment, a user 400 operates a controller 402 with an attached or integrated projector 404. As described above, the projected video 406 of a video game is stabilized to enhance the user's viewing experience during gameplay.

[0059] In some implementations, a zone 408 is defined in the local environment that defines the approximate limits of the combined controller / projector device's ability to stabilize the projected video 406. For example, if the controller (and projector) is moved significantly up / down / left / right / forward / backward, then keystone adjustments and / or zoom adjustments, etc., can no longer stabilize the projected video in the same position on the projection surface. Thus, in some implementations, the zone 408 defines an area in the local environment, such that when the controller reaches the boundary / edge of the zone 408, feedback is provided to the user alerting them that the projected video may no longer be stabilized in the same place if the controller continues to move beyond the zone 408. In some implementations, the feedback is in the form of a pop-up display displayed on the projected video. In some implementations, the feedback is in the form of an audio warning / sound. In some implementations, the feedback is in the form of haptic feedback provided through the controller 402.

[0060] Similarly, if the direction / tilt (pitch / yaw / roll) of the controller is rotated beyond a certain amount (e.g., off a specified central axis), the projected video may no longer be stable in place. Thus, in some implementations, a maximum amount of rotation off axis may be defined, and similar feedback may be provided to the user when such maximum is reached.

[0061] In some implementations, rather than stabilizing the projected video to maintain the video in a stationary position, the orientation of the controller is used to control the orientation of the virtual view. In this way, as the controller moves and / or rotates, the physical position of the projected video also moves, resulting in a corresponding movement of the orientation of the virtual view to display different portions of the virtual environment. The effect is similar to moving / orienting a virtual camera by moving / orienting the controller, so that wherever the virtual camera is moved (depending on the controller movement), that area of ​​the virtual environment captured by the virtual camera is rendered and projected for the user to see. In such implementations, the projected video image functions as a movable window into the virtual environment. In some implementations, this projector operation mode can be characterized as a movable virtual display mode, as opposed to the fixed display mode described above. In some implementations, the stabilization described above is not performed during the movable virtual display mode. However, the lack of any stabilization of the projector can lead to unwanted shaking of the projected video due to a user's shaking hands, or an overall lack of smoothness in the video viewing experience, as even small movements of the projector can cause large movements of the projected video from a distance.

[0062] Thus, in some implementations, the projector is stabilized during movable virtual display mode in a manner that counteracts user hand shake and smooths sudden projector movements. It will be appreciated that in such implementations, controller movement is used for both stabilization and virtual camera control. This allows the user to comfortably view the projected video, but also allows the virtual camera to be directed to view different parts of the virtual environment.

[0063] In some implementations, the movable virtual display mode and the fixed display mode are enabled at different times. In some implementations, a user can switch between the two modes using an input mechanism such as pressing a button or activating a trigger. In some implementations, the fixed display mode is active by default, and the system switches to the movable virtual display mode in response to continuously applied input, such as holding a button or trigger, and returns to the stable display mode when the input is no longer applied, such as releasing the button or trigger.

[0064] In some implementations, a fixed display mode is active when the controller is oriented substantially in a first direction (e.g., facing the front wall), but the system switches to a movable virtual display mode when the controller is oriented substantially in a second direction (e.g., facing the ceiling). In some implementations, the switch to the movable virtual display mode occurs in response to the orientation or position of the controller changing by more than a predefined threshold amount.

[0065] Although embodiments of the present disclosure have been described with reference to video games, it will be appreciated that in other embodiments, the principles of the present disclosure may be applied to other types of interactive applications capable of projecting video in accordance with the above.

[0066] In some implementations, the projected video is configured to take into account the three-dimensional geometry of the projection surface. For example, if the projection surface is not planar, the projected video may be pre-distorted so that the video does not appear distorted to the user when projected onto the projection surface.

[0067] FIG. 5 is a block diagram of a gaming system 1600 according to various embodiments of the present disclosure. The gaming system 1600 is configured to provide video streams to one or more clients 1610 over a network 1615. The gaming system 1600 typically includes a video server system 1620 and an optional game server 1625. The video server system 1620 is configured to provide video streams to one or more clients 1610 with a minimum quality of service. For example, the video server system 1620 may receive game commands that change the state of a video game or a viewpoint within the video game and provide an updated video stream reflecting the change to the client 1610 with minimal latency. The video server system 1620 may be configured to provide video streams in a variety of alternative video formats, including formats that have not yet been defined. Furthermore, the video streams may include video frames configured for presentation to a user at a variety of frame rates. Typical frame rates are 30 frames per second, 60 frames per second, and 120 frames per second. However, alternative embodiments of the present disclosure include higher or lower frame rates.

[0068] The clients 1610, individually referred to herein as 1610A, 1610B, etc., may include HMDs, terminals, personal computers, game consoles, tablet computers, phones, set-top boxes, kiosks, wireless devices, digital pads, standalone devices, handheld gameplay devices, projectors, and / or the like. Typically, the clients 1610 are configured to receive encoded video streams, decode the video streams, and present the resulting video to a user (e.g., a player of a game). The process of receiving the encoded video stream and / or decoding the video stream typically involves storing individual video frames in a receive buffer of the client. The video stream may be presented to the user on a display integrated into the client 1610 or on a separate device such as a monitor or television. The clients 1610 are optionally configured to support multiple game players. For example, a game console may be configured to support two, three, four, or more simultaneous players. Each of these players may receive a separate video stream, or a single video stream may include a region of frames generated specifically for each player (e.g., generated based on each player's perspective). Clients 1610 are optionally geographically distributed. The number of clients included in gaming system 1600 may vary widely, from one or two to thousands, tens of thousands, or more. As used herein, the term "game player" is used to refer to a person who plays a game, and the term "gameplay device" is used to refer to a device used to play a game. In some implementations, gameplay devices may refer to multiple computing devices that cooperate to provide a gaming experience to a user. For example, a gaming console and an HMD / MR headset may cooperate with video server system 1620 to deliver the game viewed through the HMD / MR headset.In one implementation, the gaming console receives the video stream from the video server system 1620, and the gaming console forwards the video stream or updates to the video stream to the HMD for rendering.

[0069] Client 1610 is configured to receive the video stream via network 1615. Network 1615 can be any type of communications network, including a telephone network, the Internet, a wireless network, a power line network, a local area network, a wide area network, a private network, and / or the like. In a typical implementation, the video stream communicates via a standard protocol such as TCP / IP or UDP / IP. Alternatively, the video stream is communicated via a proprietary standard.

[0070] A typical example of a client 1610 is a personal computer that includes a processor, non-volatile memory, a display, decoding logic, network communication capabilities, and input devices. The decoding logic may include hardware, firmware, and / or software stored on a computer-readable medium. Systems for decoding (and encoding) video streams are well known in the art and vary depending on the particular encoding scheme used.

[0071] Additionally, the clients 1610 may, but need not, include systems configured to modify the received video. For example, the clients may further be configured to perform rendering, overlay one video image on another, crop video images, and / or the like. For example, the clients 1610 may be configured to receive various types of video frames, such as I-frames, P-frames, and B-frames, and process these frames into images for display to a user. In some implementations, members of the clients 1610 are configured to perform further rendering, shading, conversion to 3D, or similar manipulation of the video stream. Members of the clients 1610 are optionally configured to receive multiple audio or video streams. Input devices for the clients 1610 may include, for example, a single-handed game controller, a two-handed game controller, a gesture recognition system, an eye-gaze recognition system, a voice recognition system, a keyboard, a joystick, a pointing device, a force-feedback device, a motion and / or position sensing device, a mouse, a touchscreen, a neural interface, a camera, an input device under development, and / or the like.

[0072] The video stream (and optionally the audio stream) received by client 1610 is generated and provided by video server system 1620. As further described elsewhere herein, this video stream includes video frames (and the audio stream includes audio frames). The video frames are configured to meaningfully contribute to the image displayed to the user (e.g., the video frames include pixel information in a suitable data structure). As used herein, the term "video frame" is used to refer to a frame that primarily includes information configured to contribute to, e.g., act on, the image shown to the user. Most of the teachings herein regarding "video frames" can also be applied to "audio frames."

[0073] Client 1610 is typically configured to receive input from a user. These inputs may include game commands configured to change the state of a video game or otherwise affect game play. Game commands may be received using an input device and / or may be generated automatically by computer processing instructions executing on client 1610. Received game commands are communicated from client 1610 over network 1615 to video server system 1620 and / or game server 1625. For example, in some embodiments, game commands are communicated to game server 1625 via video server system 1620. In some embodiments, separate copies of game commands are communicated from client 1610 to game server 1625 and video server system 1620. Communication of game commands optionally depends on the identity of the command. Game commands are optionally communicated from client 1610A over a different route or communication channel than is used to provide audio or video streams to client 1610A.

[0074] The game server 1625 is optionally operated by a different entity than the video server system 1620. For example, the game server 1625 may be operated by a publisher of a multiplayer game. In this example, the video server system 1620 is optionally viewed as a client by the game server 1625 and is optionally configured to appear, from the perspective of the game server 1625, to be a conventional client running a conventional game engine. Communication between the video server system 1620 and the game server 1625 optionally occurs over the network 1615. Thus, the game server 1625 may be a conventional multiplayer game server that transmits game state information to multiple clients, one of which is the video server system 1620. The video server system 1620 may be configured to communicate with multiple instances of the game server 1625 simultaneously. For example, the video server system 1620 may be configured to provide multiple different video games to different users. Each of these different video games may be supported by a different game server 1625 and / or published by a different entity. In some implementations, several geographically distributed instances of video server system 1620 are configured to provide game videos to multiple different users. Each of these instances of video server system 1620 may communicate with the same instance of game server 1625. Communication between video server system 1620 and one or more game servers 1625 optionally occurs over a dedicated communication channel. For example, video server system 1620 may be connected to game server 1625 over a high-bandwidth channel dedicated to communication between these two systems.

[0075] Video server system 1620 comprises at least a video source 1630, an I / O device 1645, a processor 1650, and non-transitory storage 1655. Video server system 1620 may include one computing device or may be distributed among multiple computing devices, optionally connected via a communication system such as a local area network.

[0076] Video source 1630 is configured to provide a video stream (e.g., a series of video frames forming a streaming video or movie). In some implementations, video source 1630 includes a video game engine and rendering logic. The video game engine is configured to receive game commands from a player and maintain a copy of the video game state based on the received commands. This game state includes the positions of objects within the game environment and, typically, the viewpoint. The game state may also include object properties, images, colors, and / or textures. The game state is typically maintained based on game rules and game commands for movement, turning, attacking, focusing, interaction, use, and / or the like. Portions of the game engine are optionally located within game server 1625. Game server 1625 may maintain a copy of the game state based on game commands received from multiple players using geographically distributed clients. In these cases, the game state is provided by game server 1625 to video source 1630, where a copy of the game state is stored and rendered. Game server 1625 may receive game commands directly from client 1610 via network 1615 and / or may receive game commands via video server system 1620.

[0077] Video Source 1630 typically includes rendering logic (e.g., hardware, firmware, and / or software stored on a computer-readable medium, such as Storage 1655). This rendering logic is configured to create video frames of the video stream based on the game state. All or part of the rendering logic is optionally located within a graphics processing unit (GPU). The rendering logic typically includes processing stages configured to determine three-dimensional spatial relationships between objects and / or apply appropriate textures, etc., based on the game state and viewpoint. The rendering logic generates raw video, which is then typically encoded before being communicated to Client 1610. For example, raw video may be encoded according to the Adobe Flash® standard, .wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith, MPG-x, Xvid, FFmpeg, x264, VP6-8, realvideo, or mp3. The encoding process generates a video stream that is optionally packaged for delivery to a decoder on a remote device. The video stream is characterized by a frame size and a frame rate. Typical frame sizes include 800x600, 1280x720 (e.g., 720p), and 1024x768, although any other frame size may also be used. The frame rate is the number of video frames per second. A video stream may include various types of video frames. For example, the H.264 standard includes "P" frames and "I" frames. I-frames contain information to update all macroblocks / pixels on a display device, while P-frames contain information to update a subset of them. P frames typically have a smaller data size than I frames. As used herein, the term "frame size" is meant to refer to the number of pixels in a frame. The term "frame data size" is used to refer to the number of bytes required to store a frame.

[0078] In an alternative embodiment, video source 1630 includes a video recording device such as a camera. The camera may be used to generate delayed or live video that can be included in the video stream of a computer game. The resulting video stream optionally includes both rendered images and images recorded using a still camera or a video camera. Video source 1630 may also include a storage device configured to store pre-recorded video that is included in the video stream. Video source 1630 may also include a motion or position sensing device configured to detect the movement or position of an object (e.g., a person) and logic configured to determine a game state or to produce video based on the detected movement and / or position.

[0079] Video source 1630 is optionally configured to provide overlays configured to be set over other video. For example, these overlays may include a command interface, login prompts, messages for the game player, images of other game players, and video feeds of other game players (e.g., webcam video). In implementations of client 1610A that include a touchscreen interface or an eye-gaze interface, the overlays may include a virtual keyboard, joystick, touchpad, and / or the like. In one example of an overlay, a player's voice is overlaid on an audio stream. Optionally, video source 1630 further includes one or more audio sources.

[0080] In embodiments in which video server system 1620 is configured to maintain game state based on input from multiple players, each player may have a different viewpoint, including viewing position and direction. Video source 1630 is optionally configured to provide a separate video stream for each player based on the player's viewpoint. Furthermore, video source 1630 may be configured to provide different frame sizes, frame data sizes, and / or encoding to each of clients 1610. Video source 1630 is optionally configured to provide 3D video.

[0081] The I / O devices 1645 are configured to allow the video server system 1620 to send and / or receive information such as video, commands, requests for information, game state, gaze information, device movement, device location, user movement, client identification information, player identification information, game commands, security information, audio, and / or the like. The I / O devices 1645 typically include communications hardware such as a network card or modem. The I / O devices 1645 are configured to communicate with the game server 1625, the network 1615, and / or the clients 1610.

[0082] The processor 1650 is configured to execute logic (e.g., software) included within the various components of the video server system 1620 described herein. For example, the processor 1650 may be programmed with software instructions to perform the functions of the video source 1630, the game server 1625, and / or the client qualifier 1660. The video server system 1620 optionally includes multiple instances of the processor 1650. The processor 1650 may also be programmed with software instructions to execute commands received by the video server system 1620 or to coordinate the operation of the various elements of the game system 1600 described herein. The processor 1650 may include one or more hardware devices. The processor 1650 is an electronic processor.

[0083] Storage 1655 includes non-transitory analog and / or digital storage devices. For example, storage 1655 may include an analog storage device configured to store video frames. Storage 1655 may include computer-readable digital storage (e.g., a hard drive, optical drive, or solid-state storage). Storage device 1615 is configured (e.g., via a suitable data structure or file system) to store video frames, artificial frames, video streams including both video frames and artificial frames, audio frames, audio streams, and / or the like. Storage 1655 is optionally distributed among multiple devices. In some implementations, storage 1655 is configured to store software components of video source 1630, described elsewhere herein. These components may be stored in a format that allows them to be readily available as needed.

[0084] Optionally, video server system 1620 further includes a client qualifier 1660. Client qualifier 1660 is configured to remotely determine the capabilities of a client, such as client 1610A or 1610B. These capabilities may include both the capabilities of client 1610A itself and the capabilities of one or more communication channels between client 1610A and video server system 1620. For example, client qualifier 1660 may be configured to test the communication channel via network 1615.

[0085] The Client Qualifier 1660 can manually or automatically determine (e.g., learn) the capabilities of the Client 1610A. Manual determination includes communicating with a user of the Client 1610A and prompting the user to provide the capabilities. For example, in some embodiments, the Client Qualifier 1660 is configured to display images, text, and / or the like within the browser of the Client 1610A. In one embodiment, the Client 1610A is an HMD / MR headset that includes a browser. In another embodiment, the Client 1610A is a gaming console with a browser, and the browser may be displayed on the HMD / MR headset. The displayed objects request the user to enter information such as the Client 1610A's operating system, processor, video decoder type, network connection type, display resolution, etc. The information entered by the user is transmitted back to the Client Qualifier 1660.

[0086] The automated determination may be made, for example, by running an agent on the client 1610A and / or by sending a test video to the client 1610A. The agent may include computing instructions, such as JavaScript, embedded in a web page or installed as an add-on. The agent is optionally provided by the client qualifier 1660. In various implementations, the agent may discover the processing capabilities of the client 1610A, the decoding and display capabilities of the client 1610A, the latency reliability and bandwidth of the communication channel between the client 1610A and the video server system 1620, the display type of the client 1610A, any firewalls present on the client 1610A, the hardware of the client 1610A, software running on the client 1610A, registry entries within the client 1610A, and / or the like.

[0087] Client Qualifier 1660 includes hardware, firmware, and / or software stored on a computer-readable medium. Client Qualifier 1660 is optionally located on a computing device separate from one or more other elements of Video Server System 1620. For example, in some implementations, Client Qualifier 1660 is configured to determine characteristics of communication channels between Client 1610 and multiple instances of Video Server System 1620. In these implementations, information known by Client Qualifier 1660 can be used to determine which instance of Video Server System 1620 is best suited for delivering streaming video to one of Clients 1610.

[0088] Embodiments of the present disclosure can be practiced with a variety of computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. The present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0089] With the above embodiments in mind, it should be understood that the present disclosure can employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the present disclosure are useful machine operations. The present disclosure also relates to devices or apparatus for performing these operations. Apparatus can be specially constructed for the required purposes, or the apparatus can be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines can be used with computer programs written in accordance with the teachings herein. Alternatively, it may be more convenient to construct a more specialized apparatus to perform the required operations.

[0090] The present disclosure may also be embodied as computer-readable code on a computer-readable medium. The computer-readable medium is any data storage device that can store data which can then be read by a computer system. Examples of computer-readable media include hard drives, network-attached storage (NAS), read-only memory, random-access memory, optical media, non-optical data storage devices, etc. The computer-readable medium may include computer-readable tangible media distributed over network-coupled computer systems so that the computer-readable code is stored and executed in a distributed fashion.

[0091] Although the method operations have been described in a particular order, it should be understood that other housekeeping operations may be performed between operations, or operations may be adjusted to occur at slightly different times, or may be distributed within a system that allows processing operations to occur at various intervals relative to processing, so long as the processing of the overlay operation is performed in the desired manner.

[0092] Although the foregoing disclosure has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered illustrative rather than restrictive, and the disclosure is not limited to the details given herein, but can be modified within the scope of the disclosure and equivalents.

Claims

1. a controller device, at least one motion sensor; a wireless communication device for receiving video from a remote execution session over a network; a projector associated with the controller device; Equipped with the motion sensor generates motion data in response to a movement of the controller device held by a user; the projector is configured to project the video onto a projection surface in a local environment; the video projection is stabilized based on the motion data; 10. A controller device, wherein the stabilization of the video includes adjusting the projector in response to movement of the controller toward or away from the projection surface.

2. the remote execution session is a remote execution session of a video game; the movements of the controller device are generated from interactive gameplay of the video game; The controller device of claim 1 , wherein the motion data is provided as input to a remote execution session of the video game.

3. The controller device of claim 1 , wherein the remote execution session is executed by a cloud gaming service.

4. The controller device of claim 1 , wherein the remote execution session is executed by a remotely located gaming console.

5. The controller device of claim 1 , wherein the video depicts a view of a virtual environment of a video game.

6. The controller device of claim 1 , wherein the stabilizing the video includes zooming the projector or adjusting the direction of the projector toward the location of the projection surface.

7. The controller device of claim 1 , wherein the video stabilization includes adjusting a keystone of the projector in response to lateral movement of the controller relative to the projection surface.

8. The controller device of claim 1 , wherein the at least one motion sensor includes an accelerometer, a gyroscope, or a magnetometer.

9. The controller device of claim 1 , further comprising at least one input device selected from a button, a trigger, a joystick, or a touchpad.

10. 1. A system comprising: a controller device; and a projector associated with the controller device, the controller device is configured to be handheld by a user and has at least one motion sensor; the motion sensor generates motion data in response to movement of the controller device held by the user; the projector has a wireless communication device configured to receive video from a remote execution session over a network and to project the video onto a projection surface in the local environment; the video projection is stabilized based on the motion data; The system, wherein the stabilizing the video includes adjusting the projector in response to movement of the controller toward or away from the projection surface.

11. the remote execution session is a remote execution session of a video game; the movements of the controller device are generated from interactive gameplay of the video game; The system of claim 10, wherein the motion data is provided as input to a remote execution session of the video game.

12. The system of claim 10 , wherein the remote execution session is executed by a cloud gaming service.

13. 11. The system of claim 10, wherein the remote execution session is executed by a remotely located gaming console.

14. The video depicts a view of a virtual environment of a video game. The system of claim 10.

15. The system of claim 10, wherein the stabilizing the video includes zooming the projector or adjusting the direction of the projector toward the location of the projection surface.

16. 11. The system of claim 10, wherein the video stabilization includes adjusting a keystone of the projector in response to lateral movement of the controller relative to the projection surface.

17. The system of claim 10 , wherein the at least one motion sensor includes an accelerometer, a gyroscope, or a magnetometer.

18. The system of claim 10 , wherein the at least one motion sensor includes an accelerometer, a gyroscope, or a magnetometer.

Images