Remote play using a local projector

The integration of motion sensors in a controller-projector device stabilizes gameplay video projections, addressing the challenge of remote gameplay stability and enabling portable, stable video projection for remote gaming.

JP2024542774AActive Publication Date: 2024-11-15SONY INTERACTIVE ENTERTAINMENT LLC
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
JP2024532955
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2021-12-15
Filing Date
2022-11-22
Publication Date
2024-11-15
Estimated Expiration
2042-11-22

AI Technical Summary

Technical Problem

Existing video game technologies do not effectively enable remote gameplay using a local projector, particularly in maintaining a stable projection of gameplay video on various surfaces, especially when the projector is moved or oriented differently.

Method used

A controller device integrated with a projector that stabilizes the projected gameplay video using motion sensors, such as accelerometers and gyroscopes, adjusting the projector's orientation and zoom based on the controller's movements to maintain a stable projection on a surface.

Benefits of technology

Enables stable and portable remote gameplay by maintaining a consistent projection of gameplay video on various surfaces, allowing users to play video games remotely with a smooth viewing experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024542774000001_ABST
    Figure 2024542774000001_ABST
Patent Text Reader

Abstract

Provided is a controller device for interactive gameplay of a video game, the controller device being configured to be handheld by a user; at least one motion sensor that generates motion data in response to movement of the controller device when held in the user's hand during interactive gameplay of the video game; a wireless communication device that receives gameplay video over a network from a remote execution session of the video game; and a projector that is integrated with the controller device and configured to project the gameplay video onto a projection surface in a local environment, wherein the projection of the gameplay video is stabilized based on the motion data.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure relate to a method and system for remote play using a local projector. [Background technology]

[0002] 2. Description of Related Art A projector can receive a video signal and project the video onto a surface, allowing a viewer to view a potentially large image that is appealing, and even allowing an existing surface to be used as a video display area. Continuing advances in technology are leading to smaller projectors, including the development of small form factors such as those seen in today's state-of-the-art picoprojectors.

[0003] Video game technology continues to evolve, and advances in computing and networking technology now enable gameplay of video games running on computing devices in remote locations.

[0004] It is in this context that embodiments of the present disclosure arise. Summary of the Invention

[0005] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure relate to a method and system for remote play using a local projector.

[0006] In some embodiments, a controller device for interactive gameplay of a video game is provided, the controller device being configured to be held in a user's hand; at least one motion sensor, the motion sensor generating motion data responsive to movement of the controller device when held in the user's hand during interactive gameplay of the video game; a wireless communication device that receives gameplay video over a network from a remote execution session of the video game; and a projector, the projector integrated with the controller device and configured to project the gameplay video onto a projection surface in a local environment, wherein the projection of the gameplay video is stabilized based on the motion data.

[0007] In some implementations, movements of the controller device are generated from interactive gameplay of a video game, and the motion data is provided as input to a remote execution session of the video game.

[0008] In some implementations, the remote execution session is executed by a cloud gaming service.

[0009] In some implementations, the remote execution session is executed by a remotely located gaming console.

[0010] In some implementations, the gameplay video depicts a view of the virtual environment of the video game.

[0011] In some implementations, stabilizing the gameplay video includes adjusting the orientation of the projector toward the location of the projection surface.

[0012] In some implementations, stabilizing the gameplay video includes adjusting a projector's keystone in response to lateral movement of the controller relative to the projection surface.

[0013] In some implementations, the stabilization of the gameplay video includes adjusting the zoom of the projector in response to movement of the controller towards or away from the projection surface.

[0014] In some embodiments, the at least one motion sensor includes one or more of an accelerometer, a gyroscope, or a magnetometer.

[0015] In some implementations, the controller device further includes at least one input device selected from a button, a trigger, a joystick, or a touchpad.

[0016] In some embodiments, a system for interactive gameplay of a video game is provided that includes a controller device configured to be held in a user's hand, the controller device having at least one motion sensor that generates motion data in response to movement of the controller device held in the user's hand during interactive gameplay of the video game; a projector configured to be removably connected to the controller device, the projector having a wireless communication device that receives gameplay video over a network from a remote execution session of the video game, the projector configured to project the gameplay video onto a projection surface in a local environment, and the projection of the gameplay video is stabilized based on the motion data.

[0017] In some implementations, movements of the controller device are generated from interactive gameplay of a video game, and the motion data is provided as input to a remote execution session of the video game.

[0018] In some implementations, the remote execution session is executed by a cloud gaming service.

[0019] In some implementations, the remote execution session is executed by a remotely located gaming console.

[0020] In some implementations, the gameplay video depicts a view of the virtual environment of the video game.

[0021] In some implementations, stabilizing the gameplay video includes adjusting the orientation of the projector toward the location of the projection surface.

[0022] In some implementations, stabilizing the gameplay video includes adjusting a projector's keystone in response to lateral movement of the controller relative to the projection surface.

[0023] In some implementations, stabilizing the gameplay video includes adjusting the zoom of the projector in response to movement of the controller towards or away from the projection surface.

[0024] In some embodiments, the at least one motion sensor includes one or more of an accelerometer, a gyroscope, or a magnetometer.

[0025] In some implementations, the controller device further includes at least one input device selected from a button, a trigger, a joystick, or a touchpad.

[0026] Other aspects and advantages of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the disclosure.

[0027] The present disclosure, together with further advantages thereof, may best be understood by reference to the accompanying drawings and the following description, wherein: [Brief description of the drawings]

[0028] [Figure 1] 1 conceptually illustrates a system for providing remote gameplay via a projector according to an embodiment of the present disclosure.

[0029] [Figure 2A] 1 illustrates a front view of a controller with a projector attached, according to an embodiment of the present disclosure.

[0030] [Figure 2B] 1 illustrates a front view of a controller with a projector attached, according to an embodiment of the present disclosure.

[0031] [Figure 2C] 1 illustrates a front view of a controller with integrated projector according to an embodiment of the present disclosure.

[0032] [Diagram 3] 1 conceptually illustrates a system for enabling remote game play using a projector attached to or integrated into a controller in accordance with an embodiment of the present disclosure.

[0033] [Figure 4] 1 conceptually illustrates a user operating a controller with a projector attached or integrated therein, according to an embodiment of the present disclosure.

[0034] [Diagram 5] FIG. 1 is a block diagram of a gaming system according to various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] Broadly speaking, embodiments of the present disclosure relate to implementations of a projector for enabling gameplay of a video game. In some implementations, the projector is a standalone device that can be enabled for remote gameplay of a video game run by a cloud video game service or by a remote computing device such as a game console or personal computer. In other implementations, the projector is integrated with a controller device that a user operates for gameplay. In related implementations, the projector is configured as an attachment that can be connected to an existing controller device. In such implementations, the image projected by the projector is stabilized using motion data captured by the controller device, so that the position of the image on a particular projection surface is maintained in a consistent manner, making it easy for the player to view.

[0036] FIG. 1 conceptually illustrates a system for providing remote gameplay via a projector, according to an embodiment of the present disclosure. In the illustrated embodiment, a user 100 manipulates a controller 102 to provide input to a video game. The video game is executed at a location / device remote from the user's location. For example, in some embodiments, the video game is executed in the cloud by a cloud gaming service 110. In such embodiments, the video game may be executed by a cloud gaming machine, a virtualized gaming machine, or other hardware or server resource capable of executing a video game session. In some embodiments, the video game is executed by a remote computing device (e.g., a gaming console, a computer, a gaming machine, etc.) and made available through a remote gaming service. The remote gaming service facilitates remote gameplay of the video game, such as facilitating the transmission of video from the remote computing device and the transmission of input to the remote computing device.

[0037] Projector 104 is configured to receive video from a video game, e.g., a cloud / remote gaming service 110, and display / project the video onto surface 106, which in the illustrated implementation is conceptually shown as projected video 108. By way of non-limiting example, surface 106 may be a wall, a ceiling, a projection screen, or any other surface suitable for projecting an image. In some implementations, controller 102 communicates with projector 104 via a wired or wireless connection, and input from controller 102 is transmitted from controller 102 to projector 104, which passes the input to a running video game for processing and to facilitate execution of the game.

[0038] It will be appreciated that in some implementations, the projector 104 can have a portable form factor for ease of transport. Additionally, the projector 104 includes all necessary components to enable the projection of an image onto a surface, such as electronics, light sources, and optics for receiving and processing video signals. In some implementations, the projector 104 includes a battery so that it can be used without being plugged into an outlet. In some implementations, the projector 104 includes wireless communication electronics to enable wireless communication over various networks / wireless protocols, such as 5G cellular, WiFi, Bluetooth, etc.

[0039] It will be appreciated that the projector 104 as described allows a user to play video games remotely. For example, a user may be away from their home where their gaming console is located, but the projector 104 may be used to allow the user to remotely play a video game running on the gaming console at the user's home. That is, the video game may be run remotely on the user's gaming console, and the video may be transmitted over one or more networks to the projector 104 for display on a surface in the user's local environment. In some implementations, the transmission of video may be enabled through a remote gaming service. In other implementations, the video game is run by a cloud gaming service as described above. With appropriate connectivity and a suitable projection surface for viewing, the projector 104 constitutes a portable device that allows a user to enjoy gameplay of a video game anywhere.

[0040] In some implementations, the projector 104 is configured to project an anchor image onto the projection surface that is used to track the controller, i.e., the position / orientation of the controller 102 in the local environment can be determined, at least in part, based on images captured by a camera included in the controller, by identifying and tracking at least a portion of the projected anchor image in the captured images.

[0041] In some implementations, projector 104 includes a video pass-through connector, such as an HDMI input, a DVI input, or other video input, through which projector 104 can receive video from another source or device and project it onto the projection surface.

[0042] In some implementations, the projector is connected to or integrated with a controller device that provides input to a video game. The projector projects a video of the game and can utilize motion data from the controller to stabilize the projected image of the projector.

[0043] 2A illustrates a front view of a controller with a projector attached thereto, according to an embodiment of the present disclosure. In the illustrated embodiment, the projector 206 is attached to the underside of the controller 200. The controller 200 includes various input devices, such as a joystick 202 and triggers / buttons 204. The configuration of attaching the projector 206 to the underside of the controller 200 exposes the top surface of the controller 200, allowing easy access to the joystick 202 and any other input devices disposed on the top surface, such as buttons or touch pads. It will be appreciated that the projector 206 is configured to be removably connected to and communicate with the controller 200. As described herein, the projector 206 receives motion data from the controller 200 to enable stabilization of the projected image.

[0044] 2B shows a front view of the controller with a mounted projector, according to an embodiment of the present disclosure. In the embodiment shown, the projector 208 is mounted on the top side of the controller 200. The mounting configuration of the projector 208 is such that the projector 208 is elevated and supported above the top surface of the controller 200, thereby allowing input devices on the top surface of the controller to remain accessible. It will be appreciated that the projector 208 is configured to be removably connected to the controller 200, to communicate with the controller 200, and to receive motion data from the controller 200 to enable stabilization of the projected image.

[0045] 2C shows a front view of a controller with an integrated projector according to an embodiment of the present disclosure. In the embodiment shown, the controller 210 includes a projector 212. The image from the projector is stabilized as described herein.

[0046] 3 conceptually illustrates a system for enabling remote gameplay using a projector attached to or integrated into a controller, according to embodiments of the present disclosure. In the embodiment shown, projector 300 is communicatively connected to controller 320. Projector 300 can be removably connected to the controller via an attachment mechanism, or in other embodiments, projector 300 can be integrated with controller 320.

[0047] The controller 320 includes one or more input devices 330, such as a joystick, buttons, directional pad, touch pad, biometric sensors, etc., for generating video game input. The controller 320 further includes one or more motion sensors 326 for detecting the position, orientation, and movement of the controller 320. In some implementations, the motion sensors 326 may include one or more accelerometers, magnetometers, and gyroscopes. In some implementations, the motion sensors 326 are defined in the form of an inertial measurement unit (IMU). In some implementations, the controller 326 includes one or more cameras 328, which can be used for tracking, such as by using simultaneous location detection and mapping (SLAM) techniques.

[0048] The controller 326 includes at least one processor 322 configured to execute controller tracking logic 324, which determines and tracks the position / location and orientation of the controller and the movement of the controller based on motion data from the aforementioned motion sensor 326 and / or image data from the camera 328. In some implementations, the controller 320 further includes one or more haptic devices 332 that can be activated by the controller to provide haptic feedback to the user, such as by generating vibrations or other haptic sensations.

[0049] Projector 300 is connected to or integrated with controller 320. Projector includes battery 316 for powering its components. Projector includes wireless communication module 314 that enables wireless communication (e.g., WiFi, 5G, etc.) with a remote game session 342 of a video game over network 340. As mentioned above, in some implementations, remote game session 342 is performed by a cloud gaming service, and in some implementations, remote game session 342 is performed by a user's remote gaming device, such as a gaming console or computer. Video from the remote game session is streamed over network 340 to projector 300 for display.

[0050] The projector includes a processor 302 that executes video logic 306 to process and modify input video for projection. The processor 302 further executes stabilization logic 304 configured to stabilize the projected image of the projector 300 based on motion data from the controller 320 to maintain the projected image in a substantially stationary configuration (e.g., stationary position, orientation, magnification, keystone, brightness / color consistency, etc.). In some implementations, the stabilization logic 304 communicates with an orientation controller 308 to control and adjust the projection orientation / tilt (pitch, yaw, roll, etc.) to stabilize the projected image. In some implementations, the stabilization logic 304 communicates with an optical controller 312 to control and adjust the optical properties of the projector, such as zoom (size) and keystone, to stabilize the projected image. In some implementations, the stabilization logic 304 communicates with an image generation controller 310 to adjust the generation of the projected image to stabilize the projected image, for example by adjusting the brightness, color intensity, or other image properties of various areas of the projection to stabilize the image.

[0051] It will be appreciated that the motion and / or image data from the controller 320 can be used as input for the remote game session 342 of the video game. In such an implementation, the motion / image data can be communicated to the remote game session 342 over the network 340 via the projector, or by the controller itself to the remote game session over the network. In such an implementation, the motion data of the controller 320 serves a dual purpose: as input to the game and as data that enables stabilization of the projected video image of the projector 300.

[0052] In some implementations, image data from camera 328 is processed and used as feedback to the system to determine and verify that the projected video is properly stabilized. For example, the image data can be analyzed to identify the projected video on the projection surface, as well as to identify other anchors in the local environment, and to determine whether the projected video is maintained in a stable position on the projection surface.

[0053] In some implementations, a motion sensor 326 and / or a camera 328 are included in the projector 300, and data therefrom is processed by the stabilization logic 304 in the projector.

[0054] In some implementations, projector 300 receives motion data and / or image data generated by motion sensor 326 and camera 328, respectively, from controller 320. In some implementations, projector 300 receives such data after it is first processed in the controller by controller tracking logic. In some implementations, projector 300 receives processed motion information derived from the motion / image data, indicating motion detected based on the motion / image data.

[0055] In some implementations, the remote gaming session 342 is configured to optimize the rendering of gameplay video based on the detected position / orientation and / or movement of the controller or projector. For example, the remote gaming session 342 can adjust the brightness or color intensity or other graphical properties of areas of the video based on such information, thereby improving the stability of the image quality of the projected video and compensating for projector movement.

[0056] In some implementations, in response to a sudden change in projector / controller orientation, the virtual view is temporarily reoriented in a similar direction (in the virtual environment of the video game), thereby rendering an augmentation of the view in the direction the projector moved, and then when the projector stabilizes and returns to its original orientation, the virtual view is also reoriented back to its original direction in the virtual environment.

[0057] Therefore, in view of the above, a hybrid projector stabilization technique is provided in which stabilization is performed by both the projector 300 and the remote game session 342.

[0058] 4 conceptually illustrates a user operating a controller having an attached or integrated projector in accordance with an embodiment of the present disclosure. In the embodiment shown, a user 400 operates a controller 402 having an attached or integrated projector 404. As described above, the projected video 406 of a video game is stabilized to enhance the user's viewing experience during gameplay.

[0059] In some implementations, a zone 408 is defined in the local environment that defines the approximate limit of the combined controller / projector device's ability to stabilize the projected video 406. For example, if the controller (and projector) is moved up / down / left / right / forward / backward significantly, then the projected video can no longer be stabilized in the same position on the projection surface, such as by adjusting keystone and / or adjusting zoom. Thus, in some implementations, the zone 408 defines an area of ​​the local environment, such that when the controller reaches the boundary / edge of the zone 408, feedback is given to the user to alert the user that the projected video may no longer be stabilized in the same place if the controller continues to move beyond the zone 408. In some implementations, the feedback is in the form of a pop-up display displayed on the projected video. In some implementations, the feedback is in the form of an audio warning / sound. In some implementations, the feedback is in the form of haptic feedback provided through the controller 402.

[0060] Similarly, if the direction / tilt (pitch / yaw / roll) of the controller is rotated beyond a certain amount (e.g., off a specified central axis), the projected video may no longer be stable in place. Thus, in some implementations, a maximum amount of rotation off axis may be defined, and similar feedback may be given to the user when such maximum is reached.

[0061] In some implementations, rather than stabilizing the projected video to keep it in a stationary position, the orientation of the controller is used to control the orientation of the virtual view. In this way, as the controller moves and / or rotates, the physical location of the projected video also moves, resulting in a corresponding movement of the orientation of the virtual view to display different portions of the virtual environment. The effect is similar to moving / orienting the controller to move / orient a virtual camera, so that wherever the virtual camera is moved (depending on the controller movement), that area of ​​the virtual environment captured by the virtual camera is rendered and projected for the user to see. In such implementations, the projected video image acts as a moveable window into the virtual environment. In some implementations, this projector operation mode can be characterized as a moveable virtual display mode, as opposed to the fixed display mode discussed above. In some implementations, during the moveable virtual display mode, the stabilization discussed above is not performed. However, lack of any stabilization of the projector can lead to unnecessary shaking of the projected video due to a user's shaking hands, or an overall lack of smoothness in the video viewing experience as even small movements of the projector can result in large movements of the projected video from a distance.

[0062] Thus, in some implementations, during the movable virtual display mode, the projector is stabilized in a manner that counteracts the user's shaking and smooths out rapid projector movements. It will be appreciated that in such implementations, the controller movements are used for both stabilization and also for controlling the virtual camera. This allows the user to comfortably view the projected video, but also allows the virtual camera to be oriented to view different parts of the virtual environment.

[0063] In some implementations, the movable virtual display mode and the fixed display mode are enabled at different times. In some implementations, a user can switch between the two modes using an input mechanism such as pressing a button or activating a trigger. In some implementations, the fixed display mode is active by default, with the system switching to the movable virtual display mode in response to continuously applied input, such as holding down a button or trigger, and switching back to the stable display mode when the input is no longer applied, such as releasing the button or trigger.

[0064] In some implementations, a fixed display mode is active when the controller is oriented substantially in a first direction (e.g., facing a front wall), but when the controller is oriented substantially in a second direction (e.g., facing a ceiling), the system switches to a movable virtual display mode. In some implementations, the switch to the movable virtual display mode occurs in response to the orientation or position of the controller changing by more than a predefined threshold amount.

[0065] Although embodiments of the present disclosure have been described with reference to video games, it will be appreciated that in other embodiments, the principles of the present disclosure may be applied to other types of interactive applications capable of projecting video in accordance with the above.

[0066] In some implementations, the projected video is configured to take into account the three-dimensional geometry of the projection surface. For example, if the projection surface is not planar, the projected video may be pre-distorted so that the video does not appear distorted to the user when projected onto the projection surface.

[0067] FIG. 5 is a block diagram of a gaming system 1600 according to various embodiments of the present disclosure. The gaming system 1600 is configured to provide video streams to one or more clients 1610 over a network 1615. The gaming system 1600 typically includes a video server system 1620 and an optional game server 1625. The video server system 1620 is configured to provide video streams to one or more clients 1610 with a minimum quality of service. For example, the video server system 1620 may receive game commands that change the state of a video game or a viewpoint within a video game, and provide an updated video stream reflecting the change to the client 1610 with minimal latency. The video server system 1620 may be configured to provide video streams in a variety of alternative video formats, including formats that are yet to be defined. Additionally, the video streams may include video frames configured to be presented to a user at a variety of frame rates. Typical frame rates are 30 frames per second, 60 frames per second, and 120 frames per second. However, alternative implementations of the present disclosure include higher or lower frame rates.

[0068] The clients 1610, individually referred to herein as 1610A, 1610B, etc., may include HMDs, terminals, personal computers, game consoles, tablet computers, phones, set-top boxes, kiosks, wireless devices, digital pads, standalone devices, handheld game play devices, projectors, and / or the like. Typically, the clients 1610 are configured to receive encoded video streams, decode the video streams, and present the resulting video to a user (e.g., a player of a game). The process of receiving an encoded video stream and / or decoding a video stream typically involves storing individual video frames in a receive buffer of the client. The video stream may be presented to the user on a display integrated with the client 1610 or on another device such as a monitor or television. The clients 1610 are optionally configured to support multiple game players. For example, a game console may be configured to support two, three, four or more simultaneous players. Each of these players may receive a separate video stream, or a single video stream may include a region of frames generated specifically for each player (e.g., generated based on each player's viewpoint). Clients 1610 are optionally geographically distributed. The number of clients included in gaming system 1600 may vary widely, from one or two to thousands, tens of thousands, or more. As used herein, the term "game player" is used to refer to a person who plays a game, and the term "game playing device" is used to refer to a device used to play a game. In some implementations, game playing devices may refer to multiple computing devices that cooperate to provide a gaming experience to a user. For example, a gaming console and an HMD / MR headset may cooperate with video server system 1620 to deliver a game viewed by the HMD / MR headset.In one implementation, the gaming console receives the video stream from the video server system 1620 and the gaming console forwards the video stream or updates to the video stream to the HMD for rendering.

[0069] Client 1610 is configured to receive the video streams via network 1615. Network 1615 can be any type of communications network, including a telephone network, the Internet, a wireless network, a power line network, a local area network, a wide area network, a private network, and / or the like. In an exemplary implementation, the video streams are communicated via standard protocols such as TCP / IP or UDP / IP. Alternatively, the video streams are communicated via proprietary standards.

[0070] A typical example of a client 1610 is a personal computer with a processor, non-volatile memory, a display, decoding logic, network communication capabilities, and input devices. The decoding logic may include hardware, firmware, and / or software stored on a computer-readable medium. Systems for decoding (and encoding) video streams are known in the art and vary depending on the particular encoding scheme used.

[0071] Additionally, the clients 1610 may, but need not, include systems configured to modify the received video. For example, the clients may be further configured to render, overlay one video image on another, crop video images, and / or the like. For example, the clients 1610 may be configured to receive various types of video frames, such as I-frames, P-frames, and B-frames, and process these frames into images for display to a user. In some implementations, members of the clients 1610 are configured to perform further rendering, shading, conversion to 3D, or similar manipulation of the video stream. Members of the clients 1610 are optionally configured to receive multiple audio or video streams. The input devices of the clients 1610 may include, for example, a one-handed game controller, a two-handed game controller, a gesture recognition system, an eye gaze recognition system, a voice recognition system, a keyboard, a joystick, a pointing device, a force feedback device, a motion and / or position sensing device, a mouse, a touch screen, a neural interface, a camera, an input device under development, and / or the like.

[0072] The video stream (and optionally the audio stream) received by the client 1610 is generated and provided by the video server system 1620. As further described elsewhere herein, the video stream includes video frames (and the audio stream includes audio frames). The video frames are configured to meaningfully contribute to an image displayed to a user (e.g., a video frame includes pixel information in a suitable data structure). As used herein, the term "video frame" is used to refer to a frame that primarily includes information configured to contribute to, e.g., act on, an image shown to a user. Most of the teachings herein regarding "video frames" can also be applied to "audio frames."

[0073] The client 1610 is typically configured to receive inputs from a user. These inputs may include game commands configured to change the state of a video game or otherwise affect game play. The game commands may be received using an input device and / or may be generated automatically by computer processing instructions executing on the client 1610. The received game commands are communicated from the client 1610 to the video server system 1620 and / or the game server 1625 over the network 1615. For example, in some implementations, the game commands are communicated to the game server 1625 via the video server system 1620. In some implementations, separate copies of the game commands are communicated from the client 1610 to the game server 1625 and the video server system 1620. The communication of the game commands optionally depends on the identity of the command. The game commands are optionally communicated from the client 1610A over a different route or communication channel used to provide the audio or video stream to the client 1610A.

[0074] The game server 1625 is optionally operated by a different entity than the video server system 1620. For example, the game server 1625 may be operated by a publisher of a multiplayer game. In this example, the video server system 1620 is optionally viewed as a client by the game server 1625 and is optionally configured to appear from the perspective of the game server 1625 to a conventional client running a conventional game engine. Communication between the video server system 1620 and the game server 1625 is optionally performed over the network 1615. Thus, the game server 1625 may be a conventional multiplayer game server that transmits game state information to multiple clients, one of which is the video server system 1620. The video server system 1620 may be configured to communicate with multiple instances of the game server 1625 simultaneously. For example, the video server system 1620 may be configured to provide multiple different video games to different users. Each of these different video games may be supported by a different game server 1625 and / or published by a different entity. In some implementations, several geographically distributed instances of the video server system 1620 are configured to provide game videos to multiple different users. Each of these instances of the video server system 1620 may communicate with the same instance of the game server 1625. Communication between the video server system 1620 and one or more game servers 1625 optionally occurs over a dedicated communication channel. For example, the video server system 1620 may be connected to the game server 1625 over a high-bandwidth channel dedicated to communication between these two systems.

[0075] The video server system 1620 comprises at least a video source 1630, an I / O device 1645, a processor 1650, and non-transitory storage 1655. The video server system 1620 may include one computing device or may be distributed among multiple computing devices, optionally connected via a communication system such as a local area network.

[0076] The video source 1630 is configured to provide a video stream (e.g., a series of video frames forming a streaming video or a movie). In some implementations, the video source 1630 includes a video game engine and rendering logic. The video game engine is configured to receive game commands from a player and maintain a copy of the video game state based on the received commands. The game state includes the positions of objects in the game environment and typically the viewpoint. The game state may also include object properties, images, colors, and / or textures. The game state is typically maintained based on game rules and game commands for movement, turning, attacking, setting focus, interacting, using, and / or the like. A portion of the game engine is optionally located in a game server 1625. The game server 1625 may maintain a copy of the game state based on game commands received from multiple players using geographically distributed clients. In these cases, the game state is provided by the game server 1625 to the video source 1630, and a copy of the game state is stored and rendered. The game server 1625 may receive game commands directly from the clients 1610 via the network 1615 and / or via the video server system 1620.

[0077] Video source 1630 typically includes rendering logic (e.g., hardware, firmware, and / or software stored in a computer-readable medium such as storage 1655). This rendering logic is configured to create video frames of a video stream based on the game state. All or part of the rendering logic is optionally located in a graphics processing unit (GPU). The rendering logic typically includes processing stages configured to determine three-dimensional spatial relationships between objects and / or apply appropriate textures, etc., based on the game state and viewpoint. The rendering logic generates raw video, which is then typically encoded before being communicated to client 1610. For example, the raw video may be encoded according to the Adobe Flash® standard, .wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith, MPG-x.Xvid.FFmpeg, x264, VP6-8, realvideo, or mp3, etc. The encoding process produces a video stream that is optionally packaged for delivery to a decoder on a remote device. The video stream is characterized by a frame size and a frame rate. Typical frame sizes include 800x600, 1280x720 (e.g., 720p), 1024x768, although any other frame size may be used. The frame rate is the number of video frames per second. A video stream may include various types of video frames. For example, the H.264 standard includes "P" frames and "I" frames. An I frame contains information that updates all macroblocks / pixels on a display device, while a P frame contains information that updates a subset of them. P frames are typically smaller in data size than I frames. As used herein, the term "frame size" is meant to refer to the number of pixels in a frame. The term "frame data size" is used to refer to the number of bytes required to store a frame.

[0078] In an alternative embodiment, video source 1630 includes a video recording device, such as a camera. The camera may be used to generate delayed or live video that may be included in the video stream of a computer game. The resulting video stream optionally includes both rendered images and images recorded using a still camera or a video camera. Video source 1630 may also include a storage device configured to store pre-recorded video that is included in the video stream. Video source 1630 may also include a motion or position sensing device configured to detect the movement or position of an object (e.g., a person) and logic configured to determine a game state or to produce video based on the detected movement and / or position.

[0079] Video source 1630 is optionally configured to provide overlays that are configured to be placed over other video. For example, these overlays may include command interfaces, login prompts, messages to the game player, images of other game players, and video feeds of other game players (e.g., webcam videos). In implementations of client 1610A that include a touchscreen interface or an eye-gaze detection interface, the overlays may include a virtual keyboard, joystick, touchpad, and / or the like. In one example of an overlay, the player's voice is overlaid on the audio stream. Optionally, video source 1630 further includes one or more audio sources.

[0080] In embodiments in which video server system 1620 is configured to maintain game state based on input from multiple players, each player may have a different viewpoint, including viewing position and direction. Video source 1630 is optionally configured to provide separate video streams for each player based on the player's viewpoint. Additionally, video source 1630 may be configured to provide different frame sizes, frame data sizes, and / or encoding to each of clients 1610. Video source 1630 is optionally configured to provide 3D video.

[0081] The I / O devices 1645 are configured for the video server system 1620 to transmit and / or receive information such as video, commands, requests for information, game state, gaze information, device movement, device location, user movement, client identification information, player identification information, game commands, security information, audio, and / or the like. The I / O devices 1645 typically include communications hardware such as a network card or modem. The I / O devices 1645 are configured to communicate with the game server 1625, the network 1615, and / or the clients 1610.

[0082] The processor 1650 is configured to execute logic (e.g., software) included within the various components of the video server system 1620 described herein. For example, the processor 1650 may be programmed with software instructions to perform the functions of the video source 1630, the game server 1625, and / or the client qualifier 1660. The video server system 1620 optionally includes multiple instances of the processor 1650. The processor 1650 may also be programmed with software instructions to execute commands received by the video server system 1620 or to coordinate the operation of the various elements of the game system 1600 described herein. The processor 1650 may include one or more hardware devices. The processor 1650 is an electronic processor.

[0083] Storage 1655 includes non-transitory analog and / or digital storage devices. For example, storage 1655 may include an analog storage device configured to store video frames. Storage 1655 may include computer-readable digital storage (e.g., a hard drive, optical drive, or solid-state storage). Storage device 1615 is configured (e.g., by a suitable data structure or file system) to store video frames, artificial frames, video streams including both video frames and artificial frames, audio frames, audio streams, and / or the like. Storage 1655 is optionally distributed among multiple devices. In some implementations, storage 1655 is configured to store software components of video source 1630 described elsewhere herein. These components may be stored in a format that allows them to be readily provided as needed.

[0084] Optionally, the video server system 1620 further includes a client qualifier 1660. The client qualifier 1660 is configured to remotely determine the capabilities of a client, such as client 1610A or 1610B. These capabilities may include both the capabilities of the client 1610A itself and the capabilities of one or more communication channels between the client 1610A and the video server system 1620. For example, the client qualifier 1660 may be configured to test the communication channel via the network 1615.

[0085] The Client Qualifier 1660 can determine (e.g., know) the capabilities of the Client 1610A manually or automatically. Manual determination includes communicating with a user of the Client 1610A and asking the user to provide the capabilities. For example, in some implementations, the Client Qualifier 1660 is configured to display images, text, and / or the like in a browser of the Client 1610A. In one implementation, the Client 1610A is an HMD / MR headset that includes a browser. In another implementation, the Client 1610A is a gaming console with a browser, and the browser may be displayed in the HMD / MR headset. The displayed objects request the user to input information such as the operating system, processor, video decoder type, network connection type, display resolution, etc. of the Client 1610A. The information entered by the user is transmitted back to the Client Qualifier 1660.

[0086] The automated determination may be made, for example, by running an agent on the client 1610A and / or by sending a test video to the client 1610A. The agent may include computing instructions, such as JavaScript, embedded in a web page or installed as an add-on. The agent is optionally provided by the client qualifier 1660. In various implementations, the agent may discover the processing capabilities of the client 1610A, the decoding and display capabilities of the client 1610A, the latency reliability and bandwidth of the communication channel between the client 1610A and the video server system 1620, the display type of the client 1610A, any firewalls present on the client 1610A, the hardware of the client 1610A, software running on the client 1610A, registry entries in the client 1610A, and / or the like.

[0087] Client Qualifier 1660 includes hardware, firmware, and / or software stored on a computer-readable medium. Client Qualifier 1660 is optionally located on a computing device separate from one or more other elements of Video Server System 1620. For example, in some implementations, Client Qualifier 1660 is configured to determine characteristics of communication channels between Client 1610 and multiple instances of Video Server System 1620. In these implementations, information known by the Client Qualifier can be used to determine which instance of Video Server System 1620 is best suited for delivery of streaming video to one of the Clients 1610.

[0088] Implementations of the present disclosure can be practiced with a variety of computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. The present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0089] In view of the above embodiments, it should be understood that the present disclosure can employ various computer-implemented operations involving data stored in computer systems. These operations are operations requiring physical manipulation of physical quantities. Any of the operations described herein that form part of the present disclosure are useful machine operations. The present disclosure also relates to devices or apparatus for performing these operations. The apparatus can be specially constructed for the required purpose or the apparatus can be a general-purpose computer selectively activated or configured by a computer program stored in the computer. In particular, various general-purpose machines can be used with computer programs written in accordance with the teachings herein. Alternatively, it may be more convenient to construct a more specialized apparatus to perform the required operations.

[0090] The present disclosure may also be embodied as computer readable code on a computer readable medium. The computer readable medium is any data storage device that can store data which can then be read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read only memory, random access memory, optical media, non-optical data storage devices, etc. The computer readable medium may include computer readable tangible media distributed over network coupled computer systems such that the computer readable code is stored and executed in a distributed fashion.

[0091] Although the method operations have been described in a particular order, it should be understood that other housekeeping operations may be performed between the operations, or the operations may be adjusted to occur at slightly different times, or may be distributed within a system that allows processing operations to occur at various intervals related to processing, so long as the processing of the overlay operation is performed in the desired manner.

[0092] Although the foregoing disclosure has been described in some detail for purposes of clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Thus, the present embodiments should be considered as illustrative rather than restrictive, and the disclosure is not limited to the details given herein, but can be modified within the scope of the disclosure and equivalents.

Claims

1. 1. A controller device for interactive game play of a video game, comprising: the controller device configured to be hand-held by a user; at least one motion sensor, the motion sensor generating motion data responsive to movement of the controller device when held by the user during the interactive gameplay of the video game; a wireless communication device for receiving gameplay video from a remote execution session of said video game over a network; A controller device comprising: a projector, the projector integrated with the controller device, the projector configured to project the gameplay video onto a projection surface of a local environment, the projection of the gameplay video being stabilized based on the motion data.

2. The controller device of claim 1 , wherein the movements of the controller device are generated from interactive gameplay of the video game, and the motion data is provided as input to the remote execution session of the video game.

3. The controller device of claim 1 , wherein the remote execution session is executed by a cloud gaming service.

4. The controller device of claim 1 , wherein the remote execution session is executed by a remotely located gaming console.

5. The controller device of claim 1 , wherein the gameplay video depicts a view of a virtual environment of the video game.

6. The controller device of claim 1 , wherein the stabilization of the gameplay video includes adjusting an orientation of the projector toward a location of the projection surface.

7. The controller device of claim 1 , wherein the stabilization of the gameplay video includes adjusting a keystone of the projector in response to lateral movement of the controller relative to the projection surface.

8. The controller device of claim 1 , wherein the stabilization of the gameplay video includes adjusting a zoom of the projector in response to movement of the controller toward or away from the projection surface.

9. The controller device of claim 1 , wherein the at least one motion sensor includes one or more of an accelerometer, a gyroscope, or a magnetometer.

10. The controller device of claim 1 , further comprising at least one input device selected from a button, a trigger, a joystick, or a touchpad.

11. 1. A system for interactive game play of a video game, comprising: a controller device configured to be held in a hand of a user, the controller device having at least one motion sensor that generates motion data in response to movement of the controller device held in the hand of the user during the interactive gameplay of the video game; 11. A system comprising: a projector configured to be removably connected to the controller device, the projector having a wireless communication device that receives gameplay video from a remote execution session of the video game over a network, the projector configured to project the gameplay video onto a projection surface in a local environment, the projection of the gameplay video being stabilized based on the motion data.

12. The system of claim 11 , wherein the movement of the controller device is generated from interactive gameplay of the video game, and the motion data is provided as input to the remote execution session of the video game.

13. The system of claim 11 , wherein the remote execution session is executed by a cloud gaming service.

14. The system of claim 11 , wherein the remote execution session is executed by a remotely located gaming console.

15. The system of claim 11 , wherein the gameplay video depicts a view of a virtual environment of the video game.

16. The system of claim 11 , wherein the stabilization of the gameplay video includes adjusting an orientation of the projector toward a location of the projection surface.

17. The system of claim 11 , wherein the stabilization of the gameplay video includes adjusting a keystone of the projector in response to lateral movement of the controller relative to the projection surface.

18. The system of claim 11 , wherein the stabilization of the gameplay video includes adjusting a zoom of the projector in response to movement of the controller toward or away from the projection surface.

19. The system of claim 11 , wherein the at least one motion sensor includes one or more of an accelerometer, a gyroscope, or a magnetometer.

20. The system of claim 11 , wherein the controller device further includes at least one input device selected from a button, a trigger, a joystick, or a touchpad.

Citation Information

Patent Citations

  • Portable terminal

    JP2008096457A

  • Projector

    JP2009171012A

  • Projector and control method

    JP2010164605A

  • Game system

    JP2013017548A

  • Information processing device, terminal device, method for controlling information processing device, and method for controlling terminal device

    JP2021158605A