Intermediary device for cloud gaming

Abstract

To provide an intermediary device for a player to efficiently access a cloud game through a streaming device.SOLUTION: The intermediary device includes an HDMI output port, an HDMI input port, a pass-through portion, a game streaming portion, and a wireless connection to a controller that controls the video game. When a video game is selected from the user interface, the game streaming portion is activated to receive a compressed stream of the video game from the cloud gaming system, decode the compressed stream, and provide image data for rendering. When an application not related to the video game is selected, a video stream generated by the streaming device, which is an HDMI signal related to the application, is input from the HDMI input port and passed.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates generally to streaming video games, and more particularly to a method and system for processing video games streamed over a network from a cloud gaming system. [Background technology]

[0002] 2. Description of Related Art The video game industry has undergone many changes over the years. In particular, cloud gaming-related technologies have improved, transforming how video games are played, distributed, and sold. Currently, cloud gaming systems allow players to access and play video games over a network without having to download the video game to any game console or device. In contrast to previous gaming approaches in which the video game runs locally on the player's video game console, cloud gaming systems do not require expensive video game consoles or powerful computers to run the video game. Instead, cloud gaming systems can visualize the game on a remote server and stream it directly to the player device. This has led developers to seek ways to develop advanced operations that could improve cloud gaming.

[0003] The video game industry is increasingly focused on improving and developing unique ways to enhance cloud gaming systems and provide efficient ways for players to access cloud games. While third-party streaming devices, such as Apple® TV, Roku® streaming media players, Amazon® Fire TV Sticks, and Google® Chromecast®, are becoming increasingly popular, and streaming devices may provide access to a variety of media content, they are unable to support and facilitate cloud gaming. For example, consumers may use Roku streaming media players to stream a variety of media content and access a limited number of video games. Unfortunately, streaming devices, such as Roku® streaming media players, may not be able to support cloud gaming, which requires ultra-low latency video decoding and high processing power to effectively decode video game streams. Because traditional streaming devices have limited processing power, they are unable to effectively decode high-quality video game streams transmitted from cloud gaming systems. As a result, consumers who may wish to access video games from cloud gaming systems through streaming devices may be unable to access cloud gaming.

[0004] It is in these situations that embodiments of the present disclosure arise. Summary of the Invention

[0005] Implementations of the present disclosure include methods, systems, and devices related to processing video games streamed from a cloud gaming system over a network. In one embodiment, a pass-through device can be integrated into a user's streaming device (e.g., Apple® TV, Roku® Streaming Media Player, Amazon® Fire TV Stick, Google® Chromecast®, etc.) to provide the user with access to video games on a cloud gaming system. In this manner, the user's streaming device does not need to be modified with any customized hardware or software integration. With the pass-through device connected to the user's streaming device, the user can quickly access cloud games through the streaming device's user interface. In some embodiments, the user can continue to use the user's preferred streaming device and its corresponding user interface to access various streaming entertainment services (e.g., Netflix®, Hulu®, Amazon® Prime Video, YouTube®, etc.) and also access video games from the cloud gaming system.

[0006] Thus, a user can seamlessly switch between viewing media content and cloud gaming efficiently using a pass-through device connected to their preferred streaming device. In some embodiments, various operations necessary for cloud gaming may be supported and performed by the pass-through device rather than the streaming device. For example, the pass-through device may incorporate operations that may facilitate low-latency video decoding, high processing power, wireless connection to the device via Bluetooth, wireless connection to the cloud gaming server, low-latency processing of controller inputs and video frames, providing video frames for rendering by a video display device, etc.

[0007] In one embodiment, a method for processing a video game streamed from a cloud gaming system over a network is provided. The method includes receiving an indication of a selection of the video game for a game session through a user interface associated with a streaming device connected to a display screen via a pass-through device. The method includes receiving a signal from game streaming logic of the pass-through device that the game session is active. In one embodiment, the game streaming logic is configured to receive a compressed stream of the video game, decode the compressed stream of the video game, and provide image data for rendering by the display screen, and receive user input by the game streaming logic to drive interactivity of the video game while the cloud gaming system is running the video game.

[0008] In another embodiment, a pass-through device for streaming a video game from a cloud gaming system over a network is provided. The pass-through device includes an HDMI® output port. The pass-through device includes an HDMI® input port. The pass-through device includes pass-through logic. The pass-through device includes game streaming logic. The game streaming logic is configured to receive a compressed stream of the video game, decode the compressed stream of the video game, and provide image data for rendering by a display screen.

[0009] Other aspects and advantages of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the disclosure.

[0010] The present disclosure can be better understood by referring to the following description in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0011] [Figure 1] 1 illustrates an embodiment of a system configured to stream video games from cloud computing and game providers and media content from entertainment services provided to a user using a streaming device connected to a display screen via a pass-through device, in accordance with embodiments of the present disclosure.

[0012] [Figure 2] 1 illustrates an embodiment of a pass-through device and a streaming device connected to a display screen for facilitating cloud gaming and streaming media content, according to an embodiment of the present disclosure.

[0013] [Figure 3] 1 illustrates an embodiment of a pass-through device according to an embodiment of the present disclosure.

[0014] [Figure 4] 1 illustrates an embodiment of an HDMI® signal flowing from a streaming device to a pass-through device and a display screen when the pass-through device is not activated, according to an embodiment of the present disclosure.

[0015] [Figure 5] 10 illustrates an embodiment of a streaming device user interface and a corresponding PlayStation Now interface after a user selects a PlayStation Now application to start a gaming session, according to an embodiment of the present disclosure.

[0016] [Figure 6] 1 illustrates an embodiment of a system configured for cloud computing and streaming a video game selected by a user from a game provider, according to an embodiment of the present disclosure.

[0017] [Figure 7] 1 illustrates an embodiment of streaming a selected video game from a cloud computing and game provider and displaying the selected video game on a user's display screen, according to an embodiment of the present disclosure.

[0018] [Figure 8] 1 illustrates an embodiment of a controller wirelessly connected to a pass-through device, according to an embodiment of the present disclosure.

[0019] [Figure 9] 1 illustrates a method for processing video games streamed from a cloud gaming system over a network, according to an embodiment of the present disclosure.

[0020] [Figure 10] 1 illustrates components of an exemplary device that can be used to implement aspects of various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0021] The following embodiments of the present disclosure provide methods, systems, and devices for processing video games streamed from a cloud gaming system over a network. In one embodiment, a pass-through device can be integrated into a user's streaming device (e.g., Apple® TV, Roku® streaming media player, Amazon® Fire TV Stick, Google® Chromecast®, etc.) to provide the user with access to video games on the cloud gaming system. In one embodiment, the pass-through device and streaming device are connected to each other so that the devices can work in conjunction to provide the user with access to cloud games from the cloud gaming system and access to media content from various streaming entertainment services (e.g., Netflix®, Hulu®, Amazon® Prime Video, YouTube®, etc.). This facilitates a seamless way for consumers to stream video games from the cloud gaming system without giving up their preferred streaming device. For example, a user may own a Roku® streaming media player that is used to stream media content from various entertainment services. To access video games from the cloud gaming system, the user can simply connect the pass-through device to their Roku streaming media player.

[0022] In one embodiment, the pass-through device can be launched by selecting a video game or a video game-related application (e.g., a PlayStation®Now application) through a user interface associated with the Roku streaming media player. Once a video game is selected, a game session is activated, and the pass-through device is configured to receive, decode, and display a compressed stream of the video game on the user's display screen. Generally, the methods described herein provide a way to integrate a pass-through device with a user's streaming device so that cloud gaming is accessible without any hardware or software modifications to the user's streaming device.

[0023] By way of example, a method is disclosed that enables processing a video game streamed from a cloud gaming system over a network. The method includes receiving an indication of a selection of a video game for a game session through a user interface associated with a streaming device connected to a display screen via a pass-through device. In one example, a user may be watching a movie on a Roku® streaming device. After the movie ends, the user may want to play a video game on the cloud gaming system. To start a game session, the user may select the video game on the interface of the Roku® streaming device and begin gameplay.

[0024] In another embodiment, the method may further include receiving a signal from the game streaming logic of the pass-through device that the game session is active. In one example, the game streaming logic is configured to receive a compressed stream of the video game, decode the compressed stream of the video game, and provide image data for rendering by a display screen, and receive user input by the game streaming logic to drive interactivity in the video game while the cloud gaming system is running the video game. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without some or all of the specific details currently described. In other examples, well-known process operations have not been described in detail so as not to unnecessarily obscure the present disclosure.

[0025] According to one embodiment, a system for streaming online video games to a user over a network from a cloud gaming system is disclosed. In one embodiment, the system includes a streaming device connected to a display screen via a pass-through device. In one embodiment, the pass-through device may provide the user with access to cloud gaming. For example, when the pass-through device is not activated, the user may stream and view media content from any of the entertainment streaming services available through the streaming device (e.g., Netflix®, Hulu®, Amazon® Prime Video®, YouTube®, etc.). To activate and enable the pass-through device for cloud gaming, the user may select a video game or a video game-related application through a user interface of the streaming device.

[0026] In one embodiment, when the pass-through device is activated, a user can access and play multiple video games available on the cloud gaming system. In some embodiments, the pass-through device is configured to perform multiple operations that may facilitate cloud gaming, such as receiving a compressed stream of the video game, decoding the compressed stream of the video game, providing image data for rendering by a display screen, and receiving user input to drive interactivity in the video game. In this manner, a user can use the pass-through device for cloud gaming while still being able to use their streaming device to stream media content from various streaming entertainment services.

[0027] With the above summary in mind, the following provides several illustrative figures to facilitate understanding of example embodiments.

[0028] FIG. 1 illustrates an embodiment of a system configured to stream video games from a cloud computing and gaming provider 114 and media content from an entertainment service to a user 102 using a streaming device 106 connected to a display screen 116 via a pass-through device 104, according to embodiments of the present disclosure. In one embodiment, FIG. 1 illustrates a user 102, a pass-through device 104, a streaming device 106, a network 110, a cloud computing and gaming provider 114, and a display screen 116. As shown in FIG. 1, a user 102 is shown using a controller 108 to explore various features on a user interface 112 associated with the streaming device 106. In some embodiments, the user interface 112 may be displayed on another mobile device or any other device, such as a personal computer, a laptop, a tablet computer, a monitor and console / PC setup, a television and console setup, a peripheral, a tablet, a thin client, a set-top box, a network device / appliance, etc.

[0029] In one embodiment, streaming device 106 is wirelessly connected to an entertainment service provider (not shown) via network 110. In some embodiments, the entertainment service provider may stream and provide users with access to multiple media content. For example, streaming device 106 may be a media streaming device such as an Apple® TV, a Roku® streaming media player, an Amazon® Fire TV stick, a Google® Chromecast®, or the like. Each streaming device 106 may offer different entertainment services and provide users with different user interfaces.

[0030] For example, in one embodiment, as shown in FIG. 1 , user interface 112 of streaming device 106 may include multiple applications for accessing different types of streaming entertainment services. As shown, home menu 124 may include various applications 118a-118n that user 102 can access. Home menu 124 includes PlayStation Now application 118a, music application 118b, movie application 118c, news application 118d, sports application 118e, TV show application 118f, anime application 118g, and internet application 118n. In one embodiment, user interface 112 may include a search 122 feature that allows user 102 to quickly search for a particular application, media content, or video game. In other embodiments, user interface 112 may include a settings 120 feature that allows a user to configure various features associated with the streaming device, such as audio settings, display settings, account settings, etc. It should be understood that the home menu 124 and user interface 112 can take a variety of forms and layouts, and this example is provided to illustrate the interactivity associated with the streaming device 106 interface.

[0031] In some embodiments, the pass-through device 104 may enable cloud gaming for the user 102. As shown in FIG. 1 , in one embodiment, the pass-through device 104 may be connected to an input of the display screen 116, and the streaming device 106 may be connected to an input of the pass-through device 104. When the user 102 interacts with the user interface 112 of the streaming device 106, the user may have the option of selecting a video game or video game-related application (e.g., PlayStation®Now 118a or other video game access application) capable of initiating a game session or generating a signal requesting the simulation of a game session. When a video game or video game-related application is selected, the pass-through device 104 is activated and a signal is sent to the cloud computing and gaming provider 114, which initiates or triggers a request to initiate processing of the selected video game.

[0032] In one embodiment, cloud computing and game provider 114 maintains and runs the video game being played by user 102. In some embodiments, cloud computing and game provider 114 is configured to receive input from controller 108 operated by user 102 over network 112. As user 102 interacts with the video game, the user provides input via controller 108, and game commands are sent to cloud computing and game provider 114. In some embodiments, cloud computing and game provider 114 is configured to process the input to affect the game state of the running video game. Output from the running video game, such as video data, audio data, etc., is compressed and sent to pass-through device 104 for decoding. In some embodiments, pass-through device 104 is configured to receive and decode the compressed stream of the video game. In one embodiment, the video game stream may be presented to user 102 on display screen 116 and / or on a separate device, such as a monitor or television.

[0033] In some embodiments, the pass-through device 104 may operate without connecting the streaming device 106 to the pass-through device 104. In one embodiment, when the pass-through device 104 is connected to a display screen 116, a user interface 112 associated with the pass-through device 104 is displayed on the display screen. In one embodiment, the user interface 112 associated with the pass-through device 104 may include a game catalog application that provides a user with a list of video games available for cloud gaming. For example, if a user decides to remove the streaming device 106 from the pass-through device 104, the user interface associated with the pass-through device 104 will appear on the display in place of the user interface associated with the streaming device 106. In this manner, the pass-through device 104 is configured to operate independently, without connecting the streaming device 106 to the pass-through device 104. In other words, a user can access cloud gaming even when the pass-through device 104 is connected to a display screen 116 without the streaming device 106 being connected to the pass-through device.

[0034] 2 illustrates an embodiment of a pass-through device 104 and a streaming device 106 connected to a display screen 116 for facilitating cloud gaming and streaming media content. In some embodiments, the pass-through device 104 may be integrated with the streaming device 106 to provide users with access to video games on a cloud gaming system. As shown in FIG. 2, the HDMI output port of the pass-through device 104 is connected to the HDMI input port of the display screen 116, and the HDMI output port of the streaming device 106 is connected to the HDMI input port of the pass-through device 104.

[0035] In one embodiment, streaming device 106 may include antenna 206, device streaming service 208, and device logic 210. In one embodiment, antenna 206 is configured to provide wireless Internet access. In some embodiments, device streaming service 208 of streaming device 106 may provide user 102 with access to various content via multiple applications 118a-118n. In one embodiment, applications 118a-118n may include content such as video games, movies, sports, music, live TV, etc. In some embodiments, device logic 210 is configured to process data received from a server of device streaming service 208. In one embodiment, device logic 210 is configured to determine and decode which of applications 118a-118n is selected by the user. In response to the selected application 118, device logic 210 sends a signal to the corresponding device streaming service provider to request a video stream of the selected content.

[0036] For example, user 102 may select application 118 to watch a movie, in which case a request for a stream of the movie is sent to the corresponding device streaming service provider. The movie stream is then transmitted and received by streaming device 106 for decoding, where it may be displayed on display screen 116. In another embodiment, user 102 may select a video game (e.g., a PSNow® application) to initiate a gaming session, in which a request for a stream of the selected video game is sent to cloud computing and gaming provider 114.

[0037] 2, when the user 102 selects the PlayStation Now 118a application or video game to begin a gaming session, a first HDMI® signal 212a is transmitted from the streaming device 106 and received by the game streaming logic 204 of the pass-through device 104 for processing. In one embodiment, the selection of the PlayStation Now 118a application or video game launches the pass-through device 104, where the game streaming logic 204 of the pass-through device 104 receives and processes the first HDMI® signal 212a. In one embodiment, the first HDMI® signal 212a may include a compressed stream of the video game, which may be decoded by the game streaming logic 204 and provided to the display screen 116 for rendering of the selected video game.

[0038] As further shown in FIG. 2, if the user 102 selects an application that does not include a cloud gaming video game (e.g., Netflix®, Hulu®, Amazon® Prime Video, etc.), a second HDMI® signal 212b is transmitted from the streaming device 106 and received by the pass-through logic 202 of the pass-through device 104. In one embodiment, the second HDMI® signal 212b may include a compressed stream of the media content selected by the user. Because the second HDMI® signal 212b passes through the pass-through logic 202, the pass-through device 104 is not activated, and as a result, the second HDMI® signal 212b remains untouched, as if the streaming device 106 had not been connected to the pass-through device 104. In other words, the pass-through logic 202 allows the second HDMI® signal 212b to pass through without any processing by the pass-through device 104. In this embodiment, the user can interact with their streaming device 106 as before, as if the pass-through device 104 did not exist. When the second HDMI® signal 212b is received by the display screen 116, the display screen may render the selected media content.

[0039] In some embodiments, the pass-through device 104 may include a high-resolution upscaling module. In one embodiment, the high-resolution upscaling module is configured to receive an HDMI® signal from the streaming device 106 for processing to generate a higher resolution output of the HDMI® signal. In one embodiment, the high-resolution upscaling module may incorporate an artificial intelligence algorithm configured to use the HDMI® signal to predict a higher resolution output. For example, the HDMI® signal from the streaming device may be a movie at 1080p resolution. When receiving the HDMI® signal, the high-resolution upscaling module is configured to convert the HDMI® signal to a higher quality, such as 4K resolution.

[0040] 3 illustrates an embodiment of a pass-through device 104 in accordance with an embodiment of the present disclosure. In one embodiment, the pass-through device 104 includes an HDMI® input port 302, an HDMI® output port 304, pass-through logic 202, and game streaming logic 204. In one embodiment, the HDMI® output port 304 of the pass-through device 104 may be connected to an HDMI® input port of a display screen 116. In another embodiment, the output of a streaming device 106, such as an Apple® TV, Roku®, Fire TV®, cable box, or Blu-ray player, may be connected to the HDMI® input port 302 of the pass-through device 104. Thus, an existing streaming device owned or preferred by a user 102 may be integrated with the pass-through device 104 to provide the user 102 with access to video games from the cloud gaming system.

[0041] In some embodiments, during operation of non-video game-related media applications streamed from the cloud gaming system, such as movies, music, live TV, etc., the HDMI signal of the media content passes through the pass-through logic 202. In one embodiment, when the HDMI signal passes through the pass-through logic 202, the pass-through device 104 remains inactive, and the streaming device 106 operates without any interruption by the pass-through device 104. In other words, the streaming device 106 operates as if the pass-through device 104 and the streaming device 106 were not connected. In other embodiments, the pass-through logic 202 can process data associated with the HDMI signal. For example, the user 102 may be using the streaming device 106 to watch live TV. The HDMI signal may include data indicating that the user 102 regularly watches college basketball or hockey at night. To that end, the data received by the pass-through logic 202 can be processed to determine user behavior and the types of media content the user may be interested in viewing.

[0042] In some embodiments, when an application related to cloud gaming video games is selected by a user, the pass-through device 104 is activated, resulting in an HDMI signal passing through and being processed by the game streaming logic 204. In one embodiment, when the pass-through device 104 is activated, the pass-through device 104 is configured to connect to and communicate with the cloud computing and game provider 114. In one example, the game streaming logic 204 may include multiple operations configured to support cloud gaming, such as a Bluetooth module 306, operation of a video decoder 308, operation of a streaming player 310, and operation of a game catalog 312. In one embodiment, the Bluetooth module 306 may enable wireless connections to various types of devices. For example, the Bluetooth module 306 may enable the controller 108 of the user 102 to wirelessly connect with the pass-through device 102.

[0043] In one embodiment, the operation of the video decoder 308 may be configured to decode the compressed stream of the video game to render decompressed video images to be displayed on the display screen 116. In some embodiments, the video decoder 308 may include operations for storing individual video frames in a receive buffer of the pass-through device 104. In some embodiments, the operation of the streaming player 310 may be configured to communicate with the cloud computing and game provider 114 to request the video game selected by the user 102. For example, when the user 102 makes a video game selection through a user interface associated with the streaming device, the cloud computing and game provider 114 automatically begins streaming the selected video game to the pass-through device 104.

[0044] In one embodiment, the games catalog 312 may serve as the primary user interface for the user 102. For example, if the streaming device is disconnected or removed from the pass-through device, the pass-through device is configured to operate independently without the streaming device. In this embodiment, the user interface associated with the pass-through device may include a user-selectable games catalog 312 application.

[0045] 4 illustrates an embodiment of an HDMI® signal 302 flowing from a streaming device 106 to the pass-through device 104 and to a display screen 116 when the pass-through device 104 is not activated. In one embodiment, the streaming device 106 may be a device such as an Apple® TV, a Roku® media player, or a Fire TV® configured to receive a video stream of media content (e.g., movies, music, live TV, etc.) selected by a user. In some embodiments, the streaming device 106 is configured to process the video stream and output an HDMI® video signal 302. As shown in FIG. 4 , the HDMI® video signal 302 flows from the streaming device 106 and is received by the pass-through device 104. After the pass-through device receives the HDMI® video signal 302, the HDMI® video signal 302 flows to the display screen 116, such as a TV, monitor, or projector, where it is processed and rendered by the display screen 116. 4, the HDMI® video signal 302 flows unimpeded through the pass-through device 104. In the example embodiment, a user can connect their existing streaming device 106 to the pass-through device 104 and continue to use their streaming device 106 as they did before without any interference from the pass-through device 104.

[0046] In another embodiment, selection of an application not related to video games (e.g., music, live TV, movies, etc.) causes the game session to be inactive (e.g., the pass-through device 104 is not activated), which in some embodiments allows the HDMI signal 302 associated with the selected application to pass through the pass-through logic 202 unimpeded for rendering by the display screen 116.

[0047] In one embodiment, turning on the display screen 116 triggers the pass-through device 104 to provide power and turn on the streaming device 106. In one embodiment, when the streaming device 106 is turned on, the streaming device can generate an output HDMI® video signal 302 that flows through the pass-through device 104 to the display screen 116, where it is processed and rendered by the display screen 116. As a result, the display screen 116 can include a user interface associated with the streaming device 106, and a user can view and interact with multiple applications available through the streaming device 106.

[0048] For example, user 102 owns a Roku streaming device, which is connected to display screen 116 via pass-through device 104. When user 102 turns on display screen 116, a user interface associated with the Roku streaming device is displayed, and user 102 can interact with one or more applications through the user interface, e.g., launching applications, watching videos, listening to music, etc. If the user wants to watch a movie using a movie application (e.g., Netflix®), the user can select a movie from the movie application. Once a movie is selected, streaming of the movie occurs from the respective server associated with the movie application. As the selected movie is streamed to the Roku streaming device and displayed on display screen 116, pass-through device 104 remains inactive and does not interfere with the video stream.

[0049] FIG. 5 illustrates an embodiment of the user interface 112 of a streaming device 106 and a corresponding PlayStation Now interface 506 after a user 102 selects a PlayStation Now application 118n to begin a gaming session. In one embodiment, the user interface 112 of a streaming device 106 (e.g., a Roku streaming media player) is presented on the user's mobile phone, display screen 116, or any other device. In one embodiment, a device streaming service 208 may provide the user interface 112. In the illustrated example, the user interface 112 includes multiple applications 118a-118n. As shown, the applications may include a sports application 118a, a movie application 118b, a video application 118c, and a PlayStation®Now application 118n. In other embodiments, the user interface 112 may include a menu 502 including multiple categories 504a-504n. It should be understood that the user interface 112 can take on a variety of forms and layouts, and this illustration is provided to illustrate the interactivity of a particular type of streaming device.

[0050] 5, after the PlayStation®Now application 118n is selected by the user 102, the selection of the PlayStation®Now application 118n launches the pass-through device 104 and presents a PlayStation®Now interface 506 on the user's display screen 116. In some embodiments, the PlayStation®Now interface 506 includes a plurality of video games 508a-508n that are presented to the user for selection. In one embodiment, the plurality of video games 508a-508n are stored and may be retrieved from a video game server associated with cloud computing and the game provider 114. In other embodiments, the plurality of video games 508a-508n are stored and may be retrieved from a game catalog 312 stored in the game streaming logic 204 of the pass-through device 104.

[0051] 5, PlayStation Now interface 506 includes six different video games 508a-508n (e.g., Snowboarding Game 2021, Rescue Mission, Mortal Fighting Tournament, You Got Games, World Series Baseball, and Japan Racing) that a user can select. In some embodiments, when a user selects a video game 508 to begin a gaming session, the selection of the video game activates pass-through device 104. Pass-through device 104 can receive and begin processing a compressed stream of the selected video game from a video game server associated with cloud computing and game provider 114.

[0052] FIG. 6 illustrates an embodiment of a system configured to stream a video game selected by a user from a cloud computing and game provider 114. As shown, a streaming device 106 is connected to a display screen 116 via a pass-through device 104. In one embodiment, after a user 102 selects a video game 508c (e.g., Mortal Fighting Tournament) through a PlayStation Now interface 506 (or user interface 112) to initiate a game session, the selection results in a signal 602 being sent to the pass-through device 104, causing the pass-through device 104 to activate. In some embodiments, the streaming player 310 operation of the pass-through device 104 includes a communications interface 604. In some embodiments, the communications interface 604 may include operations configured to facilitate communication with the cloud computing and game provider 114 over the network 110.

[0053] For example, when a user selects a video game, the communications interface 604 may be configured to communicate with the cloud computing and gaming 114, including sending an indication to the cloud computing and gaming 114 notifying the cloud computing and gaming 114 that the user has requested a particular video game 508 for the game session. In response to the request, the cloud computing and gaming 114 transmits a compressed stream of the video game to the pass-through device 104 for decoding by the video decoder 308 operation of the pass-through device 104. In one embodiment, the selection of the video game may trigger transmission of the compressed stream of the video game from a server of the cloud computing and gaming provider 114 to a receive buffer for decoding by the game streaming logic. This may help eliminate latency and delays that may occur during transmission of the video game stream.

[0054] 6, the controller 108 of the user 102 may connect wirelessly to the pass-through device 104 via a Bluetooth module 306 of the pass-through device 104. In one embodiment, the user may provide input via the controller 108, and game commands are sent to the cloud computing and game provider 114 for processing of the input.

[0055] 7 illustrates an embodiment in which a selected video game is streamed from a cloud computing and gaming provider 114 and the selected video game is displayed on a user's display screen 116. In the embodiment illustrated in FIG. 7, a pass-through device 104 is connected to the display screen 116, and a streaming device 106 is connected to the pass-through device 104. In one embodiment, after a request to initiate a game session for the selected video game is sent to the cloud computing and gaming 114 provider, the cloud computing and gaming 114 provider sends a stream of the video game to the pass-through device 104, which is received and decoded. In one embodiment, after decoding the video game stream, the pass-through device 104 is configured to provide image data for rendering by the display screen 116.

[0056] 7, a display screen 116 is shown displaying a game scene 702 of a video game (e.g., Game 3: Mortal Fighting Tournament). Specifically, game scene 702 illustrates a martial arts fight between a first fighter and a second fighter, where a user 102 controls the game actions of the first fighter. In some embodiments, during the user's gameplay, when the user provides input to control the game actions of the first fighter via the controller 108, the user's input is received by the pass-through device 104, thereby driving interactivity in the video game while a cloud computing and game 114 provider is running the video game.

[0057] In some embodiments, after a game session ends, pass-through device 104 may be inactive and user interface 112 is displayed to the user on display screen 116. In one embodiment, a user may start another game session by selecting a video game on user interface 112 or application 118 to view a particular type of media content provided by one or more streaming entertainment services.

[0058] 8 illustrates an embodiment of a controller 108 wirelessly connected to a pass-through device 104. As noted above, in one embodiment, the pass-through device 104 may include a Bluetooth module 306 that enables wireless connection to various devices, such as the controller 108. During a user's gameplay, the user provides input via the controller 108, which is transmitted to the pass-through device 104 and processed by the game streaming logic 204 of the pass-through device 104 to drive interactivity in the video game. The input provided by the user 102 is further transmitted to a cloud computing and gaming 114 provider, which is configured to process the input to affect the game state of the running video game.

[0059] 8 may include a user-selectable input button (e.g., a PSNow® button). In one embodiment, when the input button (e.g., a PS Now button) is selected by the user, the pass-through device 104 is activated, such that the video game server may send a compressed stream of the video game to the pass-through device 104 for processing by game streaming logic of the pass-through device 104. The PS Now button, in one embodiment, is a user interface selection icon selectable using the device.

[0060] 9 illustrates a method for processing video games streamed from a cloud gaming system over a network. In one embodiment, the method of FIG. 9 provides a user with access to cloud gaming by integrating the user's existing streaming device 106 with a pass-through device 104. In one embodiment, the method includes an operation 902 configured to receive an indication of a selection of a video game for a gaming session through a user interface 112 associated with the streaming device 106 connected to a display screen 116 via the pass-through device 104. As noted above, the user interface may be displayed on other mobile devices or any other device, such as a personal computer, laptop, tablet computer, monitor and console / PC setup, television and console setup, peripheral device, tablet, thin client, set-top box, network device / appliance, etc.

[0061] In some embodiments, streaming device 106 may be a media streaming device such as an Apple TV, Roku® streaming media player, Amazon® Fire TV Stick, Google® Chromecast®, etc. Pass-through device 104 may include multiple operations configured to process and facilitate cloud gaming when streaming device 106 is connected to pass-through device 104. In one embodiment, streaming device 106 includes a corresponding device streaming service 208 that may provide a user with access to multiple applications 118 and a user interface 112. In some embodiments, user interface 112 may include various applications for accessing different types of streaming entertainment services, e.g., video games, music, movies, news, sports, TV shows, etc. In one embodiment, selecting a video game or any other type of application may result in a signal being sent to a cloud gaming system or a streaming entertainment service system notifying the respective system that the application has been selected by the user.

[0062] The method proceeds to operation 904, which is configured to receive a signal from game streaming logic of the pass-through device that a game session is active. In one embodiment, operation 904 may include game streaming logic 204 configured to receive a compressed stream of the video game. In some embodiments, game streaming logic 204 may include a video decoder 308 operation configured to decode the compressed stream of the video game. After decoding the compressed stream of the video game, operation 904 is configured to provide image data to display screen 116 for rendering a scene of the video game. In other embodiments, operation 904 is configured to receive user input for driving interactivity in the video game while the cloud gaming system is executing said video game as a user plays the video game and provides input via controller 106.

[0063] FIG. 10 illustrates components of an exemplary device 1000 that can be used to implement aspects of various embodiments of the present disclosure. The block diagram illustrates device 1000, which can incorporate or be a personal computer, video game console, personal digital assistant, server, or other digital device suitable for implementing embodiments of the present disclosure. Device 1000 includes a central processing unit (CPU) 1002 for executing software applications and optionally an operating system. CPU 1002 may be comprised of one or more homogeneous or heterogeneous processing cores. For example, CPU 1002 is one or more general-purpose microprocessors with one or more processing cores. Further embodiments can be implemented using one or more CPUs with a microprocessor architecture particularly suited for highly parallel and computationally intensive applications, such as processing operations for interpreting queries, identifying contextually relevant resources, and immediately implementing and rendering the contextually relevant resources within a video game. The device 1000 may be localized to the player playing the game segment (e.g., a game console), or remote from the player (e.g., a back-end server processor), or one of many servers that use virtualization in a game cloud system for remote streaming of gameplay to clients.

[0064] Memory 1004 stores applications and data used by CPU 1002. Storage 1006 provides non-volatile storage and other computer-readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices, as well as signal transmission and storage media. User input device 1008 communicates user input from one or more users to device 1000; examples of user input device 1008 may include a keyboard, mouse, joystick, touchpad, touchscreen, still or video recorder / camera, gesture-recognizing tracking device, and / or microphone. Network interface 1014 enables device 1000 to communicate with other computer systems over electronic communications networks, which may include wired or wireless communications over local area networks and wide area networks such as the Internet. The audio processor 1012 is adapted to generate analog or digital audio output from instructions and / or data provided by the CPU 1002, memory 1004, and / or storage 1006. The components of the device 1000, including the CPU 1002, memory 1004, data storage 1006, user input device 1008, network interface 1010, and audio processor 1012, are connected via one or more data buses 1022.

[0065] A graphics subsystem 1020 is further coupled to the data bus 1022 and the components of the device 1000. The graphics subsystem 1020 includes a graphics processing unit (GPU) 1016 and a graphics memory 1018. The graphics memory 1018 includes display memory (e.g., a frame buffer) used to store pixel data for each pixel of an output image. The graphics memory 1018 may be integrated into the same device as the GPU 1008, connected as a separate device from the GPU 1016, and / or incorporated within the memory 1004. Pixel data may be provided to the graphics memory 1018 directly from the CPU 1002. Alternatively, the CPU 1002 may provide data and / or instructions defining a desired output image to the GPU 1016, from which the GPU 1016 generates pixel data for one or more output images. The data and / or instructions defining the desired output image may be stored in the memory 1004 and / or the graphics memory 1018. In an embodiment, GPU 1016 includes 3D rendering functionality for generating pixel data for output images from instructions and data defining scene geometry, lighting, shading, texturing, motion, and / or camera parameters. GPU 1016 may further include one or more programmable execution units capable of executing shader programs.

[0066] Graphics subsystem 1014 periodically outputs image pixel data from graphics memory 1018 for display on display device 1010. Display device 1010 may be any device capable of displaying visual information in response to signals from device 1000, including CRT, LCD, plasma, and OLED displays. Device 1000 may provide analog or digital signals to display device 1010, for example.

[0067] It should be noted that access services distributed over a wide geographic area, such as providing access to games in the current embodiment, often use cloud computing. Cloud computing is a computing paradigm in which dynamically scalable, often virtualized resources are provided as a service over the Internet. Users do not need to be experts in the technical infrastructure of the "cloud" that supports them. Cloud computing can be categorized into different services, such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services often provide common applications, such as video games, online, accessed through a web browser, but the software and data are stored on servers in the cloud. The term cloud is used as a metaphor for the Internet, based on how the Internet is depicted in computer network diagrams, and is an abstract concept that hides a complex infrastructure.

[0068] In some embodiments, a game server may be used to perform the operation of a persistent information platform for video game players. Most video games played over the Internet operate through a connection to a game server. Typically, the game uses a dedicated server application that collects data from players and distributes the collected data to other players. In other embodiments, the video game may be executed by a distributed game engine. In these embodiments, the distributed game engine may run on multiple processing entities (PEs), such that each PE executes a functional segment of the given game engine on which the video game is executed. Each processing entity is viewed by the game engine as simply a computational node. A game engine typically performs a functionally diverse set of operations to execute a video game application along with additional services that the user experiences. For example, the game engine implements game logic and performs game calculations, physics, geometry transformations, rendering, lighting, shading, audio, and additional in-game or game-related services. The additional services may include, for example, messaging, social utilities, voice communication, gameplay playback functions, help functions, etc. The game engine may run on an operating system virtualized by a hypervisor on a particular server, but in other embodiments the game engine itself may be distributed across multiple processing entities, each residing on a different server unit in a data center.

[0069] According to this embodiment, for execution, each processing entity may be a server unit, a virtual machine, or a container, depending on the needs of each game engine segment. For example, if a game engine segment is responsible for camera transformations, which involve a large number of relatively simple mathematical operations (e.g., matrix transformations), that particular game engine segment may be provisioned with a virtual machine associated with a graphics processing unit (GPU). Other game engine segments, which require fewer but more complex operations, may be provisioned with processing entities associated with one or more more powerful central processing units (CPUs).

[0070] By distributing the game engine, the game engine has elastic computational characteristics that are not bound by the capabilities of a physical server unit. Instead, the game engine is provisioned with more or fewer computational nodes as needed to meet the demands of the video game. From the perspective of the video game and the video game player, a game engine that is distributed across multiple computational nodes is indistinguishable from a non-distributed game engine running on a single processing entity, as a game engine manager or supervisor distributes the workload and seamlessly integrates the results to provide the video game output component to the end user.

[0071] Users access remote services through client devices that include at least a CPU, display, and I / O. The client devices may be PCs, cell phones, netbooks, PDAs, etc. In one embodiment, a network running on the game server recognizes the type of device used by the client and adjusts the communication method employed. In another instance, the client devices access applications on the game server over the Internet using standard communication methods such as HTML.

[0072] It should be understood that a given video game or game application may be developed for a particular platform and a particular associated controller device. However, when such games are made available through a game cloud system as presented herein, users may access the video game with different controller devices. For example, a game may be developed for a game console and its associated controller, but a user may be accessing a cloud-based version of the game from a personal computer using a keyboard and mouse. In such a scenario, the input parameter configuration may define a mapping from inputs that can be generated by the user's available controller device (in this case, a keyboard and mouse) to inputs that are acceptable for execution of the video game.

[0073] In another example, a user may access the cloud gaming system via a tablet computing device, a touchscreen smartphone, or other touchscreen-driven device. In this case, the client device and the controller device are integrated together within the same device, and input is provided via detected touchscreen input / gestures. For such devices, the input parameter configuration may define specific touchscreen inputs corresponding to game inputs for the video game. For example, buttons, directional pads, or other types of input elements may be displayed or overlaid during execution of the video game to indicate locations on the touchscreen that the user can touch to generate game input. Gestures such as swipes in specific orientations or specific touch motions may also be detected as game inputs. In one embodiment, to familiarize the user with control operations on the touchscreen, a tutorial showing how to provide gameplay inputs via the touchscreen may be provided to the user, for example, before beginning gameplay of the video game.

[0074] In some embodiments, the client device serves as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection and transmits inputs from the controller device to the client device. The client device may then process these inputs and then transmit the input data to a cloud gaming server over a network (e.g., a network accessed through a local network device such as a router). However, in other embodiments, the controller itself may be a networked device capable of communicating inputs directly to the cloud gaming server over a network, without the need for such inputs to first be communicated through the client device. For example, the controller could connect to a local network device (such as the router mentioned above) and transmit data to and receive data from the cloud gaming server. Thus, while the client device may still be required to receive video output from the cloud-based video game and render it on a local display, input latency may be reduced by allowing the controller to transmit inputs directly to the cloud gaming server over the network, bypassing the client device.

[0075] In one embodiment, the networked controller and client device can be configured to transmit certain types of input directly from the controller to the cloud gaming server and other types of input via the client device. For example, inputs detected without relying on any additional hardware or processing, aside from the controller itself, can be transmitted directly from the controller to the cloud gaming server over the network, bypassing the client device. Such inputs can include button inputs, joystick inputs, embedded motion-sensing inputs (e.g., accelerometers, magnetometers, gyroscopes), etc. However, inputs that utilize additional hardware or require processing by the client device can be transmitted by the client device to the cloud gaming server. These can include video or audio captured from the game environment, which can be processed by the client device before transmission to the cloud gaming server. Additionally, inputs from the controller's motion-sensing hardware can be processed by the client device in conjunction with the captured video to detect the position and movement of the controller, which is then communicated by the client device to the cloud gaming server. It should be understood that controller devices according to various embodiments can also receive data (e.g., feedback data) from the client device or directly from the cloud gaming server.

[0076] It should be understood that the various embodiments defined herein may be combined or assembled into specific implementations that use various features disclosed herein. Thus, the examples provided are only some of the possible examples and are not intended to limit the various implementations that may be defined by combining various elements. In some instances, an implementation may include fewer elements without departing from the spirit of the disclosed or equivalent implementations.

[0077] Embodiments of the present disclosure may be practiced with a variety of computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0078] Although the operations of the method are described in a particular order, it should be understood that other housekeeping operations may be performed between operations, or operations may be coordinated to occur at slightly different times, or operations may be distributed within the system to allow processing operations to occur at various intervals relative to processing, so long as the processing of telemetry and game state data to generate modified game states is performed in the desired manner.

[0079] One or more embodiments may also be fabricated as computer-readable code on a computer-readable medium. The computer-readable medium may be any data storage device that can store data, which can then be read by a computer system. Examples of computer-readable media include hard drives, network-attached storage (NAS), read-only memory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tape, and other optical and non-optical data storage devices. The computer-readable medium may include tangible computer-readable media distributed over network-connected computer systems so that the computer-readable code is stored and executed in a distributed fashion.

[0080] In one embodiment, the video game is executed locally on a game console, personal computer, or server. In some cases, the video game is executed by one or more servers in a data center. When the video game is executed, some instances of the video game may be a simulation of the video game. For example, the video game may be executed by an environment or server that generates a simulation of the video game. A simulation, in some embodiments, is an instance of the video game. In other embodiments, the simulation may be generated by an emulator. In either case, when the video game is represented as a simulation, the simulation may be executed to render interactive content that can be interactively streamed, executed, and / or controlled by user input.

[0081] Although the foregoing embodiments have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. The present embodiments are therefore to be considered as illustrative and not restrictive, and the present embodiments should not be limited to the details set forth herein but may be modified within the scope of the appended claims and their equivalents.

Claims

1. 1. A relay device for streaming video games from a cloud gaming system over a network, comprising: an HDMI output port; an HDMI (registered trademark) input port; A pass-through unit; a game streaming unit, the game streaming unit being activated in response to selection of the video game from a user interface during use of the relay device, the game streaming unit being configured to receive a compressed stream of the video game from the cloud gaming system, decode the compressed stream of the video game, and provide image data for rendering by a display screen, the game streaming unit providing a wireless connection via the game streaming unit to a controller that provides input for controlling the video game; Equipped with the pass-through unit, when an application not related to the video game is selected, passes an HDMI signal related to the selected application for rendering by the display screen, and the relay device remains inactive so as not to interfere with a video stream of the HDMI signal; an HDMI output port of a streaming device is connected to the HDMI input port of the relay device, and the streaming device generates the video stream; Intermediate device.

2. the HDMI output port of the relay device connects to the HDMI input port of the display screen; The relay device according to claim 1 .

3. the user interface is rendered on a mobile device or on the display screen connected to the relay device; The relay device according to claim 1 .

4. a server of the cloud gaming system sending the compressed stream of the video game to a receive buffer of the relay device for decoding by a video decoder of the game streaming unit; The relay device according to claim 1 .

5. 1. A relay device for streaming video games from a cloud gaming system over a network, comprising: an HDMI output port; an HDMI (registered trademark) input port; A pass-through unit; a game streaming unit, the game streaming unit being activated in response to selection of the video game from a user interface during use of the relay device, the game streaming unit being configured to receive a compressed stream of the video game from the cloud gaming system, decode the compressed stream of the video game, and provide image data for rendering by a display screen, the game streaming unit providing a wireless connection via the game streaming unit to a controller that provides input for controlling the video game; Equipped with the pass-through unit allows a video stream generated by a streaming device to be passed through to be rendered by the display screen when an application not related to the video game is selected, while the relay device remains inactive without interference from the relay device; an HDMI output port of the streaming device is connected to the HDMI input port of the relay device; Intermediate device.

6. the relay device further comprises a high-definition upscaling module, the high-definition upscaling module configured to receive an HDMI signal from a streaming device and process the HDMI signal to generate a higher quality image. The relay device according to claim 5 .

Images