Adaptive graphics for cloud gaming

Abstract

A method is provided that includes the following operations: executing, by a cloud game machine, a video game, the execution of the video game including rendering a play game video; processing, by a streaming media server, the play game video for streaming over a network to a client device; monitoring a quality of a connection between the streaming media server and the client device over the network; responsive to detecting a change in the quality of the connection between the streaming media server and the client device, then adjusting, by the cloud game machine, the rendering of the play game video.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to systems and methods for adaptive graphics for cloud gaming. BACKGROUND

[0002] Description of Related Art

[0003] A current trend in the gaming industry is a move towards cloud gaming. Cloud gaming provides advantages to end users by enabling remote execution of video games in data centers that can guarantee the resources of the video games. The video generated by the remotely executed video games is streamed to the user’s equipment, and inputs from the user are sent back to the data center. This allows end users to execute the game itself without the need to have specific hardware. Instead, end users only need to have enough hardware to stream the game play and can still enjoy a high quality gaming experience. Moreover, in theory, cloud gaming enables gaming anywhere there is a network connection.

[0004] One continuing trend in the video game industry is the increase in graphics complexity and availability of computing resources to meet the needs of modern game engines. As video games evolve, their resolution and frame rate are constantly increasing, enabling very realistic and detailed virtual environments to be rendered. Moreover, the popularity of cloud gaming continues to grow, and the shift to cloud execution of video games enables greater opportunities for high quality gaming experiences.

[0005] Broadly speaking, today’s cloud gaming architectures rely on the streaming of existing console or PC game titles. Using existing games unmodified, and without their “knowledge”, their video and audio are captured and streamed to the user. During such streaming, it is noted that the user is provided with the best experience based on their internet connection. If the connection is poor, the image quality can be reduced by changing the video encoder settings, such as video resolution, frame rate, or compression settings. However, such adjustments are made on the video / audio that has already been output by the video game.

[0006] It is against this background that embodiments of the present disclosure are presented. SUMMARY

[0007] Implementations of the present disclosure provide systems and methods for adaptive graphics for cloud gaming. While current cloud gaming systems address different internet conditions by changing video encoder settings, such changes only affect existing video / audio that the video game has already generated, without involving the video game itself. Better image quality and improved user experience can be achieved in cases where the video game is involved in quality adjustment. For example, the video game can render lower resolution textures that are easier to compress or not render unnecessary objects at all. Implementations of the present disclosure provide better coordination between the video game and the video streaming solution in order to obtain higher quality streaming for the user.

[0008] In some implementations, a method is provided that includes the following operations: executing, by a cloud gaming machine, a video game, the execution of the video game including rendering gameplay video; processing, by a streaming media server, the gameplay video for streaming over a network to a client device; monitoring a quality of connection between the streaming media server and the client device over the network; in response to detecting a change in the quality of connection between the streaming media server and the client device, then adjusting, by the cloud gaming machine, the rendering of the gameplay video.

[0009] In some implementations, processing, by the streaming media server, the gameplay video includes encoding the gameplay video in a compressed video format.

[0010] In some implementations, monitoring the quality of connection includes monitoring one or more of bandwidth or latency.

[0011] In some implementations, monitoring the quality of connection includes monitoring packet loss that occurs during streaming of the gameplay video.

[0012] In some implementations, adjusting, by the cloud gaming machine, the rendering of the gameplay video includes adjusting a frame rate or resolution of the gameplay video.

[0013] In some implementations, adjusting, by the cloud gaming machine, the rendering of the gameplay video includes adjusting a level of detail of the rendering of the gameplay video.

[0014] In some implementations, adjusting, by the cloud gaming machine, the rendering of the gameplay video includes adjusting an amount of content included in the gameplay video.

[0015] In some implementations, the change in the quality of connection is defined by a degradation in the quality of connection; wherein adjusting, by the cloud gaming machine, the rendering of the gameplay video is configured to reduce a bitrate of the streaming of the gameplay video over the network.

[0016] In some implementations, the change in connection quality is defined by an improvement in connection quality; wherein the adjustment of the rendering of the gameplay video by the cloud game machine is configured to increase a bitrate of the streaming of the gameplay video over the network.

[0017] In some implementations, the detection of the change in connection quality triggers the transmission of a message from the streaming media server to the cloud game machine, and wherein the adjustment of the rendering of the gameplay video by the cloud game machine is in response to the message.

[0018] In some implementations, a method is provided that includes the following method operations: executing, by a cloud game machine, a video game, the execution of the video game including rendering gameplay video; processing, by a streaming media server, the gameplay video for streaming to a client device over a network, wherein the processing of the gameplay video by the streaming media server includes encoding the gameplay video in a compressed video format; monitoring connection quality between the streaming media server and the client device over the network, wherein the monitoring of the connection quality includes monitoring packet loss occurring during the streaming of the gameplay video; in response to detecting a change in connection quality between the streaming media server and the client device, then adjusting, by the cloud game machine, the rendering of the gameplay video, wherein the adjusting of the rendering of the gameplay video by the cloud game machine includes adjusting a frame rate or a resolution of the gameplay video.

[0019] In some implementations, a system is provided that includes: a cloud game machine that executes a video game, the execution of the video game including rendering gameplay video; a streaming media server that processes the gameplay video for streaming to a client device over a network, wherein the streaming media server monitors connection quality between the streaming media server and the client device over the network; wherein in response to detecting a change in connection quality between the streaming media server and the client device, then adjusting, by the cloud game machine, the rendering of the gameplay video.

[0020] Other aspects and advantages of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrated by way of example of the principles of the disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0021] The disclosure can best be understood by reference to the following description taken in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A cloud gaming system configured to base video game self-involvement video / audio quality adjustments on network conditions is conceptually illustrated in accordance with implementations of the present disclosure.

[0023] Figure 2 A game stream quality feedback loop in which video / audio quality is adjusted based on stream quality is conceptually illustrated in accordance with implementations of the present disclosure.

[0024] Figure 3 Conceptually illustrates reduction in frame rate natively performed by an executing video game relative to frame rate performed by a streaming server after rendered by the video game, according to implementations of the present disclosure.

[0025] Figure 4 Conceptually illustrates a method for controlling or setting frame rate of a video game, according to implementations of the present disclosure.

[0026] Figure 5 Conceptually illustrates variable rendering of content in a virtual environment, according to implementations of the present disclosure.

[0027] Figure 6 Conceptually illustrates several content settings based on network bandwidth feedback, according to implementations of the present disclosure.

[0028] Figure 7 Conceptually illustrates feed forward optimization from video game to streaming logic, according to implementations of the present disclosure.

[0029] Figure 8A Illustrates an exemplary system for loading game files for games available via a cloud gaming site, according to implementations of the present disclosure.

[0030] Figure 8B Is a flowchart conceptually illustrating various operations performed to stream a cloud video game to a client device, according to implementations of the present disclosure.

[0031] Figure 9 Illustrates an implementation of an information service provider architecture, according to implementations of the present disclosure. DETAILED DESCRIPTION

[0032] In the following description, numerous specific details are set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure can be practiced without some or all of these specific details. In other instances, well known process steps have not been described in detail in order not to obscure the present disclosure.

[0033] Implementations of the present disclosure improve audio / video quality of a cloud gaming session by including a video game into a quality adjustment feedback loop. Typically, quality is adjusted based on fluctuations in bandwidth, latency, and packet loss of an internet connection. In existing cloud gaming systems, the game is typically not involved in these adjustments. However, by involving the video game in adjusting the quality of the video / audio, more informed decisions can be made to improve the quality and responsiveness of the cloud gaming session.

[0034] Broadly speaking, the internet connection between the client and the server fluctuates in terms of available bandwidth, latency, and packet loss. In current cloud gaming systems, the connection is monitored during the session and the audio / video quality is adjusted based on this. For example, when the user's bandwidth becomes lower than the bandwidth configured on the server side for the audio and video encoders, an adjustment is needed. If no adjustment is made, the user will not be able to receive all the data because their connection cannot handle it, which will result in packet loss and a degraded experience. Additionally, if the available bandwidth increases, the quality can be increased by increasing the encoder bit rate.

[0035] Currently, quality adjustments are made through a feedback loop between the client and the server. There are a few settings that can be adjusted, such as the bit rate of the audio or video encoders, the frame rate, and the video resolution. However, these knobs are suboptimal because the streaming software receives the finished audio or video frame to be encoded. While the server can adjust the resolution, it does so by scaling the already rendered image; and while it can adjust the frame rate, it does so by dropping video frames. These adjustments are not ideal because they do not affect the original native rendering of the video and audio, but rather further process the already rendered video and audio.

[0036] However, better results can be achieved by involving the video game itself in the quality adjustment. Thus, implementations of the present disclosure extend the streaming quality feedback loop to involve the video game itself. The existing feedback loop can be extended by a channel from the streaming software to the video game. This channel can be used to trigger quality adjustments, but also for other purposes, such as pausing the game or the game providing hints to the streaming software (e.g., when a scene change, which is typically difficult for video encoders to handle).

[0037] Figure 1 A cloud gaming system according to implementations of the present disclosure is conceptually illustrated as being configured to involve the video game itself in video / audio quality adjustments based on network conditions. As shown, the cloud gaming infrastructure is typically located in a data center 100 and consists of a game console 102 executing a video game 104 and a streaming server computer 108 implementing streaming logic / streaming server 110. The game console 102 is a device with sufficient hardware to execute the video game 104. By way of example and not limitation, the game console can be a game console or its hardware equivalent (e.g., a blade version), one or more server computers, or any other device capable of providing an appropriate execution environment for the video game 104. In some implementations, the game console implements a virtual machine (e.g., using a hypervisor) on which the video game is executed, cloud computing, or other virtualization resources (e.g., including an operating system).

[0038] Broadly, the executing video game 104 generates video and audio reflecting gameplay of the video game (e.g., renderings of the virtual environment of the video game, such as perspective views of virtual characters, along with accompanying sound effects and soundtrack) in a native uncompressed format, for example. The streaming logic / streaming server 110 encodes the video and audio in a compressed format before streaming the video and audio to the client device 122 over the network 116. To this end, the streaming logic 110 includes a video encoder 112 that is capable of encoding video in a compressed video format, e.g., using a video codec or according to a video compression protocol. More specifically, the streaming logic can adjust the encoding of the video or audio (e.g., adjust bitrate, frame rate, resolution, etc.) based on network conditions or feedback from the client. In some implementations, the video encoder 112 also includes an audio encoder that encodes audio in a compressed audio format, e.g., using an audio codec or according to an audio compression protocol. In some implementations, the audio encoder is separate from the video encoder 112.

[0039] As noted, the encoded video / audio is streamed to the client device 122 over the network 116. By way of example and not limitation, the client device 122 can be a game console, personal computer, laptop computer, tablet computer, mobile phone, or any other device capable of streaming gameplay of a video game from a cloud game provider as in this disclosure. The client device 122 executes a client application 124 that receives the encoded video / audio and decodes the video to render to a display 128 viewed by the user 118, and also decodes the audio to play through one or more speakers to be heard by the user 118. In various implementations, the display 128 and / or speakers can be integrated with or separate from the client device 122. In some implementations, the display 128 is a television, monitor, screen, projector, or other device through which the user 118 can view video. Additionally, the user 118 can operate an input device 120 to provide input to the video game 104. By way of example and not limitation, the input device 120 can be a game controller, motion controller, keyboard, mouse, touchpad, trackball, camera, depth camera, microphone, or any other sensing device through which interactive input can be provided for the video game.

[0040] As shown, the stream logic 110 includes or calls a stream monitor 114 that is configured to monitor the quality of the stream to the client device 122. By way of example and not limitation, the stream monitor 114 can monitor packet loss or retransmission, positive acknowledgements from the client device (e.g., for data / packets received by the client device), negative acknowledgements from the client device (e.g., for data / packets not received by the client device), network latency, ping times to the client device 122, network bandwidth, or any other indicator of the quality of data transmission between the stream logic 110 and the client device 122.

[0041] Additionally, in some implementations, the client application 124 can include or call a connection monitor 126. The connection monitor 126 can be configured to monitor network conditions similar to the stream monitor 114 described above but from the client side, such as packet / data loss, network bandwidth, network latency, etc. In some implementations, the client application 114 sends feedback to the stream logic 110 regarding such network conditions.

[0042] In response to sensed network conditions, such as network latency or network bandwidth conditions, the stream logic 110 is configured to extend the stream quality feedback loop backward to include the video game itself. That is, the stream logic 110 is configured to communicate with the video game 104 to adjust some aspect of the execution of the video game, such as its video / audio rendering or gameplay process, to compensate for the changing network conditions. For example, this can include adjusting the video / audio rendering of the video game in a manner that reduces the bandwidth requirements of the game stream. In some implementations, the stream logic 110 can perform adjustments to the video encoding in addition to those made to the encoding of the video game 114. Additionally, the video game 104 can communicate to the stream logic 110 and / or to the client device 122 via the stream logic 110 information related to video / audio encoding or decoding, such as information about regions of interest in the video or hints about upcoming events in the video, such as scene changes. Additionally, this communication can establish a path from the video game 104 to the client application 114 via the stream logic 110 for other purposes, such as sending information for client-side rendering or interface operation.

[0043] To facilitate communication between the video game 104 and the streaming logic 110, the video game 104 can display, by way of example and without limitation, an API 106 or shared memory or sockets. In some implementations, the streaming logic 110 accesses the API 106 of the video game 104 in order to transmit messages requesting adjustments to the video / audio output by the video game 104. Generally, the video game 104 executes a game loop, which typically involves processing inputs (e.g., user interaction inputs such as controller / action / voice inputs, network requests, or inputs from other players in a multiplayer game, etc.), updating game state, and rendering video / audio using the current game state. Thus, in various implementations, messages from the streaming logic 110 that can be received through the API 106 or sockets can constitute another input to the game’s game loop and / or can be processed to affect rendering of video / audio based on the current game state.

[0044] Further, the communication between the streaming logic 110 and the video game 104 can be part of a feedback loop in which adjustments made by the video game 104 are implemented, the results of such adjustments are monitored (e.g., by continuously monitoring stream quality), and additional adjustments are made if necessary. For example, if the streaming logic 110 causes the video game 104 to reduce video / audio quality in response to a deterioration in network conditions, it can not be fully known until after video encoding by the video encoder 112 of the streaming logic is performed whether the effect is significant enough to overcome the deterioration in network conditions. And thus, if the effect is not great enough, additional instructions can be needed by the streaming logic 110 to the video game 104 to further reduce video quality. In a sense, it can take several video frames to evaluate the situation, and multiple rounds of adjustments by the video game 104 before a proper video / audio quality level is reached. It should be appreciated that the opposite scenario, where network conditions improve, can be handled in a similar manner, but with the streaming logic instructing the video game to increase video / audio quality.

[0045] While implementations of the present disclosure generally involve configurations in which the streaming logic 110 is implemented on a computing device separate from the video game 104, it should be appreciated that in other implementations, the functionality of the streaming logic 110 as described herein can be implemented on the same device as the video game 104, such as the game console 102, or in some implementations, can even be logically incorporated as part of the video game 104.

[0046] Figure 2A game stream quality feedback loop is conceptually illustrated in accordance with implementations of the present disclosure, in which video / audio quality is adjusted based on stream quality. At method operation 200, a cloud video game is initiated, which can include assigning a user to a data center and a cloud game console, executing the cloud video game on the cloud game console, establishing a connection between a streaming media server and a client device of the user, etc. At method operation 202, video / audio of the executing video game is streamed to the client device, and as the user interacts with the streamed video / audio, an interactive gameplay process of the cloud video game is commenced, e.g., using a controller device or other input device.

[0047] At method operation 204, game stream quality or connection quality or network conditions are monitored. By way of example and not limitation, this can entail monitoring conditions between a data center from which the game is streamed and the client device, such as packet loss, latency, bandwidth, etc.

[0048] At method operation 206, it is determined whether the game stream quality is acceptable, and further whether the quality settings of the video game should be increased or decreased. Generally, if the game stream quality is below a given threshold, then at method operation 208, the video game is instructed to lower the game quality settings. By way of example and not limitation, game stream quality below a given threshold can be defined by various metrics, such as packet loss exceeding a predefined threshold, bandwidth falling below a predefined threshold, latency exceeding a predefined threshold, etc. The resulting lowering of the game quality settings can include various measures described in further detail herein, such as lowering resolution or frame rate, lowering texture quality, lowering detail settings, reducing the extent of the rendered virtual environment, etc.

[0049] Similar to the above, if the game stream quality rises above a given threshold, then in some implementations no action can be taken, while in other implementations, then at method operation 210, the video game is optionally instructed to increase the game quality settings. By way of example and not limitation, game stream quality rising above a given threshold can be defined by various metrics, such as packet loss falling below a predefined threshold, bandwidth rising above a predefined threshold, latency falling below a predefined threshold, etc. In execution, the resulting increase of the game quality settings can include various measures described in further detail herein, such as increasing resolution or frame rate, increasing texture quality, increasing detail settings, increasing the extent of the rendered virtual environment, etc.

[0050] In light of the above, it can be appreciated that in some implementations, the game stream quality can have a target range, in which the game quality settings are adjusted upward or downward when the game stream quality is above or below an upper or lower limit of the target range, respectively.

[0051] It can be appreciated that after the game quality setting is increased or decreased, the method returns to method operation 204, where the stream quality is further monitored and additional adjustments are made as needed, thereby forming a feedback loop that optimizes the stream quality for changing network conditions, and this is accomplished by involving the video game in adjusting aspects of the rendering, logic, or other settings of the video game.

[0052] Figure 3 Conceptually illustrated is the reduction in frame rate by an executing video game natively performed relative to the frame rate performed by a stream media server after being rendered by the video game, in accordance with implementations of the present disclosure. It should be appreciated that one way to reduce the amount of data required to stream a video, in accordance with implementations of the present disclosure, is to reduce the frame rate of the video. Existing video encoders can reduce the frame rate; however, this works best when reducing between even multiples, such as receiving a video at 60 frames per second (FPS) and encoding it at a rate of 30 FPS (and possibly performing interpolation at the client), such that the interval between frames remains the same.

[0053] However, dropping from 60 FPS to 30 FPS can reduce the amount of data more than is necessary for the available bandwidth. However, if an attempt is made to reduce the video from 60 FPS to, for example, 50 FPS, the video encoder can drop every sixth frame; however, the interval between frames is not consistent, and thus motion can appear to be unsmooth because the frame smoothness is not consistent. This scenario is conceptually illustrated by a series of image frames that have been generated by the video game 104 at 60 FPS, shown at reference numeral 300, and have a 16.67 millisecond (ms) interval between each frame. However, if the video encoder drops every sixth frame, the interval between the preceding frame and the subsequent frame for each dropped frame doubles to 33.33 ms, and no longer matches the interval between the other frames. While client-side interpolation can be performed, such interpolation is only a best guess at the content of the missing frame, and can not match the actual dropped frame, potentially creating artifacts or artificial looking effects.

[0054] Better results can be obtained if the video game itself reduces the frame rate to 50 FPS natively. This achieves video frames (as shown at reference numeral 302) with constant 20ms intervals, so that there is no inconsistency between frames. By having the video game itself adjust the video frame rate, any custom frame rate can be generated natively, so that the intervals between frames remain consistent. As noted, the frame rate can be adjusted in response to sensed network conditions or stream quality. For example, the video game 104 can be caused to reduce its frame rate when the stream quality or network conditions worsen; and the video game 104 can be caused to provide its frame rate when the stream quality or network conditions improve. By having the video game involved in the frame rate adjustment, this allows the game to adjust its internal timing and the rate at which frames are transmitted, thereby removing the frame smoothing limitations of current approaches, making it possible for the frame rate to be reduced (or increased) to other rates.

[0055] Figure 4 A method for controlling or setting the frame rate of a video game is conceptually illustrated in accordance with implementations of the present disclosure. Some existing PC / console games are configured to wait for the vertical sync (Vsync) of the monitor / display. That is, when the game passes a video frame to the GPU driver, the GPU driver waits until the display starts a new line of images, during which the game loop is blocked until the GPU driver accepts the frame, thereby limiting the game loop and synchronizing it to the refresh rate of the display (e.g., 60Hz, 120Hz, etc.).

[0056] But in the case of a cloud video game, there is no actual display that can be synchronized with the cloud video game. Thus, in some implementations, a virtual Vsync is created that simulates the effect of an actual Vsync as described above. For example, in some implementations, an artificial Vsync timing signal is generated at a given frame rate that is set or adjusted to achieve a desired effect on the video stream, such as reducing or increasing data usage. In some implementations, the effect of the virtual Vsync is created by the GPU driver.

[0057] In some implementations, a similar effect to using a Vsync timing signal as described above is achieved using a timer / clock, as shown in the illustrated implementation. A game loop is shown, in which at method operation 400, inputs are processed, such as inputs generated from user interaction input by operating input devices such as game controllers or other devices. At method operation 402, the game state is updated based on the game’s logic, such as a real-time physics engine or other process for determining the state of objects in the virtual environment of the game.

[0058] At method operation 404, the next video frame is then rendered according to the updated game state. At method operation 406, the game loop is paused and the game loop is resumed after a timer / clock 408. The timer / clock 408 sets the interval between frames and is thus configured to limit the game loop to a desired frame rate based on feedback 410 received from the streaming logic as described above. In some implementations, the timer is provided by the operating system or the video game logic or game engine. While in the illustrated implementation, the game loop is shown as being paused by the timer after rendering the last frame, in other implementations, the game loop can be paused after any of the other steps of processing input and updating the game state. By implementing the pause of the game loop, the frame rate of the video output of the game loop can be set to any custom rate such that the interval between video frames remains uniform, consistent, and thus motion appears smooth.

[0059] Another aspect of the gameplay video that can be adjusted by the video game itself is the resolution of the video. In current cloud gaming systems, the streaming logic / streaming server receives the completed video frames from the video game and can only scale the video frames in size and color depth after the fact. However, quality can be improved by instructing the video game itself to render the video at a lower resolution rather than scaling after the fact. For example, if the generated video frame is 1280 x 720 pixels, scaling the generated video frame to other resolutions that are not good fractions of 1280 x 720 can be problematic and text quality can be affected.

[0060] However, rather than scaling such video frames after they have been generated, the video game itself can be instructed (or caused) to render the video frames at different resolutions, which enables the video game to optimize the rendering for the selected resolution, e.g., such that text quality is optimized. In this way, the resolution can be dynamically changed to any resolution without suffering from post-scaling. For example, the video game 104 can be caused to lower the video resolution when streaming quality or network conditions deteriorate; and the video game 104 can be caused to provide the video resolution when streaming quality or network conditions improve.

[0061] In some implementations, the video game can be instructed to change the aspect ratio of the video from an existing first aspect ratio to a (different) second aspect ratio (e.g., 16:9 to 4:3, etc.). Changing the aspect ratio can be useful for devices with displays whose orientation can be changed, such as mobile devices (e.g., cell phones, tablets, etc.) or other devices whose displays the user can rotate during typical operation. Thus, in some implementations, a sensed display rotation (or orientation change) can trigger the streaming logic to instruct the video game to change the aspect ratio. In this way, the video is transmitted from the video game itself in an aspect ratio that is appropriate for the viewing display, rather than relying on the streaming logic to change the aspect ratio after the video has been generated. It will be appreciated that changing the aspect ratio by the video game can require adjusting the field of view rendered through the video, such as adjusting the horizontal / vertical angle of view. In some implementations, the aspect ratio is changed to match the full screen area of the display with the correct or preferred orientation for viewing by the user.

[0062] In some implementations, detecting a device rotation (or orientation change, e.g., portrait to landscape or vice versa) can trigger any of the other changes described elsewhere in this disclosure that are influenced by the video game.

[0063] In some implementations, detecting a switch from one device to another device can trigger an adjustment performed by the video game, such as the above-described changes to video resolution, frame rate, or aspect ratio. For example, if a user switches a game play session from a television to a cell phone, the system can detect this and change the video resolution, frame rate, or aspect ratio to accommodate the display screen of the cell phone.

[0064] Figure 5 Variable rendering of content in a virtual environment according to implementations of the present disclosure is conceptually illustrated. In some implementations, another aspect that can be adjusted by the video game itself (e.g., in response to feedback about streaming quality or network conditions) is the amount of content in the virtual environment that is rendered or shown in a video frame. One way to accomplish this is to adjust the viewing distance in the virtual environment.

[0065] For example, in the illustrated implementation, a virtual character 500 is conceptually shown in a virtual environment. The view of the virtual environment that is rendered for viewing by a user of a video game can be a portion that falls within a field of view 502 from the perspective of the virtual character 500 (and has a view direction in the virtual environment that is controllable by the user). Generally, the field of view 502 is defined by a horizontal and vertical angular quantity to be rendered. At a given time, the rendered view of the virtual environment can extend from the position of the virtual character 500 to a depth D2 (or radius or distance) within the virtual environment. That is, objects in the virtual environment that fall within the field of view 502 and the depth D2 (i.e., within regions 504 and 506 in the illustrated implementation) will be rendered in video frames output by the video game. However, objects beyond the depth D2, such as those falling within region 508, will not be rendered. This effectively limits the distance that objects in the virtual environment will be rendered for viewing.

[0066] In some implementations, the rendering depth can be decreased, e.g., in response to stream quality or network conditions deteriorating, so that fewer objects are rendered from the virtual environment. For example, the depth can be decreased from D2 to a depth Dl, as shown in the illustrated implementation. And thus in this case, objects in region 504 are rendered that fall within the field of view 502 and the depth Dl. Whereas objects beyond the depth Dl, such as those in regions 506 and 508, are not rendered for viewing by the user.

[0067] In some implementations, the rendering depth can be increased, e.g., in response to stream quality or network conditions improving, so that a greater number of objects are rendered from the virtual environment. For example, the depth can be increased from D2 to a depth D3, as shown in the illustrated implementation. And thus in this case, objects in regions 504, 506, and 508 are rendered because they fall within the field of view 502 and the depth D3. It should be appreciated that objects beyond the depth D3 are not rendered for viewing by the user.

[0068] It will be appreciated that by adjusting the amount of virtual objects in the scene rendered by the video game in the video frames, the complexity of the video frames can be increased or decreased, thereby changing the amount of data bandwidth required to faithfully transmit the video. While in the above implementations the amount of virtual objects included in the view has been determined based on distance or depth, it will be appreciated that other ways of adjusting the amount of virtual objects to render can be employed. For example, in some implementations, certain types or classes of objects can be included / excluded from rendering based on stream quality and / or network conditions. In some implementations, objects are assigned a priority ranking or priority value that defines their order of inclusion in the rendered view of the virtual environment. That is, when reducing the amount of objects to include in the rendered video frames, the order in which objects are dropped is from lower priority objects to higher priority objects. Or conversely, when increasing the amount of objects to render, the order in which objects are added is from higher priority objects to lower priority objects.

[0069] Thus, virtual objects can be turned on or off with respect to their inclusion in the rendered image frames of the virtual environment. And when network conditions worsen or stream quality degrades, objects with the lowest rendering priority will be the first to be dropped from the frame rendering. It will be appreciated that generally, objects expected to be excluded from rendering will be objects whose removal will not significantly impact the game play process.

[0070] Another aspect of the video game that can be adjusted is the image quality. For example, in the case of deteriorating network conditions, the goal of adjusting the image quality will be to adjust the scene complexity to obtain video frames that are easier to encode. The video game can be instructed to adjust the image quality in response to given network conditions. This can include adjusting the quality of textures applied to surfaces in the virtual environment. High quality textures can be very complex to encode, and so by reducing the quality of the textures, this reduces the amount of detail on surfaces in the virtual environment, making them easier to encode and reducing the bandwidth of the streamed video. In some implementations, the resolution of the textures can be adjusted in response to monitored network conditions. For example, the texture resolution can be reduced (e.g., 4K x 4K textures to 1K x 1K textures) in response to a reduction in network bandwidth. This will result in textures that look less sharp, but are easier for the encoder to encode, and can have fewer encoding artifacts, resulting in a better overall user experience than if the higher resolution textures were simplified at the encoder level.

[0071] In some implementations, the color depth or color space or color gamut or number of colors used for textures is adjusted. For example, such color parameters can be reduced or adjusted down in response to a reduction in bandwidth, or scaled up in response to an increase in bandwidth.

[0072] In some implementations, the number of polygons or points used to model a given surface or object in the virtual environment can be adjusted (e.g., reduced in response to a decrease in network bandwidth).

[0073] In some implementations, lighting effects, particle effects, fog effects, reflections, or any other rendering parameters can be simplified or reduced in response to a decrease in network bandwidth or a decrease in stream quality. In some implementations, the video game can not render certain geometry (e.g., reduce the amount or level of detail of vegetation in an outdoor scene).

[0074] In some implementations, depending on the game scenario or functionality, the game can make decisions on what visual content / scenario to provide based on network conditions such as bandwidth. For example, for a character walking in a foggy forest, the game can be configured to increase the amount of fog so that the user cannot see very far in the virtual environment.

[0075] In some implementations, the video game can implement different levels of content settings that determine the amount of content or detail to include. Figure 6 Several content settings based on network bandwidth feedback are conceptually illustrated in accordance with implementations of the present disclosure. At operation 600, the available network bandwidth is detected. At operation 602, a level of detail is set for the video game from several options. For example, a low bandwidth option is illustrated at reference numeral 604, in which two characters are shown with a simple background consisting of a simple texture representing grass on which the characters are standing and a sky without showing clouds or other details.

[0076] For a medium bandwidth option shown at reference numeral 606, the grass is represented by polygons so that individual blades of grass are visible. In addition, some clouds are shown, but the rendering is simple and the amount of detail is low.

[0077] However, for a high bandwidth option shown at reference numeral 608, the number of individual blades of grass is greater and denser. In addition, the blades of grass can be configured to sway or sway. Furthermore, the amount of detail in the rendering of the clouds is very rich. Also, trees are visible in the high bandwidth background that were not visible in the lower level of detail settings. Additional vegetation such as flowers, shrubs, or other types of plants can be shown.

[0078] Accordingly, the video game can be configured to adjust the amount of content or the level of detail of the content in response to changes in network conditions or stream quality. Various adjustments can be made, such as adjusting the viewing distance, textures, details, the amount of virtual elements such as vegetation, characters, vehicles, etc. Accordingly, the level of quality of the virtual environment can be dynamically modified, which is different from previous approaches because implementations of the present disclosure provide modifications to the content itself, rather than merely modifying the frame rate after they have been rendered. In some implementations, the content modification of the video game can be set to various levels that are dynamically switched as needed, such as high / medium / low bandwidth alternatives. In some implementations, user preference settings can be provided to enable the user to set preferences regarding content adjustments, such as setting their relative importance levels. For example, the user can indicate relative preference levels for aspects such as texture quality, number of objects, resolution, frame rate, etc., and the video game will prioritize the aspects accordingly.

[0079] In some implementations, a machine learning approach is applied to determine the user’s preferences. For example, in some implementations, the system is configured to analyze the user’s game history to determine their preferences. Factors such as the player’s level, how often he / she dies, experience level, difficulty setting selected, combination of game inputs used by the player, etc. can be considered to determine the user’s preferences.

[0080] In addition to making adjustments that affect the video images, the video game can also adjust the audio that the video game is rendering. In existing cloud gaming systems, the video game transmits audio that is rendered in, for example, stereo, 5.1 surround sound, or other high-fidelity audio settings, where such settings are fixed at game startup. The audio encoder of the streaming server encodes the audio stream, and there are limited options available to adjust the audio quality. For example, if network conditions drop (e.g., a user who was streaming in 5.1 now only has bandwidth for stereo), the audio encoder will need to drop channels or downmix to stereo. In addition, the audio encoder can reduce the audio bit rate. However, such adjustments are made after the audio has been rendered. By adjusting the audio of the video game itself, rather than relying on downstream encoding adjustments, better results can be obtained.

[0081] In some implementations, the user’s audio capabilities can change. For example, the user can be using a 5.1 audio setup, but switches to using a pair of headphones halfway through the game. In the case of such a change in audio capabilities, the system can trigger the video game to change its audio settings, and for example, switch from 5.1 to stereo. It will be appreciated that in various implementations, in response to a detected change in the user’s audio capabilities or the user’s listening device, the video game can be triggered to change the audio settings to an audio setting / configuration that is appropriate for the new audio capabilities or listening device.

[0082] By involving the video game in the audio adjustment, more intelligent audio rendering decisions can be made, providing a better user experience. Considering the example above, the video game rendering of 5.1 surround sound can switch to stereo rendering, which provides a better experience than doing a downmix afterwards. Other options that can be applied in response to a reduction in bandwidth include limiting audio / sound effects or disabling background audio, such as background music. In some implementations, sound effects can be prioritized for the purpose of being turned on / off dynamically in response to user connection quality. Generally, in the example above, the video game reduces audio quality in response to network condition degradation, such as a reduction in bandwidth. Likewise, it can be appreciated that in the reverse situation, when network conditions improve, the video can responsively increase audio quality.

[0083] In some implementations, the video game can adjust game difficulty based on network connection or stream quality, such as latency and bandwidth, in order to ensure that the video game remains playable and enjoyable for the user. For example, the video game can be configured to respond to latency between the client and the streaming server. In some implementations, latency is determined based on ping time. In some implementations, the video game can adjust its internal timing, such as when its cutoff times for performing a given action for a given input (e.g., a cutoff time for pressing a button to jump or kill an enemy). For example, in some implementations, when latency increases, the internal timing is relaxed. That is, for a given action, more time is allowed or the cutoff time is extended. Conversely, when latency decreases, the internal timing is tightened, i.e., for a given action, less time is allowed or the cutoff time is shortened. This can have the effect, for example, of making it easier to perform actions such as attacking or killing an enemy when latency increases, and vice versa when latency decreases.

[0084] In some implementations, when connection quality decreases, the video game ramps up the effect of in-game player assist features (e.g., auto-aim, turning assist, etc.), e.g., because the game can become more difficult when latency is relatively high. Conversely, in some implementations, when connection quality improves, the video game decreases the effect of in-game player assist features. Additionally, the above dynamic adjustment of in-game assist features has the effect of making the game easier when network conditions decrease and more difficult when network conditions improve. This helps to maintain a given level of fairness for the user.

[0085] In some implementations, game difficulty can be adjusted in response to the type of device used by the user. For example, playing a video game on a phone using touch controls can be more difficult than playing a video game on a game console using a controller. Thus, when a user switches from using one type of device to a different type of device, this helps to adjust (lower) the game difficulty.

[0086] In some implementations, the video game can be configured to temporarily pause the game when the user connection quality or stream quality is particularly poor or drops below a minimum threshold level (e.g., the user is using a 4G / 5G cellular data network on their phone and is temporarily in a place where reception is poor, such as on a train going through a tunnel). Likewise, the video game can be configured to un-pause the game when the connection / stream quality recovers to an acceptable level. In some implementations, an advertisement can be shown while the game is paused, where the advertisement has been transmitted to the user device and stored there prior to encountering the poor connection / stream quality situation.

[0087] In other implementations, the video game is intentionally paused by the system in order to be able to present an advertisement (rather than in response to poor connection quality).

[0088] In various implementations, the streaming server and the video game can interact in different ways to adjust the video or audio based on changes in connection / stream quality. As an example and not by way of limitation, there can be a target bitrate for the video encoder or audio encoder based on monitored packet loss. In some implementations, the video game is first caused to make adjustments, and then the video encoder or audio encoder is caused to make further adjustments if the adjustments made by the video game are insufficient (e.g., scale, drop frames, adjust encoder settings to reduce bitrate, QP settings, etc.). In a sense, the streaming server can fall back to its own mechanisms if the adjustments made by the game itself are insufficient to meet the current connection quality.

[0089] In some implementations, the streaming server first performs certain adjustments, and then the video game is caused to perform adjustments if the adjustments made by the streaming server are insufficient.

[0090] In some implementations, both the video game and the streaming server are configured to initiate adjustments in response to connection quality changes at substantially the same time.

[0091] In some implementations, there can be a threshold for causing either or both of the video game and the streaming server to perform adjustments in response to connection quality. For example, in some implementations, if the latency exceeds a given threshold, this triggers the video game to begin performing adjustments.

[0092] In some implementations, the streaming server can be set according to game play logic. That is, each game can define settings that control the type and amount of adjustments the video game will make relative to the streaming server. In some implementations, the game provides a configuration file that contains presets for a particular game. The configuration file can provide priority for certain settings to either or both the game engine of the video game and the streaming server. For example, a first person shooter game can prioritize frame rate over resolution, while a puzzle game can prioritize resolution over frame rate.

[0093] In some implementations, as the streaming server monitors the quality of the connection with the client, the streaming server determines when to invoke adjustments in the video game play, and in some implementations, further determines which types of adjustments the video game is to make.

[0094] While the implementations described herein provide for streaming optimization based on feedback returned to the video game from the client or the streaming software, there can also be a feed forward path where the video game provides information to the streaming server or the client to enable streaming optimization. Figure 7 A feed forward optimization from the video game to the streaming logic according to implementations of the disclosure is conceptually illustrated. A scene 700 of a video game is conceptually illustrated, including a virtual character 702 and a door / elevator 706.

[0095] In some implementations, the video game can inform the video encoder of regions of interest. The video game can be configured to determine which portions of a video frame are changing, and can inform the streaming server so that the video encoder can focus on the portions that are changing. In existing game streaming systems, the video encoder allocates a certain amount of bits to all different portions of a video frame, as it is unaware of which portions can change. However, if a frame does not change, such as three-quarters of a frame, the video game will not send data for those portions that did not change, and the bits allocated to those portions are unnecessary. Thus, in some implementations, the video encoder can allocate bits that would otherwise be unnecessarily allocated to portions that did not change to portions of the frame that are experiencing changes, and thus these regions can have better image quality compared to existing systems. This allocation of bits is based on the streaming server receiving information from the video game itself that identifies regions of interest or regions that are changing. Furthermore, even if the entire scene can be changing (e.g., due to a pan of the view), identifying regions of interest can be useful to enable the video encoder to prioritize certain regions over others, such as regions that can be the focus of the user, e.g., where their character is, where their target is, where activity is occurring, etc.

[0096] Continuing with reference to Figure 7In the illustrated implementation, the region of interest 704 is identified (reference 708) by the video game as the region in which the virtual character 702 is located. This region can be identified by the video game based on the location of the virtual character 702 and further based on movement of the virtual character. For example, the region of interest can be identified as the region in which the virtual character is walking towards or in which an interaction is likely to occur. The identified region of interest 704 is communicated to the streaming logic 110 and used by the video encoder 112 to prioritize the encoding quality of the region of interest.

[0097] Another challenge for the video encoder in existing streaming systems occurs at scene changes, as the encoder is unaware and only "sees" a large amount of change in the frame (e.g., the size of the received p-frame is much larger). As a result, scene changes are challenging because the frame can become too large to conveniently handle and the image can take some time to stabilize due to the large amount of change. The result is that many frames can be required to restore the image quality to the state prior to the scene change, during which time the user experience is degraded. However, according to implementations of the present disclosure, the video game can inform the video encoder that a scene change is imminent or that the current frame is part of a scene change. This information can be used by the video encoder, for example, to determine how to allocate its bit budget.

[0098] In the illustrated implementation, the virtual character 702 is moving towards the door / elevator 706. When the virtual character 702 reaches the door 706, a scene change will occur. The video game determines that a scene change is likely to occur soon and communicates this information to the streaming logic 110 (reference 710). The video encoder 112 uses this information to adjust its encoding settings (e.g., allocate a bit budget to handle the upcoming scene change).

[0099] In some implementations, the operating system of the game console / game system can also export certain "feed forward" information. This is particularly useful for unmodified games and / or can provide additional cues to the streaming logic. For example, the process can monitor which files on the disk the game is accessing (e.g., can indicate loading of a new level or other event) or the throughput of the hardware / ssd (e.g., a sudden change in throughput can indicate a scene change or other event). In some implementations, the process can monitor which memory regions the video game is accessing.

[0100] In some implementations, a game profile can be generated (e.g., by a QA operator) based on these system-level indicators. The system-level indicators can be used as markers to determine where the user is in the game. Thus, appropriate reference settings for the video game can be picked based on the indicators and stored into the profile. For example, the profile can map various ones of the indicators to particular reference settings, such that when an indicator has a particular value, then it indicates that the video game is to adjust its video / audio rendering according to the reference settings. Different areas of the game can have different settings. For example, if the user is in a battle (e.g., inferred from storage access, machine learning of video images, machine learning of audio data, rate at which the user presses buttons / joy sticks), then frame rate can be prioritized. Whereas if the user is in a fairly static area or exhibiting low activity, then resolution or other video properties can be prioritized.

[0101] In some implementations, a data channel can extend from the video game to the streaming server to the client, such that the video game can send content to the client, e.g., to render overlays or send code / scripts to provide some client-side logic. The video game feeds forward to the streaming server, which then feeds forward to the client. For example, the video game can send shared assets, e.g., to render overlays for items such as status information, player health, achievements like defeating a boss, etc. In some implementations, data of the data channel can be sent to enable client-side rendering of touch interfaces. In some implementations, code sent to the client through the data channel can be configured to render a timer, e.g., a clock that counts down that the game will be completed in ten seconds. In some implementations, the data channel can be used for peripheral devices (such as microphones, cameras, or game-specific peripheral devices) to provide ways for the video game to control processing of the peripheral devices.

[0102] Such data channels will consume an amount of available bandwidth, so in some implementations, the streaming logic can communicate to the video game an amount of bandwidth available for the data channel. In some implementations, the video game can set a priority setting or a deadline for information to be communicated to the client. In some implementations, the rate at which packets of data for the data channel are propagated can be adjusted, if necessary, e.g., to maintain an amount of bandwidth available for streaming of video / audio.

[0103] Implementations of the present disclosure can be included as part of a game engine. Broadly, a game engine is a software development framework that provides features to enable efficient development of video games. A game engine can include a software library with reusable modules to handle various aspects of game functionality, including (by way of example and not limitation) graphics rendering (e.g., including vertex processing, polygon processing, shading, lighting, texturing, etc.), sound, physics effects (including collision processing), animation, scripting, artificial intelligence, networking, streaming, memory management, threading, localization support, scene graph, cutscenes, etc.

[0104] A game engine can be optimized for different hardware platforms, such as game consoles, mobile devices, personal computers, etc. By way of example and not limitation, a game engine can optimize memory usage (e.g., how various tasks are prioritized in an image pipeline, etc.) according to the platform. In some implementations, the hardware can be a blade version of some specific processing entity, such as a game console. Thus, a user can be assigned to a specific blade, which gives the same hardware that a console game has been optimized for.

[0105] It can be appreciated that there can also be game server logic to provide streaming and / or other services (packetization, encoding, quality of service (QOS) monitoring, bandwidth testing, access to social networks / friends, etc.).

[0106] In some implementations, a cloud infrastructure can run a hypervisor, which abstracts the hardware and provides a virtual machine framework on which an operating system (OS) can be loaded. Thus, the stack can include an application / video game running on the OS, which is loaded on a virtual machine (VM) instantiated by the hypervisor, which is loaded on the underlying hardware. In this way, the execution of the application is not necessarily coupled to a specific hardware.

[0107] In some implementations, an application / video game can execute on a container, which abstracts at the application layer, packaging code and dependencies together, enabling software development that is independent of the OS or hardware platform, and facilitating cross-platform software portability.

[0108] In some implementations, a distributed game engine is employed, in which different parts of the game engine can be processed by different computing entities. For example, functions of the game engine such as physics engine, rendering engine (2D / 3D graphics), sound, scripting, animation, AI, networking, streaming (encoding), memory management, thread handling, etc. can be divided into different function processing blocks and / or services that are distributed among many different computations. It will be appreciated that for a distributed game engine, low latency communication is needed to avoid latency issues. In order to maintain a desired frame rate, the total time of computation and communication should meet certain constraints. Thus, it can or can not be efficient to divide certain tasks depending on whether it is possible to complete the process in a shorter time.

[0109] One advantage of using a distributed game engine is that elastic computing can be utilized, in which computing resources can be scaled up or down as needed. For example, in a large multiplayer game that is traditionally executed on a single hardware server, after, for example, about 100 players, the hardware resources become limited, so that no more players can be added. The game can queue additional players, meaning that players have to wait to join the game. However, using a distributed game engine, by using elastic cloud computing resources, more computing nodes can be added to meet the demand, supporting, for example, thousands of players. The game is no longer limited by a particular hardware server.

[0110] Thus, a cloud game engine can distribute functionality to different processing entities. It can be appreciated that different functionality can be executed in different frameworks. For example, some functionality (e.g. social) can be more easily run in a container, while graphics can be better run using a VM connected to a GPU.

[0111] To facilitate distribution of cloud game engine functionality, a distribution / synchronization layer can manage distribution of jobs, e.g. issuing jobs, retrieving data, identifying which tasks to perform and when, handling queuing (in cases where a job completes faster than needed). In some implementations, a given task can be dynamically subdivided if needed. For example, an animation can have lighting, and if the lighting is particularly complex, the lighting can be subdivided into three lighting jobs that are issued to be computed and reassembled upon return. Thus, if a game engine function needs more work, it can be subdivided.

[0112] A cloud service provider provides computing at specified performance levels, e.g. in terms of input / output operations per second (“IOPS”). Thus, a game provider can specify VMs, dedicated processing capabilities, amounts of memory, etc. from a cloud service provider, and use the cloud service provider’s system to instantiate a distributed cloud game engine.

[0113] In some implementations, the library module and the update handler can be one or more components or modules of the game engine. In some implementations, the library module and the update handler can be standalone components, or integrated. In some implementations, the library module and the update handler can operate in addition to the game engine. In some implementations, the game engine can be a distributed game engine, as described above.

[0114] As noted, implementations of the present disclosure can be applied to a cloud gaming system. One example of a cloud gaming system is Now cloud gaming system. In such systems, the client device can be a game console, such as 4game console, or can be another device, such as a personal computer, laptop computer, tablet computer, phone, mobile device, etc.

[0115] Broadly, to implement cloud gaming, when a user request for a game title is received, a number of operations are performed by one or more servers within a data center associated with the cloud gaming site. When the cloud gaming site receives the user request, a data center hosting a game associated with the selected game title is identified, and the request is sent to the identified data center to instantiate the game for the selected game title. In response to the request, the server at the data center identifies game code, loads the identified game code and initializes files related to the game code in preparation for presenting game content to the user. Game data associated with the game can include generic game data and user-specific game data. Thus, initializing the files can include identifying, loading and initializing the generic game data and the user-specific game data. Initializing the generic game data can include initializing a graphics engine, installing graphics data, initializing sound files, installing original artwork, etc. Initializing the user-specific data can include locating, transmitting and installing user data, user history, game history, etc.

[0116] Upon loading and initializing the generic game data, a "splash" screen can be provided for rendering at the client device. A splash screen can be designed to provide a representative image of the game being loaded to allow the user to preview the type of game being loaded. Once the generic game data is loaded, certain initial content can be rendered, and a selection / navigation screen can be presented for the user to select and customize. User selection inputs provided at the selection / navigation screen can include game level selection, one or more game icon selections, game mode selection, game rewards, and other user-related data that can require uploading of additional game content. In some embodiments, the game content is made available for viewing and interaction by streaming the game content from the game cloud system to the user's computing device. In some implementations, the game content is available for playing after the user-specific data is loaded.

[0117] Figure 8A An exemplary system for loading game files for games available through a cloud gaming site is shown. The system includes a plurality of client devices 800 that are communicatively connected to a cloud gaming site 804 through a network 802, which can include a LAN, a wired, wireless, cellular (e.g., 4G, 5G, etc.) network, or any other type of data network, including the Internet. When a request to access the cloud gaming site 804 is received from a client device 800, the cloud gaming site 804 accesses user account information 806 stored in a user data store 808 to identify a user associated with the client device that initiated the request. In some embodiments, the cloud gaming site can also verify the identified user in order to determine all games that the user is authorized to view / play. After the user account is identified / verified, the cloud gaming site accesses a game name data store 810 to identify game names available at the game cloud site for the user account that initiated the request. The game name data store 810 in turn interacts with a game database 812 to obtain game names for all games available to the cloud gaming site. When a new game is launched, the game database 812 will be updated with the game code and the game name data store 810 will be provided with game name information for the newly launched game. When a request is initiated, the initiating client device can or can not be registered with the cloud gaming site. If the user of the initiating client device is not a registered user, the cloud gaming site can identify the user as a new user and select a game name appropriate for the new user (e.g., a default set of game names). The identified game names are returned to the client device for presentation on a display screen 800-a as shown. Figure 8A

[0118] ​A user interaction on one of the game titles presented on the client device is detected and a signal is sent to the cloud gaming site. The signal includes the game title information for which the user interaction was detected and the user interaction registered at the game title. In response to the signal received from the client device, the cloud gaming site proactively determines a data center hosting the game and sends a signal to the identified data center to load the game associated with the game title for which the user interaction was detected. In some embodiments, there can be more than one data center hosting the game. In such embodiments, the cloud gaming site can determine a geographic location of the client device that initiated the request and identify a data center that is geographically close to the client device and signal that data center to preload the game. The geographic location of the user can be determined using a global positioning system (GPS) mechanism within the client device, the IP address of the client, the ping information of the client, etc. Of course, the foregoing ways of detecting the geographic location of the user can be exemplary and other types of mechanisms or tools can be used to determine the geographic location of the user. Identifying a data center close to the client device can minimize latency during the user's interaction with the game. In some embodiments, the identified data center can not have the bandwidth / capacity required to host the game or can be over-utilized. In these embodiments, the cloud gaming site can identify a second data center that is geographically close to the client device. The loading of the game includes loading the game code and executing an instance of the game.

[0119] In response to receiving the signal from the cloud gaming site, the identified data center can select a server at the data center to instantiate the game on the server. The server can be selected based on available hardware / software capabilities and the game requirements. The server can include a plurality of game consoles and the server can determine which one of the plurality of game consoles to use to load the game. The game console can be similar to a standalone game console or can be a rack-mounted server or a blade server. The blade server in turn can include a plurality of server blades, where each blade has the circuitry required to instantiate a single dedicated application, such as a game. Of course, the foregoing game console is exemplary and should not be considered limiting. Other types of game consoles, including the Xbox, etc., and other forms of blade servers can also be used to host the identified game.

[0120] Once the game console is identified, the generic game-related code for the game is loaded onto the game console and a signal is returned to the client device via the cloud gaming site over the network identifying the game console on which the game is instantiated. Thus, the loaded game is available for use by the user.

[0121] Figure 8Bis a flowchart conceptually illustrating various operations performed to stream a cloud video game to a client device, according to implementations of the present disclosure. A game system 818 executes a video game and generates raw (uncompressed) video 820 and audio 822. The video 820 and audio 822 are captured and encoded for streaming purposes, as indicated at reference numeral 824 in the illustrated figure. The encoding can be used for compression of the video and audio streams to reduce bandwidth usage and optimize the gaming experience. Examples of encoding formats include H.265 / MPEG-H, H.264 / MPEG-4, H.263 / MPEG-4, H.262 / MPEG-2, WMV, VP6 / 7 / 8 / 9, etc.

[0122] The encoded audio 826 and encoded video 828 are further packetized into network packets, as indicated at reference numeral 832, for transmission over a network, such as the Internet. The network packet encoding process can also employ data encryption processes, providing enhanced data security. In the illustrated implementation, audio packets 834 and video packets 836 are generated for transmission over the network, as indicated at reference numeral 840.

[0123] The game system 818 additionally generates haptic feedback data 830, which is also packetized into network packets for network transmission. In the illustrated implementation, haptic feedback packets 838 are generated for transmission over the network, as further indicated at reference numeral 840.

[0124] The above-described operations of generating raw video and audio and haptic feedback data, encoding the video and audio, and packetizing the encoded audio / video and haptic feedback data for transmission are performed on one or more servers that collectively define a cloud gaming service / system. As indicated at reference numeral 840, the audio, video, and haptic feedback packets are transmitted over a network, such as and / or including the Internet. As indicated at reference numeral 842, the audio packets 834, video packets 836, and haptic feedback packets 838 are decoded / reassembled by a client device to define encoded audio 846, encoded video 848, and haptic feedback data 850 at the client device. If the data has been encrypted, the network packets will also be decrypted. The encoded audio 846 and encoded video 848 are then decoded by the client device, as indicated at reference numeral 844, to generate client-side raw audio and video data for rendering on a display device 852. The haptic feedback data 850 can be processed / communicated to produce haptic feedback effects at a controller device 856 or other interface device that can present haptic effects. One example of a haptic effect is a vibration or rumble of the controller device 856.

[0125] It should be understood that the video game responds to user input and thus can perform a similar but in the opposite direction from the client device to the server of the program flow described above for the transmission and processing of user input. As shown, a user operating a controller device 856 can generate input data 858. This input data 858 is packetized at the client device for transmission over the network to the cloud gaming system. The input data packets 860 are unpacked and reassembled by the cloud gaming server to define input data 862 on the server side. The input data 862 is fed to the game system 818, which processes the input data 862 to update the game state of the video game.

[0126] During the transmission (reference 840) of the audio data packets 834, the video data packets 836, and the haptic feedback data packets 838, the data transmission over the network can be monitored to ensure the quality of service of the cloud game stream. For example, the network conditions, including both upstream and downstream network bandwidth, can be monitored as indicated by reference 864, and the game streaming can be adjusted in response to changes in available bandwidth. That is, the encoding and decoding of the network data packets can be controlled based on the current network conditions, as indicated by reference 866.

[0127] Figure 9 An embodiment of an information service provider architecture is shown. An information service provider (ISP) 970 provides a number of information services to users 1082 that are geographically dispersed and connected via a network 986. An ISP can provide only one type of service, such as stock price updates, or can provide multiple services, such as broadcast media, news, sports, games, etc. In addition, the services provided by each ISP are dynamic, i.e., services can be added or removed at any point in time. Thus, the ISP that provides a particular type of service to a particular individual can change over time. For example, when a user is at her home town, the user can be provided services by an ISP that is close to the user, and when the user travels to a different city, the user can be provided services by a different ISP. The local ISP will transfer the required information and data to the new ISP so that the user information "follows" the user to the new city, making the data closer to the user and more easily accessible. In another embodiment, a master-slave relationship can be established between a master ISP that manages information for the user and a server ISP that interfaces directly with the user under the control of the master ISP. In another embodiment, as the client moves around the world, data is transferred from one ISP to another so that the ISP that is in a better position to provide services to the user becomes the ISP that provides those services.

[0128] The ISP 970 includes an application service provider (ASP) 972 that provides computer-based services to customers over a network (e.g., including but not limited to any wired or wireless network, LAN, WAN, WiFi, broadband, cable, fiber, satellite, cellular (e.g., 4G, 5G, etc.), the Internet, etc.). Software provided using the ASP model is sometimes called on-demand software or software as a service (SaaS). A simple form of providing access to a particular application, such as customer relationship management, is using a standard protocol such as HTTP. The application software resides on a vendor's system and is accessed by a user using a web browser via HTML, by a dedicated client software provided by the vendor, or by other remote interface such as a thin client.

[0129] Services provided over a wide geographic area often use cloud computing. Cloud computing is a style of computing in which dynamically scalable and often virtualized resources are provided over the Internet as a service. Users do not need to be an expert in the technology infrastructure in the "cloud" that supports them. Cloud computing can be divided into different services, e.g., infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). Cloud computing services often provide online, general business applications that are accessed from a web browser, while the software and data are stored on servers. The term cloud is used as a metaphor for the Internet (e.g., using servers, storage and logic), based on how the Internet is depicted in computer network diagrams and is an abstract concept that conceals the complexity of the infrastructure it is hiding.

[0130] Further, the ISP 970 includes a game processing server (GPS) 974 that is used by game clients to play single and multiplayer video games. Most video games played over the Internet are run through a connection with a game server. Typically, the game uses a dedicated server application that collects data from the players and distributes it to the other players. This is more efficient and effective than a peer-to-peer arrangement, but it requires a separate server to host the server application. In another embodiment, the GPS establishes communication between the players, and the respective game playing devices of the players exchange information without relying on a centralized GPS.

[0131] A dedicated GPS is a server that runs independently of the client. Such a server is typically run on dedicated hardware located within a data center, providing more bandwidth and dedicated processing power. For most PC-based multiplayer games, a dedicated server is the preferred method of hosting a game server. Large multiplayer online games run on dedicated servers, which are typically hosted by the software company that owns the game title, allowing them to control and update the content.

[0132] A broadcast processing server (BPS) 976 distributes audio or video signals to viewers. Broadcasting to a very small range of viewers is sometimes called narrowcasting. The last leg of broadcast distribution is how the signal reaches the listener or viewer, and it can come over the air as in radio broadcasting or television broadcasting, or can come through cable TV or cable radio (or "wireless cable"), either directly to the reception antenna / speaker / scope, or a workstation connected to a network. The Internet can also bring either radio or TV to the receiver, especially using technologies like Streaming media to allow a single signal to be shared by a large number of recipients. Historically, broadcasts have been defined as being divided by geographic regions, such as national broadcasts or regional broadcasts. However, with the popularization of fast Internet, broadcasts are not defined by geographic conditions, as content can reach almost any country in the world.

[0133] A storage service provider (SSP) 978 provides computer storage space and related management services. The SSP also provides periodic backups and archives. By providing storage as a service, users can order more storage as needed. Another major advantage is that the SSP includes a backup service, and if the computer's hard drive fails, the user will not lose all of their data. In addition, multiple SSPs can have full or partial copies of a user's data, allowing the user to access the data in an efficient manner, regardless of the location the user is in or the device used to access the data. For example, a user can access personal files in a home computer as well as in a mobile phone (as the user moves around).

[0134] A communication provider 980 provides connectivity to users. One type of communication provider is an Internet service provider (ISP), which provides access to the Internet. ISPs connect their customers using data transmission technologies suitable for delivering Internet Protocol datagrams, such as dial-up, DSL, cable modems, fiber, wireless, or dedicated high-speed interconnections. Communication providers can also provide messaging services, such as email, instant messaging, and SMS texting. Another type of communication provider is a network service provider (NSP), which sells bandwidth or network access by providing direct backbone access to the Internet. Network service providers can include telecommunications companies, data carriers, wireless communication providers, Internet service providers, cable television operators that provide high-speed Internet access, and the like.

[0135] A data exchange 988 interconnects several modules within the ISP 970 and connects the modules to users 982 via the network 986. The data exchange 988 can cover a small area where all of the modules of the ISP 970 are in very close proximity, or it can cover a large geographic area when the different modules are geographically dispersed. For example, the data exchange 988 can include fast gigabit Ethernet (or faster Ethernet) within the racks of a data center, or an intercontinental virtual local area network (VLAN).

[0136] A user 982 accesses the remote service using a client device 984, which includes at least a CPU, memory, a display, and I / O. The client device can be a PC, mobile phone, notebook computer, tablet computer, game system, PDA, etc. In one embodiment, the ISP 970 recognizes the type of device used by the client and adjusts the communication method employed. In other cases, the client device uses a standard communication method, such as html, to access the ISP 970.

[0137] Embodiments of the disclosure can be practiced with various computer system configurations including hand-held devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. The disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire- or wireless network.

[0138] In some embodiments, wireless technology can be used to facilitate communication. Such technology can include, for example, 5G wireless communication technology. 5G is the fifth generation of cellular network technology. A 5G network is a digital cellular network in which a provider covers a service area divided into small geographic areas called cells. Analog signals representing sounds and images are digitized, converted by an analog-to-digital converter, and transmitted as bit streams in the phone. All 5G wireless devices in a cell communicate by radio waves with a local array of antennas and low-power automatic transceivers (transmitters and receivers) in the cell through a frequency channel allocated by a transceiver from a pool of frequencies that are reused in other cells. The local antennas are connected with the phone network and the Internet through high-bandwidth optical fiber or wireless backhaul. Like other cell networks, a mobile device automatically transfers to a new cell when it crosses from one cell to another. It should be understood that 5G networks are just one example type of communication network, and embodiments of the present invention can utilize earlier generations of wireless or wired communication, as well as newer generations of wired or wireless technology after 5G.

[0139] With the above embodiments in mind, it should be understood that the disclosure can employ various computer-implemented operations that manipulate data stored as bits in computer memories. This manipulation is for purposes of facilitating the operations and is carried out by physical manipulations of physical quantities. Any of the operations described herein that form part of the disclosure are useful machine operations. The disclosure is also related to the apparatus or devices used to perform the operations. The required structure for a device can be created using a general purpose computer, or a special purpose computer, or a special purpose device that is customized to perform the operations described herein. Specifically, various general purpose machines can be used with computer programs written in accordance with the teachings herein, or it can be more convenient to construct a more specialized machine to perform the required operations.

[0140] The present disclosure can also be implemented as computer-readable code on a computer-readable medium. Alternatively, the data exchange interworking described above can be used to download the computer-readable code from a server. The computer-readable medium is any data storage device that can store data which can thereafter be read by a computer system. Examples of computer-readable media include hard drives, network attached storage devices (NAS), read-only memory, random-access memory, CDs, DVDs, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and non-optical data storage devices. The computer-readable medium can include computer-readable tangible medium distributed over a network-coupled computer system so that the computer-readable code is stored and executed in a distributed fashion.

[0141] Although the method operations were described in a specific order, it should be understood that other housekeeping operations can be performed in between operations, or operations can be adjusted so that they occur at slightly different times, or can be distributed in different order, or distributed across different systems, etc. without departing from the scope of the present disclosure.

[0142] While the foregoing disclosure has been described in some detail for purposes of clarity and the specific embodiments, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Accordingly, the present embodiments are to be considered illustrative and not restrictive, and the disclosure is not to be limited to the details given herein, but can be modified within the scope and equivalents of the described embodiments.

Claims

1. A method for cloud gaming, the method comprising: executing, by a cloud game machine, a video game, the execution of the video game including rendering a play game video; processing, by a streaming server, the play game video for streaming over a network to a client device; monitoring a quality of a connection between the streaming server and the client device over the network; in response to detecting a change in the quality of the connection between the streaming server and the client device, then adjusting, by the cloud game machine, the rendering of the play game video and a difficulty of the video game.

2. The method of claim 1, wherein processing, by the streaming server, the play game video includes encoding the play game video in a compressed video format.

3. The method of claim 1, wherein monitoring a quality of a connection includes monitoring one or more of a bandwidth or a latency.

4. The method of claim 1, wherein monitoring a quality of a connection includes monitoring packet loss occurring during the streaming of the play game video.

5. The method of claim 1, wherein adjusting, by the cloud game machine, the rendering of the play game video includes adjusting a frame rate or a resolution of the play game video.

6. The method of claim 1, wherein adjusting, by the cloud game machine, the rendering of the play game video includes adjusting a level of detail of the rendering of the play game video.

7. The method of claim 1, wherein adjusting, by the cloud game machine, the rendering of the play game video includes adjusting an amount of content included in the play game video.

8. The method of claim 1, wherein the change in the quality of the connection is defined by a degradation of the quality of the connection; wherein adjusting, by the cloud game machine, the rendering of the play game video is configured to decrease a bit rate of the streaming of the play game video over the network.

9. The method of claim 1, wherein the change in the quality of the connection is defined by an improvement of the quality of the connection; wherein adjusting, by the cloud game machine, the rendering of the play game video is configured to increase a bit rate of the streaming of the play game video over the network.

10. The method of claim 1, wherein detecting the change in the quality of the connection triggers a transmission of a message from the streaming server to the cloud game machine, and wherein the adjusting, by the cloud game machine, of the rendering of the play game video is responsive to the message.

11. A method for cloud gaming, the method comprising: executing, by a cloud game machine, a video game, the execution of the video game including rendering a play game video; processing, by a streaming server, the play game video for streaming over a network to a client device, wherein processing, by the streaming server, the play game video includes encoding the play game video in a compressed video format; monitoring connection quality between the streaming media server and the client device over the network, wherein monitoring connection quality comprises monitoring packet loss occurring during the streaming of the gameplay video; adjusting, by the cloud game machine, the rendering of the gameplay video and difficulty of the video game in response to detecting a change in the connection quality between the streaming media server and the client device.

12. The method of claim 11, wherein adjusting, by the cloud game machine, the rendering of the gameplay video comprises adjusting a level of detail of the rendering of the gameplay video.

13. The method of claim 11, wherein the change in the connection quality is defined by a degradation of the connection quality; wherein adjusting, by the cloud game machine, the rendering of the gameplay video is configured to reduce a bitrate of the streaming of the gameplay video over the network.

14. The method of claim 11, wherein the change in the connection quality is defined by an improvement of the connection quality; wherein adjusting, by the cloud game machine, the rendering of the gameplay video is configured to increase a bitrate of the streaming of the gameplay video over the network.

15. The method of claim 11, wherein detecting the change in the connection quality triggers transmission of a message from the streaming media server to the cloud game machine, and wherein the adjusting of the rendering of the gameplay video by the cloud game machine is in response to the message.

16. A system for cloud gaming, the system comprising: a cloud game machine that executes a video game, the execution of the video game comprising rendering a gameplay video; a streaming media server that processes the gameplay video for streaming over a network to a client device, wherein the streaming media server monitors connection quality between the streaming media server and the client device over the network; wherein the rendering of the gameplay video and difficulty of the video game are adjusted by the cloud game machine in response to detecting a change in the connection quality between the streaming media server and the client device.

17. The system of claim 16, wherein processing the gameplay video by the streaming media server comprises encoding the gameplay video in a compressed video format.

18. The system of claim 16, wherein monitoring connection quality comprises monitoring one or more of bandwidth or latency.

19. The system of claim 16, wherein monitoring connection quality comprises monitoring packet loss occurring during the streaming of the gameplay video.

20. The system of claim 16, wherein adjusting, by the cloud game machine, the rendering of the gameplay video comprises adjusting a frame rate or resolution of the gameplay video.

Images