Streaming systems and methods

Abstract

A method for streaming interactive content includes the following steps: initiating the generation of a first frame of content reflecting the current state of an application, the generation including a multi-stage process; during the generation of the first frame, receiving input data affecting the state of the application via a network from a client device; evaluating the progress of the generation against the first criterion; stopping the generation of the first frame of content in response to the evaluation against the first criterion; updating the current state of the application in response to the input data; generating a second frame of content reflecting the updated state of the application; and outputting the generated second frame without outputting the stopped first frame.

Description

Technical Field

[0001] This invention relates to a streaming system and method. Background Technology

[0002] Traditionally, video games have been played using PCs or consoles that generate game content locally. However, so-called cloud gaming or game streaming has recently become popular. In this case, the game is executed in the cloud and streamed to the player's client device. Therefore, it's worth noting that the client device itself does not need to be able to execute the game locally, thus expanding access to the game.

[0003] However, one problem with game streaming is the potential for lag or delay between user input uploaded to the cloud server and the updated game images streamed from the cloud server.

[0004] The present invention seeks to solve or mitigate this problem. Summary of the Invention

[0005] Various aspects and features of the invention are defined in the appended claims and in the text of the appended description.

[0006] In a first aspect, claim 1 provides a method for streaming interactive content.

[0007] In another aspect, claim 13 provides a streaming server for interactive content.

[0008] In another aspect, claim 15 provides a streaming system for interactive content. Attached Figure Description

[0009] A more complete understanding of this disclosure and its many accompanying advantages will become readily apparent when considered in conjunction with the accompanying drawings, and by referring to the following detailed description, in which:

[0010] Figure 1 This is a schematic diagram of an entertainment device suitable as a client or server according to an embodiment of this specification.

[0011] Figure 2 This is a schematic diagram of a streaming system for interactive content according to an embodiment of this specification.

[0012] Figure 3A and Figure 3B This is a schematic diagram of frame and input timing according to an embodiment of this specification.

[0013] Figure 4A and Figure 4B This is a schematic diagram of frame and input timing according to an embodiment of this specification.

[0014] Figure 5 This is a schematic diagram of a multi-stage rendering process according to an embodiment of this specification.

[0015] Figure 6A , Figure 6B and Figure 6C This is a schematic diagram of frame and input timing according to an embodiment of this specification.

[0016] Figure 7 This is a schematic diagram of three views of a handheld controller according to an embodiment of this specification.

[0017] Figure 8 This is a flowchart of a streaming method for interactive content according to an embodiment of this specification. Detailed Implementation

[0018] A streaming system and method are disclosed. Numerous specific details are set forth in the following description to provide a thorough understanding of embodiments of the invention. However, it will be apparent to those skilled in the art that these specific details are not necessary to practice the invention. Instead, specific details known to those skilled in the art have been omitted where appropriate for clarity.

[0019] Referring now to the accompanying drawings, in which the same reference numerals denote the same or corresponding parts in several views. Figure 1 An example of an entertainment system 10, such as a computer or console, is shown.

[0020] Depending on its configured role (e.g., via appropriate software instructions), an entertainment device can operate as a cloud gaming server or as a client device, and is used as an example of both in this article.

[0021] The entertainment system 10 includes a central processing unit or CPU 20. The entertainment system also includes a graphics processing unit or GPU 30 and RAM 40. Two or more of the CPU, GPU, and RAM can be integrated into a system-on-a-chip (SoC). Further storage can be provided by a disk 50.

[0022] The entertainment device can send or receive data via one or more data ports 60. It can also optionally receive data via an optical driver 70. Audio / video output from the entertainment device is typically provided via one or more A / V ports 90 or one or more data ports 60. Where the components are not integrated, they can be appropriately connected via a dedicated data link or via a bus 100.

[0023] Examples of devices used to display images output by an entertainment system include a head-mounted display “HMD” 120 worn by user 1, a television (not shown), and a portable screen 140.

[0024] In the case of HMD, one or more handheld controllers 130, 140 and / or one or more VR controllers (130A-L, 130A-R) are typically used to provide interaction with the system.

[0025] Now for reference Figure 2 The entertainment device 10 collaborates with a controller, such as a handheld controller 130, and communicates with a cloud server 200 via a network 150, such as the Internet. The entertainment device receives input signals from the controller and relays them (or their parsed versions, such as input IDs and values ​​instead of the original signals) to the cloud server 200. The cloud server receives these inputs and uses them in games running on the server, typically as if the inputs were regular local inputs, allowing the game to operate transparently. The cloud server's output (video images and often audio) is then streamed back to the entertainment device via the network. Similarly, a device 140 with built-in controller inputs can interact with the server in a similar manner.

[0026] However, compared to what would occur during local playback, the round trips over the network from the entertainment device to the cloud server and from the cloud server back to the entertainment device can introduce lag. Typically, the round-trip network latency between player input and server output can be on the order of 10-20 milliseconds per path.

[0027] This may seem small, but video games are typically designed to run at 60 frames per second to make the game feel responsive to the user. Therefore, there is usually only a 16-millisecond interval between frames during which the game state is updated, so a variable input latency of similar magnitude can significantly affect the apparent responsiveness of a game.

[0028] Now for reference Figure 3A The diagram shows three example frames 310, 320, and 330, each with a duration of 16 ms (the shading is a conceptual indication of the progress of the corresponding frame's completion). The diagram also shows the input action time at 0 ms when the user provides input to their controller. If the controller is local to the game, the input will be provided to the game at 0 ms and reflected in the first output image generated at 16 ms.

[0029] However, in this case, input [A] is packaged and transmitted to the game's server instance, arriving at 17ms. As a result, it is received after the first output image has already been generated and after the second output image has begun to be generated. Therefore, it must wait until the start of the third output image at 32ms, and the input is only reflected in the third output image at 48ms (which can then be received by the user after another 10-20ms network delay).

[0030] Therefore, the local response time of about 16ms has become the network response time of about 48-68ms.

[0031] We really hope to reduce this response delay.

[0032] The variability of network lag can also cause other problems. Now refer to Figure 3B In this case, the next input [B] from the controller is packaged and sent at the next frame interval of 16ms (for example), but arrives at the server within 15ms instead of within 17ms as the first input packet did.

[0033] Therefore, both inputs A and B can be applied to update the game state at 32ms. However, compatibility is not guaranteed; [A] could indicate a button was pressed, and [B] could indicate it wasn't—which input is the correct one to use now? Should [A] be used (otherwise the button press will never be registered), or is [B] better suited to more accurately reflect the player's timing if it's simply the end of a longer button press as conveyed in an earlier input package? Furthermore, if the game itself operates transparently (i.e., assuming it runs on a normal, locally controlled system without additional code handling these situations), which part of the system makes this decision?

[0034] Similarly, the goal is to avoid this input contention problem.

[0035] In short, because there is no correlation between the time it takes for the game to need input to update the game state and the time it actually takes for the input to arrive over the network, inputs may cause unwanted delays waiting for the next frame, or conflict with each other within a given frame.

[0036] As mentioned above, network latency can cause an additional 32 milliseconds of delay. In networks with ping times of 10-20ms per path, this can result in a total latency that feels too unresponsive. Meanwhile, for users with fast ping times of 2-10ms per path, the additional latency represents a significant portion of the total latency and can still be frustrating. Furthermore, in any competitive environment (such as esports), any additional latency is undesirable.

[0037] Therefore, in the embodiments of this specification, when the server receives an input packet after the generation of a new frame has already begun, the new frame is discarded, and frame replacement begins based on the received input. This frame discarding scheme can be implemented by the operating system intervening in the rendering pipeline to reset it, or by resetting the frame generation loop in the game. It is possible that the operating system requires configuration data for a specific game title, indicating which variables or memory values ​​should be changed to apply such modifications to the game. Alternatively, the game can be simply coded or updated to reset its frame generation loop in response to API calls from the OS indicating the arrival of new input, or when new input data is received in a designated memory area monitored by the game.

[0038] In any case, for the first approximation, when a new input is received, the current frame generation is abandoned, the game state is updated in response to the input, and a new frame generation begins.

[0039] Therefore, now refer to Figure 4A At time 16ms, frame 320 was started being generated, but a new input [A] was received at time 17ms. As a result, frame 320 was discarded (by...). Figure 4A The X in the image represents the frame, and the new frame starts at 320'. Therefore, the total latency before generating an image reflecting user input has been reduced from 48 milliseconds to 33 milliseconds, making the game more responsive to the user.

[0040] Still referencing Figure 4B For the second approximation, the method is subject to a timing threshold. The timing threshold is set such that the input can cause a small delay to avoid long delays between output images or a lack of response to previous output images.

[0041] therefore, Figure 4B copied Figure 3B The input timing is as follows. In this case, input [B] is received at time 31ms after 15ms. However, in this case, discarding frame 320' would not be beneficial; it would mean that the image reflecting input [A] would again not be output for a total of about 48 or 49ms. This also means that a large amount of rendering data needs to be cleared from the system, all to avoid the additional delay of 2ms for input [B].

[0042] Therefore, if the latency caused by not discarding the current frame is less than a predetermined amount, the current frame is not discarded. The predetermined amount can be any amount up to, for example, half the frame generation duration.

[0043] To avoid the possibility that inputs arrive consistently early enough during the frame generation cycle (e.g., if the network latency is consistently 10ms and the drop threshold is after 12ms), and therefore there is a consistent dropping of the current frame and no frames are actually output, the above method can apply a hysteresis so that it can only happen in isolation, rather than for two or more frames in a row (or for a predetermined maximum number of frames in a row).

[0044] Using the above techniques for the first approximation, or preferably the second approximation, the average delay between user input and the result seen by the user is advantageously reduced by about one frame (typically 16 ms), and also avoids contention issues.

[0045] It is worth noting that the above methods can be further improved to increase overall efficiency and potentially reduce latency even more.

[0046] Go to Figure 5 The process of generating an image frame within 16ms involves multiple stages.

[0047] After the game logic (game state) has been updated, the pipeline generally, but not necessarily in the given order, includes some or all of the following (as well as other possible steps not included in this article).

[0048] Geometric update (510)

[0049] - If new elements need to be displayed, or if elements have changed, the geometry of the environment to be rendered can be updated. Similarly, changes to character animations / poses can be defined.

[0050] Z-buffer update (520)

[0051] - Update which aspects of the environment are visible based on geometry and virtual camera viewpoint - unseen parts do not need to be processed, or are processed only to a limited extent.

[0052] Texture (530)

[0053] Texture maps are used to apply basic graphic images to the geometry of a scene. Typically, different textures are used as a function of distance from the viewpoint.

[0054] Static lighting (540)

[0055] - Light maps can be used for direct and indirect lighting, shadows, and reflections.

[0056] Dynamic lighting (550)

[0057] - It can calculate direct and indirect lighting, shadows, and reflections for light sources in dynamic games (such as the player's flashlight).

[0058] Atmosphere (560)

[0059] Effects such as mist or fog, particle effects, and translucency (water) are also possible.

[0060] Post-processing (570)

[0061] - Various effects that can typically be applied to the entire image include halos, exposure, depth of field bokeh, lens effects, etc.

[0062] Zoom in (580)

[0063] - If rendering has been performed at a lower resolution than expected in order to achieve the preferred frame rate, some systems have a machine learning-based AI upgrade step to restore the rendered image to the target resolution.

[0064] Although these stages are Figure 5 The stages are shown as equal in size, but in reality, some stages take longer than others, and some stages may overlap or be revisited during the rendering process. Similarly, as mentioned above, some stages may occur in a different order than those listed. However, sequential stages are generally used to form the image.

[0065] Therefore, in the embodiments of this specification, for the third approximation when a discarded frame is generated, only those stages of the generation process that will change in response to the received input are discarded.

[0066] As will be discussed later in this article, “response change” may mean a threshold amount of response change, but for the third approximation, the threshold may not be considered.

[0067] Therefore, for example, in the case of movement input that changes the game's viewpoint (e.g., moving the player character), the geometry and animation of at least the rest of the environment can usually be preserved 510 and do not need to be regenerated because the viewpoint is the same environment.

[0068] Stages 520-540 can typically be retained; z-buffer updates (z-culling) are usually viewpoint-dependent, but for small changes in the viewpoint, such as those occurring in 16ms time increments, several strategies are possible:

[0069] The -z buffer can be intentionally set to not cull surfaces up to the occluded edges, but instead preserve small occluded boundaries so that a small portion of the unseen image is rendered as it is revealed by the updated viewpoint.

[0070] - The z-buffer can be rerun only for those parts of the environment that were previously marked as unseen, and textures can be added for any newly exposed pixels.

[0071] - Assuming any marginal errors are small and visible only within 16ms, the z-buffer can remain as is; alternatively, interpolation or other patching techniques can be used to hide these errors.

[0072] If dynamic lighting is currently present, the dynamic lighting phase 550 may only be relevant to changes in the input, and further, it may only be relevant if the lighting will be affected by such changes—for example, if the light source is a flashlight held by the player's avatar and causes it to move on its own or as part of an avatar animation that itself changes due to the input. Therefore, for example, dynamic lighting may be preserved or ignored unless the input will cause the light to move laterally within the image.

[0073] Phase 560 can also be retained, optionally for a subset of atmospheric effects; for example, fog is unlikely to change significantly in response to a 16ms input. Meanwhile, particle effects can be noticeable, especially if they are in the foreground or used to visually fill a volume of space and therefore have a clear parallax when the viewpoint moves laterally. Thus, all atmospheres, some atmospheres, or no atmospheres can be retained, depending on which atmospheres are used and what inputs have been received (or equivalently and potentially more precisely, what changes they have caused in the game state, since a left turn input would have no effect if a left turn is currently impossible in the game).

[0074] A similar approach can be used for post-processing, where there is a set of effects, not all of which are applied to a given image, and not all of which will be substantially affected by every type of input.

[0075] At this point, the rendered image is essentially complete; therefore, as relative to... Figure 3B Regarding the discussion of frame 320, it is generally more efficient to proceed to the end and output the image at some point during the frame generation phase.

[0076] However, alternatively, in this case, for some inputs (especially moving inputs that change the viewpoint), reprojection (for lateral changes in the viewpoint) and / or scaling (for depth changes in the viewpoint) can be used to update the final image to incorporate changes caused by motion.

[0077] Additionally, magnification is typically performed only on the final image. Generally, if an image has already reached an optional magnification stage, it would be more efficient to let it complete the magnification and be output (e.g., for reasons related to...). Figure 3B (Similar reason given in output frame 320).

[0078] Alternatively, reprojection and / or scaling can be used after magnification if it has not been used before, but potentially, if input is only received at this stage, it can be considered preferred to use it for the next image frame in any case.

[0079] Therefore, for the third approximation, those parts of the current frame generation process that can be reused based on newly received inputs and / or their effects on the game state can be retained for replacing image frames, which typically means that the next image frame can be generated in less than 16ms.

[0080] Therefore, now refer to Figure 6A In the example where input [C] is delayed by 24ms (half the generation time of the next frame), the second frame 320 has already been generated and, for example, far enough to apply textures. In this example, geometry and textures (represented by the large arrow between 320 and 320') can be reused, and since the user is currently running forward in the game, z-culling does not change significantly. Therefore, all the rendering processes up to this point can be looped, and new image 320' can continue to be rendered, but the viewpoint moves forward in response to the input. As a result, the image is output, for example, at 34ms (instead of 32ms, assuming a small overhead of, for example, 2ms when checking the rendering pipeline for reuse, and requiring any small corrections).

[0081] At the same time, Figure 6B In this case, the input [C] is delayed again by 24ms, and the texture has been applied again in the second frame 320, for example. However, in this case, the input means that the z buffer should be assumed to have expired (for example, the input may indicate that the crouch button has been pressed, which may affect foreground occlusion, or the jump button, which changes the viewpoint position by a considerable amount).

[0082] Therefore, in this case, only the geometry should be recycled, and a new image 320' should be rendered from that point. Thus, only the geometry generation time is saved in this case (e.g., 4ms processing time), so assuming an overhead of +2ms, the new image is output in 24ms + 14ms = 38ms.

[0083] Therefore, while time can be saved by partially reusing rendering pipelines of previously discarded images, the amount of time saved can vary between instances based on a combination of factors related to which stages of the pipeline have been executed, what they include, and what the inputs received (or their impact on the game state).

[0084] Therefore, for the fourth approximation, the rendering pipeline can be adjusted to improve the performance obtained from the third approximation.

[0085] As mentioned earlier, for the third approximation, when generating a discarded frame, only those stages of the generation process that will change in response to the received input are discarded.

[0086] In the fourth approximation, "response variation" may mean a response variation threshold or assumed amount, and the rendering pipeline can be restructured to achieve the amount that occurs later in the process of generating frames so that more rendering pipelines can usually be reused.

[0087] It should be understood that some elements of a scene are more likely to change or change more in response to input than others—at least within a 16ms window.

[0088] Specifically, environmental elements farther from the user's viewpoint are less likely to need to be changed. Due to parallax, viewpoint changes caused by player movement result in a smaller change in rendered position as distance increases. Optionally, a threshold level of change can be selected, such as less than one pixel, or less than two, three, or four pixels, etc. For a given allowed movement speed of the user, this can then be used to calculate the effective distance from the viewpoint, beyond which any change in viewpoint within a frame will be less than the threshold level. This distance can then represent the beginning of the "background" region of the scene.

[0089] Alternatively, it is well known that in games, scenes are typically rendered using corresponding levels of detail (LoD), which are usually functions of the distance from the virtual camera's viewpoint, with LoD 0 being the most detailed and detail decreasing at LoD 1, LoD 2, and so on.

[0090] Therefore, as an alternative or supplement, the distance to the virtual camera viewpoint can be selected based on the process used to select each LoD, which represents the start of the scene's "background" area.

[0091] Therefore, optionally, in order to increase the chance that a portion of the rendering pipeline is re-looped when a frame interruption occurs upon receiving new input data, the pipeline may render as much of the environment in the background as possible, wherein the “background” is determined to be one of the following: a distance beyond which the inter-frame positional change may be less than a threshold amount of N pixels, where N is, for example, 1, ..., 10; a distance using a selected lower LoD (e.g., LoD 1, LoD 2, LoD 3, LoD 4, etc.), or a distance determined empirically as a good trade-off between visual fidelity and reduced latency.

[0092] Therefore, for example, the geometry of part or all of the scene can be calculated, and z-culling can often be performed as well, but textures and other processes can then be applied first to the "background" portion of the scene before applying them to the foreground portion. In this way, if the rendering pipeline for a frame is interrupted (e.g., in the first half of the frame cycle), it's likely that most of the rendering up to that point can be reused. If the game state updates the position of some foreground objects, then it may be necessary to repeat the foreground z-culling. For example, the most frequently updated parts of the scene are often the user's own virtual avatar and / or the non-player characters they are interacting with—both of which tend to be in the foreground.

[0093] Therefore, in Figure 6C In, it is shown Figure 6B In this scenario, the rendering pipeline has been configured to render the background first. As a result, when a new input is received at 24ms, all rendering up to that point can be reused with the updated game state, which (for example) only causes the avatar's pose to change to change weapons, while also moving forward.

[0094] Compare Figure 6C and Figure 4B (Where late-arriving inputs wait for the next frame), clearly, a well-scheduled rendering pipeline can avoid discarding part or all of the current frame's rendering at most of the timings when new inputs might arrive—either because the background is rendered first and unaffected by the change—or because the effect is below a threshold—or because the inputs arrive late enough that the delay in reflecting the new inputs does not exceed the delay imposed by reflecting the current inputs in the rendered output. As mentioned above, this balance point is roughly halfway through the generation of the current frame, but can be chosen to be later.

[0095] Therefore, in the fourth approximation, rendering at least part of the background portion of the scene first can reduce the amount of rendering that needs to be discarded when switching to a new frame in response to received input. The background is defined in one of several ways, but is usually defined as the portion of the scene that is unaffected or unaffected by changes in the game state caused by input within the frame cycle.

[0096] Finally, alternatively or additionally, a fifth approximation can be considered. As mentioned above, parts of the scene may not be significantly affected by changes in the game state due to inputs within a frame period. With the fifth approximation, it should be understood that the amount of change in the visible game state will differ for different inputs. This has already been mentioned in part elsewhere in this paper, for example, referring to crouching versus moving forward, but it can also be generalized based on the nature of the input assignment and / or the physics controller inputs themselves.

[0097] Now for reference Figure 7The illustration shows examples of a handheld controller 130 viewed from three different viewpoints. Such a controller typically has two handle portions 131L, 131R and a central body 131C. Various controls are distributed across the controller, typically in local groups. Examples include a left button group 132L that may include directional controls, and similarly a right button group 132R that may include function controls. The controller also includes a left joystick 134L and / or a right joystick 134R, and shoulder buttons 136L, 136R, which may optionally also be operated as buttons by pressing them. These may then include shoulder buttons 136S and trigger buttons 136T, providing a total of four buttons. Other buttons (not shown), such as buttons or handles on the rear body of the controller, may be included.

[0098] The controller (typically located in the center of the device) may also include one or more system buttons 137, which typically trigger interaction with the entertainment device's operating system rather than with the game or other application currently running on it; such buttons may bring up a system menu or allow recording or sharing of displayed content. Additionally, the controller may include one or more other elements, such as a touchpad 138, a light (not shown) for optical tracking, a screen (not shown), haptic feedback elements (not shown), motion detection (139), etc.

[0099] For system button 137 that is not related to gameplay, optionally, the system does not need to abandon the current frame at all, and the system can wait until the next frame to generate the system interface image.

[0100] For the functional controls 132R, these may not affect the virtual camera viewpoint (which could be a game-related aspect) or not all of them may affect the virtual camera viewpoint. Therefore, when one of these is pressed, the system may, for example, discard any foreground processing (according to the fourth approximation) or more generally any affected processing (according to the second approximation), unless input is received late in frame generation (according to the second approximation). Therefore, more generally, the system may discard only or partially discard frames for reuse in specific controls within these controls, without discarding frames for other controls, depending on the functions assigned to those controls. Thus, some functional controls may be intentionally excluded from the frame discarding scheme.

[0101] It should also be understood that, particularly for function controls and other "binary" (active / inactive) controls, changes in input state are correlated with changes in game state. Therefore, for example, it is unusual for a user to simply press the "jump" button for 16 milliseconds—but once the jump is triggered, continuing to press the button for the duration of multiple subsequent frames will not further alter the game behavior and can therefore be disregarded. Understandably, for some games, there are inputs corresponding to long presses or short presses, but the transition from a short press to a long press can be recognized as the start of a new (type) of input.

[0102] Meanwhile, for the direction control 132L, as previously described, forward motion can be adapted relatively easily by scaling the resulting image frame by an amount proportional to the distance moved forward during the frame or optionally by the amount reached by the input to the represented portion.

[0103] As mentioned before, leftward or rightward movement (translation) can cause the virtual camera position to move sufficiently within a frame period to alter some or all of the rendering pipeline. As previously stated, this may require flushing the entire pipeline, or it may allow one or more of stages 510-540 to be retained. If the pipeline is configured to process background elements first, more pipeline work can be reserved for those elements because the change to the background due to movement is below a predetermined acceptable threshold.

[0104] Therefore, for “binary” motion input (i.e., motion that is on or off), the frame dropping scheme may depend on the direction of travel, the travel rate, whether the input is a redundant continuation of the input, and whether the rendering is arranged at least partially according to distance so that the foreground is rendered at least partially later than the background.

[0105] In addition to binary inputs such as simple buttons, the controller may also include analog or continuous inputs. Such inputs may include joysticks (134), triggers (136T), touchpads (138), mice, and motion detectors.

[0106] Typically, these inputs provide activation or location values ​​that vary over time. Therefore, for any approximation described, the frame dropping scheme in this paper may also optionally apply an incremental threshold to these analog / continuous inputs.

[0107] In other words, if a value is received from one of these inputs (e.g., as part of input data [A], [B], or [C] mentioned elsewhere in this document), it is compared with the corresponding previously received input value. If the difference (increment) in the value is less than a threshold amount, it means that the user is still doing the same thing, or that the input has not been changed enough to be noticeable within the current frame period, so the input can wait until the next frame to be used.

[0108] Roughly speaking, a threshold can serve as a test to determine whether an input is a continuation of a previous input or should be considered a new input within the current frame's timeframe. Therefore, for example, if the value changes significantly enough to have a visually noticeable effect, it can be manipulated to interrupt the current frame according to the scheme presented in this paper. It should be understood that the threshold can be different for each type of input.

[0109] This test can also be used, for example, in relation to a second approximation of the timing thresholding method. Thus, for instance, if the last input is visually significant but the new input is not, the timing threshold can be adjusted to favor reducing the delay in representing the last input (i.e., continuing the current frame even if the new input is relatively early). In contrast, if the last input is visually insignificant but the new input is visually significant, the opposite weighting can be considered (i.e., interrupting the current frame even if the new input is relatively late).

[0110] Therefore, in summary, in the embodiments described in this specification, some inputs from the controller, such as system buttons and some function buttons, can be ignored in order to determine whether to interrupt the generation of the current frame. Meanwhile, some inputs from the controller can, in whole or in part, cause parts of the rendering pipeline to be discarded, depending on how they affect the virtual camera position.

[0111] Individually, the above considerations can also be applied to analog inputs, but alternatively or additionally, these may be subject to an incremental threshold that indicates whether the user has changed their analog / continuous input by more than a threshold amount.

[0112] And in all these cases, optionally, these interruptions may be subject to global or input-specific coverage rules, such as timing thresholds for second approximations.

[0113] As noted elsewhere in this document, the interruption of frames before the generation of updated images, and for some approximations, the complete or partial discarding of existing rendering pipelines, can optionally depend on factors such as the functionality of a given input button or other input, and optionally on in-game factors such as the effective speed of the virtual camera position during movement. Therefore, optionally, a profile can be associated with the game title and bundled with the title or downloaded separately, for example, by the OS or an auxiliary application, to provide this information to the frame-dropping scheme.

[0114] Similarly, depending on the implementation of the frame dropping scheme, such a configuration file may also include information indicating which variables or memory values ​​to change in order to trigger frame interruptions in the game based on any approximation described herein, or specifying API values ​​where the game can communicate with the operating system or auxiliary applications for this purpose.

[0115] In this way, the operating system or its auxiliary applications can reduce the overall latency of cloud streaming games, which is caused by the irrelevant time between frame generation by the cloud server and input sent from the client device over the network. It should be understood that this can be applied to any game (or indeed, any streaming application or interactive content), thereby improving the general effective responsiveness of cloud streaming servers.

[0116] Summarize

[0117] Now for reference Figure 8 In the overview embodiments of this specification, the streaming method for interactive content includes the following steps.

[0118] In the first step s810, as described elsewhere in this document, the generation of the first frame of content reflecting the current state of the application (e.g., a game) begins, and this generation includes a multi-stage process (e.g., stages of the rendering pipeline).

[0119] In the second step s820, as described elsewhere in this document, during the generation of the first frame, input data affecting the state of the application is received from the client device via the network.

[0120] In the third step s830, as described elsewhere in this document, progress is evaluated against the first standard. It should be understood that, depending on the standard and / or how it is developed for the evaluation, the following steps may respond to whether the standard is met or not.

[0121] In a further step s840, and then as described elsewhere in this document, in response to the evaluation of the first criterion (e.g., triggering the following steps to interrupt and wholly or partially replace the evaluation of the first frame), the following sub-steps occur.

[0122] In substep s842, as described elsewhere in this document, the generation of the first frame of the content is stopped.

[0123] In substep s844, as described elsewhere in this document, the current state of the application is updated in response to the input data.

[0124] And in substep s846, as described elsewhere in this document, a second frame is generated that reflects the updated state of the application.

[0125] Then, in step s850, as described elsewhere in this document, the generated second frame is output instead of the stopped first frame (e.g., for encoding and transmission / streaming to the client device).

[0126] It will be apparent to those skilled in the art that variations of the above-described methods corresponding to the operation of the various embodiments of the apparatus described and claimed herein are considered to be within the scope of the invention, including but not limited to:

[0127] The first criterion is whether the generation of the first frame has reached a predetermined stage in the multi-stage process, as described elsewhere in this document. It should be understood that a predetermined stage for all intents and purposes may mean that the first frame is complete and therefore the generation process does not stop, while another predetermined stage may mean that some parts of the multi-stage process can be reused, and yet another predetermined stage may mean that the process for the first frame is completely discarded.

[0128] The first criterion is whether the generation of the first frame has taken a predetermined threshold amount of time, and / or whether it has a completion time less than the threshold amount of time, as described elsewhere in this document. Such timing criteria can be used to determine a trade-off between delaying the graphical representation of newly received input and delaying the graphical representation of previously received input.

[0129] - Stopping the generation of the first frame of content involves the following steps: determining what state change the input data caused to the application (e.g., whether the virtual camera position changed, etc.), discarding the generation results of those stages affected by the state change in the multi-stage process, and at least partially reusing the generation results of those stages unaffected by the state change when generating the second frame, as described elsewhere in this document.

[0130] - In this case, alternatively, whether a stage of a multi-stage process is affected by state changes is subject to an impact threshold (i.e., the tolerable degree of change), as described elsewhere in this document.

[0131] The generation of the first frame of content involves arranging the generation of two or more different parts of the content in the first frame according to the possibility that two or more different parts of the content can be reused in the second frame, as described elsewhere in this document.

[0132] - In this case, optionally, two or more distinct parts of the content may be separated based on one or more of the following lists: distance from the virtual camera viewpoint, level of detail attributable to a part of the content, whether a part of the content corresponds to a currently animated element of the content, and whether a part of the content corresponds to a user character or non-player character with which the user is currently interacting, as described elsewhere in this document.

[0133] The method includes the following steps: evaluating the input data against a second criterion, and generating a second frame in response to the evaluation against the first and second criteria, as described elsewhere in this document. Again, it should be understood that, depending on the criteria and / or how they are developed for the evaluation, the response may be either compliant or non-compliant with the criteria.

[0134] - In this case, optionally, the second criterion includes whether a valid input is expected to cause a change in the current state of the application, thereby causing the generation of the first frame to stop, and inputs that are not expected to cause such a change include one or more selected from the following list: inputs that invoke a user interface that is not part of the currently displayed content, inputs that correspond to functions that do not affect the position or orientation of the virtual camera used to define the viewpoint of the first frame, and inputs that are a continuation of previous inputs, as described elsewhere in this document.

[0135] Similarly, in this case, optionally, the second criterion includes whether a valid input from an analog or continuous input mechanism has changed from a value associated with a previously generated content frame by a corresponding threshold amount, and if so, stopping the generation of the first frame, as described elsewhere in this document.

[0136] - The method includes the step of accessing a configuration file that provides one or more of the following: a first standard definition related to the multi-stage process, a second standard definition for input to the application, data indicating which changes should be made to runtime application values ​​to initiate the stop generation of the first frame, and data indicating which values ​​should be provided to the application interface to trigger the stop generation of the first frame, as described elsewhere in this document.

[0137] It should be understood that the above methods can be executed via software instructions or via hardware that is appropriately adapted by including or replacing dedicated hardware.

[0138] Therefore, the required adaptation to the existing portion of the equivalent device can be implemented in the form of a computer program product comprising processor-executable instructions stored on a non-transitory machine-readable medium (such as a floppy disk, optical disk, hard disk, solid-state drive, PROM, RAM, flash memory, or any combination of these or other storage media), or implemented in hardware as an ASIC (Application-Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array), or other configurable circuitry suitable for adapting to a conventional equivalent device. Separately, such a computer program can be transmitted via data signals over a network (such as Ethernet, wireless networks, the Internet, or any combination of these or other networks).

[0139] Therefore, refer to again Figure 1 and Figure 2 A streaming server for interactive content (e.g., an entertainment device 10 or the like operating as part of a server 200) includes the following.

[0140] As described elsewhere in this document, the streaming processor (e.g., CPU 20) is configured (e.g., via suitable software instructions) to implement a streaming method including the following steps:

[0141] - Start (s810) to generate the first frame of content reflecting the current state of the application, which includes a multi-stage process;

[0142] - During the generation of the first frame, input data affecting the state of the application is received from the client device via the network (s820);

[0143] - The schedule generated for the first standard assessment (s830);

[0144] - In response to the (s840) assessment failing to meet the first criterion,

[0145] - Stop the generation of the first frame of the (s842) content;

[0146] - Update the current state of the application (s844) in response to input data; and

[0147] - Generate (s846) a second frame containing content reflecting the application's updated state; and

[0148] - Output the second frame generated by (s850) instead of the first frame that was stopped.

[0149] Within the scope of this application, examples of the outlined embodiments of the methods and techniques described herein (e.g., by using suitable software instructions) are contemplated, including but not limited to the implementation of streaming methods including the step of evaluating input data according to a second criterion, as described elsewhere herein.

[0150] Finally, in the overview embodiments, streaming systems (e.g., Figure 2 The system shown herein includes a server (200) as described elsewhere herein, and one or more client devices (10, 140) as described elsewhere herein. Optionally, if separate from the client devices, the system may also include one or more controllers.

[0151] The foregoing discussion has only disclosed and described exemplary embodiments of the invention. As those skilled in the art will understand, the invention may be practiced in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the disclosure of this invention is intended to be illustrative and not to limit the scope of the invention and the other claims. This disclosure, including any readily identifiable variations of the teachings herein, partially defines the scope of the foregoing claims, so that no inventive subject matter is exclusive to the public.

Claims

1. A method for streaming interactive content, comprising the following steps: The generation of the first frame, which reflects the current state of the application, begins and includes a multi-stage process. During the generation of the first frame, input data affecting the state of the application is received from the client device via the network; The progress of the generated data is assessed against the first criterion. In response to the evaluation of the first standard, Stop generating the first frame of the content; The current state of the application is updated in response to the input data; and Generate a second frame containing content reflecting the updated status of the application; as well as Output the generated second frame instead of the stopped first frame.

2. The method according to claim 1, wherein, The first criterion is whether the generation of the first frame has reached a predetermined stage of the multi-stage process.

3. The method according to claim 1 or claim 2, wherein, The first criterion is whether the generation of the first frame has one or more items selected from a list of the following: i. The amount of time occupied within a predetermined threshold; and ii. Less than the threshold time required to complete.

4. The method according to any one of the preceding claims, wherein, Stopping the generation of the first frame of the content includes: Determine what state change in the application is caused by the input data; Discard the generation results of those stages in the multi-stage process that are affected by the state change; and When the second frame is generated, the generation results of those stages that were not affected by the state change are reused at least in part.

5. The method according to claim 4, wherein, Whether a stage of the multi-stage process is affected by the state change is subject to an influence threshold.

6. The method according to any one of the preceding claims, wherein, The generation of the first frame of the content includes: scheduling the generation of the two or more different parts of the content in the first frame based on the possibility that two or more different parts of the content can be reused in the second frame.

7. The method according to claim 6, wherein, The two or more distinct portions of the content are separated according to one or more selections from a list consisting of: i. Distance from the virtual camera viewpoint; ii. The level of detail to which the content is part; iii. Whether the portion of the content corresponds to the current animated element of the content; and iv. Whether the portion of the content corresponds to a user role or a non-player role that the user is currently interacting with.

8. The method according to any one of the preceding claims, comprising the following steps: The input data is evaluated against the second criterion; and In response to the evaluation of the first standard and the second standard, the second frame is generated.

9. The method according to claim 8, wherein: The second criterion includes: whether valid input is expected to cause a change in the application's current state, and that change should result in the cessation of the generation of the first frame; and Inputs that are not expected to cause this change include selections from one or more of the following: i. Invoke input from a user interface that is not part of the currently displayed content; ii. Inputs corresponding to a function that does not affect the position or orientation of the virtual camera used to define the viewpoint of the first frame; and iii. Inputs that are a continuation of the previous input.

10. The method according to claim 8 or claim 9, wherein: The second criterion includes whether valid input from an analog or continuous input mechanism has changed by more than a corresponding threshold amount from a value associated with a previously generated content frame, and if so, then Stop generating the first frame.

11. The method according to any one of the preceding claims, comprising the following steps: Access the configuration file, which provides selections from one or more of the following: i. A first standard definition related to the multi-stage process; ii. A second standard definition for the input to the application; iii. Indicate what changes should be made to the runtime application value to determine the data that stops the generation of the first frame; and iv. Indicate what value to provide to the application interface to trigger the cessation of data generation for the first frame.

12. A computer program comprising computer-executable instructions adapted to cause a computer system to perform the method according to any one of the preceding claims.

13. A streaming server for interactive content, comprising: A streaming processor is configured to implement a streaming method including the following steps: The generation of the first frame, which reflects the current state of the application, begins and includes a multi-stage process. During the generation of the first frame, input data affecting the state of the application is received from the client device via the network; The progress of the generated data is assessed against the first criterion. In response to the assessment not meeting the first criterion, Stop generating the first frame of the content; The current state of the application is updated in response to the input data; and Generate a second frame containing content reflecting the updated status of the application; as well as Output the generated second frame instead of the stopped first frame.

14. The streaming server according to claim 13, wherein, The streaming method includes the step of evaluating the input data against a second standard.

15. A streaming system, comprising: The server according to claim 13 or claim 14; and The client device.

Images