DYNAMIC ALLOCATION OF COMPUTING RESOURCES TO GENERATE HIGHLIGHTS IN CLOUD GAMING SYSTEMS

Abstract

Procedure, comprehensive: Playing back a game instance on a streaming device (104); Encoding the game instance into a game stream on the streaming device (104); Transmitting the game stream to a client device (102) for display on the client device; Receiving evidence to capture at least a portion of the game stream as a recording; Analyzing system information corresponding to the client device (102) to determine the allocation of recording tasks between the client device (102) and the streaming device (104); Selection based at least partially on the allocation of recording tasks, with Transmitting a character to the client device (102) to cause the client device to generate the recording, wherein: The transmission of the character to the client device (102) to cause the client device to generate the recording is selected to be performed when system information indicates that generating the recording on the client device would at least reach a performance threshold; and Creating the recording on the streaming device (104) is selected to perform when system information indicates that creating the recording on the client device would not at least meet the performance threshold.

Description

BACKGROUND

[0001] During a game session, a user may wish to record a screenshot, a highlight, or a segment of at least part of the session—for example, to share the recorded material with others, review it later, or otherwise access the recording. Traditional systems and methods have allowed players to record game sessions or portions thereof locally on their client device. However, to play back individual frames of the game session locally (e.g., on a client device), the client device must possess a significant amount of computing resources. Performing functions such as locally generating recorded material can require additional hardware and network computing resources beyond those needed for local playback of the game session's individual frames.

[0002] In other traditional systems and methods, players can stream gameplay sessions from their client device once a recording of the gameplay has been created (e.g., Streamview). With Streamview, each (or a specific number of) frames of gameplay can be copied from a client device display and then uploaded to a (usually) separate service or application for distribution to a Streamview audience (e.g., on social media). When a user views their gameplay session as a stream, each frame must be copied on the client device and sent to a third party for distribution—requiring significant bandwidth and processing power. Therefore, similar to the above, these capabilities may not be available to client devices lacking sufficient memory capacity, GPU capability, or other hardware functionality to effectively generate, store, and / or share highlights.Consequently, client devices with limited hardware or bandwidth capabilities may be unable to create and / or share snapshots, highlights, or recordings of gameplay, thus impacting the user experience. In this area, CN 1 08 600 669 A and US 9 473 758 B1 are also known. US 2016 / 0 361 646 A1 describes a method for saving a game along with commentary made by spectators. SUMMARY

[0003] This disclosure relates to the allocation of computing resources for generating highlights in cloud gaming systems. Systems and methods according to the independent claims are disclosed that distribute processing, including generating and overlaying user interfaces, analyzing game streams for meaningful events, generating highlights, storing highlights, and sharing highlights between and among different devices. As used here, a highlight can comprise any capture of a game session stream, including a snapshot or screenshot of gameplay, a recording of an entire game session, or a recording of a portion (e.g., a segment) of a game session—such as a recording of a specific action or event within a game session.

[0004] In contrast to conventional systems, such as those described previously, the system and methods of this disclosure analyze the hardware, software, and / or network functionality of client devices to determine how processing tasks within the system (e.g., between client device(s), one or more streaming servers, one or more secondary systems, etc.) are to be assigned. For example, a proxy—responsible for various tasks within the climax generation system—can be flexibly moved between a client device, a streaming device, and / or a secondary system (e.g., a remote system to which processing tasks are offloaded). The proxy can record the game stream (e.g.,in a storage device, such as a buffer), generate and play back user interface elements related to highlight generation, analyze the game stream for meaningful events (e.g., events or actions within the game that trigger highlight generation), respond to highlight generation commands (e.g., from the client device, the streaming device, the secondary system, or a combination thereof), and display highlight generation and storage.

[0005] Because the proxy can be responsible for a variety of tasks within the system, different proxy processes can be moved between the various devices in the system. In some examples, the streaming device (or another device in the system) can query the client device to determine hardware, software, and / or bandwidth capabilities (referred to here as system information). As a result of determining this system information, the various tasks of the proxy can be moved—dynamically in some implementations—between the devices in the system to improve the cloud gaming experience for the end user.As a non-restrictive example, each of the devices in the system can benefit from improved bandwidth utilization, higher processing speeds, lower latency, and higher resolution of the game stream by performing some processing tasks on the client device (e.g., generating and storing highlights) and some processing tasks on the streaming device and / or secondary device (e.g., analyzing the game stream for meaningful events). BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The present systems and procedures for allocating computing resources to generate highlights in cloud gaming systems are described in detail below with reference to the attached drawings, whereby Fig. Figure 1A is an example system diagram for generating highlights in cloud gaming systems, in accordance with some embodiments of the present disclosure; Fig. Figure 1B is an example system diagram for generating highlights in cloud gaming systems with a client device and a streaming device, in accordance with some embodiments of the present disclosure; Fig. 1C is another example system diagram for generating highlights in cloud gaming systems, in accordance with some embodiments of the present disclosure; Fig. Figure 2 is an example screen image from a display of a game session with a graphic overlay for generating highlights, in accordance with some embodiments of the present disclosure; Fig. Figures 3-4 are flowcharts showing methods for generating highlights in cloud gaming systems, in accordance with some embodiments of the present disclosure; and Fig. Figure 5 is a block diagram of an example computer environment suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0007] Systems and methods related to the allocation of computing resources for generating highlights in cloud gaming systems are disclosed. As described herein, and in contrast to conventional systems, this disclosure, in an effort to efficiently distribute computing resources, network utilization, and storage requirements within the system, involves distributing one or more tasks of a highlight generation system between and among different devices. As a result, one or more streaming devices (e.g., a server responsible for playback, encoding, and streaming of game sessions of one or more users), one or more client devices (e.g., a tablet, computer, smartphone, streaming device, game console, or other device that displays the game stream and receives and sends input), and / or one or more secondary devices (e.g., a server, a game console, or a game console) can be distributed among different devices.B, a server responsible for one or more overflow tasks) for task execution within the system in a manner that enables a high-quality, low-latency game stream for each of the system's users.

[0008] In some examples, the tasks associated with climax generation within a cloud gaming environment can be described as the tasks of a proxy. Therefore, the proxy can be distributed between and among the various devices of the climax generation system. In some examples, the proxy can dynamically move tasks between the different devices based on the system information of each client device. This system information can include storage capacity, encoding capability, processing capability, game stream recording settings, network information, and / or other system information. The system information can relate to the streaming device(s), the client device(s), and / or the secondary device(s).In contrast to conventional systems where the client device(s) is / are tasked with playing the game, generating highlights, storing highlights, sharing highlights, and performing each task related to highlight generation, the highlight generation system of this disclosure takes into account the limitations of the client device(s). For example, one or more first client devices may have greater storage and / or processing capabilities than one or more second client devices, so the proxy representative tasks distributed across the different devices may be different (e.g., adapted) for the first client device(s) than an adaptation used for the second client device(s). As a non-limiting example, the user interface elements (e.g., an overlay) for controlling the generation of a highlight or a recording (e.g.,(based on recording settings), which is performed by the streaming device(s), on which the client device(s) are generated. By generating the user interface elements on the client device(s) within a client application, the streaming device can require less bandwidth and processing resources, and the image quality of the user interface elements in the overlay can be improved. As another, non-restrictive example, where the client device(s) have sufficient memory capacity and / or processing capabilities, the generation and storage of game stream highlights can be performed by the client device(s) – thus offloading the memory and processing requirements from the streaming device(s) to the client device(s) to enable the streaming device(s) to maintain a game stream with high image quality and low latency.Thus, in comparison to conventional systems where a single device is responsible for generating highlights, and in an effort to reduce the individual load on each device, the storage, processing, computation and bandwidth loads of the systems can be distributed by - in some embodiments dynamically - shifting the tasks of the proxy representative between and among the streaming device(s), the client device(s) and / or the secondary device(s).

[0009] As described above, conventional systems do not work in game streaming environments. Even when a game recording is streamed (e.g., in Streamview), the original game playback and the recording both run on the client device(s). Therefore, in a game streaming environment, additional limitations arise because the streaming device(s) is responsible for receiving input from the client device(s), playing back the game in response to that input, encoding the game stream, and transmitting the game stream to the client device(s).For example, game streaming often experiences bandwidth issues when high-quality, low-latency game content is streamed over the internet, while bandwidth is also required to send input back from the client device(s) to the streaming device(s).

[0010] Beyond these inherent limitations of the game streaming system, introducing a highlight generation system within the game streaming environment leads to further restrictions. For example, if a recorded highlight is generated and stored on the streaming device(s), the file size of the highlight may become prohibitively large and / or require the use of additional bandwidth to transfer or stream the highlight back to the client device(s) for display (e.g., during or after gameplay). Furthermore, the processing capacity of the streaming device(s) may be affected (reduced), as the streaming device(s) is responsible for managing and streaming game sessions to multiple end users in addition to generating and storing highlights for one or more end users.A reduction in the processing capacity(s) of the streaming device(s) can cause one or more end users to experience lower-quality game streams, higher latency, and / or other stream quality issues while playing. Therefore, by shifting one or more tasks of the proxy representative of the climax generation system between and among different devices within the game streaming environment, the system's limitations can be addressed in such a way that end users are able to experience game streams with high image quality and low latency, along with the associated seamless climax generation functionality.

[0011] Now on Fig. 1A referring is Fig. Figure 1A shows an example system diagram of a peak generation system 100 for a cloud gaming environment in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described here are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described here are functional units that can be implemented as discrete or distributed components, or in conjunction with other components, and in any suitable combination and arrangement. Various functions described here as executable by units can be performed by hardware, firmware, and / or software.For example, different functions can be performed by a processor that executes instructions stored in memory.

[0012] The peak generation system 100 may include, among other things, one or more client devices 102, one or more streaming devices 104, one or more secondary devices 106, and / or one or more network devices 108. The peak generation system 100 (and its components and / or features) may be operated using one or more computing devices, such as the computing device 500 of the Fig. 5, which is described in more detail here, will be implemented.

[0013] The devices or components of the peak generation system 100 can each communicate with and among themselves via the network(s) 108. The network(s) 108 can include a wide area network (WAN) (e.g., the Internet, a public switched telephone network (PSTN), etc.), a local area network (LAN) (e.g., Wi-Fi, Ethernet, etc.), and / or another type of network.

[0014] The client device(s) 102 may be a smartphone, a laptop, a tablet computer, a desktop computer, a clothing-worn device, a game console, a virtual reality system (e.g., a headset, a computer, a game console, remote control(s), controller(s), and / or other components), a streaming device (e.g., an NVIDIA SHIELD), a smart home device, which may include an intelligent personal assistant, and / or any other type of device that supports at least a display of a game stream from game sessions 126 and / or inputs for the game sessions 126 from input device(s) 112.

[0015] The client device(s) 102 may include a display 110, one or more input devices 112, a client application 114, one or more data stores 116, and / or at least part of the functionality of a proxy 118. In some examples, at least part of the functionality of the proxy 118 may be executed within the client application 114. Although certain components and / or features of the client device(s) 102 in Fig. The fact that the number of components mentioned in point 1 is not intended to be a limitation is not a restriction. For example, the client device(s) may have 102 additional or alternative components, such as, without limitation, the computing device 500 mentioned here. Fig. 5 described. The features and functionality of the client device(s) 102 are described herein with regard to Fig. 1B and Fig. 1C will be described in more detail.

[0016] The client application 114 can be a mobile application, a computer application, a console application, a game application, and / or another type of application. The client application 114 can include instructions that, when executed by one or more processors of the client device(s) 102, cause the processor(s) to perform one or more operations (such as, but not limited to, the operations described herein in relation to the climax generation system 100). The client application 114 can act as an intermediary to enable the playing of a game instance.For example, the client application 114 can, without restriction, display a game stream received from the streaming device(s) 104, receive and / or process inputs from one or more of the input devices 112 of the client device(s) 102, and / or initiate the transmission of input data representative of the inputs to the streaming device(s) 104. In some examples, as described here, the client application 114 can include some of the features or functions of the proxy 118, such as generating highlights and displaying a graphic overlay 128. Fig. 2) for generating highlights, recordings, or snapshots, for programming recording settings, etc. Additionally, in some examples, the client application 114 can initiate the generation of highlights, the storage of highlights, the analysis of the game stream to determine when highlights should be generated, the sharing of highlights, etc. In some embodiments, the client device(s) 102 can comprise any number of client applications 114, in which features and functionality of the highlight generation system 100 are distributed.

[0017] The client device(s) 102 may include one or more components (e.g., a communication component, network interface, etc.) and features for communication across one or more networks, such as the network(s) 108. As a non-limiting example, the client device(s) 102 may, in order to communicate within the peak generation system 100, use a wired Ethernet connection and / or a Wi-Fi connection via a router to access the internet in order to communicate with the streaming device(s) 104, the secondary device(s) 106, and / or other client devices 102.

[0018] The display 110 can include any type of display capable of showing the game (e.g., a light-emitting diode (LED) display, an organic LED display (OLED) display, a liquid crystal display (LCD), an active-matrix OLED display (AMOLED), a quantum dot display (QDD), a plasma display, and / or another type of display). In some examples, depending on the configuration of the client device(s) 102, the display 110 can include more than one display (e.g., a dual-monitor display for computer games, an initial display for setting up a game, and a virtual reality display for playing the game, etc.). If the display 110 is a touchscreen display, such as the touchscreen of a smartphone, tablet PC, laptop, and / or similar device, the display 110 can be used as at least one of the input devices 112 of the client device(s) 102 (e.g.,One of the input devices 112 for generating inputs for a game instance 132 for transmission to the streaming device(s) 104 for updating the game playback). The display 110 can show the game stream of one or more game sessions, such as a game session 126(A), a game session 126(B), and / or a game session 126(C) (hereafter collectively referred to as "game sessions 126"). The game sessions 126 can comprise any number of game sessions in which the user of the client device(s) 102 participates.

[0019] The input device(s) 112 may include any type of device capable of providing user input to the game. The input device(s) may include a keyboard, mouse, joystick, touchscreen display, controller, remote control(s), headset (e.g., sensors of a virtual reality headset), another type of input device, and / or a combination thereof.

[0020] The data storage device(s) 116 can comprise any type of storage device. In embodiments where the client device(s) 102 are capable of storing and / or generating highlights, the data storage device(s) 116 can store part or all of the game stream. For example, the data storage device(s) 116 can store the entire game stream, and snapshots, highlights, and / or recordings can be generated from the stored game stream. After generation, the snapshots, highlights, and / or recordings can be stored separately in the data storage device(s) 116, or reference points (e.g., timestamps) for the snapshots, highlights, and / or recordings (e.g., as metadata) can be stored so that the snapshots, highlights, and / or recordings can be copied or retrieved from the stored game stream using the reference point.

[0021] In some examples, datastore(s) 116 can only store a portion of the game stream. For example, datastore(s) 116 can only store snapshots, highlights, and / or recordings. A buffer 130 can be used to store a portion of the game stream, and the highlights, snapshots, or recordings can be copied from or retrieved from the buffer (and stored separately in datastore(s) 116). Buffer 130 can only store one portion of the game stream (e.g., the last thirty seconds of the game stream can be stored in buffer 130). Therefore, if a snapshot, highlight, or recording can be created, in some examples at least part of the game stream portion can be stored in buffer 130 in addition to a subsequent portion of the game stream (e.g.,The next ten seconds) are copied or retrieved to create the snapshot, peak, and / or recording. Buffer 130 may include a cyclic or looping buffer in some non-restrictive examples.

[0022] The streaming device(s) 104 may include a streaming application 120 for receiving input data representative of inputs to the client device(s) 102, for playing back a game instance 132, for encoding the game instance 132 into a stream, and for transmitting the stream to the client device(s) 102. In some examples, as described here, the streaming application 120 may include some of the features or functions of the proxy representative 118, such as generating highlights and encoding the graphic overlay 128 ( Fig. 2) into the game stream (e.g., if the client device(s) 102 are capable of limited processing). Additionally, in some examples, the streaming application 120 can initiate the generation of highlights, the storage of highlights, the analysis of the game stream to determine when highlights should be generated, the sharing of highlights, etc. In some embodiments, the streaming device(s) 104 can comprise any number of streaming applications 120 in which the features and functionality of the highlight generation system 100 are distributed. The features and functionality of the streaming device(s) 104 are described here with respect to Fig. 1B and Fig. 1C is described in more detail.

[0023] In some embodiments, the streaming device(s) 104 can play back a game instance 132 and transmit the game stream to two or more devices as a plurality of instances of the game. In these embodiments, the two or more instances of the game can be copies of the played-back game instance. In these embodiments, sending two instances of the game over the network(s) 108 can be advantageous in order to simultaneously display the game instance on client device(s) 102 and / or capture at least a portion of the game stream on secondary device(s) 106.

[0024] The streaming device(s) 104 may include one or more components (e.g., a communication component, a network interface, etc.) and features for communicating across one or more networks, such as the network(s) 108. As a non-limiting example, the streaming device(s) 104 may, in order to communicate within the peak generation system 100, send and receive data over the Internet to communicate with the secondary device(s) 106, the client device(s) 102, and / or another streaming device(s) 104.

[0025] The streaming device(s) 104 may include the data storage device(s) 116, similar to the data storage device(s) 116 of the client device(s) 102 and / or the secondary device(s) 106 described here. For example, depending on the distribution of tasks by the proxy representative 118, the streaming device(s) 104 may be tasked with generating, storing, and / or sharing highlights, snapshots, and / or recordings of the game stream.

[0026] The secondary device(s) 106 may include a game analytics application 124 for receiving a game stream from the streaming device(s) 104 and / or the client device(s) 102 (e.g., via the streaming device(s) 104) and for analyzing the game stream for events and / or actions in the game instance 132. The events and / or actions may be recordable events that, upon detection, cause the secondary device(s) 106, the client device(s) 102, and / or the streaming device(s) 104 to generate a climax, a snapshot, and / or a recording. Although depicted in a non-restrictive example as being performed by the game analytics application 124, the analysis of the game stream may also be included as a task performed by the proxy agent 118.Therefore, in some examples, the game analysis application 124 of the secondary device(s) 106 may include some of the features or functions of the proxy representative 118, such as game stream analysis, determining when to generate highlights, and / or instigating the generation of highlights (e.g., by generating and transmitting characters to the client device(s) 102 and / or the streaming device(s) 104 indicating that a highlight should be generated). The features and functionality of the secondary device(s) 106 are described here in relation to... Fig. 1B and Fig. 1C is described in more detail.

[0027] The secondary device(s) 106 may, in some embodiments, be third-party services or devices for recording and / or live-streaming the game stream. In some embodiments, the secondary device(s) 106 may be stream-viewing devices configured to receive at least a portion of the game stream and transmit it to a plurality of display devices. For example, in some embodiments, the secondary device(s) may receive the game stream from the streaming device(s) 104 and forward the game stream to a plurality of viewers (e.g., "live streaming").In some embodiments, similar to other embodiments disclosed herein, streaming to secondary device(s) 106, such as third-party services or devices, can reduce the processing resources and network resources for client device(s) 102 compared to conventional game streaming systems.

[0028] The game analytics application 124 can analyze the game stream to determine if a recording event has occurred and to instruct the proxy representative 118 characters to capture (or buffer) at least a portion of the game stream. Based on artificial intelligence, machine vision, text recognition, and / or other programs and analytical methods, the game analytics application 124 can determine that a recording event has occurred. The recording event can be any instance indicated by a game as a significant event. A significant event indicated by the game can be the elimination of another character in the game, the picking up of a specific item, the scoring of a goal, the hitting of a home run, the climbing of a tall building or mountain, the completion or fulfillment of a user-defined task or goal, and / or any other type of event.For example, in some embodiments, the game analytics application 124 can detect changes in a measurement scale of the game sessions 126 that indicate the elimination of a character in the game and provide the proxy agent 118 with characters to capture at least a portion of the game stream. In another example, the game analytics application 124 can (e.g., using optical character recognition (OCR)) detect text in the game instance 126 that means "Player 1 eliminated Player 4" or "Player 1 scored a touchdown" and provide the proxy agent 118 with characters to capture at least a portion of the game stream.

[0029] Additionally, although the game analysis application 124 is listed as a feature of the secondary device(s) 106, this is not intended to be a limitation. For example, in some embodiments, the secondary device(s) 106 may not be included in the climax generation system 100. Therefore, the game analysis application 124 and / or its features and functions can be executed on the client device(s) 102, the streaming device(s) 104, and / or a combination thereof. For example, the game stream analysis, similar to other features and functions of the proxy agent 118, can be performed by one or more other devices of the climax generation system 100.

[0030] The secondary device(s) 106 may include one or more components (e.g., a communication component, network interface, etc.) and features for communicating across one or more networks, such as the network(s) 108. As a non-limiting example, the secondary device(s) 104, in order to communicate within the climax-generating system 100, may send and receive data over the Internet to communicate with the streaming device(s) 104, the client device(s) 102, and / or other secondary device(s) 106.

[0031] The secondary device(s) 106 may include the data store(s) 116, similar to the data store(s) 116 of the client device(s) 102 and / or the streaming device(s) 104 described here. For example, depending on the distribution of tasks by the proxy representative 118, the secondary device(s) 106 may be tasked with generating, storing, and / or sharing highlights, snapshots, and / or recordings of the game stream.

[0032] Proxy 118 can encompass various functions for generating highlights within System 100. As described here, Proxy 118 can be used to overcome many of the limitations of other systems by shifting or allocating resources used by Client Device(s) 102, Secondary Device(s) 106, and / or Streaming Device(s) 104 within the highlight generation system 100 to perform the various highlight generation functions effectively and efficiently. Although Proxy 118 is a function of each of the Client Device(s) 102, Secondary Device(s) 106, and Streaming Device(s) 104 in Fig. The fact that the proxy is listed in section 1A is not intended to be a limitation. For example, the tasks of the proxy 118 can be performed on the client device(s) 102, the secondary device(s) 106, the streaming device(s) 104, and / or a combination thereof. As a non-restrictive example, and considering the functionality of the proxy 118 within the system 100, the client device(s) 102 can capture at least one segment of the game stream as a climax, and the streaming device(s) 104 can analyze the game stream using the game analytics application 124 to provide the client device(s) 102 with a clue (e.g., via a message) to capture the climax.In another non-restrictive example, the generation of the graphic overlay 128 can take place on the streaming device(s) 104, and the capture of at least a section of the game stream can take place on the client device(s) 102. In yet another non-restrictive example, the generation of the graphic overlay 128 can take place on the client device(s) 102, and the capture of at least a section of the game stream can take place on the streaming device(s) 104.

[0033] The proxy agent 118 can be configured to capture at least a portion of the game instance as a climax, which in some embodiments may be a screenshot, a short recording of a game stream, and / or a substantial (even complete) recording of gameplay. A determination of when to capture the instance may be based on an indication that at least a portion of the game stream is being captured. This indication may be based on analysis of the game stream using machine vision, artificial intelligence (e.g., inputting frames of the game session into a neural network trained to recognize events, actions, or other recording events in the game session), OCR, game state analysis, user input (e.g., in a graphical overlay 128), and / or other criteria.Proxy 118 can receive the game instance and store a portion of the game instance on datastore(s) 116. In some examples, proxy 118 can continuously store at least a portion of the game stream in buffer 130 of datastore(s) 116. In such an example, buffer 130 can store the most recent portion of the game instance, allowing highlights, snapshots, and / or recordings to be generated from buffer 130. As a non-restrictive example, the most recent duration (e.g., thirty seconds) of gameplay from a given game instance can be recorded in buffer 130, and when an indication of generating a highlight is received, the recording in buffer 130 can be moved or copied to datastore(s) 116 for permanent storage.

[0034] The proxy 118 can be configured to generate a graphical overlay 128 for setting recording settings, triggering highlight generation, setting user-defined recording events for highlight generation, and / or programming other recording functions. For example, the graphical overlay 128, as described in relation to Fig. 2 described, encompass a variety of functions, such as controlling aspects of capturing at least a portion of the game stream. Playing back the graphical overlay 128 may require computing resources and / or hardware that some client devices 102 lack. Therefore, in conventional systems, these client devices 102 may have been unable to generate highlights due to their inability to play back the graphical overlay 128. However, the proxy 118 can overcome these limitations by distributing the playback of the graphical overlay 128 between the client device(s) 102, the secondary device(s) 106, and / or the streaming device(s) 104. Therefore, the client device(s) 102 can generate the graphical overlay 128 (e.g.,within the client application 114), when it is determined that the client device(s) 102 have sufficient resources to generate the graphical overlay 128. In other embodiments, the streaming device(s) 104 and / or the secondary device(s) 106 can generate the graphical overlay 128 and include the graphical overlay in the game stream, for example, when it is determined that the client device(s) 102 do not have sufficient resources to generate the graphical overlay 128. In some embodiments, the graphical overlay 128 can include controls for capturing snippets of the game stream, filtering generated highlights, searching for generated highlights, runtime status, gallery selection, and / or keyboard shortcut management.

[0035] Graphic overlay 128 can be generated by an application separate from the application that renders the game (e.g., the game application). For example, where client device(s) 102 render(s) graphic overlay 128, a client application of client device(s) 102 (e.g., a client application performing the functions of proxy 118 on client device(s) 102) can render and display graphic overlay 128 on display 110 separately from the game stream. Although rendered separately, graphic overlay 128 can be displayed simultaneously with the game stream (e.g., on a section of the display, such as a left side, a top fold-out, a right side, a pop-up on the display, etc.).As another example, where the streaming device(s) 104 render(s) the graphical overlay 128, a streaming application of the streaming device(s) 104 (e.g., a streaming application performing the functions of the proxy representative 118 on the streaming device(s) 104) can render the graphical overlay 128 separately from the game and capture both the graphical overlay 128 and the game in the game stream (e.g., with the graphical overlay 128 superimposed on the game in the game stream). Because the graphical overlay 128 can be rendered separately, as part of an application other than the one that uses game state data to render the game, the graphical overlay 128 can be referred to here as a supplementary graphical overlay. Therefore, the graphical overlay 128 can be displayed to complement the game-based displayed information (e.g.,(the game itself and the game's graphical overlays and GUIs). Consequently, if a user makes an input in the graphical overlay 128, the input may be sent to the proxy representative 118 instead of being made available to the game itself to influence the game, regardless of whether it is included in the game stream by the streaming device(s) 104 and / or played back in a client application on the client device(s) 102.

[0036] The proxy 118 can be configured to transmit the game stream to additional devices and / or a third-party service for stream viewing. In some embodiments, the transmission of the game stream for stream viewing can occur simultaneously with the game stream or with a delay. The transmission of the game stream for stream viewing can occur separately from the recording of the game stream by the proxy 118. For example, the proxy 118 can record the game stream on the client device(s) 102 and transmit the game stream to additional devices and / or a third-party service for stream viewing. In this example, the streaming device(s) 104 can stream the game stream to both the client device(s) 102 and the secondary device(s) 106 over the network(s) 108.

[0037] Therefore, the proxy 118 and its associated functionality can be distributed between and among different devices within the system 100. However, since the game is streamed from the streaming device(s) 104, and the proxy 118 can be distributed across different devices within the system 100, implementing the recording functionality may be prohibitively expensive for game developers. For example, without using the proxy 118, game developers may need to provide different functionality depending on the distribution of computing resources within the system 100. This functionality may require generating different types of messages and / or making application programming interface (API) calls to different devices, components, and / or features, depending on the distribution of computing resources.To avoid or reduce this burden on game developers, game developers within System 100 should only need to program the generation of the same message type and / or the same API call type, regardless of the distribution of computing resources within System 100. Therefore, the proxy 118 can be configured to receive the same message type and / or API call type from the game application being played back on the streaming device(s) 104, and can determine and / or facilitate task assignment within System 100. For example, a user can input a keyboard shortcut that provides a clue to a request to generate a climax, and the game application can generate a message and / or API call that represents the clue to the request.The proxy 118 can receive the message and / or API call and determine which devices within System 100 should perform the various tasks for generating the climax. As a result, it may appear to the game developers or the game application that game generation and playback are performed on the local hardware of the client device(s) 102, even though the game is actually being generated and played back on the streaming device(s) 104. Therefore, by using the proxy 118, the implementation of System 100 can be essentially seamless for the game developer, thus reducing the implementation effort for game developers when integrating climax capture functionality into a game streaming environment.

[0038] Additionally, since the game is streamed to client device(s) 102 and the proxy 118 can be used to distribute computing resources based on system information, the requirement for client device(s) 102 to have specific hardware and / or software configurations—such as specific operating systems, GPU(s), etc.—can be eliminated. Therefore, the climax generation system 100 can be used for all client device(s) 102 regardless of their respective hardware and / or software configurations.

[0039] Now on Fig. 1B referring is Fig. 1B Another example system diagram for a climax generation system 140 in cloud gaming systems using client device(s) 102 and streaming device(s) 104, in accordance with some embodiments of the present disclosure. The climax generation system 140 may, among other things, include the client device(s) 102 (which have similar components, features, and / or functionality to the client device(s) 102 of the Fig. 1A may include) and the streaming device(s) 104 (which have similar components, features and / or functionality to the streaming device(s) 104 of the Fig. 1A may include) which are interconnected via network(s) 108 (which are part of network(s) 108 of the Fig. (They can be similar to 1A) communicate.

[0040] In the climax generation system 140, the streaming device(s) 104 can include the game instance 132. The game instance 132 is shown in bidirectional communication with and under a camera control application 144, a real-time streaming protocol (RTSP) application 142, and the climax capture application 164. The game instance 132 can be available to or transmitted to the camera control application 144 and / or the climax capture application 164. Upon generation of the game instance 132, the camera control application 144 and / or the climax capture application 164 can perform various functions—such as, but not limited to, those listed here—using data from the game instance 132.

[0041] The camera control application 144 can include in-game photography services. For example, the camera control application 144 can capture in-game footage from a variety of angles (e.g., 180 degrees, 360 degrees, etc.). The camera control application 144 can enable the capture of in-game footage from any position (e.g., even positions from a different perspective than the user's current perspective). The camera control application 144 (e.g., NVIDIA's Ansel) can capture screenshots during gameplay and apply one or more post-processing filters (e.g., to adjust color, saturation, add predefined filters such as sepia, retro, night mode, and / or change other properties) (e.g., with NVIDIA's Freestyle). Although separate from the highlight capture application 164 in Fig. As 1B illustrates, this is not intended to be a limitation. In some examples, at least some of the features and functions of the camera control application 144 can be performed by the peak detection application 164.

[0042] The highlight capture application 164 can include a highlight capture service on the streaming device(s) 104. The highlight capture application 164 can directly interpret the API requests of the game instance 132 to capture recordings of gameplay. In some embodiments, the highlight capture application 164 can be a non-streaming, single-system platform that directly interprets the API requests of the generated game to capture highlights on the streaming device(s) 104. In some embodiments, highlight capture on the streaming device(s) 104 can be performed by software (e.g., software representing hardware). In other embodiments, highlight capture on the streaming device(s) 104 can be performed by GPU hardware systems (such as NVIDIA's hardware-based SHADOWPLAY system).

[0043] The camera control application 144 can communicate with data storage device(s) 116, which may include buffer 130. The camera control application 144 can also communicate with buffer 130 if it is used. In some embodiments, captured game stream data can be transferred to data storage device(s) 116 for storage. Data storage device(s) 116 may include databases containing game information, graphic overlay information, game name information for highlighting key moments, and system information (e.g., for the client device(s) 102, such as whether certain features or functionality are supported).

[0044] The RTSP application 142 can receive the game instance 132 directly and / or via the camera control application 144 and / or the highlight capture application 164 and encode the game instance 132 into a game stream that is communicated over the network(s) 108. In some embodiments, the RTSP application 142 can act as a router for communication between the camera control application 144, the highlight capture application 164, the game instance 132, and the network(s) 108.

[0045] In some embodiments, the RTSP application 142 can communicate a captured game stream to the client device(s) 102 in order to display, share and / or store the captured game stream.

[0046] The climax generation system 140 may further include the client device(s) 102 with a Chromium Embedded Framework (CEF) client 146 (or another embedded framework browser client type). The CEF client 146 may communicate bidirectionally with the graphical overlay 148 and / or a display 110. The CEF client 146 may also communicate with a share server 166 and data storage 176, which may include a buffer 178. The CEF client 146 may receive the game stream from the RTSP application 142 over the network 108. The CEF client 146 may generate the climax capture graphical overlay 128 and / or perform one or more game streaming functions. The CEF client 146 can accept input from input device(s) 112 (shown in Fig. 1A) communicate with the streaming device(s) 104. In some embodiments, the CEF client 146 can receive input from the graphical overlay 128 and instructions can (in some embodiments) accordingly be sent to the ShareServer 166 and / or the streaming device(s) 104. The game stream received by the CEF client 146 can be transmitted to the display 110. The CEF client 146 can decode the game stream before transmitting it to the display 110 or act as a decoder for the game stream.

[0047] The CEF client 146 may also be associated with a highlight capture application 172. Like the highlight capture application 164, the highlight capture application 172 can use software and / or hardware systems to capture highlights on the client device(s) 102. In some embodiments, the game stream can be captured and stored in the data storage device(s) 176 and / or the buffer 180. The highlight capture application 172 can receive decoded game stream data from the streaming device(s) 104 and provide for the automatic generation of highlights. The highlight capture application 172 can transform the game stream into a format that can be stored, processed, and / or otherwise used by the client device(s) 102, as specified by the highlight request. The highlight capture application 172 can generate metadata from the game and associate it with the highlight files.

[0048] Datastore(s) 176 may include buffer 180 on client device(s) 102. Datastore(s) 176 and buffer 180 may be similar to datastore(s) 116 and buffer 130. Datastore(s) 176 and / or buffer 180 may be configured to capture the game stream in temporary or permanent storage and may include hard disks, solid-state drives, RAM, or other computer storage media.

[0049] The ShareServer 166 can be a client plugin that syntactically analyzes data between the CEF client 146 and the peak capture application 172. Data can be syntactically analyzed and stored for information such as keyboard shortcuts and saved settings between multiple game instances 132.

[0050] In some embodiments, the climax generation system 140 may include other plugins 170. The other plugins 170 may be additional plugins that can be implemented in the climax generation system 140. The other plugins 170 may interact with the climax capture application 172, the data storage(s) 176, the share server 166, and / or the CEF client 146. In some embodiments, the other plugins 170 may be third-party stream viewing services, local client software for online chats, and / or post-processing filtering software (e.g., NVIDIA's Freestyle). The other plugins 170 may enable colorization and filtering of game stream data, inform others that a player is in-game, or provide system information to the client device(s) 102.

[0051] The highlight generation system 140 can include a multiplexing application 174. The multiplexing application 174 can include operating system-specific components for generating highlight files, such as by standardizing the interface for multiplexing video and audio data into the files.

[0052] Although the system's components are 140 in Fig. The fact that 1B is represented with a specific distribution is not intended to be a limitation. For example, some or all components, features, and / or functionality of System 140 can be considered components, features, and / or functionality of a proxy representative (e.g., the proxy representative 118). Therefore, the arrangement of the components, features, and / or functionality within System 140 can differ—e.g., depending on the system information of the client device(s) 102 and / or the streaming device(s) 104—in various embodiments, as described here with regard to the proxy representative. Additionally, the client device(s) 102 and the streaming device(s) 104 are in Fig. 1B is shown, although this is not intended to be a limitation. In some embodiments, one or more secondary devices may also be included in the system 140 (e.g., to perform game analysis for the automatic generation of highlights, such as by using a game analysis application as one of the other plugins 170).

[0053] Now on Fig. 1C referring is Fig. Figure 1C shows another example system diagram for generating highlights in cloud gaming systems, in accordance with some embodiments of the present disclosure. In a highlight generation system 150 in a game streaming environment, the client device(s) 102 can receive input data as a result of inputs to the input device(s) 112, transmit the input data to the streaming device(s) 104 and / or the secondary device(s) 106, receive an encoded game stream from the streaming device(s) 104, and display the encoded game stream on the display 110. The requirements for generating a game instance (e.g., the game instance 132 of the Fig. 1A or Fig. 1B) The required computationally intensive calculation and processing can be offloaded to the streaming device(s) 104 (e.g., so that the game session is rendered by the GPU(s) of the streaming device(s) 104). In some examples, a game session can be streamed from the streaming device(s) 104 to the client device(s) 102 and / or the secondary device(s) 106. Offloading the generation of the gameplay can reduce the graphics processing and rendering requirements on the client device(s) 102 and the secondary device(s) 106. By offloading game generation and playback tasks from the client device(s) 102, the client device(s) 102 can use its computing resources to capture at least a portion of the in-game highlights, store the highlights, and generate the graphical overlay (e.g., the graphical overlay 128 of the Fig. 1A) and / or to perform one or more other tasks to generate highlights (e.g. tasks assigned here to proxy representative 118).

[0054] For example, based on receiving a game stream from the streaming device(s) 104, the client device(s) 102 can display a single frame of a game instance on a screen. The client device(s) 102 can receive input from one of the input devices 112. Input data can be transmitted via the network(s) 108 to the streaming device(s) 104 and / or the secondary device(s) 106, and the streaming device(s) 104 and / or the secondary device(s) 106 can receive the input data. The input data can be received at the communication interfaces 152. The CPU(s) 154 can receive the input data, process the input data and transfer data to the GPU(s) 156, which causes the GPU(s) 156 to render the game instance (e.g., game instance 132 of the Fig. 1A) to generate. For example, in some embodiments, the input data can represent the movement of a user character in a game, firing a weapon, reloading, passing a ball, turning a vehicle, etc. The playback component 158 ​​can reproduce the game session (e.g., representative of the result of the input data), and the proxy 118 can capture the reproduction of the game instance as display data in data memory(s) 116 and / or buffer 130.

[0055] The encoder 160 can then encode the game instance to generate an encoded game stream, and the encoded game stream can be transmitted via communication interface 152 to the client device(s) 102 and / or the secondary device(s) 106 via the network(s) 108. The client device(s) 102 can receive the encoded game via a communication interface 182, and the decoder 162 can decode the encoded game to generate the game to be displayed. The client device(s) 102 can then display the game stream via the display 110.

[0056] In some embodiments, the climax generation system can comprise 150 secondary device(s) 106. The game stream can be transmitted—e.g., simultaneously in some examples—to the secondary device(s) 106 and received by a communication interface 184. The communication interface 184 can send the encoded game stream to a decoder 162, a game analysis application 124, and / or a proxy 118. The proxy 118 can store at least a section of the game instance in data memory(s) 116 and / or buffers 130.

[0057] Now on Fig. 2 referring is Fig. 2 An example screenshot of a graphical overlay for generating and customizing the game stream and for generating highlights, in accordance with some embodiments of the present disclosure. The graphical overlay 128 may include recording settings 204, network utilization 206, stream settings 208, storage location 210, an auto-recording option 212, and / or other settings. The graphical overlay 128 may also include capture controls 214.

[0058] The capture controls 214 can, based on a selection, send a signal to record at least a portion of the game stream based on the recording settings 204. Various types of game stream capture can be offered in the capture controls 214. For example, the capture controls 214 can include a selection to record gameplay (e.g., a snapshot, a highlight, an entire recording, etc.) by signaling the proxy to record at least a portion of the game stream. In operation, gameplay can be recorded from the moment the proxy is signaled to capture the game stream. In another example, the capture controls 214 can include a selection to capture a "replay" (e.g., a highlight) by signaling the proxy to a buffer (e.g., the buffer of the Fig. 1A) to retain a time interval within the game session and / or an additional non-buffered recording of a time interval after the signal has been received by the proxy. In other words, and as a non-restrictive example, a user may select a repeat button, and the proxy may capture a time interval before and / or after the button selection.

[0059] The capture controls 214 can also include a user-selectable element for streaming the game stream. If a user selects the option to stream the game stream, client device(s) 102 can transmit the game instance generated by the streaming application 120 to additional devices or to a third-party server to be sent to further devices (e.g., secondary device(s) 106, the streaming device(s) 104, other client device(s) 102, etc.).

[0060] The recording settings 204 can control various aspects of the proxy representative (e.g., as proxy representative 118 of the Fig. 1A). A user can select one of the various options for recording gameplay from user-selectable elements. Recording settings 204 can include, for example, resolutions of 720, 1080, or 4K. Recording gameplay at each of these resolutions requires increasingly more resources (e.g., network or computing resources) and / or storage space in data storage 116. Fig. 1A), and therefore a user can choose from many different recording settings, depending on whether the proxy is recording the gameplay on a client device, a secondary device, or a streaming device (e.g., client device(s) 102, secondary device(s) 106, and streaming device(s) 104). Fig. 1A).

[0061] In some embodiments, the recording settings may include manual controls for selecting the client, secondary, and / or streaming device(s) (such as those described in Fig. (as described in 1A) include the features that activate the proxy. This allows a user to select their preferred recording location for gameplay.

[0062] The network usage setting (206) of the stream can include a variety of user-selectable elements to control the network usage of the game stream. In some aspects and with certain settings, streaming a game to a client device can consume a large amount of data over a network. A user can select the game stream usage in (as a non-restrictive example) MB / s to limit the network usage of the game stream on the client device.

[0063] The stream settings 208 can include a variety of user-selectable elements for controlling the stream quality. Each of the selectable elements can provide an indication of the streaming application (e.g., the streaming application 120 of the Fig. 1A) Send to customize the game instance generation. A user can choose whether the game instance should be generated at 720p, 1080p, 4K, or a custom resolution. Additionally, the user can select whether the game should be generated at 60 fps, 120 fps, or a custom frame rate.

[0064] In cases where the proxy is configured to record gameplay locally on the client device, the user can select storage location 210 of the data storage on the client device. For example, a user can select a download folder on drive "C", as shown in Fig. 2 shown.

[0065] In some aspects, the graphical overlay 128 may include an auto-recording option 212. When enabled, the auto-recording option 212 (which may be referred to as "automatic highlights") can be used with a game analysis application (e.g., the game analysis application 124 of the Fig. 1A) Enable. When enabled, the game analytics application can detect that a recording event has occurred and signal the proxy to capture at least a portion of the game stream.

[0066] Now on Fig. Referring to 3-4, each block of procedures 300 and 400 described herein comprises a computational process that can be performed with any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. The procedures can also be embodied as computationally usable instructions stored on computer storage media. The procedures can be provided by a standalone application, service, or data center service (standalone or in combination with another data center service), or by a plug-in for another product, to name just a few. Additionally, procedures 300 and 400 are illustrated by way of example with reference to the peak generation system 100 of the Fig. 1A described. However, these procedures can additionally or alternatively be carried out by any system or any combination of systems, including but not limited to those described here.

[0067] Fig. Figure 3 is a flowchart showing a procedure 300 for generating a climax in a game streaming environment. Procedure 300 includes, in block B302, the playback of a game instance. For example, game instance 132 can be played back by streaming device(s) 104.

[0068] Procedure 300, in block B304, involves encoding the game instance into a game stream. For example, the game stream can be encoded using various methods. Encoding a game stream can aid in transmitting the game instance 132 over the network(s) 108, generating highlights, and / or displaying it on the client device(s).

[0069] Procedure 300 includes, in block B306, the transmission of the game stream to a client device. For example, the game stream can be transmitted via network(s) 108 to client device(s) 102.

[0070] Procedure 300, in block B308, includes receiving an indication to capture at least a section of the game stream as a recording. For example, an indication to create a climax can be received as a result of user input (e.g., a keyboard shortcut) on the client device(s) 102, as a result of analysis of the game stream by the game analysis application 124, and / or as a result of another action or determination by the system 100.

[0071] Method 300, in block B310, includes determining system information corresponding to a client device. For example, the system information can be determined for the client device(s) 102, the secondary device(s) 106, and / or the streaming device(s) 104. The system information can include, without limitation, the storage capacity, encoding capability, and / or processing capability of the client device(s) 102, the secondary device(s) 102, and / or the streaming device(s) 104. In some embodiments, the system information can be used to determine a playback and / or streaming bitrate and / or frame rate (e.g., a minimum quality of service (QoS)) achieved by the distribution—dynamically in some embodiments—of the proxy representative's processing tasks between and among the various devices of the system 100.In these examples, the system information can represent a capability measure based on the capabilities of one or more of the devices. The system information can additionally or alternatively include the game stream recording settings and / or the network information of network(s) 108.

[0072] Method 300, in block B312, comprises a selection, based at least partially on system information, between generating the recording on the streaming device and transmitting a character to the client device to cause the client device to generate the recording. For example, the system information of the client device(s) 102 can be used to determine whether certain features and / or functionalities should be executed on the client device(s) 102, the streaming device(s) 104, and / or the secondary device(s) 106. In some examples, a minimum capability measure (e.g., in some embodiments, a threshold) of one or more of the devices in the system 100 can be used to select which device should generate the recording.In another example, configurations that achieve QoS (measured in non-restrictive examples by a playback and / or streaming bitrate and / or frame rate) can be used to select which device should create the recording, perform the analysis of the recording, share the recording, store the recording, and / or perform one or more other proxy representative tasks.

[0073] Procedure 300 includes in block B314 generating the recording on the streaming device(s) 104. For example, if the client device(s) 102 do not meet the minimum capability (and / or QoS) or requirements to effectively enable highlight generation and / or game streaming (e.g., with low latency and high quality), the recording can be generated on the streaming device(s) 104.

[0074] Procedure 300, in block B316, involves transmitting a character to the client device to cause the client device to generate the recording. For example, where the client device(s) 102 meet a minimum set of capability standards or requirements to effectively enable highlight generation and / or game streaming (e.g., with low latency and high quality), the recording can be generated on the client device(s) 102, and the streaming device(s) 104 and / or the secondary device(s) 106 can transmit the character to the client device(s) 102 to cause the client device(s) 102 to generate the recording.

[0075] Now on Fig. 4 referring is Fig. Figure 4 shows another flowchart illustrating Procedure 400 for generating a climax in a game streaming environment. Procedure 400 includes, in block B402, receiving a game stream from a streaming device. For example, Client Device(s) 102 can receive the game stream from Streaming Device(s) 104.

[0076] Procedure 400, in block B404, includes displaying an initial instance of the game stream on a client device display. For example, an initial instance of the game stream can be displayed on a client device display (or displays) 102.

[0077] Procedure 400, in block B406, involves storing at least a portion of a second instance of the game stream in a buffer. For example, a second instance of the game stream can be stored in buffer 130 of client device(s) 102.

[0078] Procedure 400, in block B408, includes receiving an indication of a request to generate a recording of at least a section of the second instance of the game stream. For example, an indication to generate a climax can be received as a result of user input (e.g., a keyboard shortcut) on the client device(s) 102, as a result of analysis of the game stream by the game analytics application 124, and / or as a result of another action or determination by the system 100.

[0079] Procedure 400, in block B410, involves generating the recording using at least the portion of the second instance of the game stream in the buffer. For example, at least a part of the second instance of the game stream, stored in buffer 130, can be copied to create a highlight (e.g., a snapshot, a recording, a short clip, etc.).

[0080] Procedure 400 includes, in block B412, storing the recording in a data storage device on the client device. For example, the recording (e.g., the climax) can be stored in the data storage device(s) 116 of the client device(s) 102.

[0081] Fig. Figure 5 shows a block diagram of an example computing environment suitable for implementing some embodiments of the present disclosure. The computing device 500 may comprise a bus 502 that directly or indirectly couples the following devices: memory 504, one or more central processing units (CPUs) 506, one or more graphics processing units (GPUs) 508, a communication interface 510, input / output (I / O) ports 512, input / output (I / O) components 514, a power supply 516, and one or more display components 518 (e.g., display(s)).

[0082] Although the various blocks of the Fig. Where components 5 are shown as connected via lines over bus 502, this is not intended to be a limitation and serves only for clarity. For example, in some embodiments, a display component 518, such as a display device, can be considered an I / O component 514 (e.g., if the display is a touchscreen). As another example, the CPU(s) 506 and / or the GPU(s) 508 can include memory (e.g., the memory 504 can represent an additional memory device to the memory of the GPU(s) 508, the CPU(s) 506, and / or other components). In other words, the computing device of the Fig. Figure 5 is merely an illustration. No distinction is made between categories such as "workstation", "server", "laptop", "desktop", "tablet", "client device", "mobile device", "portable device", "game console", "electronic control unit (ECU)", "virtual reality system" and / or other device or system types, as all are included within the scope of the computing device of the Fig. 5 should be considered.

[0083] The 502 bus can refer to one or more buses, such as an address bus, a data bus, a control bus, or a combination thereof. The 502 bus can encompass one or more bus types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standard Unification (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Express (PCIe) bus, and / or another bus type.

[0084] Memory 504 can comprise any of a wide variety of computer-readable media. Computer-readable media can be any available media that the computing device 500 can access. Computer-readable media can include both volatile and non-volatile media, as well as removable and non-removable media. By way of example, and not as a limitation, computer-readable media can include computer storage media and communication media.

[0085] Computer storage media can include both volatile and non-volatile media, and / or removable and non-removable media, implemented in any method or technology for storing information such as computer-readable instructions, data structures, program applications, and / or other data types. For example, Memory 504 can store computer-readable instructions (which represent, for example, a program and / or program element(s), such as an operating system). Computer storage media can include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, Digital Versatility Discs (DVDs) or other optical disk storage, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that Computing Device 500 can access.As used here, computer storage media do not contain signals per se.

[0086] Communication media can include computer-readable instructions, data structures, program applications, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and encompass all information delivery media. The term "modulated data signal" can refer to a signal in which one or more of its properties have been defined or modified in such a way that information is encoded within the signal. Communication media can include, for example, but are not limited to, wired media such as a wired network or a direct-wired connection, as well as wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of the aforementioned media are also considered to fall within the scope of computer-readable media.

[0087] The CPU(s) 506 can be configured to execute the computer-readable instructions for controlling one or more components of the Computing Device 500 in order to perform one or more of the procedures and / or processes described herein. The CPU(s) 506 can each comprise one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing a multitude of software threads concurrently. The CPU(s) 506 can comprise any type of processor and can include different types of processors depending on the type of Computing Device 500 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers).For example, depending on the type of computing device 500, the processor can be an ARM processor implemented using reduced instruction set (RISC) computing, or an x86 processor implemented using complex instruction set (CISC) computing. The computing device 500 can include one or more CPU(s) 506 in addition to one or more microprocessors or supplementary coprocessors, such as mathematical coprocessors.

[0088] The GPU(s) 508 can be used by the computing device 500 to render graphics (e.g., 3D graphics). The GPU(s) 508 can include hundreds or thousands of cores capable of processing hundreds or thousands of software threads concurrently. The GPU(s) 508 can generate pixel data for output images as a result of rendering instructions (e.g., rendering instructions from the CPU(s) 506 received via a host interface). The GPU(s) 508 can include graphics memory, such as display memory, for storing pixel data. The display memory can be included as part of the memory 504. The GPU(s) 508 can include two or more GPU(s) operating in parallel (e.g., via a link). When combined, each GPU can generate 508 pixel data points for different parts of an output image or for different output images (e.g., a first GPU for a first image and a second GPU for a second image).Each GPU can have its own dedicated memory or share memory with other GPUs.

[0089] In examples where the computing device 500 does not include the GPU(s) 508, the CPU(s) 506 can be used for graphics rendering.

[0090] The Communication Interface 510 can include one or more receivers, transmitters, and / or transceivers that enable the Computing Device 500 to communicate with other Computing Devices over an electronic communication network, including wired and / or wireless communication. The Communication Interface 510 can include components and functionality that enable communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., Ethernet communication), low-power wide-range networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0091] The I / O ports 512 enable the computing device 500 to be logically coupled with other devices, including the I / O components 514, the display component(s) 518, and / or other components, some of which may be built into (e.g., integrated with) the computing device 500. Illustrative I / O components 514 include a microphone, mouse, keyboard, joystick, gamepad, game controller, satellite dish, scanner, printer, wireless device, etc. The I / O components 514 may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some cases, inputs may be transferred to a suitable network element for further processing.A NUI can implement any combination of speech recognition, pen recognition, facial recognition, biometric recognition, gesture recognition both on-screen and near-screen, air gestures, head and eye tracking, and touch recognition (as further described below) in conjunction with a display of the Computing Device 500. The Computing Device 500 can include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof for gesture capture and recognition. Additionally, the Computing Device 500 can include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit (IMU)) that enable motion detection. In some examples, the output from the accelerometers or gyroscopes can be used by the Computing Device 500 to render immersive augmented reality or virtual reality.

[0092] The power supply 516 can be a hardwired power supply, a battery power supply, or a combination thereof. The power supply 516 can supply power to the computing device 500 to enable the components of the computing device 500 to operate.

[0093] The display component(s) 518 can include a display (e.g., a monitor, a touchscreen, a television screen, a heads-up display (HUD), other display types, or a combination thereof), speakers, and / or other display components. The display component(s) 518 can receive data from other components (e.g., the GPU(s) 508, the CPU(s) 506, etc.) and output the data (e.g., as an image, video, sound, etc.).

[0094] The disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program applications, that are executed by a computer or other machine, such as a personal data assistant or other portable device. Generally, program applications, including routines, programs, objects, components, data structures, etc., refer to code that performs specific tasks or implements specific abstract data types. The disclosure can be executed in a variety of system configurations, including portable devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The disclosure can also be executed in distributed computing environments where tasks are performed by remote processing devices interconnected via a communication network.

[0095] As used here, a statement of "and / or" in relation to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Furthermore, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0096] The subject matter of this disclosure is described here based on specific features in order to meet legal requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have considered that the claimed subject matter could also be embodied in other ways to include various steps or combinations of steps similar to those described in this document, in conjunction with other present or future technologies. Although the terms "step" and / or "block" may be used in this document to denote various elements of the methods employed, these terms should not be interpreted as implying a particular sequence among or between the various steps disclosed herein, unless the sequence of each step is expressly described.

Claims

[1] Procedure, encompassing: Playing back a game instance on a streaming device (104); Encoding the game instance into a game stream on the streaming device (104); Transmitting the game stream to a client device (102) for display on the client device; Receiving evidence to capture at least a portion of the game stream as a recording; Analyzing system information corresponding to the client device (102) to determine the allocation of recording tasks between the client device (102) and the streaming device (104); Selection based at least partially on the allocation of recording tasks, with Transmitting a character to the client device (102) to cause the client device to generate the recording, wherein: The transmission of the character to the client device (102) to cause the client device to generate the recording is selected to be performed when system information indicates that generating the recording on the client device would at least reach a performance threshold; and Creating the recording on the streaming device (104) is selected to perform when system information indicates that creating the recording on the client device would not at least meet the performance threshold. [2] Method according to claim 1, wherein the indication is received by a game stream analysis representative based at least partially on a finding by the game stream analysis representative that a recording event has occurred in the game instance. [3] Method according to one of claims 1 or 2, wherein the reference point is received by the client device (102) at least partially based on an input received by the client device in order to generate the recording. [4] Method according to any one of claims 1 to 3, wherein the system information includes at least one of the following features: storage capacity, encoding capability, processing capability, recording settings or network information. [5] Method according to any one of claims 1 to 4, wherein game stream data is maintained in a buffer by the client device (102) or the streaming device (104) at least partially based on system information, and the generation of the recording by the client device or the streaming device comprises using at least the portion of the game stream from the buffer. [6] Method according to any one of claims 1 to 5, further comprising: Transmitting the game stream to a secondary device (106), where the selection to execute is made from at least one of the following: Creating the recording on the streaming device (104); or Transmitting a character to the client device (102) to cause the client device to generate the recording; Generating the recording on the secondary device (106); or Sending a character to the secondary device to cause the secondary device to generate the recording. [7] Method according to any one of claims 1 to 6, further comprising: at least partially based on system information, on the streaming device (104), a supplementary graphical overlay separate from the playback of the game instance, wherein the supplementary graphical overlay is configured to enable the control of one or more recording functions of the game stream; and encoding the supplementary graphical overlay within the game stream, wherein the game stream includes the graphical overlay. [8] Method according to claim 7, further comprising: Receiving a further indication of input that corresponds to an interaction with at least one user interface element of the supplementary graphical overlay; and at least partially based on the further indication of the act of creating the recording. [9] System (100) for generating a recording of a game instance in a game stream, comprising: a streaming module running on a streaming device (104), wherein the streaming module is configured to create the game instance in the game stream on the streaming device (104), to encode the game instance in the game stream to create an encoded game instance, and to transmit the encoded game instance in the game stream to a client device (102) remote from the streaming device (104); a client module running on the client device (102) with associated system information, wherein the client module is configured to receive the encoded game instance in the game stream on the client device; and a proxy agent (118) configured to receive a clue to generate the recording and, based at least in part on the system information and the clue, to dynamically assign recording tasks to the client device (102) to generate the recording by transmitting a character to the client device (102) to induce the client device to generate the recording, wherein the transmission of the character to the client device (102) to induce the client device to generate the recording is selected to be performed when the system information indicates that generating the recording on the client device (102) would meet at least a performance threshold; and Creating the recording on the streaming device (104) is selected to perform when system information indicates that creating the recording on the client device (102) would not at least reach the performance threshold. [10] System (100) according to claim 9, wherein the proxy agent is configured to run on at least one of the streaming device, the client device (102) or a secondary device (106) different from the client device and the streaming device (104). [11] System (100) according to claim 10, wherein the secondary device (106) is a live streaming device configured to receive at least a portion of the game instance in the game stream and send it to a plurality of display devices. [12] System (100) according to any one of claims 9 to 11, wherein the client module is further configured to display the game stream on a display of the client device (102). [13] System (100) according to any one of claims 9 to 12, further comprising a game analysis module configured to detect that a recording event has taken place and to generate the indication to create the recording, wherein the game analysis module is configured to be executed on at least one of the client device, the streaming device or a secondary device.

Images