Dynamically allocating computing resources to generate highlights in cloud gaming systems

Abstract

The present invention discloses dynamically allocating computing resources to generate highlights in a cloud gaming system. In various examples, computing resources are allocated for generating highlights in a cloud gaming system. The present invention discloses systems and methods distributed between and among various devices, processing including generation and overlay of user interfaces, game stream analysis of actionable events, generation of highlights, storage of highlights, and sharing of highlights. The allocation of processing resources or computing resources in a cloud gaming system depends on system information of various devices and / or networks. Records, snapshots, and / or other highlights can be generated in a cloud gaming system using the determined allocation of computing resources.

Description

Background Art

[0001] During a gaming session, a user may wish to have screenshots, highlights, or clips of at least a portion of the gaming session, for example so that the user can share the recorded material, review the material later, or otherwise access the recorded content. Conventional systems and methods allow players to record a gaming session or portions thereof on their local client devices. However, in order to be able to render frames of the gaming session locally (e.g., on the client device), the client device must include significant computing resources. In addition to rendering frames of the gaming session locally, performing functions such as generating recorded material locally requires additional hardware and network computing resources.

[0002] In other conventional systems and methods, once a gameplay recording is generated (e.g., stream viewing), the player is able to stream the game session from their client device. In stream viewing, each (or several) frames of the game can be copied from the client device's display and then uploaded to a (usually) separate service or application for distribution to the stream viewing audience (e.g., on social media). If the user is streaming their game streaming session, each frame must be copied at the client device and sent to a third party for distribution, thus requiring a large amount of bandwidth and processing power. Therefore, similar to the above, if there is not enough storage capacity, GPU power, or other hardware capabilities to effectively perform the generation of highlights, storage, and / or sharing, the client device cannot use these capabilities. Therefore, client devices with limited hardware or bandwidth capacity are unable to generate and / or share snapshots, highlights, or game recordings, thereby affecting the user experience. Summary of the invention

[0003] The present disclosure relates to allocating computing resources in a cloud gaming system to generate highlights. The present invention discloses systems and methods for allocating processing between and among various devices, including user interface generation and overlay, game stream analysis of actionable events, highlight generation, highlight storage, and highlight sharing. As used herein, highlights may include any capture of a game session stream, including game snapshots or screenshots, records of an entire game session, or records of a portion of a game session (e.g., a clip), such as records of specific actions and events in a game session.

[0004] Compared to conventional systems (such as the above-described systems), the systems and methods of the present disclosure analyze the hardware, software, and / or network capabilities of the client device to determine how to allocate processing tasks within the system (e.g., allocate processing tasks among one or more client devices, one or more streaming media servers, one or more auxiliary systems, etc.). For example, a proxy agent (responsible for various tasks within the highlight generation system) can be variably transferred between client devices, streaming media devices, and / or auxiliary systems (e.g., remote systems that offload processing tasks). The proxy agent can record the game stream (e.g., in a memory device (such as a buffer)), generate and present user interface elements related to generating highlights, analyze actionable events in the game stream (e.g., events or actions in the game that trigger the generation of highlights), respond to commands for generating highlights (e.g., from a client device, a streaming media device, an auxiliary system, or a combination thereof), and send signals to generate and store highlights.

[0005] Therefore, because the proxy agent is responsible for various tasks within the system, the various processes of the proxy agent can be transferred between various devices in the system. In some examples, the streaming device (or another device of the system) can query the client device to determine the hardware, software and / or bandwidth capacity (referred to herein as system information). In response to determining the system information, in some embodiments, the various tasks of the proxy agent can be dynamically transferred among the system devices to enhance the cloud gaming experience of the end user. As a non-limiting example, by performing certain processing tasks on the client device (e.g., generating highlights and storing highlights) and performing certain processing tasks on the streaming device and / or auxiliary device (e.g., analyzing the game stream for actionable events), each device in the system will benefit from increased bandwidth utilization, increased processing speed, low latency, and higher game stream resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The system and method for allocating computing resources for generating highlights in a cloud gaming system will be described in detail below with reference to the accompanying drawings, wherein:

[0007] Figure 1A is an example system diagram for generating highlights in a cloud gaming system according to some embodiments of the present disclosure;

[0008] Figure 1B is an example system diagram for generating highlights in a cloud gaming system (including a client device and a streaming device) according to some embodiments of the present disclosure;

[0009] Figure 1C is another example system diagram for generating highlights in a cloud gaming system according to some embodiments of the present disclosure;

[0010] Figure 2is an example screenshot taken from a display of a gaming session including a graphical overlay for generating highlights according to some embodiments of the present disclosure;

[0011] Figure 3-Figure 4 is a flowchart illustrating a method of generating highlights in a cloud gaming system according to some embodiments of the present disclosure; and

[0012] Figure 5 is a block diagram of an example computing environment suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0013] Disclosed are systems and methods for allocating computing resources to generate highlights in a cloud gaming system. As described above, compared to conventional systems, the present disclosure provides for transferring one or more tasks of a highlight generation system between and among various devices in an effort to efficiently allocate computing resources, network usage, and storage requirements in the system. Thus, one or more streaming devices (e.g., a server responsible for rendering, encoding, and streaming a game session for one or more users), one or more client devices (e.g., a tablet, computer, smartphone, streaming device, game console, or other device that displays a game stream and receives and sends input), and / or one or more auxiliary devices (e.g., a server responsible for one or more overflow tasks) may be responsible for performing tasks within the system in a manner that allows high-quality, low-latency game streaming to be achieved for each user of the system.

[0014] In some examples, tasks associated with generating highlights in a cloud gaming environment may be referred to as tasks of a proxy agent. Therefore, the proxy agent may be distributed between and among various devices of the highlight generation system. In some examples, the proxy agent may dynamically transfer tasks between various devices based on the system information of each client device in one or more client devices. The system information may include storage capacity, encoding capability, processing capability, recording settings for game streams, network information, and / or other system information. The system information may be related to one or more streaming media devices, one or more client devices, and / or one or more auxiliary devices. Therefore, compared further to a conventional system, in which the tasks of one or more client devices are to render games, generate highlights, store highlights, share highlights, and perform each generation highlight task, the highlight generation system of the present disclosure takes into account the limitations of one or more client devices. For example, one or more first client devices may have greater storage and / or processing capabilities than one or more second client devices, so the tasks of the proxy agents distributed among the various devices may be different for the first client device (e.g., customized) and for the customization of one or more second client devices. As a non-limiting example, user interface elements can be generated on one or more client devices, rather than generating user interface elements (e.g., overlays) for controlling the generation of highlights or recordings performed by a streaming device (e.g., based on recording settings). By generating user interface elements at one or more client devices in a client application, the streaming device requires less bandwidth and computing resources, and the image quality of the user interface elements in the overlay can be improved. As another non-limiting example, if one or more client devices have sufficient storage capacity and / or processing power, the generation and storage of highlights of the game stream can be performed by one or more client devices, thereby offloading storage and processing requirements from one or more streaming devices to one or more client devices to allow one or more streaming devices to maintain high image quality, low-latency game streams. Therefore, compared to traditional systems in which a single device is responsible for generating highlights, by dynamically (in some embodiments, between and among one or more streaming devices, one or more client devices, and / or one or more auxiliary devices) transferring the tasks of the delegate agent, the storage, processing, computing, and bandwidth burdens of the system can be distributed in an effort to reduce the individual burden of any one device.

[0015] As described above, conventional systems do not operate in a game streaming environment. Even if a recording of a game is streamed (e.g., in stream viewing), the original game rendering is performed on one or more client devices, and the recording is performed on one or more client devices. Therefore, in a game streaming environment, additional constraints are implemented because one or more streaming devices are responsible for receiving input from one or more client devices, rendering the game based on the input, encoding the game stream, and sending the game stream to one or more client devices. For example, game streaming typically encounters bandwidth issues when streaming high-quality game content with low latency over the Internet while also allowing bandwidth for the input to be sent back from one or more client devices to one or more streaming devices.

[0016] In addition to these constraints of a separate game streaming system, introducing a highlight generation system into a game streaming environment introduces further constraints. For example, if recorded highlights are generated and stored on one or more streaming devices, the file size of the highlights may become costly, and / or additional bandwidth may be required to send or stream the highlights back to one or more client devices for viewing (e.g., during or after the game). In addition, since one or more streaming devices are responsible for managing the game session and streaming it to multiple end users in addition to generating and storing highlights for one or more end users, the processing power of one or more streaming devices may be affected (reduced). The reduced processing power of one or more streaming devices may cause one or more end users to experience lower quality game streams, higher latency, and / or other stream quality issues during the game. Therefore, by transferring one or more tasks of the delegate agent of the highlight generation system between and among various devices in the game streaming environment, the constraints of the system are interpreted in a manner that allows end users to experience high image quality, low latency game streams, and related seamless highlight generation functionality.

[0017] Reference now Figure 1A , Figure 1A is an example system diagram of a highlight generation system 100 in a cloud gaming environment according to some embodiments of the present disclosure. It should be understood that this arrangement and other arrangements described herein are only set forth as examples. In addition to or in place of the arrangements and elements shown, other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groupings, etc.) may be used, and some elements may be omitted entirely. In addition, many of the elements described herein are functional entities that may be implemented as discrete or distributed components, or in combination with other components, and in any suitable combination and location. The various functions performed by the entities described herein may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory.

[0018] The highlight generation system 100 may include one or more client devices 102, one or more streaming devices 104, one or more auxiliary devices 106, and / or one or more networks 108, etc. The highlight generation system 100 (and its components and / or its features) may be implemented using one or more computing devices, such as Figure 5 The computing device 500 in FIG. 1 is described in more detail herein.

[0019] Each device or component of the highlight generation system 100 can communicate between and among each other using one or more networks 108. The one or more networks 108 can include a wide area network (WAN) (e.g., the Internet, a public switched telephone network (PSTN), etc.), a local area network (LAN) (e.g., Wi-Fi, Ethernet, etc.), and / or other network types.

[0020] One or more client devices 102 may include a smartphone, a laptop, a tablet, a desktop computer, a wearable device, a gaming console, a virtual reality system (e.g., a headset, a computer, a gaming console, one or more remote controls, one or more controllers, and / or other components), a streaming device (e.g., NVIDIA SHIELD), a smart home device that may include an intelligent personal assistant, and / or another type of device capable of supporting at least display of a game stream of a gaming session 126 and / or input to the gaming session 126 (from one or more input devices 112).

[0021] One or more client devices 102 may include a display 110, one or more input devices 112, a client application 114, one or more data stores 116, and / or at least some functionality of a proxy agent 118. In some examples, at least some functionality of a proxy agent 118 may be performed in a client application 114. Although certain components and / or features of one or more client devices 102 are illustrated in FIG. 1 , this is not intended to be limiting. For example, one or more client devices 102 may include additional or alternative components, such as, and without limitation, the components described herein. Figure 5 The components of the computing device 500 described in FIG. Figure 1B and Figure 1C Features and functionality of one or more client devices 102 are described in greater detail herein.

[0022] The client application 114 may be a mobile application, a computer application, a console application, a gaming application, and / or other types of applications. The client application 114 may include instructions that, when executed by one or more processors of one or more client devices 102, cause the one or more processors to perform one or more operations (such as, but not limited to, the operations described herein with respect to the highlight generation system 100). The client application 114 may operate as a facilitator for enabling a game instance to be played. For example and without limitation, the client application 114 may display a game stream received from a streaming device 104, receive and / or process input from one or more input devices 112 of one or more client devices 102, and / or cause input data representing the input to be sent to one or more streaming devices 104. In some examples, as described herein, the client application 114 may include certain features or functionality of the proxy agent 118, such as generating highlights and displaying a graphical overlay 128 ( Figure 2 ), to generate highlights, recordings or snapshots, and program recording settings, etc. In some examples, the client application 114 may cause highlights to be generated, store highlights, analyze the game stream for determining when highlights should be generated, share highlights, etc. In some embodiments, one or more client devices 102 may include any number of client applications 114, among which the features and functions of the highlight generation system 100 are distributed.

[0023] One or more client devices 102 may include one or more components (e.g., communication components, network interfaces, etc.) and features for communicating over one or more networks, such as one or more networks 108. As a non-limiting example, to communicate within the highlight generation system 100, one or more client devices 102 may access the Internet through a router using a wired Ethernet connection and / or a Wi-Fi connection to communicate with one or more streaming devices 104, one or more auxiliary devices 106, and / or one or more other client devices 102.

[0024] Display 110 may include any type of display capable of displaying a game (e.g., a light emitting diode display (LED), an organic LED display (OLED), a liquid crystal display (LCD), an active matrix OLED display (AMOLED), a quantum dot display (QDD), a plasma display, and / or other types of displays). In some examples, depending on the configuration of one or more client devices 102, display 110 may include more than one display (e.g., a dual display display for computer games, a first display for configuring games, and a virtual reality display for playing games, etc.). When display 110 is a touch screen display, such as a touch screen of a smartphone, tablet computer, laptop computer, etc., display 110 may be used as at least one of one or more input devices of one or more client devices 102 (e.g., for generating inputs of game instances 132 to be sent to one or more streaming devices 104 to update the rendering of the game). Display 110 may display a game stream of one or more game sessions, such as game session 126 (A), game session 126 (B), and / or game session 126 (C) (collectively referred to herein as "game sessions 126"). Game sessions 126 may include any number of game sessions in which users of one or more client devices 102 participate.

[0025] The one or more input devices 112 may include any type of device capable of providing user input to the game. The one or more input devices may include a keyboard, a mouse, a joystick, a touch screen display, one or more controllers, one or more remote controls, a headset (e.g., sensors of a virtual reality headset), other types of input devices, and / or combinations thereof.

[0026] One or more data stores 116 may include any memory device type. In embodiments where one or more client devices 102 may be responsible for storing and / or generating highlights, one or more data stores 116 may store some or all of a game stream. For example, one or more data stores 116 may store an entire game stream, and snapshots, highlights, and / or records may be generated from the stored game stream. Snapshots, highlights, and / or records may be stored separately in one or more data stores 116 after generation, or an indication (e.g., a timestamp) of the snapshot, highlight, and / or record may be stored (e.g., as metadata) so that the snapshot, highlight, and / or record may be copied or retrieved from the stored game stream using the indication.

[0027] In some examples, one or more data stores 116 may store only a portion of the game stream. For example, one or more data stores 116 store only snapshots, highlights, and / or records. Buffer 130 may be used to store a portion of the game stream, and highlights, snapshots, or records may be copied or retrieved from the buffer (and stored separately in one or more data stores 116). Buffer 130 may store only a portion of the game stream (e.g., the last 30 seconds of the game stream may be stored in buffer 130). Therefore, when a snapshot, highlight, or record is generated, in addition to at least some portion of the game stream in buffer 130, in some examples, the next portion of the game stream (e.g., the next 10 seconds) may be copied or retrieved to generate the snapshot, highlight, and / or record. In some non-limiting examples, buffer 130 may include a circular or rolling buffer.

[0028] One or more streaming media devices 104 may include a streaming application 120 (also referred to as a streaming application) for receiving input data representing input to one or more client devices 102, rendering a game instance 132, encoding the game instance 132 into a stream, and sending the stream to the one or more client devices 102. In some examples, as described herein, the streaming application 120 may include certain features or functions of the proxy agent 118, such as generating highlights and encoding graphic overlays 128 ( Figure 2 ) (e.g., where one or more of the client devices 102 have limited processing power). In addition, in some examples, the streaming applications 120 may enable the generation of highlights, storage of highlights, analysis of the game stream for determining when highlights should be generated, sharing of highlights, etc. In some embodiments, the one or more streaming devices 104 may include any number of streaming applications 120 among which the features and functionality of the highlight generation system 100 are distributed. The features and functionality of the one or more streaming devices 104 may be described herein with reference to Figure 1B and Figure 1C Describe in detail.

[0029] In some embodiments, one or more streaming media devices 104 can render a game instance 132 and transmit the game stream as multiple game instances to two or more devices. In these embodiments, the two or more game instances can be copies of the rendered game instance. In these embodiments, sending two game instances via one or more networks 108 facilitates simultaneous display of the game instances on one or more client devices 102 and / or capturing at least a portion of the game stream on one or more auxiliary devices 106.

[0030] One or more streaming devices 104 may include one or more components (e.g., communication components, network interfaces, etc.) and features for communicating over one or more networks, such as one or more networks 108. As a non-limiting example, in order to communicate within the highlight generation system 100, one or more streaming devices 104 may send and receive data over the Internet to communicate with one or more auxiliary devices 106, one or more client devices 102, and / or one or more other streaming devices 104.

[0031] One or more streaming devices 104 may include one or more data stores 116, similar to the one or more data stores 116 of one or more client devices 102 and / or one or more auxiliary devices 106 described herein. For example, based on the task assignment by the proxy agent 118, one or more streaming devices 104 may be responsible for generating, storing, and / or sharing highlights, snapshots, and / or records of a game stream.

[0032] One or more auxiliary devices 106 may include a game analysis application 124 for receiving a game stream from one or more streaming devices 104 and / or one or more client devices 102 (e.g., via one or more streaming devices 104) and analyzing the game stream for events and / or actions in the game instance 132. The events and / or actions may be recordable events that, when detected, result in the generation of highlights, snapshots, and / or recordings by one or more auxiliary devices 106, one or more client devices 102, and / or one or more streaming devices 104. Although illustrated in the non-limiting example as being performed by the game analysis application 124, the analysis of the game stream may be included as a task of the proxy agent 118. Thus, in some examples, the game analytics application 124 of one or more auxiliary devices 106 may include some features or functions of the proxy agent 118, such as game stream analysis, determining when to generate highlights, and / or causing highlights to be generated (e.g., by generating a signal indicating that a highlight should be generated and sending the signal to one or more client devices 102 and / or one or more streaming devices 104). Features and functions of one or more auxiliary devices 106 may be referred to herein as Figure 1B and Figure 1C Describe in more detail.

[0033] In some embodiments, one or more auxiliary devices 106 may be a third-party service or device for recording and / or live streaming a game stream. In some embodiments, one or more auxiliary devices 106 may be a stream viewing device configured to receive and send at least a portion of a game stream to multiple viewing devices. For example, in some embodiments, one or more auxiliary devices may receive a game stream from one or more streaming devices 104 and redistribute the game stream to multiple viewers (e.g., "live streaming"). In some embodiments, similar to other embodiments disclosed herein, streaming to one or more auxiliary devices 106 (such as a third-party service or device) may reduce processing resources and network resources of one or more client devices 102 compared to conventional game streaming systems.

[0034] The game analysis application 124 can analyze the game stream to determine that a recording event has occurred and signal the proxy agent 118 to capture (or save from the buffer) at least a portion of the game stream. The game analysis application 124 can determine that a recording event has occurred based on artificial intelligence, computer vision, text recognition, and / or other programs and analysis methods. The recording event can be any instance indicated by the game as an important event. Important events indicated by the game may include eliminating another character in the game, collecting certain items, scoring, hitting a home run, climbing a tall building or mountain, performing or achieving a user-specified task or goal, and / or other event types. For example, in some embodiments, the game analysis application 124 can identify changes in icons (reticles) in the game session 126 that indicate the elimination of characters in the game and send a signal to the proxy agent 118 to capture at least a portion of the game stream. In another example, the game analysis application 124 can identify text in the game instance 126 that indicates "Player 1 eliminates player 4" or "Player 1 scores a touchdown" (e.g., using optical character recognition (OCR)) and send a signal to the proxy agent 118 to capture at least a portion of the game stream.

[0035] Furthermore, while the game analysis application 124 is listed as a feature of one or more auxiliary devices 106, this is not intended to be limiting. For example, in some embodiments, one or more auxiliary devices 106 may not be included in the highlight generation system 100. Thus, the game analysis application 124 and / or its features and functions may be executed on one or more client devices 102, one or more streaming devices 104, and / or a combination thereof. For example, similar to other features and functions of the proxy agent 118, analysis of the game stream may be performed by one or more other devices of the highlight generation system 100.

[0036] One or more auxiliary devices 106 may include one or more components (e.g., communication components, network interfaces, etc.) and features for communicating over one or more networks, such as one or more networks 108. As a non-limiting example, in order to communicate within highlight generation system 100, one or more auxiliary devices 106 may send and receive data over the Internet to communicate with one or more streaming devices 104, one or more client devices 102, and / or one or more other auxiliary devices 106.

[0037] One or more auxiliary devices 106 may include one or more data stores 116, similar to the one or more data stores 116 of one or more client devices 102 and / or one or more streaming devices 104 described herein. For example, based on the task assignment by the proxy agent 118, one or more auxiliary devices 106 may be responsible for generating, storing, and / or sharing highlights, snapshots, and / or recordings of a game stream.

[0038] The proxy agent 118 may include various functions for generating highlights within the system 100. As described above, the proxy agent 118 may be used to overcome many limitations of other systems by transferring or allocating resources used by one or more client devices 102, one or more auxiliary devices 106, and / or one or more streaming devices 104 in the highlight generation system 100 so as to effectively and efficiently perform the various functions of generating highlights. Figure 1A 104, but this is not intended to be limiting. For example, the tasks of the proxy agent 118 may be performed on one or more client devices 102, one or more auxiliary devices 106, one or more streaming devices 104, and / or a combination thereof. For a non-limiting example, and with respect to the functionality of the proxy agent 118 in the system 100, one or more client devices 102 may capture at least a portion of a game stream as highlights, and one or more streaming devices 104 may analyze the game stream using a game analysis application 124 to provide an indication (e.g., via a message) to one or more client devices 102 to capture highlights. In another non-limiting example, the generation of the graphic overlay 128 may be at one or more streaming devices 104, and capturing at least a portion of the game stream may occur at one or more client devices 102. In another non-limiting example, the generation of the graphic overlay 128 may be at one or more client devices 102, and capturing at least a portion of the game stream may occur at one or more streaming devices 104.

[0039] The proxy agent 118 may be configured to capture at least a portion of a game instance as a highlight, which in some embodiments may be a screenshot, a brief record of a game stream, and / or a large (or even entire) record of a game. The determination of when to capture an instance may be based on an indication of capturing at least a portion of a game stream. The indication may be based on an analysis of the game stream using computer vision, artificial intelligence (e.g., inputting frames of a game session into a neural network trained to detect events, actions, or other recorded events in a game session), OCR, game state analysis, user input (e.g., to a graphics overlay 128), and / or other criteria. The proxy agent 118 may receive the game instance and store portions of the game instance on one or more data stores 116. In some examples, the proxy agent 118 stores at least some of the game stream (e.g., continuously) in a buffer 130 of one or more data stores 116. In such an example, the buffer 130 may store recent portions of the game instance so that highlights, snapshots, and / or records may be generated from the buffer 130. As a non-limiting example, the most recent duration of play in a given game instance (e.g., 30 seconds) may be recorded in buffer 130, and when an indication of highlight generation is received, the record in buffer 130 may be moved or copied to one or more data stores 116 for permanent storage.

[0040] The proxy agent 118 may be configured to generate a graphic overlay 128 for configuring recording settings, triggering highlight generation, setting user-defined recording events for generating highlights, and / or programming other recording functions. Figure 2The graphical overlay 128 may include multiple functions, such as for controlling at least part of the capture aspect of the game stream. Rendering the graphical overlay 128 may require computing resources and / or hardware that some client devices 102 are not equipped with. Therefore, in traditional systems, these client devices 102 cannot generate highlights due to the inability to render the graphical overlay 128. However, the proxy agent 118 can overcome these limitations by transferring the rendering of the graphical overlay 128 between one or more client devices 102, one or more auxiliary devices 106, and / or one or more streaming media devices 104. Therefore, in the case where it is determined that one or more client devices 102 have sufficient resources to generate the graphical overlay 128, then one or more client devices 102 can generate the graphical overlay 128 (e.g., in the client application 114). In other embodiments, for example, when it is determined that one or more client devices 102 do not have sufficient resources to generate the graphical overlay 128, one or more streaming media devices 104 and / or one or more auxiliary devices 106 can generate the graphical overlay 128 and include the graphical overlay in the game stream. In certain embodiments, the graphical overlay 128 may include controls for capturing clips of a game stream, filtering generated highlights, searching generated highlights, runtime status, gallery selection, and / or hotkey management.

[0041] The graphic overlay 128 is generated by an application that is independent of the application that renders the game (e.g., the game application). For example, in the case where one or more client devices 102 render the graphic overlay 128, the client application of one or more client devices 102 (e.g., the client application that performs the proxy agent 118 function at one or more client devices 102) can render and display the graphic overlay 128 on the display 110 (independent of the display of the game stream). Although rendered independently, the graphic overlay 128 can be displayed simultaneously with the game stream (e.g., on a portion of the display, such as the left side, top drop-down, right side, a pop-up window on the display, etc.). In another example, in the case where one or more streaming media devices 104 render the graphic overlay 128, the streaming media application of one or more streaming media devices 104 (e.g., the streaming media application that performs the proxy agent 118 function at one or more streaming media devices 104) can render the graphic overlay 128 (independent of the rendering of the game) and can capture the graphic overlay 128 and the game in the game stream (e.g., the graphic overlay 128 overlaid on the game in the game stream). Because the graphics overlay 128 can be rendered separately, as part of an application that is different from the application that renders the game using the game state data, the graphics overlay 128 may be referred to herein as a supplemental graphics overlay. Thus, the graphics overlay 128 may be displayed to supplement the game-based display information (e.g., the game itself and the game's graphics overlay and GUI). Thus, when a user provides input to the graphics overlay 128, whether included in a game stream from one or more streaming devices 104, or / and rendered within a client application of one or more client devices 102, the input may be sent to the proxy agent 118, rather than being provided to the game itself to affect the game.

[0042] The proxy agent 118 may be configured to send the game stream to other devices and / or third-party services for stream viewing. In some embodiments, the sending of the game stream for stream viewing may occur simultaneously with the game stream, or with a delay in the game stream. The sending of the game stream for stream viewing may occur independently of the proxy agent 118 recording the game stream. For example, the proxy agent 118 may record the game stream at one or more client devices 102 and send the game stream to other devices and / or third-party services for stream viewing. In this example, one or more streaming media devices 104 may stream the game stream to both one or more client devices 102 and one or more auxiliary devices 106 via one or more networks 108.

[0043] Therefore, the proxy agent 118 and its related functions can be distributed between and among various devices within the system 100. However, since the game is being streamed from one or more streaming devices 104, and since the proxy agent 118 can be distributed among different devices within the system 100, it seems that it may be too costly for the game developer to implement the recording function. For example, without using the proxy agent 118, the game developer may need to have different functions, depending on the different allocation of computing resources within the system 100. This function may require the generation of different types of messages and / or application programming interface (API) calls to different devices, components and / or features, depending on the different allocation of computing resources. In order to avoid or reduce this burden on the game developer, the game developer within the system 100 may only have to program the generation of the same message type and / or the same API call type, regardless of the allocation of computing resources in the system 100. In this way, the proxy agent 118 can be configured to receive the same message type and / or the same API call type from the game application being rendered on one or more streaming devices 104, and can determine and / or facilitate the task allocation within the system 100. For example, a user may provide input to a hot key indicating a request to generate highlights, and the game application may generate a message and / or an API call indicating the request. The proxy agent 118 may receive the message and / or API call and determine which devices in the system 100 should perform the different tasks of generating highlights. Therefore, for a game developer or a game application, even if the game is generated and rendered on one or more streaming devices 104, the game generation and rendering may appear to be performed on the local hardware of one or more client devices 102. Therefore, by using the proxy agent 118, the implementation of the system 100 is substantially seamless to the game developer, thereby reducing the implementation burden on the game developer when integrating highlight recording functionality in a game streaming environment.

[0044] Furthermore, because the game is streamed to one or more client devices 102 and the proxy agent 118 can be used to allocate computing resources based on the system information, the requirement for one or more client devices 102 to have any specific hardware and / or software configuration (such as certain operating systems, GPUs, etc.) is eliminated. Therefore, the highlight generation system 100 can be used with any client device 102 regardless of its respective hardware and / or software configuration.

[0045] Reference now Figure 1B . Figure 1B1 is another example system diagram of a highlight generation system 140 using one or more client devices 102 and one or more streaming media devices 104 in a cloud gaming system according to some embodiments of the present disclosure. In addition, the highlight generation system 140 may include one or more client devices 102 (which may include Figure 1A and one or more streaming media devices 104 (which may include components, features and / or functionality similar to one or more client devices 102 of the present invention) ... Figure 1A 104) and the like, both of which use one or more networks 108 (which may be similar to Figure 1A One or more networks 108) communicate with each other.

[0046] In the highlight generation system 140, one or more streaming devices 104 may include a game instance 132. The game instance 132 is shown as being in bilateral communication with or in the following: a camera control application 144, a real-time streaming protocol (RTSP) application 142, and a highlight capture application 164. The game instance 132 may be used for the camera control application 144 and / or the highlight capture application 164, or may be sent to the camera control application 144 and / or the highlight capture application 164. When the game instance 132 is generated, the camera control application 144 and / or the highlight capture application 164 may use data from the game instance 132 to perform various functions, such as, but not limited to, those functions listed herein.

[0047] The camera control application 144 may include in-game photography services. For example, the camera control application 144 may capture in-game shots from any of a variety of perspectives (e.g., 180 degrees, 360 degrees, etc.). The camera control application 144 may enable in-game shots to be captured from any position (e.g., even from a position with a perspective different from the user's current perspective). The camera control application 144 (e.g., NVIDIA's ANSEL) may capture screenshots during gameplay (e.g., using NVIDIA's FREESTYLE) and apply one or more post-processing filters (e.g., to adjust color, saturation, add predefined filters such as sepia, retro, night mode, and / or change other characteristics). Although in Figure 1B 164 is shown separately from the highlight capture application 164, but this is not intended to be limiting. In some examples, some features and functions of the camera control application 144 can be performed by the highlight capture application 164.

[0048] The highlight capture application 164 may include a highlight capture service on one or more streaming devices 104. The highlight capture application 164 may directly interpret the game instance 132's API request to capture game recordings. In some embodiments, the highlight capture application 164 may be a non-streaming single-system platform that may directly interpret the generated game's API request to capture highlights on one or more streaming devices 104. In some embodiments, capturing highlights on one or more streaming devices 104 may be implemented via software (e.g., software representing hardware). In other embodiments, capturing highlights on one or more streaming devices 104 may be implemented via a GPU hardware system (e.g., a SHADOWPLAY system based on NVIDIA's hardware).

[0049] The camera control application 144 may communicate with one or more data stores 116, wherein the data stores 116 may include a buffer 130. The camera control application 144 may further communicate with the buffer 130 in use. In some embodiments, the captured game stream data may be sent to one or more data stores 116 for storage. The one or more data stores 116 may include a database containing game information, graphic overlay information, game name information for marking highlights, and system information (e.g., system information of the client device 102, such as whether certain features or functions are supported).

[0050] The RTSP application 142 may receive the game instance 132 directly and / or through the camera control application 144 and / or the highlight capture application 164, and encode the game instance 132 into a game stream for transmission over one or more networks 108. In some embodiments, the RTSP application 142 may act as a router to facilitate communication between the camera control application 144, the highlight capture application 164, the game instance 132, and the one or more networks 108.

[0051] In some example embodiments, RTSP application 142 may transmit the captured game stream to one or more client devices 102 to display, share, and / or store the captured game stream.

[0052] The highlight generation system 140 may further include one or more client devices 102 having a chromium embedded framework (CEF) client 146 (or other browser embedded framework client type). The CEF client 146 may communicate bidirectionally with the graphics overlay 128 and / or the display 110. The CEF client 146 may further communicate with a shared server 166 and one or more data stores 176 that may include a buffer 180. The CEF client 146 may receive a game stream from an RTSP application 142 via one or more networks 108. The CEF client 146 may generate a graphics overlay 128 to capture highlights and / or perform one or more game streaming functions. The CEF client 146 may receive a game stream from one or more input devices 112 (such as Figure 1A 104. In some embodiments, CEF client 146 may receive input from graphics overlay 128, and instructions may be submitted to (in some embodiments) sharing server 166 and / or one or more streaming devices 104 as appropriate. The game stream received by CEF client 146 may be sent to display 110. CEF client 146 may decode the game stream or act as a decoder for the game stream before the game stream is sent to display 110.

[0053] The CEF client 146 further communicates with a highlight capture application 172. A highlight capture application 172 (e.g., highlight capture application 164) may use software and / or hardware systems to capture highlights on one or more client devices 102. In some embodiments, a game stream may be captured and stored in one or more data stores 176 and / or buffers 180. The highlight capture application 172 may receive decoded game stream data from one or more streaming devices 104 and may provide automatic game highlight generation. The highlight capture application 172 may manipulate the game stream into a form that may be stored, edited, and / or otherwise used by one or more client devices 102 as indicated by a request for highlights. The highlight capture application 172 may generate metadata from the game and associate it with a highlight file.

[0054] The one or more data stores 176 may include a buffer 180 on one or more client devices 102. The one or more data stores 176 and buffer 180 may be similar to the one or more data stores 116 and buffer 130. The one or more data stores 176 and / or buffer 180 may be configured to capture the game stream in temporary or permanent storage, and may include a hard drive, solid state drive, RAM, or other computer storage media.

[0055] The shared server 166 may be a client-side plug-in that is used to parse data between the CEF client 146 and the highlight capture application 172. The data may be parsed and retained as information such as hotkeys and settings between multiple game instances 132 that are saved.

[0056] In some embodiments, the highlight generation system 140 may include other plug-ins 170. Other plug-ins 170 may be additional plug-ins that may be implemented into the highlight generation system 140. Other plug-ins 170 may communicate with a highlight capture application 172, one or more data stores 176, a shared server 166, and / or a CEF client 146. In some embodiments, other plug-ins 170 may be a third-party stream viewing service, a local client software for online chat, and / or post-processing filtering software (e.g., NVIDIA's FREESTYLE). Other plug-ins 170 may enable coloring and filtering of game stream data, may notify other players in the game, or provide system information to one or more client devices 102.

[0057] The highlight generation system 140 may include a muxing application 174. The muxing application 174 may include operating system specific components for generating a highlight file, such as through a standardized interface to mux video data and audio data into a file.

[0058] Although the components of system 140 are Figure 1B 104. However, this is not intended to be limiting. For example, some or all of the components, features, and / or functions of system 140 may be considered components, features, and / or functions of a proxy agent (e.g., proxy agent 118). Therefore, in different embodiments, the locations of components, features, and / or functions in system 140 may be different, as described herein with reference to the proxy agent, for example, depending on system information of one or more client devices 102 and / or one or more streaming media devices 104. In addition, although Figure 1B 104. One or more client devices 102 and one or more streaming devices 104 are illustrated, but this is not intended to be limiting. In some embodiments, one or more auxiliary devices may also be included in the system 140 (e.g., to perform game analysis for automatic highlight generation, e.g., by using a game analysis application as one of the other plug-ins 170).

[0059] Reference now Figure 1C , Figure 1C1 is another example system diagram for generating highlights in a cloud gaming system according to some embodiments of the present disclosure. In a highlight generation system 150 within a game streaming environment, one or more client devices 102 may receive input data in response to input to one or more input devices 112, send the input data to one or more streaming devices 104 and / or one or more auxiliary devices 106, receive an encoded game stream from one or more streaming devices 104, and display the encoded game stream on a display 110. Generate a game instance (e.g., Figure 1A or Figure 1B In some examples, the game session may be streamed from one or more streaming devices 104 to one or more client devices 102 and / or one or more auxiliary devices 106. Offloading the generation of the game may reduce the graphics processing and rendering requirements on one or more client devices 102 and one or more auxiliary devices 106. By removing the task of game generation and rendering from one or more client devices 102, one or more client devices 102 may use its computing resources to capture at least a portion of highlights in the game, store the highlights, generate graphical overlays (e.g., Figure 1A ) and / or perform one or more other tasks to generate highlights (eg, tasks belonging to the proxy agent 118 herein).

[0060] For example, one or more client devices 102 may display frames of a game instance on a display based on receiving a game stream from one or more streaming devices 104. One or more client devices 102 may receive input from one of the one or more input devices 112. The input data may be sent to one or more streaming devices 104 and / or one or more auxiliary devices 106 via one or more networks 108, and then one or more streaming devices 104 and / or one or more auxiliary devices 106 may receive the input data. The input data may be received at communication interface 152. One or more CPUs 154 may receive the input data, process the input data, and send the data to one or more GPUs 156, causing one or more GPUs 156 to generate a game instance (e.g., Figure 1AFor example, in some embodiments, the input data may represent movement of a user's character in the game, firing a weapon, reloading, passing a ball, steering a vehicle, etc. The rendering component 158 ​​may render the game session (e.g., representing the result of the input data), and the proxy agent 118 may capture the rendering of the game instance as display data in one or more data stores 116 and / or buffers 130.

[0061] The encoder 160 can then encode the game instance to generate an encoded game stream, which can be sent to one or more client devices 102 and / or one or more auxiliary devices 106 via the communication interface 152 through one or more networks 108. The one or more client devices 102 can receive the encoded game via the communication interface 182, and the decoder 162 can decode the encoded game to generate a game to be displayed. The one or more client devices 102 can then display the game stream through the display 110.

[0062] In some embodiments, the highlight generation system 150 may include one or more auxiliary devices 106. In some embodiments, the game stream may be simultaneously sent to one or more auxiliary devices 106 and received via the communication interface 184, for example. The communication interface 184 may submit the encoded game stream to the decoder 162, the game analysis application 124, and / or the proxy agent 118. The proxy agent 118 may store at least a portion of the game instance in one or more data stores 116 and / or the buffer 130.

[0063] Reference now Figure 2 , Figure 2 2 is an example screenshot of a graphical overlay for generating and adjusting a game stream and generating highlights according to some embodiments of the present disclosure. The graphical overlay 128 may include recording settings 204, network usage 206, stream settings 208, location 210, auto-recording options 212, and / or other settings. The graphical overlay 128 may also include capture controls 214.

[0064] The capture control 214 may send an indication based on a selection to capture at least a portion of the game stream based on the record settings 204. Various types of captured game streams may be presented in the capture control 214. For example, the capture control 214 may have a selection to record the game (e.g., snapshots, highlights, entire recording, etc.) by signaling the proxy agent to capture at least a portion of the game stream. In operation, the game may be recorded from the moment the proxy agent receives the signal to capture the game stream. In another example, the capture control 214 may have a selection to capture a "replay" (e.g., highlights) by signaling the proxy agent to retain a time interval in the game session in a buffer (e.g., Figure 1AIn other words, as a non-limiting example, a user may select a replay button, and the proxy agent may capture the time interval before and / or after the button selection.

[0065] The capture controls 214 may also have a user-selectable element to stream the game stream. When the user selects the option to stream the game stream, one or more client devices 102 may send the game instance generated by the streaming application 120 to other devices or to a third-party server for transmission to other devices (e.g., one or more auxiliary devices 106, one or more streaming media devices 104, one or more other client devices 102, etc.).

[0066] Record settings 204 can control the proxy agent (e.g., Figure 1A The user may select a recording setting 204 of one of various options from the user selectable elements to record the game. For example, the recording setting 204 may include 720, 1080, or 4K resolution. Recording the game at each of these resolutions requires increasing resources (e.g., network resources or computing resources) and / or ( Figure 1A ) space in one or more data stores 116, so that the user can select one of a number of different recording settings, depending on whether the proxy agent is on a client device, an auxiliary device, or a streaming device (e.g., Figure 1A The game is recorded on one or more client devices 102, one or more auxiliary devices 106, and one or more streaming devices 104).

[0067] In some embodiments, recording settings 204 may include options for selecting one or more client devices, one or more auxiliary devices, and / or one or more streaming media devices (e.g., Figure 1A In this way, the user can select his preferred location for game recording.

[0068] Network usage of stream 206 may include various user selectable elements to control network usage of game streams. In some aspects and certain settings, streaming a game to a client device may use a large amount of data over the network. A user may select usage of game streams in MB / S (as a non-limiting example) to limit client device usage of game streams over the network.

[0069] Stream settings 208 may include various user selectable elements to control the quality of the stream. Each selectable element may send instructions to the streaming application (e.g., Figure 1AThe streaming application 120 of the embodiment of the present invention can be used to adjust the generation of the game instance. The user can choose to generate the game instance at 720p, 1080p, 4K or a custom resolution. In addition, the user can choose to generate the game instance at 60FPS, 120FPS or a custom frame rate.

[0070] In the case where the proxy agent is set to record the game locally on the client device, the user can select the location of the data storage on the client device 210. For example, Figure 2 As shown, the user can select the download folder in the "C" drive.

[0071] In some aspects, the graphic overlay 128 may include an auto-record option 212. When the auto-record option 212 (which may be referred to as "auto highlights") is activated, it may activate a game analysis application (e.g., Figure 1A The game analytics application 124 may be activated to determine that a recording event has occurred and to signal the proxy agent to capture at least a portion of the game stream.

[0072] Now refer to Figure 3-Figure 4 Each block of the methods 300 and 400 described herein includes a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method can also be embodied as computer usable instructions stored on a computer storage medium. The method can be provided by a stand-alone application, a service or a hosted service (standalone or in combination with other hosted services), or a plug-in for another product, to name a few. In addition, for example, with respect to Figure 1A Methods 300 and 400 are described with reference to the bright spot generation system 100. However, these methods (including but not limited to the methods described herein) may additionally or alternatively be performed by any one system or any combination of systems.

[0073] Figure 3 is a flow chart showing a method 300 for generating highlights in a game streaming environment. At block B302 , the method 300 includes rendering a game instance. For example, the game instance 132 may be rendered by one or more streaming devices 104 .

[0074] At block B304, method 300 includes encoding the game instance into a game stream. For example, the game stream may be encoded using various techniques. Encoding the game stream facilitates sending the game instance 132 over one or more networks 108 for generating highlights and / or for display on one or more client devices.

[0075] At block B306 , the method 300 includes sending the game stream to the client device. For example, the game stream may be sent to one or more client devices 102 via one or more networks 108 .

[0076] At block B308, method 300 includes receiving an indication to capture at least a portion of the game stream as a recording. For example, the indication to generate highlights is received in response to user input (e.g., to a hotkey) on one or more client devices 102, in response to analysis of the game stream by game analytics application 124, and / or in response to another action or determination of system 100.

[0077] At block B310, method 300 includes determining system information corresponding to a client device. For example, system information of one or more client devices 102, one or more auxiliary devices 106, and / or one or more streaming media devices 104 may be determined. System information may include, but is not limited to, storage capacity, encoding capabilities, and / or processing capabilities of one or more client devices 102, one or more auxiliary devices 106, and / or one or more streaming media devices 104. In some embodiments, system information may be used to determine a rendering and / or streaming bit rate and / or frame rate (e.g., minimum quality of service (QoS)) achieved by distributing processing tasks of a proxy agent between and among various devices of system 100 (dynamically in some embodiments). In these examples, system information may represent a capability metric based on the capabilities of one or more devices. System information may also include or alternatively include recording settings for game streaming and / or network information of one or more networks 108.

[0078] At block B312, method 300 includes: selecting, based at least in part on system information, from generating a record at a streaming device and sending a signal to a client device to cause the client device to generate a record. For example, system information of one or more client devices 102 may be used to determine whether certain features and / or functions should be performed on one or more client devices 102, one or more streaming devices 104, and / or one or more auxiliary devices 106. In some examples, a minimum capability measurement (e.g., in some embodiments, a threshold) of one or more devices in system 100 may be used to select which device is used to generate a record. In another example, a configuration that implements QoS (measured by rendering and / or streaming bit rate and / or frame rate in a non-limiting example) may be used to select which device is used to generate a record, perform record analysis, share records, store records, and / or perform one or more other tasks of a proxy agent.

[0079] At block B314, the method 300 includes generating a record at one or more streaming devices 104. For example, a record may be generated at one or more streaming devices 104 in the event that one or more client devices 102 do not meet minimum capability measures (and / or QoS) or requirements that allow generation of highlights and / or game streaming to operate efficiently (e.g., low latency and high quality).

[0080] At block B316, the method 300 includes sending a signal to the client device to cause the client device to generate a recording. For example, in the event that one or more client devices 102 meet a minimum capability measurement or requirement that allows highlights to be generated and / or game streaming to operate effectively (e.g., low latency and high quality), a recording may be generated at the one or more client devices 102, and the one or more streaming devices 104 and / or the one or more auxiliary devices 106 may send a signal to the one or more client devices 102 to cause the one or more client devices 102 to generate a recording.

[0081] Reference now Figure 4 , Figure 4 4 is another flow chart illustrating a method 400 for generating highlights in a game streaming environment. At block B402, the method 400 includes receiving a game stream from a streaming device. For example, one or more client devices 102 may receive a game stream from one or more streaming devices 104.

[0082] At block B404 , the method 400 includes displaying a first instance of the game stream on a display of a client device. For example, the first instance of the game stream may be displayed on a display of one or more client devices 102 .

[0083] At block B406 , the method 400 includes storing at least a portion of the second instance of the game stream in a buffer. For example, the second instance of the game stream may be stored in a buffer 130 of one or more client devices 102 .

[0084] At block B408, method 400 includes receiving a request to generate a recording of at least a portion of a second instance of the game stream. For example, an indication to generate a highlight is received in response to user input (e.g., to a hotkey) on one or more client devices 102, in response to analysis of the game stream by game analytics application 124, and / or in response to another action or determination of system 100.

[0085] At block B410, method 400 includes generating a record using at least a portion of the second instance of the game stream in the buffer. For example, at least some of the second instance of the game stream stored in buffer 130 may be copied to generate highlights (e.g., snapshots, records, clips, etc.).

[0086] At block B412 , the method 400 includes storing the recording in a data store on the client device. For example, the recording (eg, the highlight) may be stored in one or more data stores 116 of one or more client devices 102 .

[0087] Figure 5 5 is a block diagram of an example computing environment suitable for implementing some embodiments of the present disclosure. Computing device 500 may include a bus 502 that directly or indirectly couples the following devices: memory 504, one or more central processing units (CPUs) 506, one or more graphics processing units (GPUs) 508, communication interface 510, input / output (I / O) ports 512, input / output components 514, power supply 516, and one or more presentation components 518 (e.g., one or more displays).

[0088] although Figure 5 The various blocks of are shown as being connected via bus 502 with wires, but this is not intended to be limiting and is for clarity only. For example, in some embodiments, presentation component 518 (such as a display device) may be considered an I / O component 514 (e.g., if the display is a touch screen). As another example, one or more CPUs 506 and / or one or more GPUs 508 may include memory (e.g., memory 504 may also represent a storage device in addition to one or more GPUs 508, one or more CPUs 506, and / or other components). In other words, Figure 5 The computing devices in the Term are illustrative only. No distinction is made between categories such as "workstations," "servers," "laptops," "desktops," "tablets," "client devices," "mobile devices," "handheld devices," "gaming consoles," "electronic control units (ECUs)," "virtual reality systems," and / or other device or system types, as all categories are in the Figure 5 is considered within the context of computing devices.

[0089] The bus 502 may represent one or more buses, such as an address bus, a data bus, a control bus, or a combination thereof. The bus 502 may include one or more bus types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or other types of buses.

[0090] Memory 504 may include any of a variety of computer-readable media. Computer-readable media may be any available media that can be accessed by computing device 500. Computer-readable media may include volatile and non-volatile media and removable and non-removable media. For example and without limitation, computer-readable media may include computer storage media and communication media.

[0091] Computer storage media may include volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 504 may store computer-readable instructions (e.g., representing one or more programs and / or one or more program elements), such as an operating system. Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 500. As used herein, computer storage media itself does not contain signals.

[0092] Communication media may contain computer-readable instructions, data structures, applications, and / or other data types in a modulated data signal such as a carrier wave or other transport mechanism, and include any information delivery media. The term "modulated data signal" may refer to a signal that has one or more characteristics set or changed in a manner that encodes information into the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, radio frequency, infrared, and other wireless media. Any combination of the above should also be included within the scope of computer-readable media.

[0093] One or more CPUs 506 may be configured to execute computer-readable instructions to control one or more components of the computing device 500 to perform one or more methods and / or processes described herein. Each of the one or more CPUs 506 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) that are capable of processing numerous software threads simultaneously. Depending on the type of computing device 500 implemented, the one or more CPUs 506 may include any type of processor and may include different types of processors (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 500, the processor may be an ARM processor implemented using reduced instruction set computing (RISC) or an x86 processor implemented using complex instruction set computing (CISC). The computing device 500 may also include one or more CPUs 506 in addition to one or more microprocessors or supplementary coprocessors (e.g., math coprocessors).

[0094] The computing device 500 may use one or more GPUs 508 to render graphics (e.g., 3D graphics). One or more GPUs 508 may include hundreds or thousands of cores that can process hundreds or thousands of software threads simultaneously. One or more GPUs 508 may generate pixel data for output images in response to rendering commands (e.g., rendering commands received from one or more CPUs 506 via a host interface). One or more GPUs 508 may include graphics memory, such as display memory, for storing pixel data. Display memory may be included as part of memory 504. One or more GPUs 508 may include two or more GPUs running in parallel (e.g., by linking). When combined together, each GPU 508 may generate pixel data for different parts of an output image or for different output images (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0095] In examples where computing device 500 does not include GPU 508 , one or more CPUs 506 may be used to render graphics.

[0096] The communication interface 510 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 500 to communicate with other computing devices via an electronic communication network (including wired and / or wireless communications). The communication interface 510 may include components and functionality for enabling communication via any number of different networks, such as wireless networks (e.g., Wi-Fi, Z-wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating via Ethernet), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet, etc.

[0097] I / O ports 512 may enable computing device 500 to be logically coupled to other devices, including I / O components 514, one or more presentation components 518, and / or other components, some of which may be built into (e.g., integrated into) computing device 500. Illustrative I / O components 514 include microphones, mice, keyboards, joysticks, game keyboards, game controllers, satellite dishes, scanners, printers, wireless devices, and the like. I / O components 514 may provide a natural user interface (NUI) for processing user-generated air gestures, voice, or other physiological inputs. In some cases, the input may be sent to appropriate network elements for further processing. The NUI may implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and near the screen, air gestures, head and eye tracking, and touch recognition (described in more detail below) associated with the display of computing device 500. Computing device 500 may include a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, touch screen technology, and combinations of these for gesture detection and recognition. In addition, computing device 500 may include an accelerometer or gyroscope (eg, as part of an inertial measurement unit (IMU)) that enables detection of motion. In some examples, the output of the accelerometer or gyroscope may be used by computing device 500 to render immersive augmented reality or virtual reality.

[0098] The power supply 516 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 516 may provide power to the computing device 500 to enable the components of the computing device 500 to operate.

[0099] The one or more presentation components 518 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The one or more presentation components 518 may receive data from other components (e.g., one or more GPUs 508, one or more CPUs 506, etc.) and output data (e.g., as images, video, sound, etc.).

[0100] The present disclosure may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions (such as applications) executed by a computer or other machine (e.g., a personal data assistant or other handheld device). Generally, an application (including routines, programs, objects, components, data structures, etc.) refers to code that performs a specific task or implements a specific abstract data type. The present disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The present disclosure may also be practiced in a distributed computing environment, where tasks are performed by remote processing devices linked through a communications network.

[0101] As used herein, the statement "and / or" for two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0102] The subject matter of the present disclosure is specifically described herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. Instead, the inventors have contemplated that the claimed subject matter may also be embodied in other ways to include different steps or combinations of steps similar to the steps described in this document in conjunction with other existing or future technologies. In addition, although the terms "step" and / or "box" may be used herein to refer to different elements of the method employed, the terms should not be interpreted as implying any particular order between or among the various steps disclosed herein unless the order of the various steps is explicitly described.

Claims

1. A method for storing highlights in a game stream, comprising: rendering, using the computing device, graphical output corresponding to the game instance; encoding, using the computing device, graphical output corresponding to the game instance into a game stream; using the computing device to transmit the game stream to a client device for display at the client device, the client device being located at a remote location relative to the computing device; receiving an instruction to store at least a portion of the game stream as a highlight; analyzing, using the computing device and based at least on the indication, system information corresponding to the client device; determining, using the computing device and based at least on the analyzing, to cause the client device to store at least a portion of the game stream as a highlight; as well as Using the computing device and based at least on the determination, a signal is sent to the client device to cause the client device to store at least a portion of the game stream as a highlight.

2. The method according to claim 1, wherein: Based at least on the analysis, determining that causing the client device to store at least a portion of the game stream reaches a performance threshold level, Wherein sending the signal is also based at least on determining that causing the client device to store at least a portion of the game stream reaches the performance threshold level.

3. The method of claim 1, wherein the indication is received from the secondary device based at least on analysis associated with the game stream executed by the secondary device indicating recorded events occurring in the game instance. The method of claim 1 , wherein the indication is received from the client device based at least on input received from the client device.

5. The method of claim 1, wherein the system information comprises at least one of the following: storage capacity, encoding capability, processing capability, recording settings, or network information.

6. The method of claim 1, wherein game stream data is stored in a buffer based at least on the system information, and the signal causes the client device to store at least a portion of the game stream from the buffer.

7. The method according to claim 1, further comprising: sending the game stream to a secondary device; as well as A signal is sent to the secondary device to cause the secondary device to store at least a portion of the game stream as a highlight.

8. The method according to claim 1, further comprising: generating, using the computing device, a supplemental graphics overlay independent of said rendering of said game instance based at least on said system information, said supplemental graphics overlay capable of controlling one or more recorded features of said game stream; as well as encoding said supplemental graphics overlay in said game stream, Wherein the game stream includes the supplemental graphics overlay.

9. A system for generating a record of a game instance in a game stream, comprising: One or more processors for: generating the game instance in the game stream; sending the game instance in the game stream to a client device remote from the system; Determining to generate a record associated with the game stream; analyzing system information corresponding to the client device based at least on a determination that a record associated with the game stream is to be generated; Based at least on the analysis of the system information, determining to cause the client device to store at least a portion of the game stream as the record; Based at least on the determination to cause the client device to store at least a portion of the game stream in the recording, a signal is sent to the client device to cause the client device to store at least a portion of the game stream in the recording.

10. The system of claim 9, the one or more processors further configured to send at least a portion of the game instance in the game stream to a plurality of viewing devices.

11. The system according to claim 9, wherein: Determining to generate a record associated with the game stream includes: An instruction to generate the record is received.

12. A method for generating a record of a game instance in a game stream, comprising: initiating a cloud gaming session between a computing device and a client device remote from the computing device, the client device comprising an input device for providing gameplay input to a game during the cloud gaming session; receiving, using the client device, a game stream of the game from the computing device during the cloud gaming session, and rendering the game stream based on at least the gameplay input; causing a first instance of the game stream to be displayed on a display of the client device; causing at least a portion of a second instance of the game stream to be stored in a buffer; using the client device, sending system information corresponding to the client device to the computing device; receiving, using the client device, from the computing device a request to generate a recording of the game stream based at least on the system information; causing, using the client device, the recording to be generated using at least a portion of the second instance of the game stream from the buffer based at least on the request; as well as The record is stored in a data store on the client device.

13. The method according to claim 12, further comprising: receiving a first input from the input device representing a request for a supplemental graphical overlay for a display showing the first instance of the game stream, the supplemental graphical overlay being associated with a recording generation feature in the cloud gaming session; as well as Based at least on the request, the supplemental graphical overlay is caused to be displayed on a display within a client application that is separate from a gaming application that displays the first instance of the gaming stream on the display.

14. The method according to claim 12, further comprising: receiving a first input from the input device representing a request for a supplemental graphical overlay for a display showing the first instance of the game stream, the supplemental graphical overlay being associated with a recording generation feature within the cloud gaming session; sending an indication of a second request to the computing device; The game stream continues to be received from the computing device, the game stream including the supplemental graphical overlay for at least a period of time after sending the indication of the second request, the computing device rendering the supplemental graphical overlay independently of the game rendering the game stream.

15. The method according to claim 12, further comprising: A supplemental graphical overlay is caused to be overlaid on a display of the first instance of the game stream, the graphical overlay comprising one or more selectable interface elements associated with the generated recording.

16. The method according to claim 15, further comprising: receiving a selection of one of the one or more selectable interface elements, Wherein receiving the request for generating a record of the game stream is also based at least on the selection.

Images