Automatically generated extended activity and event summaries of your gameplay sessions

Abstract

To provide automatically generated expansion activity summaries and expansion event summaries of a game play session.SOLUTION: A game summary is created by using an event log and game content of an event within a game on the basis of game data related to a game play session. The event log shows event time within a game, and association between events within a game and game content items taken into the events within a game. The game summary provides characteristics, such as time lines for transmitting relative timing of events within a game, a list of events within a game, a map of virtual environment of a game annotated regarding time on the basis of events within a game, game status information, statistical information, and performance information, shows a temporal tendency and / or a tendency for covering a game session, and conveys information regarding a selected set of players of a team or the like.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to automatically generated extended activity and event summaries of a gameplay session. [Background technology]

[0002] In today's world, maintaining connections with others through social media, texting, or email has become integral to how modern society functions. In several popular social media applications, users are presented with an interface that displays posts, videos, and other (usually) curated information in the form of an activity feed. Activity feeds can help users gauge the activities in which users within an individual's network are participating. Video games have become increasingly popular as a source of social connection for people to interact with friends or connect online with others who share similar gaming interests. Users often have a friends list that they use to compete against each other during gameplay sessions and to connect socially with other users. However, video games have historically had a limited place in activity feeds, which have typically served as a means to covertly or overtly encourage others to play.

[0003] Some gaming systems offer basic activity feeds for games. Users can manually capture screenshots or clips of their gameplay sessions using the gaming operating system (e.g., on a gaming console) and share them with other users within their gaming social circles. These posts may be accompanied by a brief description of the captured gameplay, but typically do not provide the viewer with much context for understanding what happened in the gameplay session. In some cases, achievements unlocked by a player during gameplay may trigger corresponding posts to the player's social feed. However, leveraging these features requires game developers to specifically code social feed interactions within the game, something not all developers have the time and resources to do. Furthermore, these features are implemented using the game system's application programming interface (API), and posting capabilities are limited to those supported by the game engine and gaming platform. Furthermore, on traditional gaming platforms that include activity feeds, social feed posts related to a gameplay session are no longer available once the session ends. Additionally, determining whether a particular section of video or video clip contains interesting content (e.g., more action) is important in determining which videos may be worth including in an activity feed, so that "interesting content" is automatically captured for later review and sharing. Summary of the Invention [Means for solving the problem]

[0004] The present disclosure relates to automatically generating and updating enhanced summaries of gameplay session activities and events. In embodiments, the game summaries can be generated from metadata extracted from game data of a gameplay session, including in-game events. Systems and methods are disclosed that use metadata to generate game summaries that collect and convey in-game events that occurred within one or more gameplay sessions, allowing viewers to access associated screenshots, video clips, and / or other game content to better understand one or more gameplay sessions.

[0005] In contrast to conventional systems, a game summary may be created using an event log of in-game events and corresponding screenshots, video clips, and / or other game content, each of which may be automatically generated based on analysis of video data, user input data, game data, data sent by the game using an application programming interface (API), and / or other data associated with the gameplay session. The event log may comprise metadata indicating the time of the in-game events within the gameplay session and the association between the in-game events and the game content items that captured the in-game events. In some embodiments, using the disclosed techniques, a user may interact with in-game events with a temporal context, thereby enabling more informed selection and a better understanding of the gameplay session. In various embodiments, the event log may be used to generate an enhanced game summary that provides features such as a timeline conveying the relative timing of in-game events, a list (e.g., thumbnails) of in-game events, a map of the game's virtual environment annotated with time based on in-game events, game status information, and statistical and performance information. One or more portions of the game summary may correspond to one or more game sessions and / or players, to show trends over time and / or across game sessions, and to convey information about a selected set of players, such as a team.

[0006] The present system and method for generating an automatic game summary of a game play session is described in detail below with reference to the figures of the accompanying drawings. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 illustrates an exemplary system diagram of a game summary system, according to some embodiments of the present disclosure. [Figure 2]FIG. 10 illustrates an example game summary of in-game events, including a map of the virtual environment in which one or more of the in-game events occurred, according to some embodiments of the present disclosure. [Figure 3] FIG. 10 illustrates an example game summary of in-game events, including a gallery of game content incorporating one or more in-game events, according to some embodiments of the present disclosure. [Figure 4] A diagram illustrating examples of additional information that may be included in a game summary of an in-game event, according to some embodiments of the present disclosure. [Figure 5] FIG. 1 is a flow diagram illustrating a method for presenting a game summary including a timeline associated with in-game events, according to some embodiments of the present disclosure. [Figure 6] FIG. 1 is a flow diagram illustrating a method for presenting a game summary including a list associated with in-game events, according to some embodiments of the present disclosure. [Figure 7] FIG. 1 is a flow diagram illustrating a method for presenting a game summary including a map of a virtual environment associated with an in-game event, according to some embodiments of the present disclosure. [Figure 8] FIG. 1 is an exemplary system diagram of a game streaming system, according to some embodiments of the present disclosure. [Figure 9] FIG. 1 is a block diagram of an exemplary computing environment suitable for use in implementing some embodiments of the present disclosure. [Figure 10] 1 is an exemplary data center suitable for use in implementing some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0008] The present disclosure relates to automatically generated game summaries of gameplay sessions. In embodiments, the game summaries may be generated from metadata extracted from game data of the gameplay sessions, including in-game events. Systems and methods are disclosed that use metadata to generate game summaries that convey in-game events that occurred within one or more gameplay sessions, allowing viewers to access, view, and curate (e.g., publish within a social network) associated screenshots, video clips, and / or other game content to better understand one or more gameplay sessions.

[0009] A game summary may be created using an event log of in-game events and corresponding screenshots, video clips, and / or other game content, each of which may be generated based on an analysis of video data, user input data, game data, data transmitted by the game using an application programming interface (API), and / or other data associated with the gameplay session. The event log may include metadata indicating the time of the in-game event within the gameplay session and the association between the in-game event and the game content item that captured the in-game event. In some embodiments, using the disclosed techniques, a user may interact with in-game events with a temporal context, thereby enabling more informed choices and a better understanding of the gameplay session. One or more portions of the game summary may be displayed in and / or accessible from the user's activity feed, providing a more robust view of the gamer's activity. In one or more embodiments, one or more portions of the game summary may correspond to one or more game sessions and / or players, to show trends over time and / or across game sessions, and to convey information about a selected set of players, such as a team.

[0010] In at least one embodiment, using an event log, a game summary may provide an interface in which interface elements (e.g., including thumbnails) corresponding to or otherwise illustrating in-game events are displayed. In some embodiments, a timeline may also be displayed to convey the relative timing of in-game events. In at least one embodiment, the interface elements may form a list of in-game events and may be displayed in temporal (e.g., chronological) order using metadata. For example, each interface element may correspond to a thumbnail of a video clip or screenshot in which one or more in-game events occurred and may be displayed in chronological order (e.g., in an image carousel) using metadata. In at least one embodiment, a user may select an interface element, which may cause an associated game content item capturing the in-game event to load into the user interface.

[0011] In embodiments including a timeline, based on the selection, the timeline may also be updated to indicate the in-game event. By way of example, the updating may include visually highlighting an icon or other indicator on the timeline that represents at least the in-game event. Additionally or alternatively, the event log may organize the in-game events by round or match of a gameplay session. When a user selects an interface element, the timeline may be updated to correspond to the round that includes the in-game event (e.g., to display the round on the timeline and / or to adapt the timeline to the in-game events of the round). Game status information displayed in the user interface may also be updated to correspond to the in-game event. For example, the scores of one or more players may be updated using the event log to reflect the state of the players during gameplay at the time associated with the in-game event.

[0012] In a further aspect, the user interface may include a map of the game's virtual environment corresponding to the gameplay session. Based on the selection of an interface element, the map may be annotated using locations and times associated with in-game events in the metadata. The locations may correspond to the player and / or other in-game objects at the time of the in-game event and / or during or immediately after the in-game event. In various embodiments, the annotating may include displaying and / or updating paths of in-game objects on the map, whose destinations may be based on these locations and times. For example, paths or other indicators of one or more locations of the player, enemies, non-player characters (NPCs), in-game events, items, and / or other elements may be displayed on the map to reflect the state of the game leading up to, during, and / or after the in-game event. In various embodiments, these elements may be displayed using corresponding symbols or icons, and different symbols or icons may be used for each type of element.

[0013] In at least one embodiment, game data associated with a gameplay session may be analyzed to automatically detect in-game events, which may trigger recording and / or saving of gameplay via screenshots, video, and / or other game content items capturing the in-game events and metadata (e.g., timestamps, in-game object positions, scores, player stats, player ability scores, gameplay participants by username, event types, kill / death / assist ratios, etc.). Analysis may be performed during the gameplay session (e.g., by analyzing streams of one or more aspects of the gameplay (e.g., video, user input, etc.), API messages from the game, and / or viewers in a video streaming platform or application) and / or after the gameplay session (e.g., on video files and / or saved replays of the gameplay session, or other game data).

[0014] In various embodiments, in-game events may be detected using an algorithm (e.g., using neural networks, computer vision, etc.) that observes the screen for visual cues. When these visual cues are detected, this may automatically trigger recording and / or saving of gameplay. In at least one embodiment, further analysis of the corresponding video data may additionally or alternatively be triggered to extract one or more elements of metadata, or at least a portion of the metadata may be obtained by detecting the corresponding visual cues. Metadata, screenshots, video clips, maps (e.g., extracted from and / or associated with the game), player icons from gameplay, etc. may be stored in a data entity for use in displaying a game summary.

[0015] Reference is made to FIG. 1, which illustrates an exemplary system diagram of a game summary system ("GSS system") 100 according to some embodiments of the present disclosure. It should be understood that this and other configurations described herein are described by way of example only. Other configurations and elements (e.g., mechanisms, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as separate or distributed components, or in conjunction with other components, in any suitable combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory.

[0016] GSS system 100 may include, among other things, client devices 104(A), 104(B), and 104(C) (collectively referred to herein as "client devices 104"), a game summary server 116, and / or a game server 126. While client devices 104(A), 104(B), and 104(C) are illustrated in FIG. 1, this is not intended to be limiting. In any instance, any number of client devices 104 may be present. GSS system 100 (and its components and / or features) may be implemented using one or more computing devices.

[0017] The components of GSS system 100 can communicate via networks 102. The networks may include a wide area network (WAN) (e.g., the Internet, a public switched telephone network (PSTN), etc.), a local area network (LAN) (e.g., Wi-Fi, ZigBee, Z-Wave, Bluetooth, Bluetooth Low Energy (BLE), Ethernet, etc.), a low-power wide-area network (LPWAN) (e.g., LoRaWAN, Sigfox, etc.), a global navigation satellite system (GNSS) network (e.g., a global positioning system (GPS)), and / or another network type. In any instance, each of the components of GSS system 100 may communicate with one or more of the other components via one or more of networks 102.

[0018] Client devices 104 may include smartphones, laptop computers, tablet computers, desktop computers, wearable devices, gaming consoles, virtual reality systems (e.g., headsets, computers, gaming consoles, remote controls, controllers, and / or other components), streaming devices, smart home devices that may include intelligent personal assistants, and / or other types of devices capable of supporting game play.

[0019] The client device 104 may include an application 106, a display 108, a communication interface 110, an input device 112, a graphical interface manager 130, a game data capture unit 138, and an interest determiner 140. The game summary server 116 may include the graphical interface manager 130, a communication interface 122, and a data store 124. The game summary server 116 may include one or more portions of the graphical interface manager 130, the game data capture unit 138, and / or the interest determiner 140 in addition to or instead of those components included on one or more of the client devices 104. The game server 126 may include a game engine 128, a communication interface 132, and a data store 134.

[0020] While FIG. 1 illustrates only some components and / or features of the client device 104, game summary server 116, and game server 126, this is not intended to be limiting. For example, the client device 104, game summary server 116, and game server 126 may include additional or alternative components. Furthermore, the configuration of components is provided by way of example only and may be highly flexible depending on the implementation of the GSS system 100. For example, one or more of the game servers 126 may be embodied as one or more game summary servers 116 and / or may include at least some of the functionality of the game summary server 116, as described herein. Examples include one or more portions of the graphical interface manager 130, game data retrieval unit 138, or interest determination unit 140. Similarly, at least some of the functionality of game summary server 116 may or may not be included in application 106, or may or may not be separately performed at least in part by client device 104. An example is shown in FIG. 1 , illustrating a graphical interface manager 130, a game data capture unit 138, and an interest determiner 140 on both game summary server 116 and client device 104.

[0021] In overview, application 106 may comprise any of a variety of possible types of software capable of presenting one or more game summaries (e.g., game summary 136) on display 108 of client device 104. In at least one embodiment, application 106 (and / or one or more different applications of client device 104) may comprise a gaming application that facilitates gameplay of a game (e.g., cloud and / or local gaming) on ​​client device 104 via input device 112. Communications interface 110 may include one or more components and features for communicating across one or more networks, such as network 102, to receive and / or transmit data corresponding to one or more game summaries (e.g., metadata, video clips, screenshots, etc.), user input to input device 112, streaming content, etc.

[0022] Input device 112 may include any type of device capable of providing user input to a game, and may include a keyboard, a mouse, a microphone, a touchscreen display, a controller, a remote controller, a headset (e.g., a sensor in a virtual reality headset), and / or other types of input devices.

[0023] The communication interface 110 may include one or more components and features for communicating across one or more networks, such as the network 102. The communication interface 110 may be configured to communicate over any number of the networks 102 described herein. For example, to communicate in the GSS system 100 of FIG. 1, the client devices 104 may access the Internet using an Ethernet connection or a Wi-Fi connection through a router to communicate with the game summary server 116, the game server 126, and / or other client devices 104.

[0024] The graphical interface manager 130 may include any of a variety of computer-readable media. Computer-readable media may be any available media that can be accessed by the client device 104(A). Computer-readable media may include both volatile and nonvolatile media, and both removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media. The graphical interface manager 130 may include additional or alternative components such as those described below with respect to memory 604 of FIG. 6.

[0025] The graphical interface manager 130 may be configured to generate a game summary and / or manage the display of the game summary in a user interface, such as the user interface 150 of the application 106. The game data capture unit 138 may be configured to capture game data from gameplay of the game, such as metadata, video clips, and screenshots. The interest determiner 140 may be configured for use by the game data capture unit 138 to select and / or identify one or more portions of the game data (e.g., game data that may be of interest to a user). In at least one embodiment, one or more portions of the game data capture unit 138, the interest determiner 140, and / or the graphical interface manager 130 may be part of the application 106 (and / or the game engine 128).

[0026] 1 , in one or more embodiments, game summary server 116 may include at least a portion of graphical interface manager 130, game data capture unit 138, and / or interest determiner 140, in addition to or instead of client device 104. Game summary server 116 may be utilized, for example, in a client-server-based implementation of GSS system 100 in which application 106 includes a client application. Communications interface 122 may be configured to communicate over any number of networks 102 described herein, for example, to receive and / or transmit data used to generate one or more game summaries (e.g., video streams of gameplay sessions, recordings of gameplay sessions, multiple recordings, video clips, screenshots, user input to input device 112, streaming content, etc., used to generate metadata and / or generated from data associated with gameplay). The data store 124 may be used to store any of a variety of information associated with the game summary server 116 (e.g., video clips, screenshots, thumbnails, game summaries, metadata, etc.).

[0027] The game engine 128 of the game server 126 may include game functionality that enables a game to be played by one or more users of the client devices 104 over a network. The game engine 128 may be provided, at least in part, on the game server 126 for a cloud-based implementation of the GSS system 100. However, in some embodiments, the game engine 128 may be included, at least in part, in the application 106. The communication interface 132 may include one or more components and features for communicating across one or more networks, such as the network 102. For example, the communication interface 132 may be used to transmit and / or receive user input from the input device 112, video data of game play (e.g., recordings and / or live streams), etc.

[0028] As described herein, application 106 may comprise any of a variety of possible types of software capable of presenting one or more game summaries (e.g., game summary 136 in user interface 150) on display 108 of client device 104. Examples of application 106 include a mobile application, a computer application, a console application, a cloud-based game streaming application, a web browser, a gaming application, a local application, a client application, a social application, a system or native application, and / or another type of application or software.

[0029] In the case of a cloud-based game streaming application, application 106 may include instructions that, when executed by a processor, cause the processor to, without limitation, receive input data representing user input to one or more input devices 112, transmit the input data to game server 126, retrieve game data from memory or local storage, receive game data from game server 126 using communication interface 110, and display the game on display 108. More generally, in some embodiments, application 106 may act as a facilitator to enable interaction between a user on client device 104 and an instance of a game.

[0030] The application 106 may additionally or alternatively include instructions that, when executed by the processor, cause the processor to send data (e.g., game data, game summaries, etc.) to and receive data from the game summary server 116. For example, the application 106 may send video recordings generated when games are played on one or more of the client devices 104 to the game play capture server 116, and may receive video clips, metadata, screenshots, game summary data, and / or other data extracted from the game data from the game play capture server 116. Game data, as used herein, may generally refer to data associated with one or more gameplay sessions of one or more games, such as video data, audio data, one or more API messages from one or more games (e.g., including or identifying one or more in-game events), and / or user input data (e.g., from the user input device 112). Additionally or alternatively, the game summary server 116 may receive at least a portion of the game data from the game server 126 .

[0031] In one or more embodiments, client device 104 may render the game using application 106 (e.g., running on client device 104), while in other instances client device 104 may receive display data (e.g., encoded display data as described with respect to FIG. 8 ) and use the display data to display the game (e.g., running on game server 126) on display 108. In some embodiments, a first client device, such as client device 104(A), may render the game, and a second client device, such as client device 104(B), may receive the display data and use the display data to display the game. In instances where the display data is received by a client device (e.g., when client device 104 does not generate the rendering), GSS system 100 may be part of a game streaming system, such as game streaming system 800 of FIG. 8 , described in more detail below.

[0032] Display 108 may include any type of display capable of displaying a game and / or a game summary (e.g., a light-emitting diode display (LED), an organic LED display (OLED), a liquid crystal display (LCD), an active matrix OLED display (AMOLED), a quantum dot display (QDD), a plasma display, an LED / LCD display, and / or another type of display). In some embodiments, display 108 may include two or more displays (e.g., a dual monitor display for computer gaming, a first display for configuring a game and a virtual reality display for playing the game, etc.). In some embodiments, the display is a touchscreen display, such as a touchscreen on a smart phone, tablet computer, laptop computer, etc., and the touchscreen is at least one of the input devices 112 of client device 104.

[0033] As described herein, application 106 of client device 104 may display one or more game summaries, such as game summary 136, in user interface 150 via display 108. To do so, graphical interface manager 130 may generate one or more game summaries and / or manage the display of the game summaries in user interface 150 of application 106. In one or more embodiments, one or more data structures representing the game summaries and / or portions thereof may be generated on the client side and / or the server side. Rendering of the data structures as a user interface (UI) may be handled by an application (e.g., application 106) running on the client device.

[0034] In at least one embodiment, the graphical interface manager 130 may generate a game summary using event log metadata for in-game events and corresponding screenshots, video clips, and / or other game content, each of which may be automatically generated by the game data capture unit 138 based on analysis of game data associated with the gameplay session. The metadata may indicate timing information for in-game events within the gameplay session and an association between the in-game events and the screenshots, video clips, and / or other game content capturing the in-game events. By way of example, and not limitation, the metadata may include timestamps for the in-game events and / or game content corresponding to the in-game events (e.g., start time, end time, time of non-continuous events, etc.). The timestamps may be used to display data corresponding to the in-game events in a temporal context (e.g., via a timeline, thumbnails on a grid, icons on a map, tables of round-by-round statistics, etc.).

[0035] In one or more embodiments, the event log may organize in-game events by round or match of a gameplay session. For example, the metadata may indicate rounds and time ranges or segments corresponding to rounds. Other examples of metadata that may be associated with time, in-game events, and / or rounds include in-game object positions, scores, player status, player ability scores, gameplay participants by username, event type, goals scored or assists scored ratios, and / or other information used to construct game summary 136.

[0036] The game summary 136 displayed in the user interface 150 may take a variety of possible forms and may be displayed in a variety of possible contexts (e.g., with an activity feed, a social feed, a game application, a game streaming application, etc.). By way of example, and not limitation, the game summary 136 of FIG. 1 includes a list of in-game events 155, visual indicators and / or interface elements, a game content display area 142, a timeline 190, a round display area 192, a game name 152 for the game corresponding to the game summary, and a time 154 for which the game was played in one or more game sessions corresponding to the game summary 136. As described herein, the game summary 136 may include less than all of these characteristics and / or different characteristics, or a combination thereof.

[0037] List 155 may correspond to screenshots, video clips, and / or other game content captured (e.g., by game data capture unit 138) from a game session. For example, list 155 includes entries 156, 158, 160, 162, and 164 that are displayed with visual indicators that may include images (e.g., thumbnails) of the corresponding screenshots and / or video clips. List 155 can take various forms, but is shown as a band or carousel. In various embodiments, list 155 may take the form of a carousel, a gallery, a rotating banner, a sliding banner, a vertical list, a horizontal list, a grid, and / or a scrolling slider.

[0038] Entries in list 155 may correspond to one or more in-game events of a gameplay session, and game content (e.g., video clips and / or screenshots) associated with the entries may capture or otherwise correspond to one or more portions of the corresponding in-game events. Graphical interface manager 130 may use the metadata to display the entries in list 155 in an order corresponding to the times at which the in-game events occurred within the gameplay session. For example, time stamps may be used to display the entries chronologically according to the corresponding in-game events.

[0039] Video clips, screenshots, and / or other game content capture highlights from gameplay sessions and are available for viewing by the user in game summary 136. For example, game summary 136 may include interface elements corresponding to in-game events, such as visual indicators and / or associated checkboxes, selection areas, and / or buttons (e.g., in or associated with list 155). A user may select one or more of the interface elements corresponding to one or more in-game events, which causes the corresponding game content to be displayed, played, and / or loaded in game content display area 142 of game summary 136.

[0040] For example, upon selection of entry 158, as shown in FIG. 1 , entry 158 may be emphasized (e.g., highlighted, visually distinguished, etc.) to indicate that the corresponding in-game event and / or game content has been selected. Also, upon selection of entry 158, game content 166 corresponding to the in-game event, such as a video clip, screenshot, and / or audio, may be displayed, played, and / or loaded within game content display area 142 of game summary 136, as shown. If game content 166 includes multiple items, the user may be able to navigate from item to item using arrows or other means shown in game content display area 142. Additionally or alternatively, arrows and / or other interface elements in game content display area 142 may be used to select another entry in list 155, such as an adjacent entry. When another entry is selected (via list 155, arrows, or otherwise), the current entry may be deselected. Upon deselection, the highlighting may be removed from the visual indicator and the corresponding game content may be removed and / or replaced with the newly selected game content in the game content display area 142.

[0041] If other elements of game summary 136 correspond to one or more of the in-game events selected using entries in list 155, those elements may also be updated based on the selection. For example, other elements may be similarly highlighted and / or de-highlighted upon selection. Illustrative examples include entry 168 in timeline 190 and / or entry 170 in round display area 192.

[0042] The game summary 136 may include various supplemental information displayed in association with the entries (e.g., in the list 155, the timeline 190, the round display area 192, etc.) according to the metadata. The supplemental information may assist a user in determining which entry to select. For example, an entry may include an indication of the time and / or duration corresponding to an in-game event within a game session. Illustratively, entry 164 may indicate that it corresponds to a 20-second video clip. As a further example, the supplemental information may include an indication of the event type of the in-game event associated with the entry. For example, entry 164 indicates that the player who is the subject of the game summary 136 scored two consecutive goals. Other event types may correspond to a player conceding a goal, assisting, winning, losing, leveling up, etc. As a further example, the timeline 190 may include symbols such as check marks 178, "x"s 180, and multipliers 144 that symbolically represent the occurrence of one or more particular event types and / or multiple occurrences or variations thereof. Other examples of supplemental information include game status information, such as score information, player information, team information, and / or other information. Illustratively, entry 170 in round display area 192 includes score information corresponding to a round (and one or more in-game events). However, game status information may be presented in association with entries in list 155 and / or timeline 190 as well.

[0043] Timeline 190 may be displayed to convey the relative timing of one or more of the in-game events captured in the event log. Similar to list 155, timeline 190 may indicate the time of an in-game event in relation to the in-game event. Entries in timeline 190 (e.g., entry 168) may be arranged according to associated time stamps. Entries corresponding to in-game events and / or game content automatically extracted by game data capture unit 138 may be visually distinguished from those manually captured by a player and / or user (e.g., a diamond for manual capture and a circle for automatic capture). In at least one embodiment, each entry in timeline 190 may correspond (e.g., respectively) to one or more entries in list 155. For example, entry 168 may correspond to the same in-game event as entry 158 in list 155. In one or more embodiments, the entries in timeline 190 may have a one-to-one correspondence with the entries in list 155.

[0044] In one or more embodiments, each entry in the timeline may correspond to one or more items of game content, similar to the entries in list 155. Additionally, one or more of the entries in timeline 190 may be selectable. For example, upon selection of entry 168, as shown in FIG. 1 , entry 168 may be emphasized (e.g., highlighted, visually distinguished, etc.) to indicate that the corresponding in-game event and / or game content has been selected. Similarly, upon selection of entry 168, game content 166 corresponding to the in-game event, such as (but not limited to) a video clip, screenshot, and / or audio, may be displayed, played, and / or loaded within game content display area 142 of game summary 136, as shown. When another entry is selected, the current entry may be deselected. Deselection may remove the highlighting from the entry's visual indicator and the corresponding game content may be removed and / or replaced with the newly selected game content in game content display area 142.

[0045] Round display area 192 may represent one or more in-game events organized by round. For example, round display area 192 may include entry 170, entry 172, entry 174, and entry 176 corresponding to respective rounds of one or more game play sessions in game summary 136. In one or more embodiments, round display area 192 may be used to select a subset of the in-game events and / or corresponding entries to be displayed in one or more other portions of game summary 136. For example, a user selecting entry 170 may limit list 155 and / or timeline 190 to the events corresponding to entry 170 (e.g., those occurring within that round) and / or cause one or more of those entries to be displayed in those portions of game summary 136, which may include replacing existing entries that do not correspond to entry 170. In one or more embodiments, the time on timeline 190 may be scaled according to the time span of the currently selected round. A start time on timeline 190 may correspond to the beginning of a round in a gameplay session, and an end time on timeline 190 may correspond to the end of a round in a gameplay session.

[0046] In one or more embodiments, more than one round may be selected at a time. Also, in one or more embodiments, deselecting an entry corresponding to a round may remove the entry corresponding to that round from other portions of the game summary 136. In some instances, selecting an entry may automatically deselect one or more currently selected entries. As shown in FIG. 1 , a selected entry may be highlighted in the game summary 136, similar to selected entries in other portions of the game summary 136. Deselecting an entry may similarly de-highlight the entry.

[0047] Other example features that may be included in the game summary 136 are described with respect to Figures 2-4, in addition to or alternatively to those described with respect to Figure 1. Referring now to Figure 2, Figure 2 illustrates an example game summary 136 of in-game events, including a map 204 of a virtual environment in which one or more of the in-game events occurred, according to some embodiments of the present disclosure. By way of example and not limitation, the game summary 136 of Figure 2 includes a list 155, a game content display area 142, and a timeline 190. The game content display area 142 is shown as including playback control elements, such as a play button and a timeline, with which a user can interact to review activities that took place during one or more selected in-game events.

[0048] Map 204 may be dynamically updated to illustrate the movement of one or more objects and / or entities throughout a gameplay session over time, as indicated by the metadata. For example, the locations of one or more players, NPCs, and / or objects may be overlaid on or otherwise used to annotate map 204 to indicate where those objects or entities were located in the virtual environment at a particular time during the gameplay session. The time may be based at least on selected in-game events, game content, and / or other selections made in game summary 136, such as entries in user interface 150. For example, if game content display area 142 corresponds to game content 202 that includes a video clip, as the video clip plays out in-game events, objects and / or entities displayed in map 204 and / or their positions or other attributes (e.g., hit points, item counts, scores, event types, etc.) may be added, removed, or modified in map 204 to reflect the game state at the corresponding time in the video clip. The corresponding information used to update map 204 may be captured in metadata, and one or more updates may occur periodically and / or continuously. Interpolation between metadata values ​​may be used to provide intermediate values ​​for updates. Map 204 may similarly be updated to reflect the game state of other game content in game content display area 142, such as a selected screenshot or audio clip.

[0049] In one or more embodiments, annotating map 204 may include adding, removing, and / or updating one or more paths of one or more entities or objects throughout the virtual environment. For example, a path 208 of a player (e.g., the subject of game summary 136) throughout a game is shown as a dashed line. The endpoint of the path may correspond to the player's location associated with a point 220 during the viewing of a video clip. In other instances, the endpoint may correspond to a timestamp of a screenshot being viewed. View indicator 206 on map 204 may indicate where the player is within the virtual environment during a current time associated with the game content being viewed and / or may correspond to a viewable area within the virtual environment at that time or an area otherwise associated with the player by the game. In one or more embodiments, map 204 may correspond to an in-game map displayed to a player during a gameplay session. Annotations on map 204 may be limited to information known to the player account and / or viewable by the player on the in-game map (e.g., at the corresponding time within the game session), or may include information not known to the player account and / or not viewable by the player (e.g., gathered from other players or spectators).

[0050] Referring now to FIG. 3 , FIG. 3 illustrates an example game summary 136 of in-game events, including a gallery 300 of game content capturing one or more of the in-game events, according to some embodiments of the present disclosure. In one or more embodiments, the gallery 300 may be an example of the list 155 described with respect to FIGS. 1 and 2 . As shown, the game summary 136 may correspond to in-game events and / or game content from two or more games across two or more gaming platforms and / or two or more gameplay sessions. For example, a set of game content 302 is shown as being from game 2, and a set of game content 310 is shown as being from game 3. By way of example, and not limitation, an image 304 is shown to show a map associated with a game session. In one or more embodiments, a user selection of an image 304 may cause the associated game content to be displayed in game content display area 142, which may be included in game summary 136, added to game summary 136 based on the selection, and / or included in a pop-up window based on the user selection. In one or more embodiments, a user selection of an entry may cause the corresponding map 204 to be displayed, as described with respect to FIG. 2.

[0051] Referring now to FIG. 4, FIG. 4 illustrates example additional information that may be included in a game summary 136 of an in-game event, according to some embodiments of the present disclosure. The game summary 136 of FIG. 4 includes a game title and indicator 402, as well as an associated time 406 (e.g., "Today, 1:00 PM"), of several in-game events captured from the corresponding gameplay session. Below, images of five captured events are shown, such as image 404, which may correspond to the list 155 described herein. As with other example entries in list 155, notes may be included with the image or other visual indicator (e.g., two kills, points lost, etc.). These images may serve as visual previews of the in-game events and may include thumbnails of the underlying game content.

[0052] The game summary 136 also includes game status information such as the level 414 of the game played (e.g., level 1, 2, 3, etc.), the mode 416 of the game (e.g., team, capture the flag, free-for-all, etc.), and the time played. The game summary 136 may also include a performance or statistics summary 420, which may include performance and / or statistical information such as the number of eliminations, scores, points, or any other notable occurrences within one or more particular gameplay sessions or collections thereof. Game statistics 422 may also be included, which may describe important statistics such as the score for each round, how many rounds were played, the number of eliminations, or any other game statistics that may be useful to a user in tracking progress during a gameplay session. In an embodiment, the activity feed may also include a list of achievements 424. The achievements 424 will depend on the type of game being played. For example, in a medieval-themed game, achievements 424 may include the number of goals achieved (e.g., towns conquered, enemies defeated, etc.) In addition to these characteristics, the activity feed may also include a map 428 illustrating a map of the path the user took throughout the game route and captured events shown in the activity feed summary 400.

[0053] 1, further examples of how in-game events and / or game content may be captured from game data are provided. As described herein, game data may refer to data associated with one or more gameplay sessions of one or more games, such as video data, audio data, one or more API messages from one or more games (e.g., including or identifying one or more in-game events), and / or user input data (e.g., from user input device 112) generated by and / or associated with one or more games.

[0054] In one or more embodiments, application 106 and / or game data capture unit 138 may include instructions that, when executed, record game data from a game play session and store the recorded game data locally on client device 104 or transmit the recorded game data to game summary server 116 or game server 126 for storage in data stores 124 and 134, respectively. In instances where client device 104 does not generate renderings, game server 126 may record and store game data or transmit game data to game summary server 116 for storage in data store 124.

[0055] As described herein, it may be desirable to identify clips (e.g., video and / or audio) and / or other game content from within the game data that contain interesting content, such as clips during which more user input was provided and / or clips when certain important events occurred within a gameplay session, and therefore a game data capture unit 138 may be provided for this purpose.

[0056] As described herein, the game data capture unit 138 may be part of the application 106 or may be part of a separate application (e.g., one or more system services, programs, etc.). The game data capture unit 138 may include instructions that, when executed by a processor, cause the processor to record or log game data, such as, but not limited to, input device usage data, video data, and / or other data associated with a gameplay session. Examples of input device usage data include data describing or representing use of a keyboard, mouse, or other input device associated with one or more of the gameplay sessions. Examples of information that may be recorded include keyboard strokes, mouse clicks, mouse movements, microphone input, video camera input, and / or input to the client device 104 during a gameplay session. The game data capture unit 138 may also store timestamp information along with these inputs to correlate with the timestamp information of the game session video data.

[0057] The game data capture unit 138 may generally employ an interest determiner 140, which may be configured to indicate to the game data capture unit 138 game content that may be of interest to the user. It is contemplated that the interest determiner 140 may automatically identify game content or that a user and / or player may be manually instructed to identify game content (e.g., via one or more user inputs or commands, creating labels, bookmarks, capture events, etc.). In one or more embodiments, the interest determiner 140 may analyze game data of a game play session to detect game content. This analysis may be performed on recorded game data, such as after the game session is completed, and / or may be performed in real time as the game play session is occurring. The interest determiner 140 may be part of the application 106 or may be part of a separate application (e.g., one or more system services, programs, etc.). In some instances, the interest determiner 140 is part of the same application as the game data capture unit 138.

[0058] The interest determiner 140 may use a variety of possible techniques to (e.g., automatically) detect game content and / or corresponding in-game events within the game data, examples of which are described herein. To do so, the interest determiner 140 may analyze any form of game data, such as a game stream, recorded game data, and / or identified game content, to determine and / or detect that an in-game event has occurred. For example, this analysis may be used to trigger a capture event to direct the capture of at least a portion of the game content from the game stream (or saving from a buffer) and / or to classify the detected in-game event and / or one or more of its attributes (e.g., for inclusion in metadata for one or more game summaries 136). The game data capturer 138 may determine that an in-game event has occurred based on artificial intelligence, object detection, computer vision, text recognition, and / or other analytical methods. For example, the interest determiner 140 may utilize any type of machine learning model for detecting in-game events and corresponding game content, such as linear regression, logistic regression, decision trees, support vector machines (SVMs), naive Bayes, k-nearest neighbor (Knn), k-means, random forests, dimensionality reduction algorithms, gradient boosting algorithms, machine learning models using neural networks (e.g., autoencoders, convolutional, recurrent, perceptron, long / short-term memory, Hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid state machines, etc.), and / or other types of machine learning models. In some embodiments, the machine learning model includes a deep convolutional neural network.

[0059] Examples of in-game events that may be detected include defeating another character in the game, collecting a particular item, scoring a goal, hitting a home run, climbing a tall building or mountain, performing or achieving a user-specified task or objective, leveling up, winning a round, losing a round, and / or another event type. For example, in some embodiments, the interest determiner 140 may recognize a change in a reticle or other interface element of the game that indicates an in-game event, such as an in-game defeat of a character, and trigger a capture event. As a further example, the game data capturer 138 may recognize text in a game instance (e.g., using optical character recognition (OCR)) that signifies "player 1 defeated player 4," or "player 1 scored a touchdown," or other in-game event, and trigger one or more capture events. Techniques that rely on object detection may analyze visual data of a game to identify one or more capture events and / or classify game content.

[0060] In one or more embodiments, to identify one or more in-game events, the interest determiner 140 may determine periods of higher levels of activity (e.g., frequency or intensity) within a gameplay session based on the game data. For example, the game data may be used to identify time segments within the gameplay session that include higher levels of activity; these periods or time segments may be referred to as "predicted periods of interest." While the time segments are detected, specific times may be identified using the disclosed techniques (e.g., for screenshots). For example, a time segment within a gameplay session may be identified by the interest determiner 140 as high-action based, at least in part, on the time segment having a high number of keydowns per minute (KPMs), a high rate of action key selections, and / or other input device metrics. Furthermore, these input device metrics may be subject to additional analysis to reduce possible noise when identifying high-action time segments. For example, an interest algorithm may be applied to the input device metrics to convert the metrics into a measurement of motion activity over time.

[0061] Based on the value of the interest algorithm over time, time segments that correlate with potentially high in-game activity may be identified. For example, a time segment having data points (e.g., continuously) above a threshold (e.g., an average) may be identified as potentially correlating with a clip or screenshot worth highlighting. The start of the time segment may be relative to when the motion activity measure exceeds the threshold, and the end of the time segment may be relative to when the motion activity measure falls below the threshold. In this manner, a video clip or screenshot corresponding to the time segment may be identified. For example, the video clip may be stored as a separate file capturing frames spanning the time segment and / or as metadata used to identify the region within the video data of the game session that corresponds to the time segment (e.g., using a start timestamp, an end timestamp, and / or a time duration).

[0062] While user input (e.g., input device metrics) may be used to quantify in-game activity as interest, other types of data may be used in addition to or instead of user input. For example, visual and / or audio metrics of a game session corresponding to in-game camera and / or field of view movement, color changes, audio volume, audio dynamics, and / or audio variability may be used. Also, in some instances, the interest determiner 140 may identify a period of predicted interest, which is then used to generate a video clip or screenshot; in other instances, the interest determiner 140 may analyze a video clip (generated using any suitable technique) to determine whether the video clip is sufficiently interesting, such as using an interest algorithm (e.g., based on determining that the average interest level for the clip is greater than a threshold).

[0063] Once a time segment is identified as a predicted period of interest using any of the techniques described herein, a corresponding video segment can be identified from the video data of the game play session. For example, the interest determiner 140 can send a timestamp corresponding to the predicted period of interest to the application 106, the game server 126, and / or the game play capture server 116. Any combination of these components can then use the timestamp to generate and / or identify individual video clips from the game session video data (e.g., stored in data stores 124 or 134) and timestamp the video clips or other game content in metadata for one or more game summaries 136.

[0064] Game summary server 116 may include one or more servers that store, trim, sort, and / or categorize game content and / or game data from game sessions. While only some components and / or features of game summary server 138 are illustrated in FIG. 1, this is not intended to be limiting. For example, game summary server 166 may include additional or alternative components, such as those described below with respect to computing device 900 of FIG. 9.

[0065] As further illustrated in FIG. 1 , the game summary server 116 may be separate or distinct from the game server 126, although this is not intended to be limiting. In some instances, the game summary server 116 may be the same or similar server as the game server 126 (e.g., running as a task on the game server 126). In some instances, the game summary server 116 may be operated or hosted by a first entity (e.g., a first company), and the game server 126 may be operated or hosted by a second entity (e.g., a second, separate company). In such instances, the second entity may be a game developer, and the first and second entities may share data such that the first entity can identify interesting game content using data received from the second entity. In other instances, the game play capture server 116 and the game server 126 may be operated or hosted by the same entity. In further examples, the GSS system 100 may be implemented entirely on the client device 104, and / or one or more of the components and / or their functionality shown as being included in a server may be implemented at least in part on the client device 104.

[0066] The game summary server 116 may include one or more application programming interfaces (APIs) that enable communication of information (e.g., game data, timestamps, game content selection data, etc.) with the game server 126 or the client device 104. For example, the game summary server 116 may include one or more game APIs that interface with the client device 104 or the game server 126 to receive game data and / or game summary data. As a further example, the game summary server 116 may include one or more APIs that interface with the client device 104 to transmit categorized game content and / or game summary data. While different APIs are described herein, the APIs may be part of a single API, two or more APIs may be combined, different APIs other than those illustratively described herein, or a combination thereof.

[0067] The game server 126 may include one or more servers (e.g., dedicated game servers) for storing, hosting, managing, and, in some instances, rendering games. In some instances, a first game server 126 may be used to create, update, and modify games (e.g., game program code), and a second game server 126 may be used to host games (e.g., dedicated game servers). While only some components and / or features of the game server 126 are illustrated in FIG. 1, this is not intended to be limiting. For example, the game server 126 may include additional or alternative components, such as those described below with respect to the computing device 900 of FIG. 9.

[0068] The game server 126 may include one or more APIs that enable game play by the client devices 104 and / or enable communication of information (e.g., user profiles, game session data, game data, game summaries, etc.) with the game summary server 116 and / or the client devices 104. For example, the game server 126 may include one or more game APIs that interface with the applications 106 of the client devices 104 to enable game play by the client devices 104. As a further example, the game server 126 may include one or more APIs that receive categorized game content and / or other game summary data for transmission to the client devices 104. Although different APIs are described herein, the APIs may be part of a single API, two or more APIs may be combined, different APIs other than those illustratively described herein, or a combination thereof.

[0069] The game server 126 may include a game engine 128. The game engine 128 may include game functionality that enables a game to be played by one or more users over a network. The game engine 128 may include a rendering engine, a sound engine, a physics engine, an animation engine, an artificial intelligence engine, a network engine, a streaming engine, a memory management engine, and / or other components or features. The game engine 128 may be used to generate some or all of the game session data during a game session.

[0070] Referring now to FIG. 5, each block of method 500, as well as other methods described herein, includes computing processes that may be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. These methods may also be embodied as computer-usable instructions stored on a computer storage medium. These methods may be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Furthermore, the methods are described with respect to the system of FIG. 1 by way of example. However, these methods may additionally or alternatively be performed by any one system or any combination of systems, including, but not limited to, those described herein.

[0071] 5 is a flow diagram illustrating a method for presenting a game summary including a timeline associated with in-game events, according to some embodiments of the present disclosure. At block B502, method 500 includes receiving metadata indicating timing information for the in-game events. For example, graphical interface manager 130 and / or application 106 may receive metadata indicating the timing of one or more in-game events within one or more gameplay sessions and an association between the one or more in-game events and one or more video clips capturing the one or more in-game events. The in-game events may be determined based at least on an analysis of video data and / or other game data representing one or more gameplay sessions.

[0072] Method 500 includes, at block B504, using the metadata to present interface elements corresponding to one or more of the in-game events and a timeline indicating timing information associated with one or more of the in-game events. For example, graphical interface manager 130 and / or application 106 may present, in user interface 150 and using the metadata, one or more interface elements corresponding to one or more of the in-game events and a timeline 190 indicating one or more timings associated with one or more of the in-game events.

[0073] Method 500 includes, at block B506, loading game content capturing the in-game events and updating a timeline to show the in-game events based at least on the selection of an interface element. For example, graphical interface manager 130 and / or application 106 may load image data for one of the one or more video clips capturing the one or more in-game events in user interface 150 and update the timeline to show the one or more in-game events based at least on the selection of one of the one or more interface elements corresponding to the one or more in-game events and based at least on an association between the one or more in-game events and the one or more video clips.

[0074] 6, which is a flow diagram illustrating a method 600 for presenting a game summary including a list associated with in-game events, according to some embodiments of the present disclosure. The method 600 includes, at block 602, receiving metadata indicating times of the in-game events. For example, the graphical interface manager 130 and / or the application 106 may access metadata indicating one or more times of one or more in-game events within one or more gameplay sessions and including one or more associations between the one or more in-game events and one or more game content items (e.g., images, videos, etc.) capturing the one or more in-game events.

[0075] At block B604, method 600 includes presenting interface elements corresponding to the in-game events within the list of in-game events using the metadata in an arrangement corresponding to timing information of the in-game events within the gameplay session. For example, graphical interface manager 130 and / or application 106 may present one or more interface elements corresponding to one or more in-game events within list 155 of in-game events using the metadata of user interface 150 in one or more arrangements corresponding to timing information of the one or more gameplay sessions.

[0076] At block B606, method 600 includes presenting game content corresponding to the in-game events based at least on the selection of the interface element. For example, graphical interface manager 130 and / or application 106 may present one or more game content items in a user interface corresponding to the one or more in-game events based at least on one or more selections of one or more interface elements in the list of in-game events and one or more associations between the one or more in-game events and the one or more game content items.

[0077] 7, which is a flow diagram illustrating a method 700 for presenting a game summary including a map of a virtual environment associated with in-game events, according to some embodiments of the present disclosure. The method 700 includes, at block 702, receiving metadata indicating timing information and locations associated with the in-game events. For example, the graphical interface manager 130 and / or the application 106 may receive metadata indicating one or more times and one or more locations associated with in-game events of a gameplay session of the game. The in-game events may be determined based at least on an analysis of game data associated with the gameplay session.

[0078] Method 700 includes, at block B704, using the metadata to present interface elements corresponding to the in-game events along with a map of the virtual environment of the game. For example, graphical interface manager 130 and / or application 106 may use the metadata to present one or more interface elements corresponding to one or more in-game events of the gameplay session along with one or more maps of one or more virtual environments of the game.

[0079] The method includes annotating the map using the position and timing information based at least on the selection of the interface elements at block B706. For example, the graphical interface manager 130 and / or the application 106 may annotate at least one of the one or more maps using the one or more position and timing information associated with the one or more in-game events based at least on the selection of one or more of the interface elements corresponding to the one or more in-game events.

[0080] Referring now to Figure 8, Figure 8 is an example system diagram of a game streaming system 800 according to some embodiments of the present disclosure. Figure 8 includes a game server 802 (which may include similar components, features, and / or functionality as game server 126 of Figure 1 and / or computing device 900 of Figure 9), a client device 804 (which may include similar components, features, and / or functionality as client device 104 of Figure 1 and / or computing device 900 of Figure 9), and a network 805 (which may be similar to network 102 of Figure 1). In some embodiments of the present disclosure, game streaming system 800 may be implemented. The application session may correspond to a game streaming application (e.g., NVIDIA GEFORCE NOW), a remote desktop application, a simulation application (e.g., an autonomous or semi-autonomous vehicle simulation), a Computer Aided Design (CAD) application, a Virtual Reality (VR) and / or Augmented Reality (AR) streaming application, a deep learning application, and / or other application types.

[0081] In the game streaming system 800, for a game session, the client device 804 only needs to receive input data in response to input on an input device, transmit the input data to the game server 802, receive encoded display data from the game server 802, and display the display data on the display 824. As such, more computationally intensive calculations and processing are offloaded to the game server 802 (e.g., rendering of the game session is performed by the GPU of the game server 802). In other words, the game session is streamed from the game server 802 to the client device 804, thereby reducing the graphics processing and rendering requirements of the client device 804.

[0082] For example, with respect to instantiating a game session, the client device 804 may display frames of the game session on the display 824 based on receiving display data from the game server 802. The client device 804 may receive input on one of the input devices and generate input data in response thereto. The client device 804 may transmit the input data to the game server 802 via the communications interface 820 and over the network 806 (e.g., the Internet), which may receive the input data via the communications interface 818. The CPU may receive the input data, process the input data, and transmit the data to the GPU, causing the GPU to generate a rendering of the game session. For example, the input data may represent the movement of a user's character in a game, firing a weapon, reloading, passing a ball, turning a vehicle, etc. A rendering component 812 may render a game session (e.g., representing the results of input data), and a rendering capture component 814 may capture the rendering of the game session as display data (e.g., image data capturing rendered frames of the game session). An encoder 816 may then encode the display data to generate coded display data, which may be transmitted via a communications interface 818 over network 806 to client device 804. Client device 804 may receive the coded display data via communications interface 820, and a decoder 822 may decode the coded display data to generate display data. Client device 804 may then display the display data via a display 824.

[0083] 9 is a block diagram of an example computing device 900 suitable for use in implementing some embodiments of the present disclosure. The computing device 900 may include an interconnected system or bus 902 that directly or indirectly couples the following devices: memory 904, one or more central processing units (CPUs) 906, one or more graphics processing units (GPUs) 908, a communication interface 910, input / output (I / O) ports 912, input / output components 914, a power supply 916, one or more presentation components 918 (e.g., displays), and one or more logic units 920.

[0084] In at least one embodiment, computing device 900 may comprise one or more virtual machines (VMs), and / or any of its components may comprise virtual components (e.g., virtual hardware components). By way of non-limiting example, one or more of GPUs 908 may comprise one or more vGPUs, one or more of CPUs 906 may comprise one or more vCPUs, and / or one or more of logic units 920 may comprise one or more virtual logic units. As such, computing device 900 may include discrete components (e.g., a full GPU dedicated to computing device 900), virtual components (e.g., a portion of a GPU dedicated to computing device 900), or a combination thereof.

[0085] The interconnect system 902 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 902 may include one or more types of buses or links, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect express (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. Illustratively, the CPU 906 may be directly connected to the memory 904. Additionally, the CPU 906 may be directly connected to the GPU 908. Where there are direct or point-to-point connections between components, interconnect system 902 may include PCIe links to effectuate the connections. In these instances, a PCI bus need not be included in computing device 900.

[0086] Interconnect system 902 may represent one or more buses, such as an address bus, a data bus, a control bus, or a combination thereof. Interconnect system 902 may include one or more bus types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus.

[0087] While the various blocks in FIG. 9 are shown connected by lines via interconnection system 902, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 918, such as a display device, may be considered an I / O component 914 (e.g., if the display is a touch screen). In another example, CPU 906 and / or GPU 908 may include memory (e.g., memory 904 may represent storage in addition to memory for GPU 908, CPU 906, and / or other components). In other words, the computing devices in FIG. 9 are for illustrative purposes only. No distinction is made between categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “gaming console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, all intended to be within the scope of the computing devices in FIG. 9.

[0088] Memory 904 may include any of a variety of computer-readable media. Computer-readable media may be any available media that can be accessed by computing device 900. Computer-readable media may include both volatile and nonvolatile media, and both removable and non-removable media. By way of example, and not limitation, computer-readable media may comprise computer storage media and communication media.

[0089] Computer storage media may include both volatile and nonvolatile media, and / or removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 904 may store computer-readable instructions (e.g., representing programs and / or program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium used to store the desired information and that can be accessed by computing device 900. As used herein, computer storage media do not themselves comprise signals.

[0090] Computer storage media may embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transmission mechanism and includes any information delivery media. The term "modulated data signal" may refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable media.

[0091] The CPU 906 may be configured to execute computer-readable instructions to control one or more components of the computing device 900 to perform one or more of the methods and / or processes described herein. The CPUs 906 may each include one or more cores (e.g., 1, 2, 4, 8, 28, 72 cores, etc.) capable of simultaneously processing multiple software threads. The CPUs 906 may include any type of processor, and may include different types of processors depending on the type of computing device 900 implemented (e.g., a processor with fewer cores for a mobile device and a processor with more cores for a server). For example, depending on the type of computing device 900, the processor may be an ARM processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 900 may include one or more CPUs 606 in addition to one or more microprocessors or auxiliary coprocessors, such as math coprocessors.

[0092] In addition to, or instead of, the CPU 906, the GPU 908 may be configured to execute at least a portion of the computer-readable instructions to control one or more components of the computing device 900 to perform one or more of the methods and / or processes described herein. One or more of the GPUs 908 may be integrated GPUs (e.g., with one or more of the CPUs 906) and / or one or more of the GPUs 908 may be discrete GPUs. In an embodiment, one or more of the GPUs 908 may be coprocessors of one or more of the CPUs 906. The GPU 908 may be used by the computing device 900 to render graphics (e.g., 3D graphics) or to perform general-purpose computations. For example, the GPU 908 may be used for general-purpose computing on GPUs (GPGPU). The GPU 908 may include hundreds or thousands of cores capable of simultaneously processing hundreds or thousands of software threads. The GPU 908 may generate pixel data for an output image in response to a rendering command (e.g., a rendering command from the CPU 906 received via a host interface). The GPU 908 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. The display memory may be included as part of the memory 904. The GPU 908 may include two or more GPUs operating in parallel (e.g., via a link). The link may connect the GPUs directly (e.g., using NVLINK) or via a switch (e.g., using NVSwitch). When combined together, each GPU 908 can generate pixel data or GPGPU data for a different portion of the output or for a different output (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.In instances where computing device 900 does not include a GPU 908, CPU 906 may be used to render graphics.

[0093] In addition to, or instead of, the CPU 906 and / or the GPU 908, the logic unit 920 may be configured to execute at least a portion of the computer-readable instructions to control one or more components of the computing device 900 to perform one or more of the methods and / or processes described herein. In an embodiment, the CPU 906, the GPU 908, and / or the logic unit 920 may perform any combination of the methods, processes, and / or portions thereof, either discretely or cooperatively. One or more of the logic units 920 may be part of, and / or integrated into, one or more of the CPUs 906 and / or the GPUs 908, and / or one or more of the logic units 920 may be separate components or otherwise external to the CPU 906 and / or the GPU 908. In an embodiment, one or more of logic units 920 may be a co-processor for one or more of CPUs 906 and / or one or more of GPUs 908.

[0094] Examples of the logic unit 920 include a Data Processing Unit (DPU), a Tensor Core (TC), a Tensor Processing Unit (TPU), a Pixel Visual Core (PVC), a Vision Processing Unit (VPU), a Graphics Processing Cluster (GPC), a Texture Processing Cluster (TPC), a Streaming Multiprocessor (SM), a Tree Traversal Unit (TTU), an Artificial Intelligence Accelerator (AIA), a Deep Learning Accelerator (DLA), an Arithmetic-Logic Unit (ALU), an Application-Specific Integrated Circuit (ASIC), a Floating Point Unit (FPU), a Logic Controller (CCU ... The processor cores may include one or more processing cores and / or their components, such as a PCI Express (Peripheral Component Interconnect) or PCI Express (Peripheral Component Interconnect Express) (PCIe) elements, an I / O element, an I / O element, or a PCI Express (Peripheral Component Interconnect Express) element.

[0095] The communications interface 910 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 900 to communicate with other computing devices over electronic communications networks, including wired and / or wireless communications. The communications interface 910 may include components and functionality that enable communication over any of a number of different networks, such as a wireless network (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), a wired network (e.g., communication over Ethernet or InfiniBand), a low-power wide area network (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 920 and / or communications interface 910 may include one or more data processing units (DPUs) that transmit data received over a network and / or via the interconnect system 902 directly to one or more GPUs 908 (e.g., their memories).

[0096] The I / O ports 912 may enable the computing device 900 to be logically coupled to other devices, including I / O components 914, presentation components 918, and / or other components, some of which may be incorporated (e.g., integrated) into the computing device 900. Illustrative I / O components 914 include a microphone, mouse, keyboard, joystick, game pad, game controller, satellite dish, scanner, printer, wireless device, etc. The I / O components 914 may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some cases, the input may be sent to an appropriate network element for further processing. The NUI may implement any combination of voice recognition, stylus recognition, facial recognition, biometric authentication, on-screen and adjacent-screen gesture recognition, air gestures, head and eye tracking, and touch recognition associated with the display of the computing device 900 (as described in more detail below). The computing device 900 may include depth cameras, such as stereo camera systems, infrared camera systems, RGB camera systems, touch screen technology, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 900 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) to enable motion detection. In some instances, the output of the accelerometer or gyroscope may be used by the computing device 6900 to render immersive augmented or virtual reality experiences.

[0097] The power supply 916 may include a hardwired power source, a battery power source, or a combination thereof. The power supply 916 may provide power to the computing device 900 to enable the components of the computing device 900 to operate.

[0098] The presentation component 918 may include a display (e.g., a monitor, a touch screen, a television screen, a heads-up-display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The presentation component 918 may receive data from other components (e.g., the GPU 608, the CPU 906, etc.) and output the data (e.g., images, video, audio, etc.).

[0099] 10 illustrates an example data center 1000 that may be used in at least one embodiment of the present disclosure. The data center 1000 may include a data center infrastructure layer 1010, a framework layer 1020, a software layer 1030, and / or an application layer 1040.

[0100] 10 , the data center infrastructure layer 1010 may include a resource orchestrator 1012, grouped computing resources 1014, and node computing resources (“node CRs”) 1016(1) through 1016(N), where “N” represents any positive integer. In at least one embodiment, the node CRs 1016(1) through 1016(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state drives or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power supply modules, and / or cooling modules, etc. In some embodiments, one or more of the nodes CR 1016(1)-1016(N) may correspond to a server having one or more of the above-described computing resources. Additionally, in some embodiments, the nodes CR 1016(1)-1016(N) may include one or more virtual components, such as a vGPU, a vCPU, etc., and / or one or more of the nodes CR 1016(1)-1016(N) may correspond to a virtual machine (VM).

[0101] In at least one embodiment, the grouped computing resources 1014 may include distinct groups of node CRs 1016 housed in one or more racks (not shown) or multiple racks (also not shown) housed in data centers in various geographic locations. The distinct groups of node CRs 1016 within the grouped computing resources 1014 may include grouped computing resources, network resources, memory resources, or storage resources that may be configured or assigned to support one or more workloads. In at least one embodiment, several node CRs 1016, including CPUs, GPUs, DPUs, and / or other processors, may be grouped into one or more racks to provide computing resources that support one or more workloads. Additionally, the one or more racks may include any number of power supply modules, cooling modules, and / or network switches, in any combination.

[0102] The resource orchestrator 1012 may configure or otherwise control one or more nodes CR 1016(1)-1016(N) and / or grouped computational resources 1014. In at least one embodiment, the resource orchestrator 1012 may comprise a software design infrastructure (SDI) management entity for the data center 1000. The resource orchestrator 1012 may comprise hardware, software, or some combination thereof.

[0103] In at least one embodiment, as shown in FIG. 10 , framework layer 1020 may include a job scheduler 1032, a configuration manager 1034, a resource manager 1036, and / or a distributed file system 1038. Framework layer 1020 may include a framework to support software 1050 in software layer 1030 and / or one or more applications 1042 in application layer 1040. Software 1050 or applications 1042 may include web-based service software or service applications, respectively, such as those offered by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1020 may be, but is not limited to, a type of free, open-source software web application framework, such as Apache Spark™ (hereinafter, “Spark”), which may utilize distributed file system 1038 for large-scale data processing (e.g., “big data”). In at least one embodiment, the job scheduler 1032 may include a Spark driver to facilitate scheduling of workloads supported by various tiers of the data center 1000. The configuration manager 1034 may be capable of configuring various tiers, such as the software tier 1030 and the framework tier 1020, including Spark and a distributed file system 1038 to support large-scale data processing. The resource manager 1036 may be capable of managing clustered or grouped computing resources that are mapped or allocated to support the distributed file system 1038 and the job scheduler 1032. In at least one embodiment, the clustered or grouped computing resources may include the grouped computing resources 1014 in the data center infrastructure tier 1010. The resource manager 1036 may coordinate with the resource orchestrator 1012 to manage these mapped or allocated computing resources.

[0104] In at least one embodiment, software 1050 included in software layer 1030 may include software used by nodes CR 1016(1)-1016(N), grouped computational resources 1014, and / or at least a portion of distributed file system 1038 of framework layer 1020. The one or more types of software may include, but are not limited to, internet web page searching software, email virus scanning software, database software, and video content streaming software.

[0105] In at least one embodiment, the applications 1042 included in the application layer 1040 may include one or more types of applications used by the nodes CRs 1016(1)-1016(N), the grouped computational resources 1014, and / or at least a portion of the distributed file system 1038 of the framework layer 1020. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive compute, and machine learning applications including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0106] In at least one embodiment, any of configuration manager 1034, resource manager 1036, and resource orchestrator 1012 may perform any number and types of self-modifying operations based on any amount and type of data obtained in any technically feasible manner. The self-modifying operations reduce the burden on data center operators of data center 1000 to make potentially inappropriate configuration decisions and potentially avoid underutilized and / or underperforming portions of the data center.

[0107] Data center 1000 may include tools, services, software, or other resources for training one or more machine learning models or predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, the machine learning models may be trained by calculating weight parameters according to a neural network architecture using software and / or computing resources described above with respect to data center 1000. In at least one embodiment, the trained or deployed machine learning models corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 1000 using weight parameters calculated through one or more training techniques, such as, but not limited to, those described herein.

[0108] In at least one embodiment, data center 1000 may use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using such resources. Additionally, one or more of such software and / or hardware resources may be configured as services to enable users to train or infer information, such as image recognition, speech recognition, or other artificial intelligence services.

[0109] Example Network Environment

[0110] A network environment suitable for use in implementing embodiments of the present disclosure may include one or more client devices, servers, network-attached storage (NAS), other back-end devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be implemented on one or more instances of computing device 900 of FIG. 9 , e.g., each device may include similar components, features, and / or functionality of computing device 900. Furthermore, when implemented as a back-end device (e.g., server, NAS, etc.), the back-end device may be included as part of data center 1000, an example of which is described in more detail herein with respect to FIG. 10.

[0111] Components of a networked environment may communicate with each other via a network, which may be wired, wireless, or both. A network may include multiple networks or one network of multiple networks. By way of example, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks such as the Internet and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. When a network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (as well as other components) may provide wireless connectivity.

[0112] Compatible network environments may include one or more peer-to-peer network environments (wherein the server may not be included in the network environment) and one or more client-server network environments (wherein the server may be included in the network environment). In a peer-to-peer network environment, functionality described herein with respect to a server may be implemented by any number of client devices.

[0113] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include software in the software layer and / or a framework for supporting one or more applications in the application layer. The software or applications may include web-based service software or service applications, respectively. In an embodiment, one or more client devices may use the web-based service software or service applications (e.g., by accessing the service software and / or service applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open-source software web application framework, such as one that may use a distributed file system for large-scale data processing (e.g., “big data”).

[0114] A cloud-based network environment may provide cloud computing and / or cloud storage that performs any combination of the computational and / or data storage functions (or one or more portions thereof) described herein. Any of these various functions may be distributed across multiple locations from a central or core server (e.g., one or more data centers that may be distributed across a state, region, country, or the entire world). If a user (e.g., a client device) is connected to an edge server in relatively close proximity to the edge server, the core server may designate at least a portion of the functionality to be provided by the edge server. A cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0115] A client device may include at least some of the components, features, and functionality of the exemplary computing device 900 described herein with respect to Figure 9. By way of example, and not limitation, a client device may be embodied as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a video camera, a surveillance device or system, a vehicle, a boat, an airship, a virtual machine, a drone, a robot, a portable communications device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, a home appliance, a consumer electronic device, a workstation, an edge device, any combination of these outlined devices, or any other suitable device.

[0116] The present disclosure may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions, such as program modules, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs particular tasks or implements particular abstract data types. The present disclosure may be practiced in a variety of system configurations, including handheld devices, consumer electronics devices, general-purpose computers, and more specialized computing devices. The present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0117] As used herein, the term "and / or" in reference to two or more elements should be interpreted as meaning only one element or a combination of elements. For example, "element A, element B, and / or element C" may include element A only, element B only, element C only, elements A and B, elements A and C, elements B and C, or elements A, B, and C. Also, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Furthermore, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0118] The subject matter of the present disclosure has been described with particularity herein to satisfy statutory requirements. However, the specification itself is not intended to limit the scope of the present disclosure. Rather, the inventors contemplate that the claimed subject matter may be embodied in other ways to include different steps or combinations of steps similar to those described herein, in conjunction with other current or future technologies. Furthermore, while the terms "step" and / or "block" may be used herein to refer to different elements of the method employed, they should not be construed to imply any particular order among the various steps disclosed herein unless and until the order of each step is explicitly described.

Claims

1. A method performed by one or more processing units, comprising: receiving metadata indicating timing information of in-game events within a gameplay session and associations between the in-game events and interface elements corresponding to the in-game events, the in-game events being determined based at least on an analysis of video data representing the gameplay session; using the metadata to present, in a graphical user interface, interface elements corresponding to the in-game events and a timeline showing the timing information associated with the in-game events; annotating a map of the virtual environment in which the in-game event occurred using the timing information associated with the in-game event based at least on the selection of the interface element corresponding to the in-game event; A method comprising:

2. The method of claim 1 , wherein the timeline is updated to show the in-game event based at least on the selection.

3. The method of claim 1 , wherein the selection is from a list of in-game events that uses the metadata to display a plurality of the in-game events in an order corresponding to the time of the in-game events.

4. The method of claim 2 , wherein the metadata associates the in-game event with one of a plurality of rounds of the gameplay session, and the updating of the timeline is based on the round.

5. The method of claim 2 , wherein the updating comprises visually highlighting an indicator of the in-game event in the timeline.

6. 2. The method of claim 1 , wherein the metadata further associates the in-game event with an event type, the selection is from a list of in-game events, and the metadata is used to display the event type with a corresponding one of the in-game events in the list of in-game events.

7. The method of claim 1 , wherein the annotating comprises presenting a path of an in-game object of the game on the map, an endpoint of the path being based at least on the timing information.

8. accessing metadata indicating timing information corresponding to in-game events within a gameplay session and including associations between said in-game events and one or more images capturing said in-game events, said one or more images including at least screenshots; presenting, in a list of in-game events in a user interface using the metadata, interface elements corresponding to the in-game events in a position corresponding to the timing information within the gameplay session; presenting within the user interface one or more images corresponding to the in-game event based at least on the selection of the interface element from the list of in-game events and the association between the in-game event and the one or more images; using the metadata to annotate a map of a virtual environment in which the in-game event occurred based at least on the timing information corresponding to the in-game event. A processor comprising one or more circuits.

9. The processor of claim 8 , wherein the one or more images include a video clip of the gameplay session.

10. The processor of claim 8 , further comprising visually highlighting the interface element of the in-game event in the list of in-game events based at least on the selection.

11. 9. The processor of claim 8, wherein the metadata further associates the in-game event with an event type, the metadata being used to display the interface element for the in-game event with an indicator of the event type.

12. 9. The processor of claim 8, wherein the metadata associates the in-game event with one of a plurality of rounds of the gameplay session, and wherein the selection further updates a timeline of in-game events displayed in the user interface to correspond to the round.

13. 10. The system of claim 8, further comprising using the metadata to update game status information displayed in the user interface to correspond to the in-game event based at least on the selection.

14. The system of claim 8 , wherein the annotating comprises presenting a path of an in-game object of the game on the map, an endpoint of the path being based at least on the timing information.

15. one or more processing units; one or more memory devices storing instructions that, when executed using the one or more processing units, cause the one or more processing units to: receiving metadata indicating timing information and in-game locations associated with in-game events of a gameplay session of a game, the in-game events being determined based at least on an analysis of game data associated with the gameplay session; using the metadata to present interface elements corresponding to in-game events of the gameplay session along with a map of the game's virtual environment; and one or more memory devices configured to execute a method including annotating the map using the location and timing information associated with the in-game event based at least on selection of one or more of the interface elements corresponding to the in-game event; and A system with.

16. The system of claim 15 , wherein the annotating comprises presenting a path of an in-game object of the game on the map, an endpoint of the path being based at least on the timing information.

17. The system of claim 15 , wherein the annotating comprises presenting a placement of in-game objects of the game on the map based at least on the location and the timing information.

18. 16. The system of claim 15, wherein the annotating comprises presenting a location of one or more in-game events of the game on the map based at least on the location and the timing information.

19. 16. The system of claim 15, wherein the annotating comprises presenting a symbol for the in-game event based at least on an event type specified for the in-game event in the metadata.

20. 16. The system of claim 15, wherein the interface elements are displayed in a user interface timeline in an arrangement that corresponds to timing information corresponding to the in-game events within the gameplay session.

Images