Method and system for generating and providing a dynamic summary queue

Abstract

A stream of media items is sent from a server computing system to one or more client computing systems. The media items have a timeline for playing the media items. Participant-generated content is received at the server computing system from one or more participants at any of the one or more client computing systems. The participant-generated content is time-indexed relative to the timeline of the media items. The participant-generated content is processed to identify key portions of the media items based on indications present in the participant-generated content. A dynamic summary queue is generated that includes key portions of the media items. The dynamic summary queue is assembled for streaming. A request is received from a requester to view the dynamic summary queue. In response to the request, a stream of the dynamic summary queue is sent to the requester's computing device.

Description

Technical Field

[0001] 1. Field of Disclosure The present disclosure relates to methods and systems for providing video games and / or media content over a network between a server computing system and a client computing system.

Background Art

[0002] 2. Description of Related Art The video game industry has undergone many changes over the years. With the increase in computing power, video game developers have also developed game software that utilizes this improved computing power. To that end, video game developers have coded games that incorporate advanced operations and mathematics to create a very realistic gaming experience.

[0003] Exemplary game platforms include Sony PlayStation®, Sony PlayStation2® (PS2), Sony PlayStation3® (PS3), Sony PlayStation4® (PS4), and Sony PlayStation5® (PS5), each sold in the form of a game console. As is well known, a game console is designed to be connected to a display (commonly a TV) and enable interaction with the user via a handheld controller. The game console is designed to be provided with dedicated processing hardware, including a CPU, a graphics synthesizer for processing intensive graphic operations, a vector unit for performing geometry transformation, as well as other groups of hardware, firmware, and software. Further, for local play by the game console, the game console may be designed to be provided with an optical disk reader for receiving game disks. Online games are also possible, and users can interactively play against or with other users through the Internet. For example, the PlayStation (registered trademark) Network is an online digital media entertainment service that provides online play of video games, streaming of video content (such as TV, movies, sports events, game play videos, etc.), and streaming of music and other audio content. Video game developers are also developing online games that not only accommodate the expansion of computing power but also enhance the interaction between the user and the game system. For example, interactive online games provide a function for users to interactively play against or with other users on the Internet and obtain such interactions. The present invention arises in this context.

Summary of the Invention

[0004] In an exemplary embodiment, a method for generating a summary of a media item is disclosed. The method includes transmitting a stream of the media item from a server computing system to one or more client computing systems. The media item has a timeline for playing the media item. The method also includes receiving participant-generated content at the server computing system from one or more participants of any of the one or more client computing systems. Participant-generated content is time-indexed with respect to the timeline of media items. The method also includes processing participant-generated content to identify the major portions of media items based on the presentations that exist within the participant-generated content. The method also includes generating a dynamic summary queue that includes the major portions of media items. The dynamic summary queue is assembled for streaming. The method also includes receiving a request from a requester to view the dynamic summary queue. The method also includes transmitting a stream of the dynamic summary queue to the requester's computing device in response to the request.

[0005] In an exemplary embodiment, a system for generating a summary of a video is disclosed. The system includes program instructions stored in a computer memory for transmitting a stream of media items from a server computing system to one or more client computing systems. The media items have a timeline for playing the media items. The system also includes program instructions stored in a computer memory for receiving participant-generated content at the server computing system from one or more participants of any of the one or more client computing systems. The content generated by the participants is time-indexed with respect to the timeline of the media item. The system also includes program instructions stored in computer memory for processing the participant-generated content to identify the main portions of the media item based on the presentations that exist within the participant-generated content. The system also includes program instructions stored in computer memory for generating a dynamic summary queue that includes the main portions of the media item. The dynamic summary queue is assembled for streaming. The system also includes program instructions stored in computer memory for receiving a request from a requester to view the dynamic summary queue. The system also includes program instructions stored in computer memory for transmitting a stream of the dynamic summary queue to the requester's computing device in response to the request.

Brief Description of the Drawings

[0006]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5A

Figure 5B

Figure 5C

Figure 5D

Figure 5E

Figure 5F

Figure 6

Figure 7

DETAILED DESCRIPTION OF THE INVENTION

[0007] In the following detailed description of the invention, some specific details are shown in order to provide a complete understanding of the present disclosure. However, it will be apparent to those skilled in the art that embodiments of the present disclosure can be practiced without some or all of these specific details. In other instances, well-known process operations have not been described in detail so as not to unnecessarily obscure the present disclosure.

[0008] Figure 1 shows a cloud game system 101 according to some embodiments. The cloud game system 101 provides online video game play for one or more users / participants A, B, C, D (A - D). For example, Figure 1 shows users / participants A - D playing an online video game provided by the cloud game system 101 via network 123. Each of the users / participants A - D corresponds to a person 121A, 121B, 121C, and 121D (121A - 121D) who respectively operate corresponding local computing systems 120A, 120B, 120C, and 120D (120A - 120D). Thus, Figure 1 shows an example of group play by multiple users / participants A - D. The example in Figure 1 includes four users / participants A - D for illustrative purposes, but it should be understood that there is no limit to the number of users / participants who can participate in group play and / or spectate an online video game provided by the cloud game system 101. As used herein, the term online video game refers to a video game provided by the cloud game system 101 to a client computing device for a user / participant to play and / or spectate on a client computing device, such as 120A - 120D. The online video games referred to herein can be single - player video games or multiplayer video games. In some embodiments, each of the users / participants A - D is engaged in playing an online video game. In some embodiments, some of the users / participants A - D are engaged in playing an online video game, and some of the users / participants A - D are spectators watching others play an online video game.

[0009] In some embodiments, in addition to providing play of online video games, the cloud game system 101 is configured to provide other media items. For example, in some embodiments, the cloud game system 101 provides audio-visual presentations as media items. In some embodiments, the audio-visual presentations provided by the cloud game system 101 are, among other things, one or more of movies, television shows, real-world sports events, e-sports events, online shows, online videos, and uploaded videos. It should be understood that the cloud game system 101 is operable to provide essentially any type of media item that can be transmitted as a stream of digital data over a network for presentation and consumption (viewing) by users / participants A - D in their respective local computing systems 120A - 120D. In various embodiments, users / participants A - D view media items provided by the cloud game system 101 as a group so that users / participants A - D can participate in a group event. During the group event, the cloud game system 101 enables users / participants A - D to provide participant-generated content so that users / participants A - D can interact with each other and express their feelings regarding the media item being consumed. Again, as shown by way of example in FIG. 1, there is no indication of any limit to the number of users / participants A - D who can participate in the group event. It should be understood that there is no limit to the number of users / participants who can participate in the group events provided by the cloud game system 101.

[0010] In some embodiments, the cloud game system 101 includes a plurality (N) of cloud game servers 103-1 to 103-N. In some embodiments, the number (N) of cloud game servers 103-1 to 103-N can be up to 100 or more. In some embodiments, the cloud game system 101 also includes a plurality (X) of management servers 105-1 to 105-X and one or more storage servers 107. The storage server 107 includes a number of data storage devices 107A such as hard disk drives (HDDs) and / or solid state drives (SSDs). The storage server 107 stores video games available for play in the cloud game system 101. In some embodiments, the storage server 107 stores various media items available for provision by the cloud game system 101. In some embodiments, the management servers 105-1 to 105-X can also access another cloud-based storage system separate from the storage server 107, and the other cloud-based storage system stores video game data and / or media item data. Each of the management servers 105-1 to 105-X is responsible for managing the cloud game sessions of users / participants A to D. Also, in some embodiments, a given management server 105-1 to 105-X is responsible for managing a plurality of cloud game sessions simultaneously. In some embodiments, each management server 105-1 to 105-X, when necessary, powers on one or more cloud game servers 103-1 to 103-N, loads the requested video game from the storage server 107 or other cloud-based storage system to a predetermined one of the cloud game servers 103-1 to 103-N, starts the execution of the requested video game on a predetermined one of the cloud game servers 103-1 to 103-N, logs in users / participants A to D to a predetermined one of the cloud game servers 103-1 to 103-N, and operates to mount user data 111 of the users / participants A to D logging in from the storage server 107 or other cloud-based storage system among other operations.

[0011] In a cloud game (or online game), users / participants A to D operate local computing systems 120A to 120D to log in to a cloud game system 101 on a network 123 such as the Internet and play an online video game remotely executed on one or more of the cloud game servers 103-1 to 103-N. In some embodiments, users / participants A to D log in to the cloud game system 101 via one of the management servers 105-1 to 105-X. However, it should be understood that there are various cloud services related to the login of users / participants A to D. For example, cloud services involved in authenticating users / participants A to D, ensuring that users / participants A to D have a sufficient quality Internet connection, and finding available cloud game servers 103-1 to 103-N near each of the users / participants A to D who can log in.

[0012] In some embodiments, the local computing systems 120A - 120D include corresponding game controllers 125A, 125B, 125C, 125D (125A - 125D) and corresponding display devices 126A, 126B, 126C, 126D (126A - 126D) such as televisions and monitors. In some embodiments, the local computing systems 120A - 120D include corresponding dongles 128A, 128B, 128C, 128D (128A - 128D) connected to the corresponding display devices 126A - 126D. The dongles 128A - 128D are configured to enable data communication between the corresponding game controllers 125A - 125D and the corresponding display devices 126A - 126D so that the network interface controller (NIC) of the display devices 126A - 126D can be used to transmit signals from the game controllers 125A - 125D to the network 123.

[0013] In some embodiments, the local computing systems 120A - 120D also include corresponding local game machines 127A, 127B, 127C, 127D (127A - 127D). In these embodiments, signals generated by the game controllers 125A - 125D can be transmitted to the network 123 via the corresponding local game machines 127A - 127D using the NICs of the corresponding local game machines 127A - 127D. Also, in some embodiments, the game controllers 125A - 125D can directly transmit signals to the network 123 through a local area network such as a WIFI network. Also, in some embodiments, the local computing systems 120A - 120D include corresponding cameras 122A, 122B, 122C, 122D (122A - 122D) for acquiring still images, videos, and / or audio of the users / participants A - D and / or the environment in which the users / participants A - D are present. The content acquired by the cameras 122A - 122D can be transmitted to the network 123 via the corresponding local game machines 127A - 127D using the NICs of the corresponding local game machines 127A - 127D.

[0014] Also, in some embodiments, users / participants A - D can play online video games on local computing devices other than the local computing systems 120A - 120D. For example, in some embodiments, any of users / participants A - D can play online video games on mobile computing devices such as, among others, mobile phones, tablets, laptops, smart car systems, etc. Also, in some embodiments, users / participants A - D can play online video games on a desktop computing system. For the sake of simplicity, references to the local computing systems 120A - 120D in this specification also convey references to any other computing device, in which users / participants A - D can play online video games and / or consume (view) media items provided by the cloud game system 101.

[0015] Furthermore, in various embodiments, data (such as video stream data or audio data or haptic feedback data, etc.) is transmitted from the cloud game system 101 to the local computing systems 120A - 120D via the network 123. In some embodiments, data is transmitted from the cloud game system 101 to the display devices 126A - 126D via the network 123. In some embodiments, data is transmitted from the cloud game system 101 to the local game machines 127A - 127D via the network 123 and also from the local game machines 127A - 127D to the corresponding display devices 126A - 126D. Also, in some embodiments, data is transmitted from the cloud game system 101 to the game controllers 125A - 125D via the network 123.

[0016] During gameplay, users / participants A - D generate game commands that are transmitted via network 123 to cloud game servers 103 - 1 to 103 - N. In some embodiments, users / participants A - D use game controllers 125A - 125D to generate game commands, and the game commands are transmitted in signal form from game controllers 125A - 125D to network 123 and then to cloud game servers 103 - 1 to 103 - N. In some embodiments, the game commands are transmitted in signal form from game controllers 125A - 125D to corresponding local game machines 127A - 127D or corresponding display devices 126A - 126D, from where the game commands are transmitted via network 123 to cloud game servers 103 - 1 to 103 - N. In various embodiments, the transmission of game commands in signal form from game controllers 125A - 125D to network 123 can be performed via a wired network connection, a wireless network connection, or a combination thereof. Cloud game servers 103 - 1 to 103 - N execute a video game according to the game commands received from users / participants A - D and generate video stream(s) that reflect the play of the video game by users / participants A - D. The video stream(s) are transmitted via network 123 from cloud game servers 103 - 1 to 103 - N to the local computing systems 120A - 120D of users / participants A - D for display on corresponding display devices 126A - 126D.

[0017] Cloud game servers 103 - 1 to 103 - N load game data 113 and game executable files 115 from storage server 107 or other cloud - based storage systems. Also, in some embodiments, cloud game servers 103 - 1 to 103 - N load user data 111 from storage server 107 or other cloud - based storage systems to management servers 105 - 1 to 105 - X. In modern video games, a large amount of user data 111 is generated. Depending on the video game and the video game platform (video game console), there can be various types of user data 111 generated for a given user / participant A - D. For example, in some embodiments, user data 111 includes, among other types of data for user / participants A - D, save data for user / participants A - D, download data for user / participants A - D, recording data 111A for user / participants A - D (records of the user / participants A - D's gameplay in the buffer or for the length of time most recently played), trophy data for user / participants A - D, profile data for user / participants A - D, video data for user / participants A - D, audio data for user / participants A - D, pause and resume data for user / participants A - D (acquires the overall state of the online game system at a specific time). Save data 111A is part of user data 111 and is typically a data image of the video game that includes all data regarding the state of the game and the state of the user / participant within the game at a specific save time. Since user / participants A - D play the video game, there may be many save data 111A images created for user / participants A - D. For example, in some embodiments, at certain points during gameplay, such as when transitioning from one level to another or when changing from one scene to another, among many other times / examples when the generation of save data 111A is triggered, the save data 111A for user / participants A - D is generated. Also, in some embodiments, the generation of save data 111A is triggered by a menu selection by user / participants A - D.

[0018] Figure 2 shows an exemplary user interface 200 that is displayed on the display device 126A for user / participant A during the play of an online video game provided by the cloud game system 101, according to some embodiments. The user interface 200 includes a player identification section 201 that indicates which users are currently participating as players or spectators in the online video game. The player identification section 201 indicates that users / participants A - D are currently participating in the online video game, and uniquely marks (e.g., by shading) the identifier of user / participant A to indicate that the user interface 200 is currently provided to user / participant A based on the game play or spectator status of user / participant A. The user interface 200 includes a video display area 203 in which a video of the game corresponding to the current play / viewing of the online game by user / participant A is displayed. The user interface 200 also includes a time display 205 that provides a display of a time mark 205A corresponding to the game video currently displayed in the video display area 203, and the time mark 205A is shown on a time bar 205B that represents the scale of time over the expected duration of the online game event.

[0019] The user interface 200 also includes a communication display area 207 in which various communications provided by users / participants A - D are displayed. In various embodiments, the communication display area 207 is provided to present (display and / or link) essentially any type of user communication (participant - generated content), such as text data, voice data, audio data, emoji data, image data, video data captured by a camera, image data captured by a camera, tag data, and input data from a controller device, among other types of communication data. In some embodiments, each item presented in the communication display area 207 is presented along with an identifier by the users / participants A - D who contributed to the communication. In some embodiments, links to audio, video, and image content are activatable from within the communication display area 207. For example, in some embodiments, selecting the audio link 209 causes the corresponding audio file to be played. Also, in some embodiments, audio communications are automatically transcribed to text messages by the cloud game system 101, and as illustrated by the text message 209A corresponding to the audio link 209, the automatically transcribed text message is displayed in the communication display area 207. The scroll control 211 is provided to move the content displayed in the communication display area 207. In some embodiments, the items presented within the communication display area 207 are time-indexed with respect to the video of the game displayed in the video display area 203. In some embodiments, the user interface 200 also includes a user communication input control 213 that enables the users / participants A - D to provide content for display within the communication display area 207, and the content is a specification of one or more of text, audio, video, emojis, and image data, among other types of content.

[0020] The user interface 200 also includes a tagging control 215 that enables the users / participants A - D to place tags on the video displayed in the video display area 203. In some embodiments, selection of the tagging control 215 initiates a process where the users / participants A - D select a location within the video display area 203 to place the tag. In various embodiments, the tag has some associated content that is, among other things, one or more of text, audio, video, emojis, and image data. The tag is placed at the pixel position selected by the user within the video displayed in the video display area 203. The tag is associated with the pixel position selected by the user. The tag is also time-indexed with respect to the video displayed in the video display area 203 when the tag is placed. In some embodiments, the tag is time-indexed with respect to a set of video frames such that the tag is displayed while the video frame of the set of video frames for which the tag is time-indexed is being displayed. In some embodiments, when the tagging control 215 is selected, the video currently displayed in the video display area 203 is frozen, allowing the users / participants A - D to specify the content of the tag. However, in some embodiments, when the users / participants A - D specify the content of the tag, selecting the tagging control 215 does not freeze the video currently displayed in the video display area 203. Also, in some embodiments, the time display 205 is configured such that the users / participants A - D can select the time mark 205A and move along the time bar 205B to display a desired frame of the video within the video display area 203, to which the tag will be attached. In some embodiments, the insertion of the tag by the users / participants A - D is shown within the communication display area 207. For example, FIG. 2 shows a tag T1 input into the communication display area 207 by the user / participant A. The tag T1 is associated with the pixel position 217 within the video currently displayed in the video display area 203, and the tag T1 contains audio content. In some embodiments, the selection of the tag in the video display area 203 and / or the communication display area 207 triggers the display / playback of the content of the tag. For example, selecting the tag T1 in the video display area 203 plays the audio content of the tag T1.

[0021] In some embodiments, the user interface 200 also includes invitation control 219 that enables users / participants A - D to invite another person into the user interface 200 to participate in an ongoing group play event (multiplayer online game) and / or a group viewing event. In some embodiments, selection of the invitation control 219 enables users / participants A - D to specify an identifier of the person being invited to participate and / or send an email, text message, or other type of message to invite that person to participate. In some embodiments, when a new user / participant joins an ongoing group play event (multiplayer online game) and / or a group viewing event within the user interface 200, the new user / participant may desire to quickly learn about what has happened or is happening by viewing an overview of the group play event (multiplayer online game) and / or the group viewing event. In this regard, the user interface 200 includes overview request control 221, which, when selected, triggers a process to provide an audiovisual overview to the overview requester from a dynamic overview queue generated by the cloud game system 101.

[0022] Figure 3 shows an example of a user interface 300 displayed on the display device 126D for users / participants A - D upon selection of the overview request control 221, according to some embodiments. In some embodiments, the user interface 300 includes a video display area 301 in which a game video corresponding to the current play of an online game by one or more of the other users / participants A - C who did not request an overview is displayed. The user interface 300 also includes a time display 305 that provides a display of a time mark 305A corresponding to the game video currently displayed in the video display area 301, and the time mark 305A is shown on a time bar 305B that represents the scale of time over the expected duration of the online game event.

[0023] The user interface 300 includes a video display area 303 in which an audio-visual summary of a group play event (multiplayer online game) and / or a group viewing event is presented to a requester of the summary from a dynamic summary queue generated by the cloud game system 101. The user interface 300 also includes a time display 307 that provides a display of a time mark 307A corresponding to the summary content currently displayed in the video display area 303, and the time mark 307A is shown on a time bar 307B that represents the time scale for the current content of the dynamic summary queue of the cloud game system 101. In some embodiments, the time mark 307A is draggable along the time bar 307B, enabling navigation by the user through the content of the dynamic summary queue. In some embodiments, the time bar 307B is segmented into segments S1 to S(L), where L is any integer greater than zero. Each of the segments S1 to S(L) corresponds to a consecutive portion of the timeline of the historical content of a group play event (multiplayer online game) and / or a group viewing event. In some embodiments, the user interface 300 includes the video display area 303 and the time display 307, but does not include the video display area 301 and the video display area 305.

[0024] The user interface 300 also includes a communication display area 309 for displaying various past user / participant-provided content related to the dynamic summary queue (summary) content currently or most recently displayed in the video display area 303. In various embodiments, the communication display area 309 is provided to present (display and / or link) essentially any type of user / participant-provided content (participant-generated content), such as text data, voice data, audio data, emoji data, image data, video data acquired by a camera, image data acquired by a camera, tag data, and input data of a controller device, among other types of communication data. In some embodiments, each item of content presented within the communication display area 309 is presented along with an identifier by the users / participants A - D who contributed to the content. In some embodiments, links to audio, video, and image content are activatable from within the communication display area 309. For example, in some embodiments, selecting an audio link 310A causes the corresponding audio file to be played. Also, in some embodiments, audio communications are automatically transcribed to text messages by the cloud game system 101, and as illustrated by the text message 310B corresponding to the audio link 310A, the automatically transcribed text message is displayed in the communication display area 309. Scroll control 311 is provided to navigate the content provided by the user / participant within the communication display area 309. In some embodiments, the user / participant provided content items presented within the communication display area 309 are time-indexed with respect to the content of the dynamic summary queue (abstract) displayed in the video display area 303.

[0025] Also, in some embodiments, the dynamic summary queue can be shared with another user / participant by selecting a shared summary control 315 provided inside the user interface 300. For example, in some embodiments, the selection of the shared summary control 315 triggers an interface that can select an identifier of the person with whom the dynamic summary queue should be shared. In some embodiments, the selection of the shared summary control 315 triggers an interface through which an email and / or message is sent to the recipient with whom the dynamic summary queue is to be shared. In some embodiments, the sharing of the dynamic summary queue is done in a private and / or anonymous manner so that the identities of the users / participants in a group play event (multiplayer online game) and / or group viewing event are kept secret. In some embodiments, the user interface 300 also includes an end summary control 317 that, when selected, closes the user interface 300 and returns the user / participant to another user interface, such as the user interface 200 of FIG. 2.

[0026] By generating a dynamic summary queue in the cloud game system 101 and providing the dynamic summary queue to the requester, a requester who joins an online group event (online video game or presentation of media item) late can quickly and briefly learn about what happened during the online group event. It should also be understood that the content of the dynamic summary queue is selected / generated based on the historical input content provided by other users / participants within the group during the group event. In this way, the content of the dynamic summary queue reflects the past thoughts and actions of other users / participants within the group during the group event. Also, although the examples in FIGS. 1-3 are mainly described in the context of group play of online video games, it should be understood that the principles of generating and providing the dynamic summary queue disclosed in this specification are essentially applicable to the online consumption (viewing) of multi-user / participant of any type of media item as well. Thus, in some embodiments, the media item for which the dynamic summary queue is generated is the game video and audio stream of a video game. Also, in some embodiments, the media item for which the dynamic summary queue is generated is an audiovisual presentation, and the audiovisual presentation is one or more of a movie, a television program, a real-world sports event, an esports event, an online show, an online video, and / or an uploaded video, and any other essentially any type of audiovisual presentation. When explaining the generation and provision of the dynamic summary queue below, the term "media item" is used to refer to the content for which the dynamic summary queue is generated.

[0027] Figure 4 shows a diagram of a system 400 that generates and provides a dynamic summary queue 425 of media items 415, according to some embodiments. When media item 415 is consumed (viewed) by users / participants A - D within an online group event, different users / participants A - D generate respective input channels 401A, 401B, 401C, 401D (401A - 401D) as a function of time. The input channels 401A - 401D are collectively referred to as input channel 401. Input channel 401 includes various types of user / participant-generated content that is provided when users / participants A - D consume media item 415. In various embodiments, the types of user / participant-generated content within input channel 401 can be any other type of data that can be associated with a given user / participant A - D when the given user / participant A - D consumes media item 415, including, among other types of data, text data, voice data, audio data, emoji data, image data, video data acquired by a camera, image data acquired by a camera, tag data, and controller device input data, one or more of which are included. In various embodiments, users / participants A - D within a group can provide essentially any type of input and associate that input with media item 415 and / or a particular function within media item 415, at a particular point in time on the timeline of media item 415. For example, in some embodiments, input channel 401 can include a chat stream where users / participants A - D provide input as a function of time as media item 415 is consumed, and the input generated by the users / participants is time - indexed to the timeline of media item 415. Since input channel 401 is generated by users / participants A - D as they consume media item 415, there is a temporal correlation between each item of content within input channel 401 and the timeline of media item 415. In some embodiments, input channel 401 includes images and / or videos captured by the users' / participants' A - D own cameras, which are analyzed within system 400 to obtain input information such as reactions, facial expressions, gestures, etc., among other types of input information.

[0028] System 400 operates to analyze the content of input channel 401 as a function of time and obtain / determine information about which portions of media item 415 are included in dynamic summary queue 425 as a function of time. Input channel 401 provided by users / participants A - D within the group is analyzed within the context of media item 415 to determine which portions of media item 415 are more relevant than other portions, and these more relevant portions of media item 415 are identified for inclusion in dynamic summary queue 425. Also, system 400 operates to analyze the content itself of media item 415 as a function of time and obtain / determine information about which portions of media item 415 are included in dynamic summary queue 425 as a function of time. Further, in some embodiments, the state data and / or metadata of media item 415 (state / metadata 417) are analyzed as a function of time to obtain / determine information about which portions of media item 415 should be included within dynamic summary queue 425 as a function of time.

[0029] The dynamic summary queue 425 includes portions of the media item 415 that the users / participants A - D within the group considered to be the most relevant based on the analysis of the content of the input channels 401 provided by the users / participants A - D within the group. Thus, the dynamic summary queue 425 of the media item 415 enables new users / participants A - D within the group to quickly and easily get to know by showing portions of the media item 415 that are considered relevant based on the analysis of the input channels 401 provided by other users / participants A - D within the group who participated in the consumption of the media item 415 before the new users / participants A - D arrived in the group. The dynamic summary queue 425 is always generated dynamically. The content of the dynamic summary queue 425 is based on the processing of the content of the input channels 401 provided by the users / participants A - D within the group as a function of time, the content of the input channels 401 can change as a function of time, and the composition of the users / participants A - D within the group is subject to change as a function of time. Therefore, when the content within the input channels 401 provided by the users / participants A - D within the group changes, the content of the dynamic summary queue 425 may change (be dynamic). For example, if the content of the input channels 401 includes changes in the actions, speech, expressions, reactions, emotions, personalities, etc. of the users / participants A - D within the group that occur in the second half of the timeline of the media item 415, that change can be used to update the content of the dynamic summary queue 425 previously determined based on the content of the input channels 401 received earlier in the timeline of the media item 415. Thus, in some embodiments, the dynamic summary queue 425 is dynamic in the sense that the content already placed in the dynamic summary queue 425 can be changed based on information received later in the input channels 401.

[0030] In some embodiments, the input channel 401 is processed through a feature selector 403 to select features from the input channel 401. In some embodiments, the feature selector 403 is an algorithm / program executed on a computing system that automatically analyzes the input channel 401 in real time to select features from the input channel 401 when the data defining the input channel 401 is received by the system 400. The features selected from the input channel 401 are various types of inputs that convey information about how users / participants A - D respond to the media items 415 they consume. In various embodiments, the features include one or more of user / participant - generated content within the input channel 401, such as text data, voice data, audio data, emoji data, image data, video data captured by a camera, image data captured by a camera, tag data, and controller device input data, among essentially any other type of data that can be associated with a given user / participant A - D when the given user / participant A - D consumes a media item 415. In some embodiments, the feature selector 403 operates to remove input items within the input channel 401 that are redundant and / or irrelevant for determining how users / participants A - D responded to the media items 415 they consumed.

[0031] The features selected from the input channel 401 by the feature selector 403 are provided as input features to a feature extractor 405 that operates to extract / derive new features from the input features, and these new features are called the extracted features 407. In some embodiments, the feature extractor 405 is an algorithm / program executed on a computing system that automatically extracts / derives features in real time from the input features received from the feature selector 403. In some embodiments, the extracted features are defined to be information indicating how users / participants A - D reacted to the media items 415 consumed by users / participants A - D. Also, in some embodiments, the extracted features are defined so as not to be redundant. In some embodiments, the extracted features 407 generally represent the real - time dynamic integration of the information received on the input channel 401 from users / participants A - D while consuming the media item 415.

[0032] In some embodiments, the extracted feature 407 is provided as an input to a feature classifier 409 that operates to determine to which of a plurality of categories the extracted feature 407 belongs. In some embodiments, information regarding the category attribute of the extracted feature 407 is added to the extracted feature 407, generating a set 411 of classified features that is provided as an input to a machine learning model 413. The machine learning model 413 is an algorithm / program executed on a computing system that automatically determines which portions of the media item 415 and / or which portions of the information within the input channel 401 are included in the dynamic summary queue 425. In some embodiments, the machine learning model 413 is a requester system that implements a content-based filtering algorithm and / or a collaborative filtering algorithm and / or other approaches for determining which portions of the media item 415 and / or which portions of the information within the input channel 401 are considered to be more relevant by users / participants A - D when consuming the media item 415. In some embodiments, the media item 415 and the state / metadata 417 of the media item 415 are provided as an input to the machine learning model 413.

[0033] Also, in some embodiments, the system 400 includes a context analysis engine 418 implemented as an algorithm / program executed by a computing system that automatically determines / identifies a context related to one or more of the media item 415, the state / metadata 417 of the media item 415, the features selected from the input channel 401 by the feature selector 403, the extracted features 407 output by the feature extractor 405, and the set 411 of classified features output by the feature classifier 409. In various embodiments, the context determined / identified by the context analysis engine 418 for a portion of the given information or a set of information is provided as an input to the machine learning model 413. For example, in some embodiments, the context determined / identified by the context analysis engine 418 for a portion of the given information or a set of information is a description or characterization of a particular part of the media item 415, such as a particular scene within the media item 415.

[0034] It should be understood that the feature selector 403, the feature extractor 405, the feature classifier 409, the context analysis engine 418, and the machine learning model 413 represent an exemplary embodiment of the system 400. In other embodiments, the system 400 is based on input generated by a user / participant provided to the input channel 401 during consumption of the media item 415 by users / participants A - D, and can implement essentially any type of machine learning approach that has the function of identifying which parts of the media item 415 are considered highly relevant to the group of users / participants A - D. Regardless of the specific type of machine learning approach implemented by the system 400, in some embodiments, the machine learning model 413 can be queried to generate content recommendations for inclusion in the dynamic summary queue 425. In some embodiments, the machine learning model 413 receives a summary request 419 that triggers a query of the machine learning model 413 to generate content recommendations for inclusion in the dynamic summary queue 425 based on information generated by the user / participant provided at the input channel 401 during consumption of the media item 415 by users / participants A - D. In some embodiments, the summary request 419 is triggered in response to the selection of the summary request control 221 of FIG. 2.

[0035] System 400 operates to collect / compile dynamic summary queue data 421 corresponding to recommendations of content for inclusion in a dynamic summary queue 425 generated by a machine learning model 413. In some embodiments, the dynamic summary queue data 421 includes video data and / or audio data of various portions of a media item 415. Also, in some embodiments, the dynamic summary queue data 421 further includes a portion of the information received on an input channel 401 corresponding to a portion of the media item 415 included in the dynamic summary queue 425. Also, in some embodiments, the system 400 includes a natural language generator 420 that operates to create computer-generated natural language text and / or audio / verbal summaries of one or more portions of a media item 415 included in the dynamic summary queue 425. In some embodiments, the natural language generator 420 is an algorithm / program executed on a computing system that receives input data from a machine learning model 413 that specifies the information to be summarized and generates a natural language text and / or audio summary of the received input data.

[0036] The dynamic summary queue data 421 is provided to a dynamic summary queue assembler 423 that operates to assemble the dynamic summary queue data 421 into a dynamic summary queue 425. Also, the system 400 is configured to send the content of the dynamic summary queue 425 as a requested summary stream 427 to the requester's computer system that received the summary request 419. In some embodiments, the requested summary stream 427 is a video stream that includes video and audio data. In some embodiments, the system 400 is configured to encode the requested summary stream 427 before sending it to the requester's computer system via the Internet. In various embodiments, the system 400 can implement essentially any technique for sending data via the Internet to send the requested summary stream 427 to the requester's computer system. In some embodiments, the requested summary stream 427 is shown in the video display area 303 of the user interface 300 of FIG. 4.

[0037] The content of the dynamic summary queue 425 at a given point in time is a function of the input provided to the machine learning model 413. In some embodiments, the requester of the dynamic summary queue 425 is provided with the ability to customize the generation of the dynamic summary queue 425. For example, referring to FIG. 3, in some embodiments, the user interface 300 includes a summary option control 313 that, when selected, triggers a display of the user interface that allows the requester of the dynamic summary queue 425 to customize various options for the generation of the dynamic summary queue 425.

[0038] FIG. 5A shows an exemplary user interface 500 that provides various controls to enable a requester of the dynamic summary queue 425 to customize the method of generating the dynamic summary queue 425 according to some embodiments. In some embodiments, when users / participants A to D select the summary option control 313, the user interface 500 is displayed. In some embodiments, the summary option user interface 500 provides one or more of a participant-based control 501, a time frame control 503, a summary breakdown control 505, an event type control 507, and a display type control 509, which are collectively shown as summary options. In some embodiments, the user interface 500 provides a display of the current settings for each of the summary options. Also, in some embodiments, the user interface 500 provides a respective "Change" control for each of the summary options, and when selected, triggers the display of another user interface for changing the current settings of the corresponding summary option. Also, in some embodiments, the user interface 500 provides a respective "Reset" control for each summary option, and when this is selected, the corresponding summary option is reset to the default settings.

[0039] FIG. 5B shows an example of a user interface 511 that is displayed when the "Change" control is selected within the participant-based control 501 of the user interface 500 of FIG. 5A according to some embodiments. The user interface 511 enables the requester of the dynamic summary queue 425 (summary: recap) to select which of the users / participants A to D within the group to use as an input source for generating the dynamic summary queue 425. In some embodiments, the user interface 511 includes a change control 513 for each of the users / participants A to D, and when this is selected, toggles the current participant-based setting of the users / participants A to D between "include" and "exclude". If a given user / participant A - D has the current participant base setting of "include", the input channels 401 associated with the given user / participant A - D will be used by the system 400 when generating the dynamic summary queue 425. Conversely, if a given user / participant A - D has the current participant base setting of "exclude", the input channels 401 associated with the given user / participant A - D will not be used by the system 400 when generating the dynamic summary queue 425. Thus, by using the participant base control 501, the requester of the dynamic summary queue 425 can customize which user / participant A - D input channels 401 are used as a basis for generating the dynamic summary queue 425.

[0040] For example, if the requester of the dynamic summary queue 425 knows that the requester has interests and / or preferences similar to those of users / participants A and C, but the requester does not know users / participants B and D, as shown in the example of FIG. 5B, the requester can use the participant-based control 501 to "include" users / participants A and C and "exclude" users / participants B and D in the generation of the dynamic summary queue 425. In this way, the requester of the dynamic summary queue 425 can be confident that the content of the dynamic summary queue 425 is more in line with his or her interests and / or preferences. In another example, if the requester of the dynamic summary queue 425 does not know the users / participants A to D in the group, the requester can use the participant-based control 501 to "include" all the users / participants A to D. When the requester views the dynamic summary queue 425, the requester can not only quickly learn the latest information about the media item 415, but also determine whether the interests and / or preferences of the users / participants A to D in the group match his or her own interests and / or preferences. The requester of the dynamic summary queue 425 can also make an information-based decision on whether to participate in the group event with the users / participants A to D. In some embodiments, the default setting of the participant-based control 501 is to "include" all the users / participants A to D in the group so that all the input channels 401A to 401D are used in the generation of the dynamic summary queue 425.

[0041] FIG. 5C shows an example of a user interface 515 that is displayed when the "Modify" control is selected within the time frame control 503 of the user interface 500 of FIG. 5A, according to some embodiments. With the time frame control 503, a requester of the dynamic summary queue 425 can select a time frame along the timeline of the media item 415 for which the dynamic summary queue 425 is generated. In some embodiments, with the time frame control 503, the user can select a time frame 516 for the generation of the dynamic summary queue 425 that is within the overall time range from the start (time 0:00) of the timeline of the media item 415 to the current time of the live playback of the media item 415 in progress for user / participant groups A - D. In some embodiments, the time frame control 503 provides a time bar 517 that represents the overall time range within which the time frame 516 for the generation of the dynamic summary queue 425 can be set. Also, in some embodiments, the time frame control 503 includes a first slider control 519 that is slidable by the requester along the time bar 517 and sets the start time along the timeline of the media item 415 for the time frame 516 used for the generation of the dynamic summary queue 425. Also, in some embodiments, the time frame control 503 includes a second slider control 521 that is slidable by the requester along the time bar 517 and sets the end time along the timeline of the media item 415 for the time frame 516 used for the generation of the dynamic summary queue 425. In this way, the dynamic summary queue 425 will be generated over a time frame 516 along the timeline of the media item 415 that extends from the start time corresponding to the setting of the first slider control 519 to the end time corresponding to the setting of the second slider control 521.

[0042] Also, in some embodiments, the time frame control 503 provides an indication of the time position along the timeline of the media item 415 where an indication of interest (I) and / or an event of interest (E) may be present, based on the input channels 401 associated with the users / participants A - D currently included in the generation of the dynamic summary queue 425 as currently set by the participant - based control 501. The time positions along the timeline of the media item 415 where an indication of interest (I) and / or an event of interest (E) may be present can be used by the claimant as an aid in identifying which portions of the media item 415 are more desirable to include within the time frame 516 for generating the dynamic summary queue 425. In some embodiments, when a second slider control 521 is set to the current time of the live playback of the media item 415, the system 400 operates to continue adding content to the dynamic summary queue 425 as the dynamic summary queue 425 is streamed to the claimant. Thus, when the claimant reaches the end of the dynamic summary queue 425, the claimant reaches the current time of the live playback of the media item 415. Also, in some embodiments, when the claimant reaches the end of the dynamic summary queue 425, the claimant immediately enters the viewing of the live playback of the media item 415 together with the other users / participants A - D in the group. For example, when the claimant reaches the end of the dynamic summary queue 425, the user interface currently displayed to the claimant automatically changes from the user interface 300 of FIG. 3 to the user interface 200 of FIG. 2.

[0043] FIG. 5D shows an example of a user interface 523 that is displayed when the "Change" control is selected within the summary breakdown control 505 of the user interface 500 of FIG. 5A, according to some embodiments. The summary breakdown control 505 allows the claimant to set the content breakdown of the dynamic summary queue 425 and control how much content is included within the dynamic summary queue 425. In some embodiments, the user interface 523 includes a breakdown bar 525 and a slider control 527 that can be slid along the breakdown bar 525 by the claimant. When the slider control 527 is placed towards the end that is "Less Detail" of the breakdown bar 525, the dynamic summary queue 425 will be generated with less content (more briefly). Conversely, placing the slider control 527 towards the end that is "More Detail" of the breakdown bar 525 causes the dynamic summary queue 425 to be generated with more content (more lengthily). Also, in some embodiments, the user interface 523 includes a field 529 in which a requester can enter an integer value indicating the number (or maximum number) of segments included in the dynamic summary queue 425, where the segments correspond to consecutive portions of the playback timeline of the media item 415. Also, in some embodiments, the user interface 523 includes a field 531 in which a requester can enter the length of time (or the maximum length of time) used for each segment within the dynamic summary queue 425.

[0044] Figure 5E shows an example of a user interface 533 that is displayed when the "Change" control is selected within the event type control 507 of the user interface 500 of Figure 5A, according to some embodiments. The event type control 507 allows a requester of the dynamic summary queue 425 to specify a particular type of event that the requester is interested in within the media item 415 and should consider including in the dynamic summary queue 425. In some embodiments, the user interface 533 includes a custom event setting chart 534 that specifies various events that can occur within the media item 415 and are currently available for selection by the requester for inclusion in the dynamic summary queue 425. In some embodiments, the custom event setting chart 534 includes an event descriptor column 543 that contains brief descriptions of various events that may occur within the media item 415. The types of events that may occur within the media item 415, and their corresponding brief descriptions, depend on the subject matter of the media item 415. For example, if the media item 415 is a video of an auto race (occurring either in the real world or in a computer game), some exemplary events that may occur within the media item 415 may include a green flag, a yellow flag, a checkered flag, a crash, a spin out, a pass, a block, a pit, etc., such as those shown in the example of the custom event setting chart 534 of FIG. 5E.

[0045] In some embodiments, the custom event setting chart 534 includes an event status column 545 that indicates whether the event described in the corresponding row of the event descriptor column 543 is active or inactive for inclusion as a basis for generating the dynamic summary queue 425. In some embodiments, a "Yes" setting in the event status column 545 for a given event indicates that if the given event occurs within the media item 415 during the time frame 516 in which the dynamic summary queue 425 is generated, it will be included as part of the basis for generating the dynamic summary queue 425. Conversely, a "No" setting in the event status column 545 for a given event indicates that if the given event occurs within the media item 415 during the time frame 516 in which the dynamic summary queue 425 is generated, it will not be included as part of the basis for generating the dynamic summary queue 425. In some embodiments, the custom event setting chart 534 includes an adjustment column 547 that includes a "Change" control that, when selected, toggles the setting in the corresponding row of the event status column 545 between "Yes" and "No", and vice versa. Also, in some embodiments, when the custom event settings chart 534 is selected, it includes a global "No" control 549 that sets the event status column for all listed events to "No". Also, in some embodiments, when the custom event settings chart 534 is selected, it includes a global "Yes" control 551 that sets the event status column for all listed events to "Yes". In some embodiments, the default setting for the event status column for all listed events is "Yes".

[0046] In some embodiments, the custom event settings chart 534 includes an event intensity column 541 that can specify an event weighting for an event as described in the corresponding row of the event descriptor column 543. In some embodiments, the event weighting for a given event is specified on a scale of 1 to 10, with 1 being the lowest weighting and 10 being the highest weighting. The event weightings listed in the event intensity column 541 are used by the system 400 to determine events that are given a higher priority than other events to include in the dynamic summary queue 425. In particular, this is when there is more content available for inclusion with the dynamic summary queue 425 than can be accommodated by the current settings of the summary breakdown control 505. In various embodiments, the scale used for event weighting can be defined in essentially any way that provides normalization of the event weighting input so that it can be determined which events are more important than others.

[0047] In some embodiments, the user interface 533 includes a custom event addition control 553 that, when selected, enables a requester to add a new event (new row) to the custom event setting chart 534. Also, in some embodiments, the user interface 533 includes a reset event control 555 that, when selected, resets all settings of the custom event setting chart 534 to their respective values that existed when the user interface 533 was displayed in accordance with the event type control 507 selection. Also, in some embodiments, the user interface 533 includes an auto-enable control 535 and a custom-enable control 537. The selection of the auto-enable control 535 is involved in the use of the machine learning model 413 within the system 400 when determining which of the events occurring within the media item 415 to include in the basis for generating the dynamic summary queue 425. The selection of the custom-enable control 537 involves the use of the custom event setting chart 534 by the system 400 when determining which of the events occurring within the media item 415 to include in the basis for generating the dynamic summary queue 425. In some embodiments, either or both of the auto-enable control 535 and the custom-enable control 537 can be selected at a given point in time. In some embodiments, when the custom-enable control 537 is deselected, the selection of the auto-enable control 535 is implemented by default.

[0048] FIG. 5F shows an example of a user interface 561 that is displayed when the "Change" control is selected within the display type control 509 of the user interface 500 of FIG. 5A, according to some embodiments. The display type control 509 enables a requester of the dynamic summary queue 425 to specify a particular type of user / participant display of interest while the user / participants A - D are consuming the media item 415. It will be considered when determining which portions of the media item 415 should be included in the basis for generating the dynamic summary queue 425. In some embodiments, the user interface 561 includes a custom display settings chart 563 that specifies various user / participant interests that may occur during consumption of the media item 415 by the user / participant. In some embodiments, the custom display settings chart 563 includes a display descriptor column 565 that includes brief descriptions of the various user / participant displays of interest. The various user / participant displays represent various types of reactions, expressions, gestures, movements, utterances, sounds, and / or transmissions that users / participants A - D can make when consuming the media item 415. For example, in some embodiments, the various user / participant displays specified in the custom display settings chart 563 may include, among any other types of reactions, expressions, gestures, movements, utterances, sounds, and / or transmissions that can be made by any of users / participants A - D during consumption of the media item 415, like, dislike, excitement, happiness, anger, scream, tension, laughter.

[0049] In some embodiments, the custom display settings chart 563 includes a display status column 567 that indicates whether the user / participant display listed in the corresponding row of the display descriptor column 565 is active or inactive for inclusion in the dynamic summary queue 425, based on which content is selected for inclusion. In some embodiments, a "yes" setting in the display status column 567 for a given user / participant display means that the given user / participant display is used, including in the criteria for which content is identified / selected, for inclusion in the dynamic summary queue 425. Conversely, a "no" setting in the display status column 567 for a given user / participant display means that the given user / participant display is not included in the criteria for which content is identified / selected for inclusion in the dynamic summary queue 425. In some embodiments, the custom display settings chart 563 includes an adjustment column 569 that includes a "change" control where, when selected, the setting of the corresponding row in the display status column 567 toggles between "yes" and "no", and vice versa. Also, in some embodiments, the custom display settings chart 563 includes a global "no" control 571, which, when selected, will set the settings in the display status column 567 to "no" for all listed user / participant displays. Also, in some embodiments, the custom display settings chart 563 includes an inclusive "yes" control 573, which, when selected, will set the settings in the display status column 567 to "yes" for all listed user / participant displays. In some embodiments, the default setting for the display status column 567 for all listed user / participant displays is "yes".

[0050] In some embodiments, the custom display settings chart 563 includes a user / participant display intensity column 575 that can specify a display weighting for the user / participant display as described in the corresponding row of the display descriptor column 565. In some embodiments, the display weighting for a given user / participant display is specified on a scale of 1 to 10, where 1 is the lowest weighting and 10 is the highest weighting. The display weightings listed in the display intensity column 575 are used by the system 400 to determine which user / participant displays should be given a higher priority over others, and to include in the criteria for identifying / selecting which portions of the media item 415 should be included within the dynamic summary queue 425. In various embodiments, the scale used for the user / participant display weighting can be defined in essentially any way that provides a normalization of the user / participant display weighting input so as to be able to determine which user / participant displays are more important than others.

[0051] In some embodiments, the user interface 561 includes a custom display addition control 577 that, when selected, enables the requester to add a new user / participant display (new row) to the custom display settings chart 563. Also, in some embodiments, the user interface 561 includes a reset display control 579 that, when selected, resets all settings of the custom display settings chart 563 to their respective values that existed when the user interface 561 was displayed in accordance with the selection of the display type control 509. Also, in some embodiments, the user interface 561 includes an auto-enable control 581 and a custom-enable control 583. Selection of the auto-enable control 581 involves the use of the machine learning model 413 of the system 400 in determining which user / participant displays occurring during consumption of the media item 415 by users / participants A - D should be included among the criteria for determining which portions of the media item 415 will be included in the dynamic summary queue 425. Selection of the custom-enable control 583 involves the use of the custom display settings chart 563 by the system 400 in determining which user / participant displays occurring during consumption of the media item 415 by users / participants A - D should be included among the criteria for determining which portions of the media item 415 will be included in the dynamic summary queue 425. In some embodiments, either or both of the auto-enable control 581 and the custom-enable control 583 can be selected at a given point in time. In some embodiments, when the custom-enable control 583 is deselected, the selection of the auto-enable control 581 is implemented by default.

[0052] FIG. 6 shows a flowchart of a method for generating a summary (dynamic summary queue 425) of media items (media item 415) according to some embodiments. The method includes an operation 601 for sending a stream of media item 415 from a server computing system (such as, for example, the cloud gaming system 101 of FIG. 1) to one or more client computing systems (such as, for example, the local computing systems 120A-120D of FIG. 1). The media item 415 has a timeline for playing the media item 415. In some embodiments, the media item 415 is an audio-visual presentation. In various embodiments, the audio-visual presentation is one or more of a movie, a television program, a real-world sports event, an e-sports event, an online show, an online video, and an uploaded video, among any other type of audio-visual presentation. In some embodiments, the media item 415 is a game video and audio stream of a video game. In some embodiments, the server computing system is a cloud gaming system 101 configured to execute a video game according to inputs received from respective controller devices 125A-125D of one or more users / participants (e.g., game players / viewers A-D) of one or more client computing systems 120A-120D. The cloud gaming system 101 is configured to direct the transmission of a game video and audio stream reflecting the play of a video game by one or more users / participants. In these embodiments, one or more users / participants provide user / participant-generated content received via an input channel 401 during the play of the video game. In some embodiments, the video game is an e-sports video game. Also, in some embodiments, one or more of the users / participants are viewers of the video game.

[0053] The method also includes an operation 603 of receiving user / participant-generated content in a server computing system from one or more users / participants of any of one or more client computing systems. The user / participant-generated content corresponds to any information received via any of the input channels 401, as described with respect to FIG. 4. The user / participant-generated content is time-indexed with respect to the timeline of the media item 415. Also, it should be understood that the user / participant-generated content is provided by any user / participant consuming (viewing) the media item 415. In various embodiments, the user / participant can be a video game player, a spectator of a video game, and / or a viewer, depending on the nature of the media item 415. In some embodiments, the user / participant-generated content includes one or more of text data, voice data, audio data, emoji data, image data, video data captured by a camera, image data captured by a camera, tag data, and controller device input data, among essentially any other type of data that can be provided by a user / participant via the input channel 401.

[0054] This method also includes an operation 605 of processing user / participant-generated content to identify a major portion of media item 415 based on the representations present in the user / participant-generated content, where the major portion of media item 415 is identified for inclusion in the dynamic summary queue 425. In some embodiments, the identification of the user / participant's representation present in the user / participant-generated content is based on the specification of the user / participant's interest that may occur during the consumption of media item 415 by the user / participant, as described with respect to FIG. 5F. In some embodiments, operation 605 also includes processing media item 415 and / or the state / metadata 417 associated with media item 415 to identify a major portion of media item 415 for inclusion in the dynamic summary queue 425. In some embodiments, the identification of the major portion of media item 415 is based on the specification of events that may occur within media item 415, as described with respect to FIG. 5E.

[0055] In some embodiments, operation 605 of processing user / participant-generated content includes extracting features from the user / participant-generated content to obtain a set of extracted features 407, as described with respect to the feature extractor 405 of FIG. 4. Also, in some embodiments, operation 605 of processing user / participant-generated content includes classifying the extracted features 407 into one or more classifications to obtain a set of classified features 411, as described with respect to the feature classifier 409 of FIG. 4. Also, in some embodiments, operation 605 of processing user / participant-generated content includes processing the set of classified features 411 through the machine learning model 413 to identify a major portion of media item 415 included in the dynamic summary queue 425. Also, in some embodiments, operation 605 includes processing the state / metadata 417 of the media item 415 via the machine learning model 413, in addition to processing the set 411 of classified features via the machine learning model 413, to identify the main portions of the media item 415 included in the dynamic summary queue 425. The state / metadata 417 of the media item 415 is associated with the set 411 of classified features with each other by a time index. In some embodiments, the state / metadata 417 of the media item 415 includes the state data of a video game executed according to the input received from each of the controller devices 125A-125D of one or more users / participants (e.g., A-D), and generates the media item 415 as a game video and audio stream.

[0056] The method also includes an operation 607 for generating a dynamic summary queue (dynamic summary queue 425) that includes the main portions of the media item 415 identified in operation 605. The method also includes assembling the dynamic summary queue 425 for streaming to a requester of the dynamic summary queue 425. In some embodiments, operation 607 includes creating an automatically generated verbal summary and / or an automatically generated written summary of the main portions of the media item 415 for inclusion in the dynamic summary queue 425. In some embodiments, the dynamic summary queue 425 includes at least a portion of user / participant-generated content associated with the main portions of the media item 415 received from the user / participant via the input channel 401.

[0057] The method also includes operation 609 for receiving a request from a requester to view the dynamic summary queue 425. In some embodiments, the requester is any one of one or more users / participants consuming the media item 415. In some embodiments, the requester is a person invited to participate with one or more users / participants in consuming the media item. In some embodiments, the requester is a person invited to participate with one or more users / participants when playing a video game executed according to input received from respective controller devices 125A - 125D of one or more users / participants (e.g., A - D) to generate the media item 415 as a game video and audio stream.

[0058] In various embodiments, operation 609 can occur at any point in the method of FIG. 6. For example, in some embodiments, the request to view the dynamic summary queue 425 can be received in operation 609 before processing user / participant generated content in operation 605. Alternatively, in another exemplary embodiment, after the dynamic summary queue 425 has already been generated in operation 607, a request to view the dynamic summary queue 425 can be received in operation 609. Also, in some embodiments, the request to view the dynamic summary queue 425 received in operation 609 can include specifications of various controls applied when generating the dynamic summary queue 425, such as the various controls described with respect to FIGS. 5A - 5F. This method also includes operation 611 for sending a stream of the dynamic summary queue 425 to the requester's computing device in response to the request received in operation 609. In some embodiments, this method includes updating the dynamic summary queue 425 when user / participant-generated content is received in the server computing system (via input channel 401) and processed to identify a major portion of the media item 415. In some embodiments, updating the dynamic summary queue 425 includes adding new content to the dynamic summary queue 425 and / or changing some of the content already present within the dynamic summary queue 425.

[0059] In some embodiments, this method includes receiving control parameters for generating the dynamic summary queue 425, such as the control parameters described with respect to FIG. 5A. In some embodiments, the control parameters include a specification of one or more types of displays present within user / participant-generated content that indicate a major portion of the media item 415, as described with respect to FIG. 5F. In some embodiments, the types of displays include one or more of reactions, expressions, gestures, motions, utterances, sounds, transmissions, and / or essentially any other type of display that can be emitted by a user / participant related to the theme within the media item 415.

[0060] In some embodiments, the control parameters for generating the dynamic summary queue 425 include the specification of one or more types of events that may occur within the media item 415, as described with respect to FIG. 5E. In some embodiments, the control parameters for generating the dynamic summary queue 425 include the specification of which one or more of the users / participants should be restricted when processing user / participant-generated content (as received via the input channel 401) to identify the main portions of the media item 415, as described with respect to FIG. 5B. In some embodiments, the control parameters for generating the dynamic summary queue 425 include the specification of the time frame 516 within the timeline for the playback of the media item 415 for which the dynamic summary queue 425 is generated, as described with respect to FIG. 5C. In some embodiments, the control parameters for generating the dynamic summary queue 425 include the specification of the breakdown of the summary of the dynamic summary queue 425, as described with respect to FIG. 5D.

[0061] Further, in some embodiments, the method for generating the dynamic summary queue 425 described with respect to FIG. 6 is implemented as a system (e.g., system 400) for generating a summary of a video. In these embodiments, the system includes program instructions stored in a computer memory for transmitting a stream of the media item 415 from a server computing system to one or more client computing systems, where the media item 415 has a timeline for the playback of the media item 415. The system also includes program instructions stored in a computer memory for receiving user / participant-generated content at the server computing system from one or more users / participants of any of the one or more client computing systems, where the user / participant-generated content is time-indexed with respect to the timeline of the media item 415. This system also includes program instructions stored in a computer memory for processing user / participant generated content and identifying the major portions of media item 415 based on the displays present within the user / participant generated content. This system also includes program instructions stored in a computer memory for generating a dynamic summary queue 425 that includes the major portions of media item 415, where the dynamic summary queue 425 is assembled for streaming. The system also includes program instructions stored in a computer memory for receiving a request from a requester to view the dynamic summary queue 425. The system also includes program instructions stored in a computer memory for transmitting a stream of the dynamic summary queue 425 to the requester's computing device in response to the request.

[0062] FIG. 7 shows a diagram of components within an exemplary computing device 700 that can be used to execute aspects of various embodiments of the present disclosure, according to some embodiments. In various embodiments, the computing device 700 can incorporate, or can be, a personal computer, video game console, personal digital assistant, server, or other digital device suitable for implementing embodiments of the present disclosure. The computing device 700 includes a central processing unit (CPU) 702 for executing software applications and optionally an operating system. The CPU 702 can include one or more homogeneous or heterogeneous processing cores.

[0063] In some embodiments, the video game is executed locally on a gaming console, a personal computer, or a server. In some cases, the video game is executed by one or more servers in a data center. When the video game is executed, some instances of the video game may be simulations of the video game. For example, the video game may be executed by an environment or server that generates a simulation of the video game. The simulation, in some embodiments, is an instance of the video game. In other embodiments, the simulation may be generated by an emulator. In any case, when the video game is represented as a simulation, the simulation can be executed to render interactive content (e.g., video frames) that can be interactively streamed, executed, and / or controlled by user input.

[0064] The CPU 702 is one or more general-purpose microprocessors that include one or more processing cores. In some embodiments, the computing device 700 is implemented using one or more CPUs 702 having a microprocessor architecture that is particularly well-suited for high-parallel and compute-intensive applications, such as the interpretation of queries, the identification of contextually relevant resources, and the immediate implementation and rendering of contextually relevant resources within a video game. In various embodiments, the computing device 700 is one that is localized to a player playing a game segment (e.g., a gaming console), or is remote from the player (e.g., a backend server processor), or is one of many servers that uses virtualization in a game cloud system for remote streaming of game play to a client.

[0065] Computing device 700 includes a computer memory 704 that stores applications and data used by CPU 702. Computing system 700 also includes a data storage device 706 that provides a non-volatile storage device for data and a computer-readable medium for applications. In various embodiments, computing device 700 may include a fixed disk drive, a removable disk drive, a flash memory device, and / or a CD-ROM, DVD-ROM, Blu-ray (registered trademark), HD-DVD, or other optical storage device, as well as signal transmission and storage media. Computing device 700 also includes one or more user input devices 708 that communicate user input from one or more users to computing device 700. For example, user input device(s) 708 may include a keyboard(s), mouse(s), joystick(s), touch pad(s), touch screen(s), still or video recorder(s) / camera(s), gesture recognition tracking device(s), and / or microphone(s). Computing device 700 also includes a network interface 714 that enables computing device 700 to communicate with other computer systems via an electronic communication network and may include wired or wireless communication over a local area network and a wide area network such as the Internet. Computing device 700 also includes an audio processor 712 that generates an analog or digital audio output from instructions and / or data provided by CPU 702, computer memory 704, and / or data storage device 706. Computing device 700 also includes one or more data buses 722 for connecting CPU 702, computer memory 704, and / or data storage device(s) 706, user input device(s) 708, network interface 714, and audio processor 712.

[0066] The graphics subsystem 720 is further connected to a data bus 722 and various components of the computing device 700. The graphics subsystem 720 includes a graphics processing unit (GPU) 716 and graphics memory 718. The graphics memory 718 includes display memory (e.g., a frame buffer) used to store pixel data for each pixel of the output image. The graphics memory 718 may be integrated into the same device as the GPU 716, connected as a separate device from the GPU 716, and / or implemented within the computer memory 704. The pixel data may be provided directly from the CPU 702 to the graphics memory 718. Alternatively, the CPU 702 provides data and / or instructions defining the desired output image to the GPU 716, and the GPU 716 generates pixel data for one or more output images therefrom. The data and / or instructions defining the desired output image may be stored in the computer memory 704 and / or the graphics memory 718. In some embodiments, the GPU 716 includes a three-dimensional (3D) rendering function that generates pixel data for the output image from instructions and data defining geometry, lighting, shading, texturing, motion, and / or camera parameters for the scene. The GPU 716 may further include one or more programmable execution units capable of executing shader programs.

[0067] The graphics subsystem 720 periodically outputs pixel data of an image from the graphics memory 718 for display on the display device 710. The display device 710 may be any device capable of displaying visual information in response to a signal from the computing device 700, including, among other types of computer display devices, CRT, LCD, plasma, and OLED displays. In various embodiments, the computing device 700 provides an analog signal and / or a digital signal to the display device 710.

[0068] In various embodiments, the methods and systems disclosed herein are implemented in a cloud computing system that provides access services such as providing access to video games and / or other media content over a wide geographic area. Cloud computing is a computing paradigm in which dynamically scalable and often virtualized resources are provided as services over the Internet. A user need not be an expert in the technical infrastructure of the "cloud" that supports the user. Cloud computing can be classified into different services such as infrastructure as a service (IaaS), platform as a service (PaaS), and software as a service (SaaS). Cloud computing services often provide applications such as video games online for access from a web browser, but the software and data are stored on servers within the cloud. The term cloud is used as a metaphor for the Internet based on how the Internet is depicted in diagrams of computer networks and is an abstract concept that hides complex infrastructure.

[0069] In some embodiments, a game server is used to execute the operation of a persistent information platform for video game players. Most video games played over the Internet operate via a connection to a game server. In some embodiments, online video games use a dedicated server application that collects data from players and distributes it to other players. In other embodiments, online video games are executed by a distributed game engine. In these embodiments, the distributed game engine may be executed on a plurality of processing entities (PEs), such that each PE executes a given functional segment of the game engine on which the online video game is executed. In some embodiments, each PE is recognized as a computing node by the game engine.

[0070] A game engine typically performs a functionally diverse set of operations to provide additional services for the user to experience and to execute a video game application. For example, a game engine implements game logic and executes game calculations, physical processes, geometry transformations, rendering, lighting, shading, audio, and additional in-game or game-related services. The additional services may include, for example, messaging, social utilities, audio communication, game play / playback functions, help functions, and the like. In some embodiments, the game engine is executed in an operating system virtualized by a hypervisor of a particular server. In some embodiments, segments of the game engine are distributed across multiple PEs, and each PE can reside in a different server unit of a data center. In some embodiments, the various PEs used to execute the game engine segments can be implemented as server units, virtual machines, or containers according to the needs of each game engine segment. For example, if a game engine segment is responsible for camera transformation, since it will perform a large number of relatively simple mathematical operations (e.g., matrix transformation), that particular game engine segment may be provisioned with a virtual machine associated with a GPU. Other game engine segments that require fewer but more complex operations may be provisioned with processing entities associated with one or more higher-power CPUs.

[0071] By distributing the game engine, the game engine has elastic computing characteristics that are not constrained by the capabilities of physical server units. Instead, the game engine is provisioned with more or fewer computing nodes as needed to meet the requirements of the video game. From the perspective of the video game and the video game player, a game engine distributed across multiple computing nodes is indistinguishable from a non-distributed game engine executed by a single processing entity because a game engine manager or supervisor distributes the workload and seamlessly integrates the results to provide the video game output components to the end user.

[0072] The user accesses the remote service using a client device that includes at least a CPU, a display, and an input / output (I / O) device. The client device can be a personal computer (PC), a mobile phone, a tablet, a netbook, a PDA, or any other type of computing device. In some embodiments, the network running on the game server recognizes the type of computing device used by the client and adjusts the data communication method employed. In another case, the client device uses a standard data communication method such as HTML (Hypertext Markup Language) to access the application on the game server via the Internet.

[0073] It should be understood that a given video game or game application can be developed for a specific platform and a specific associated controller device. However, when making such a game available via a game cloud system, a user can access the video game with different controller devices. For example, a game may be developed for a game console and its associated controller, but a user can access the cloud-based version of the game from a personal computer using a keyboard and mouse. In such a scenario, the input parameter configuration can define a mapping from inputs that can be generated by the controller devices available to the user (in this case, the keyboard and mouse) to inputs that are acceptable for the execution of the video game. In other examples, a user can access a cloud game system via a tablet computing device, a touch screen smartphone, or other touch screen-driven devices. In this case, the client device and the controller device are integrated together within the same device while the input is provided as a touch screen input / gesture where the input is detected. For such devices, the input parameter configuration can define specific touch screen inputs that correspond to game inputs for the video game. For example, buttons, a directional pad, or other types of input elements can be displayed or overlaid during the execution of the video game to indicate the positions on the touch screen that the user can touch to generate game inputs. Gestures such as swipes in a specific direction, or specific touch motions can also be detected as game inputs. In one embodiment, for example, before starting to play a video game, a tutorial showing how to provide inputs via the touch screen of the game play can be provided to the user.

[0074] In some embodiments, the client device functions as a connection point for the controller device. For example, the controller device communicates with the client device via a wireless or wired connection and transmits inputs from the controller device to the client device. The client device then processes these inputs and may subsequently transmit the input data via a network (e.g., a network accessible via a local network device such as a router) to a cloud game server. However, in other embodiments, the controller itself has the ability to communicate inputs directly to a cloud game server via a network, and it is possible for such a controller to be a networked device without the need to first communicate these inputs through a client device. For example, the controller can connect to a local network device (such as a router), send data to, and receive data from, a cloud game server. Thus, while the client device can remain required to receive video output from a cloud-based video game and render it on a local display, input latency can be reduced by enabling the controller to transmit inputs directly to the cloud game server via the network, bypassing the client device.

[0075] In some embodiments, the networked controller and client device can be configured to send certain types of input directly from the controller to the cloud game server and other types of input via the client device. For example, apart from the controller itself, input that does not depend on any additional hardware or processing can be sent directly from the controller to the cloud game server via the network, bypassing the client device. Such input can include button input, joystick input, embedded motion detection input (e.g., accelerometer, magnetometer, gyroscope) among any other types of input that can be provided by the controller device. However, input that utilizes additional hardware or requires processing by the client device can be sent to the cloud game server by the client device. These inputs can include video and / or audio obtained from the game environment that can be processed by the client device before being sent to the cloud game server. Additionally, input from the motion detection hardware of the controller can be processed by the client device in conjunction with the obtained video to detect the position and movement of the controller, and then communicated to the cloud game server by the client device. In some embodiments, the controller device can also receive data (e.g., feedback data, haptic data, etc.) from the client device or directly from the cloud game server.

[0076] Of course, the various embodiments defined herein may be combined or assembled into specific implementations using the various features disclosed herein. Accordingly, the examples provided are only a part of the possible examples and do not limit the various embodiments that can define more embodiments by combining various elements. In one example, an embodiment may include fewer elements without departing from the spirit of the disclosed or equivalent embodiments. Embodiments of the present disclosure may be implemented by various computer system configurations including, for example, handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. Further, embodiments of the present disclosure may also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked via a wired or wireless network.

[0077] Also, although the operations of some methods are described herein in a particular order / sequence, other housekeeping operations may be performed during the operations of the various methods, and / or the operations of the methods may be adjusted to occur at slightly different times, and / or the processing of game telemetry and / or game state data to generate a modified game state may be performed in a timely manner, for example, to make the game telemetry and / or game state available when required by subsequent game-executable operations, it should be understood that the operations of the methods may be distributed in a system that allows the processing of the operations of the methods to occur at various intervals.

[0078] In some embodiments, the various methods and systems disclosed herein are implemented (manufactured) as computer-readable code on a computer-readable medium. A computer-readable medium is any data storage device that can store data and then have that data read by a computer system. Examples of computer-readable media include hard drives, network attached storage (NAS), read-only memory, random access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes, and other optical and / or non-optical data storage devices. Also, computer-readable media can include computer-readable tangible media distributed across a network-connected computer system so that the computer-readable code is stored and executed in a distributed fashion.

[0079] The foregoing description of the embodiments has been provided for purposes of illustration and description and is not intended to be exhaustive or limiting. The individual elements or features of a particular embodiment are generally not limited to that particular embodiment, but, where applicable, are interchangeable and can be used in the selected embodiment even if not specifically shown or described. Thus, one or more features from one or more of the embodiments disclosed herein can be combined with one or more features from one or more of the other embodiments disclosed herein to form other embodiments not expressly disclosed herein but implicitly disclosed herein. Such other embodiments can also be varied in many ways. Such variations are not to be regarded as a departure from the disclosure herein, and all such variations and modifications are intended to be included within the scope of the disclosure provided herein.

[0080] Although the foregoing embodiments have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications can be made within the scope of the appended claims. Accordingly, the embodiments disclosed herein should be considered as illustrative rather than restrictive, and the present embodiments should not be limited to the details described herein, but may be modified within the scope of the appended claims and equivalents.

Claims

**Claim 1** A method for generating a summary of a media item, comprising: sending a stream of the media item from a server computing system to one or more client computing systems, the media item having a timeline for playback of the media item; receiving participant-generated content at the server computing system from one or more participants of any of the one or more client computing systems, the participant-generated content being time-indexed to the timeline of the media item; processing the participant-generated content to identify a major portion of the media item based on a presentation present in the participant-generated content; generating a dynamic summary queue that includes the major portion of the media item, the dynamic summary queue being assembled for streaming; receiving a request to display the dynamic summary queue from a requester; in response to the request, sending a stream of the dynamic summary queue to the requester's computing device; receiving additional participant-generated content at the server computing system from one or more participants in any of the one or more client computing systems when the stream of the media item is sent, the additional participant-generated content being time-indexed to the timeline of the media item; automatically changing some of the content already present in the dynamic summary queue based on the additional participant-generated content while continuing to send the stream of the dynamic summary queue in response to receiving the additional participant-generated content. **Claim 2** The method of claim 1, wherein the media item is an audiovisual presentation. **Claim 3** The method of claim 2, wherein the audiovisual presentation is one or more of a movie, a television program, a real-world sports event, an esports event, an online show, an online video, and an uploaded video. **Claim 4** The method of claim 1, wherein the media item is a stream of game video and audio of a video game. **Claim 5** The server computing system is a cloud game system configured to execute the video game according to inputs received from respective controller devices of the one or more participants in the one or more client computing systems, and the cloud game system is configured to instruct transmission of the game video and audio streams reflecting play of the video game by the one or more participants, and the one or more participants provide the participant-generated content during play of the video game, the method according to claim 4.

6. The method according to claim 4, wherein the video game is an e-sports video game.

7. The participant-generated content includes one or more of text data, voice data, audio data, emoji data, image data, video data acquired by a camera, image data acquired by a camera, tag data, and input data of a controller device, the method according to claim 1.

8. Processing the participant-generated content includes extracting features from the participant-generated content to obtain a set of extracted features, classifying the extracted features into one or more classifications to obtain a set of classified features, and processing the set of classified features through a machine learning model to identify a main portion of the media item, the method according to claim 1.

9. Further, processing the state data of the media item through the machine learning model, and in combination with processing the set of classified features through the machine learning model to identify the main portion of the media item, associating the state data of the media item with the set of classified features according to a time index, the method according to claim 8.

10. The state data of the media item includes state data of a video game executed according to inputs received from respective controller devices of the one or more participants for generating the media item as game video and audio streams, the method according to claim 9.

11. The method according to claim 1, wherein the dynamic summary queue includes an automatically generated oral summary and / or an automatically generated written summary of the main part of the media item.

12. The method according to claim 1, wherein the dynamic summary queue includes at least a part of the participant-generated content associated with the main part of the media item.

13. The method according to claim 1, wherein the claimant is any one of the one or more participants.

14. The method according to claim 1, wherein the claimant is a person invited to participate with the one or more participants when viewing the media item.

15. The method according to claim 1, wherein the claimant is a person invited to participate in the play of a video game that is executed according to the input received from each controller device of the one or more participants to generate the media item as a video and audio stream of the game.

16. The method according to claim 1, further comprising receiving control parameters for generating the dynamic summary queue.

17. The method according to claim 16, wherein the control parameters include the designation of one or more types of displays present in the participant-generated content that indicate the main part of the media item.

18. The method according to claim 17, wherein the type of display includes one or more of a reaction, expression, gesture, movement, speech, sound, and / or communication uttered by any one of the one or more participants regarding a theme in the media item.

19. The method according to claim 16, wherein the control parameters include the designation of one or more types of events occurring within the media item.

20. The method according to claim 16, wherein the control parameters include the designation of which of the one or more participants the participant-generated content is limited to when processing the participant-generated content to identify the main part of the media item.

21. The method according to claim 16, wherein the control parameters include the designation of a time frame within a timeline for the playback of the media item in which the dynamic summary queue is generated.

22. The method according to claim 17, wherein the control parameters include the designation of a summary breakdown of the dynamic summary queue. The method according to claim 1, wherein when the stream of the dynamic summary queue is transmitted, new content is automatically added to the dynamic summary queue based on the additional participant-generated content. The method according to claim 1, wherein when the stream of the dynamic summary queue is transmitted, some content is automatically deleted from the dynamic summary queue based on the additional participant-generated content.

25. A system for generating a summary of a video, having program instructions stored in a computer memory for transmitting a stream of media items from a server computing system to one or more client computing systems, the media items having a timeline for playback of the media items, having program instructions stored in the computer memory for receiving participant-generated content at the server computing system from one or more participants of any of the one or more client computing systems, the participant-generated content being time-indexed to the timeline of the media items, having program instructions stored in the computer memory for processing the participant-generated content to identify a major portion of the media item based on a display present in the participant-generated content, having program instructions stored in the computer memory for generating a dynamic summary queue including the major portion of the media item, the dynamic summary queue being assembled for streaming, having program instructions stored in the computer memory for receiving a request from a requester to view the dynamic summary queue, having program instructions stored in the computer memory for transmitting a stream of the dynamic summary queue to the requester's computing device in response to the request. When the stream of the media item is transmitted, it has program instructions stored in the computer memory of the server computing system for receiving additional participant-generated content from one or more participants in any of the one or more client computing systems, and the additional participant-generated content is time-indexed to the timeline of the media item. A system that responds to the reception of the additional participant-generated content by automatically changing some of the content already existing in the dynamic summary queue based on the additional participant-generated content while continuing to transmit the stream of the dynamic summary queue.

Images