Automatic visual summarization of a media content item

Abstract

A video snippet generator may receive video data associated with a media content item, the video data including a plurality of video frames. A video snippet generator may generate tokenized video data using the video data. A video snippet generator may generate a frame mask based on the tokenized video data. A video snippet generator may select a subset of the plurality of video frames based on the frame mask. A video snippet generator may generate a visual summarization of the media content item, the visual summarization including the subset.

Description

AUTOMATIC VISUAL SUMMARIZATION OF AMEDIA CONTENT ITEMBACKGROUND

[0001] Creating a program trailer may be relatively expensive in terms of skill, computing resources, and time. In some conventional approaches, a computer may execute a computationally intensive video editing software, motion graphics software, and / or sound design software, which may be operated by a person with a high level of skill for designing and creating program trailers. As such, trailers for some media content on a streaming service may not exist.SUMMARY

[0002] In some aspects, the techniques described herein relate to a method including: receiving video data associated with a media content item, the video data including a plurality of video frames; generating, by a vision transformer, tokenized video data using the video data; generating, by an encoder, a frame mask based on the tokenized video data; selecting a subset of the plurality of video frames based on the frame mask; and generating a visual summarization of the media content item, the visual summarization including the subset.

[0003] In some aspects, the techniques described herein relate to an apparatus including: at least one processor; and a non-transitory computer-readable medium storing executable instructions that cause the at least one processor to execute operations, the operations including: receiving video data associated with a media content item, the video data including a plurality of video frames; generating, by a vision transformer, tokenized video data using the video data; generating, by an encoder, a frame mask based on the tokenized video data; selecting a subset of the plurality of video frames based on the frame mask; and generating a visual summarization of the media content item, the visual summarization including the subset.

[0004] In some aspects, the techniques described herein relate to a non-transitory computer-readable medium storing executable instructions that cause at least one processor to execute operations, the operations including: receiving video data associated with a media content item, the video data including a plurality of video frames: generating, by a vision transformer, tokenized video data using the video data; generating, by an encoder, a framemask based on the tokenized video data: selecting a subset of the plurality of video frames based on the frame mask; and generating a visual summarization of the media content item, the visual summarization including the subset.

[0005] Features described herein with respect to the method can be implemented as part of the apparatus, computer-readable medium and / or computer program product, and vice versa. Moreover, features can be combined with one another in any suitable combination. The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1 A illustrates a video snippet generator according to an aspect.

[0007] FIG. IB illustrates a vision transformer according to an aspect.

[0008] FIG. 1 C illustrates a frame group calculator according to an aspect.

[0009] FIG. 2A illustrates a streaming system with a video snippet generator according to an aspect.

[0010] FIG. 2B illustrates an example of a streaming system with a chat interface for editing a visual summarization according to an aspect.

[0011] FIG. 3 is a flowchart depicting example operations of generating a visual summarization according to an aspect.DETAILED DESCRIPTION

[0012] This disclosure relates to a video snippet generator that automatically creates a visual summarization of a media content item (e.g., a movie, program, etc.) that overcomes one or more technical problems relating to computational efficiency for summarizing visual content from a larger video file. The video snippet generator provides a technical solution that tokenizes video data in a computationally efficient manner (e.g., which may use distributed, parallel tokenization techniques) and uses the tokenized video data as an input to a model to generate a frame mask, w here the frame mask includes selection information (e.g., vector data) about which video frames are to be included in the visual summarization. This technical approach to processing and manipulating digital videos can allow to generate a visual summarization of video data in a quick and efficient manner, with fewer computational resources than previous approaches.

[0013] In further detail, the video snippet generator receives video data (e.g., a video file) associated with a media content item (e.g., a movie, a program, etc.), where the video data includes a plurality of video frames. The video snippet generator includes a vision transformer that generates tokenized video data using the video data. In some examples, the vision transformer is referred to as a vision transformer (ViT) encoder. In some examples, the vision transformer includes one or more neural networks. In some examples, the vision transformer includes a neural netw ork transformer. The tokenized video data may include embedding vectors representing the video frames. An embedding vector may encapsulate visual content in a symmetric space. For example, the vision transformer may convert the imagery' of the video data to numerical values that can be processed by a neural network (e.g., an encoder, a bi-directional encoder). In some examples, the vision transformer may include a distributed computing system with a plurality of inference processors configured to execute parallel processing to tokenize the video data, which may provide one or more technical benefits of improved computationally efficiency, including reducing the time required to tokenize the video data. This reduction in time may enable the video snippet generator to batch process a plurality of video files without significantly overloading computing resources associated with a streaming platform.

[0014] The video snippet generator includes an encoder (e.g., a bidirectional encoder) that generates a frame mask based on the tokenized video data, where the frame mask includes a sequence of values (e.g., mask vector). Each value in the frame mask corresponds to a different video frame, and the numerical value (e.g., ‘T ” or “0”) of a respective value indicates the presence or omission of a respective video frame in the visual summarization. In some examples, the encoder is a bi-directional encoder configured to process the tokenized video data by using contextual data from both directions (e.g., frames before a currently processed frame and frames after the currently processed frame). In other words, the bidirectional encoder determines the inclusion of a current video frame (or a group of video frames currently being analyzed) by using frames around the current video frame(s) (e.g., video data that is before the current video frame(s) in a timeline of the video file and video data that is after the current video frame(s) in the timeline of the video file. In some examples, the use of the bi-directional encoder may provide one or more technical benefits of improved summarization content by generating a temporally coherent summarization of the video data that engages the viewing interest of viewers. In some examples, the encoder includes one or more neural networks. In some examples, the encoder includes a transformerarchitecture. In some examples, the encoder includes one or more convolutional network networks.

[0015] The encoder may be trained to generate a frame mask for a media content item using pairs of programs (e.g., media content items) and their trailers (e.g., visual summarizations). In some examples, the structure of the video snippet generator may provide one or more technical benefits of a computationally efficient training process (e.g., reducing the complexities of training a machine learning model) because there are many examples of high quality trailers, and the encoder can be configured (e.g., trained) in an unsupervised manner, which may avoid the labeling of training data. The trained encoder is used to select which video frames to include in the visual summarization using the tokenized video data as an input. The output of the encoder is a frame mask, where the frame mask may be a vector of values, where each value corresponds to a separate video frame, and the actual value (e.g., first value or second value) may indicate the presence or omission of a respective frame. In some examples, representing the visual summarization with a frame mask may provide one or more technical benefits of improved integration with a language model configured as a visual summarization editor because the frame mask may be a smaller set of data than the tokenized output or the image data of the video frames and the format of the frame mask (e.g., digital data in the form of one or more vectors) can be modified based on prompt(s) to the language model.

[0016] The video snippet generator selects a subset of video frames from the video frames of the media content item using the frame mask. In some examples, the video snippet generator removes video frames from the video data using the frame mask, where the remaining frames are the ones used for the visual summarization. In some examples, the video snippet generator includes a generative model (e.g.. a visual -to-audio model) that receives the visual summarization and generates audio data for the visual summarization. In some examples, the audio data is music, generated by the visual-to-audio model, which accompanies the visual summarization. In some examples, the video snippet generator may determine a number of frame groups within the frames of the subset, and, for each frame group, generate audio data that is specific to that frame group. These and other features are further described with reference to the figures.

[0017] FIGS. 1 A to 1C illustrate a video snippet generator 102 that receives video data 104 of a media content item and programmatically generates a visual summarization 122 of the video data 104. The video data 104 may be a movie, a program, show, or any media that can be played back on a display device. In some examples, the video data 104 can bestreamed from a streaming service to the display device. In some examples, the video data 104 is a video file that includes the video and audio data of a particular media content item (e.g., a movie, program, series, etc.). The visual summarization 122 may be a media trailer (e.g., a preview, a short promotional video) for the media content item. The visual summarization 122 includes a subset of video frames from the larger video file, which has been selected by one or more neural networks. The video snippet generator 102 may create visual summarizations 122 for media content items without user involvement (or minimal user involvement) and / or computationally expensive software executing on a client device. The visual summarization 122 may be a high quality summary (e.g., inclusion of key scenes from the movie without providing spoilers), which can be generated from the video data more quickly, and with fewer computational resources, than previous summarizations. The visual summarization may generate viewer interest, convey information, and / or create a mood. In some examples, the video snippet generator 102 may generate audio data (e.g., music) that accompanies the replay of the visual summarization 122.

[0018] The video data 104 includes a plurality of video frames 106 (e.g., thousands or millions of frames). In some examples, the video data 104 includes two-dimensional (2D) video data. In some examples, the video data 104 includes three-dimensional (3D) video data. A video frame 106 is a digital representation of an image (e.g., single image) captured at a specific point in time by a video camera or generated by a computer program. A video frame 106 includes pixels, where each pixel includes information about its color and brightness. In some examples, the video data 104 includes a video file with the video frames 106. In some examples, the video data 104 is raw video data, which may include a stream of information captured by one or more video cameras. In some examples, the video data 104 is uncompressed or unencoded (e.g., not compressed or encoded using a video codec such as H.264 or VP9). In some examples, the video data 104 is not packaged into a container format (e.g., MP4 or WebM). The container format includes the encoded video data along with additional information such as audio streams, subtitles, and metadata. In some examples, the video data 104 includes an encoded and / or packaged video stream capable of being streamed to a display device.

[0019] The video snippet generator 102 includes a vision transformer 108. The vision transformer 108 receives the video data 104 and generates tokenized video data 110 using the video data 104. In other words, the vision transformer 108 may convert the imagery (e.g., RGB data of the pixels) of the video data 104 to numerical values (e.g.. tokenized video data 110) that can be processed by one or more neural networks (e.g., an encoder 114) that is / aretrained to select the video frames 106 for the visual summarization 122. Since the video data 104 may include up to millions video frames 106, tokenizing the video data 104 may reduce (e.g., significantly reduce) the amount of information to be evaluated for inclusion in the visual summarization 122.

[0020] The tokenized video data 110 includes tokens 112. For an image of a video frame 106 (e.g., a single video frame), the vision transformer 108 generates a sequence of tokens 112 that represents the image. In some examples, the sequence of tokens 112 may be referred to as an embedding vector. In other words, an embedding vector (e g., a sequence of tokens 112) may represent an image in a video frame 106. In some examples, an embedding vector is generated for each video frame 106. An embedding vector (e.g., the sequence of tokens 112) may encapsulate visual content in a symmetric space. An individual token 112 may represent the visual information in a specific image region (e.g., area or patch or a subportion) of the image. Each token 112 may be a vector that includes numerical values, and these numerical values represent the information extracted from a corresponding image region (e.g., an image patch) of an image. The tokenized video data 110 or the tokens 112 may be referred to as embeddings or embedding vectors.

[0021] The vision transformer 108 may be referred to as a vision transformer (ViT) encoder. In some examples, the vision transformer 108 includes one or more neural networks. In some examples, the vision transformer 108 includes one or more transformerbased neural networks. In some examples, the vision transformer 108 does not include a convolutional neural network. In some examples, the vision transformer 108 includes transformer-based neural networks and one or more convolutional neural networks. In some examples, the vision transformer 108 includes one or more convolutional neural networks.

[0022] The vision transformer 108 applies a tokenization process to the visual content (e.g., the imagery) within each video frame 106. In some examples, the vision transformer 108 divides the image of a video frame 106 into smaller image regions (e.g., patches). The vision transformer 108 may convert each region into a token 112, capturing the information within that specific region of the video frame 106. The vision transformer 108 may arrange the individual tokens 112 into a sequence, representing the video frame 106 (e.g., the entire video frame). In some examples, the vision transformer 108 may apply positional encoding to the tokenized video data 110 by inserting relatively information about region locations within the image. In some examples, positional encoding may be applied using learned embeddings and / or sine / cosine functions.

[0023] As shown in FIG. IB, the vision transformer 108 may include a distributed computing system 128 with a plurality of inference processors 130 configured to execute parallel processing to tokenize the video data 104, which can decrease the time required to tokenize the video data 104. In some examples, the number of video frames 106 may be quite large, and parallel processing to tokenize the video data 104 may decrease the time required to generate a visual summarization 122. Each inference processor 130 may tokenize a portion of the video data 104 in parallel (e.g., at least partially in parallel) with one or more other inference processors 130. In some examples, the inference processors 130 include machined earning (ML) processing nodes. In some examples, the inference processors 130 include dynamically scalable ML nodes.

[0024] The inference processors 130 may include an inference processor 130-1, an inference processor 130-2, and an inference processor 130-3. Although three inference processors are depicted in FIG. IB, the number of inference processors 130 may be greater than three (e.g., four, five, six, or any number greater than six). Each inference processor 130 may receive a separate portion of the video data 104 to be tokenized (e.g., a separate portion of the video frames 106 that represent the media content item) and may perform a tokenization process at least partially in parallel with one or more other inference processors 130.

[0025] The inference processor 130-1 may receive a video data portion 104-1 (e.g., first video frames), the inference processor 130-2 may receive a video data portion 104-2 (e.g., second video frames), and the inference processor 130-3 may receive a video data portion 104-3 (e.g., third video frames). The video data portion 104-1, the video data portion 104-2, the video data portion 104-3 may be separate portions of the video data 104. The inference processor 130-1 may generate a tokenized video data portion 106-1 from the video data portion 104-1, the inference processor 130-2 may generate a tokenized video data portion 106-2 from the video data portion 104-2, and inference processor 130-3 may generate a tokenized video data portion 106-3 from the video data portion 104-3. The inference processor 130-1, the inference processor 130-2, and the inference processor 130-3 may execute at least partially in parallel with each other.

[0026] In some examples, the vision transformer 108 may increase (e.g., scale up) or decrease (e.g., scale down) the number of inference processors 130, which may depend on the size of the video data 104. In some examples, the video snippet generator 102 may be used to generate visual summarizations 122 for a plurality of media content items (e.g., a batch of media content items), and the vision transformer 108 may increase or decrease the number ofinterference processors 130 based on the estimated amount of video data 104 to be processed (e.g., the aggregated amount of video data 104 for the media content items). For example, a media platform (e.g., a streaming platform) may periodically determine (e.g., once a week, once a month, etc.) which media content items do not have a program trailer (e.g., a visual summarization 122), and, for those media content items, the vision transformer 108 may be initiated to create visual summarizations 122 so that a trailer or preview can be rendered.

[0027] In some examples, the vision transformer 108 may detect a size of the video data 104 and determine a number of inference processors 130 based on the size of the video data 104. For example, if the size of the video data 104 is equal to or greater than a first threshold, the vision transformer 108 may determine the number of inference processors 130 as a first number (e.g., two, five, six, etc.). If the size of the video data 104 is equal to or greater than a second threshold, the vision transformer 108 may determine the number of inference processors 130 as a second number, where the second number is higher than the first number. If the size of the video data 104 is equal to or greater than a third threshold, the vision transformer 108 may determine the number of inference processors 130 as a third number, where the third number is greater than the second number. The vision transformer 108 may initiate the determined number of inference processors 130. The vision transformer 108 may divide the video data 104 into the appropriate number of portions and provide each portion to their respective inference processor 130.

[0028] Referring back to FIG. 1A. the video snippet generator 102 includes an encoder 114 that receives the tokenized video data 110 (e.g., ordered tokens 1 12) as an input and generates a frame mask 116 as an output, where the frame mask 116 includes a sequence of values 118 (e.g.. a mask vector) (e.g., [0, 0, 1, 1, 1, 0, 0,. . . ]) with a length that corresponds to the number of video frames 106 in the media content item. In other words, the encoder 114 generates a frame mask 116 using the tokenized video data 1 10 as an input. Each value 118 in the frame mask 116 corresponds to a different video frame 106 in the video data 104, and the numerical value (e.g., “1” or “0”) of a respective value 118 indicates the presence or omission of a respective video frame 106 in the visual summarization 122.

[0029] In some examples, the encoder 114 is a bi-directional encoder configured to determine a value 118 (e.g., "‘0” or “1”) for a video frame 106 (e g., whether it should be included in the visual summarization 122 or not) by using contextual data from both directions (e.g., one or more video frames 106 before a currently evaluated frame and one or more video frames 106 after the currently evaluated frame in the timeline of the video data 104). In other words, the encoder 114 may analyze the tokenized video data 110 (e.g., thevideo frames in tokenized format) in two directions. For example, for a particular video frame 106 (or a group of video frames 106). the encoder 114 may analyze information from previous video frames 106 in the timeline of the media content item and information in subsequent video frames 106 in the timeline of the media content item. By considering the context around a video frame 106, the encoder 114 configured as a bi-directional encoder may achieve a deeper understanding of the meaning or relationship between image data in the sequence of video frames 106. In some examples, the encoder 114 includes one or more neural networks. In some examples, the encoder 114 includes one or more transformer-based neural networks. In some examples, the encoder 114 includes one or more convolutional neural networks. In some examples, the encoder 114 includes transformer-based neural network(s) and convolutional neural network(s). In some examples, a portion of the encoder 114 is implemented on a distributed computing system.

[0030] The encoder 114 may be trained to generate a frame mask 116 for a media content item using pairs of programs (e.g.. media content items) and their trailers (e.g., visual summarizations). In some examples, the encoder 114 may provide one or more technical benefits of a computationally efficient training process (e.g., reducing the complexities of training a machine learning model) because there are many examples of high quality trailers, and the encoder 114 can be configured (e.g., trained) in an unsupervised manner, which may avoid the labeling of training data.

[0031] The frame mask 116 includes a sequence of values 118, and the number of values 1 18 corresponds to the number of video frames 106 in the video data 104. In some examples, the frame mask 116 is a vector (e.g., a mask vector) having a sequence of values 118 (e.g.. [0, 0. 1, 1, 1, 0, 0,... ]) with a length that corresponds to the number of video frames 106 in the video data 104 of a media content item. Each value 118 in the frame mask 116 corresponds to a respective video frame 106 (e.g., value #1 corresponds to video frame #1, value #2 corresponds to video frame #2, value #3 corresponds to video frame #3, and so forth). A value 118 in the frame mask 116 may indicate a level of relevance of a respective video frame 106 to the visual summarization 122. A particular value 118 may be a first value (e.g.. "‘0”) or a second value (e.g., “1”). The first value may indicate that its corresponding video frame 106 is not selected for inclusion in the visual summarization 122. The second value may indicate that its corresponding video frame 106 is selected for inclusion in the visual summarization 122. In some examples, a particular value 118 in the frame mask 116 may be one of a series of numerical values (e.g., multiple values between 0 and 1) (e.g., a value closer to 1 may indicate a higher probability of inclusion, a value closer to 0 mayindicate a low' probability of inclusion). It is noted that a particular value 118 in the frame mask 116 is not limited to ones and zeros (or values between zero and one) but may include other values.

[0032] The video snippet generator 102 includes a frame selector 120 that selects a subset (e.g., a portion) of the video frames 106 for inclusion in the visual summarization 122 using the frame mask 116. In some examples, the frame selector 120 executes a down selection operation that omits (e.g., removes) video frames 106 from the video data 104. In some examples, the frame selector 120 determines whether a value 118 in the frame mask 116 is the first value or the second value. In response to the value 118 being the first value, the frame selector 120 does not select the corresponding video frame 106 for inclusion in the visual summarization 122. In response to the value 118 being the second value, the frame selector 120 selects the corresponding video frame 106 for inclusion in the visual summarization 122.

[0033] In some examples, the video snippet generator 102 includes a generative model 126 configured to receive the visual summarization 122 and generate audio data (e.g.. music) for the visual summarization 122. The generative model 126 may be a visual -to-audio generative model. In some examples, the audio data is music that accompanies the visual summarization 122. In some examples, the generative model 126 receives the selected video frames 106 of the visual summarization 122 as an input and generates audio data as an output.

[0034] In some examples, as shown in FIGS. 1 A and 1C, the video snippet generator 102 includes a frame group calculator 124 that computes a number of frame groups 132 in the video data 104 using the frame mask 116. In some examples, a frame group 132 is referred to as a frame cluster (e.g., a set of temporally coherent video frames). In FIG. 1C, the video frames 106 with the white boxes are selected for inclusion in the visual summarization 122, and the video frames 106 with the block boxes are not selected for inclusion in the visual summarization 122. The frame group calculator 124 may use the frame mask 116 to determine the number of frame groups 132 such as a frame group 132-1, a frame group 132- 2, and a frame group 132-3. A frame group 132 may include sequentially adjacent video frames that correspond to a portion of the media content item. The video frames 106 that form part of a frame group 132 are part of a scene or multiple scenes that includes similar subject matter.

[0035] For each frame group 132, the generative model 126 may generate audio data that is specific to that frame group 132. In other words, the generative model 126 maycompute multiple inferences, e.g., one inference per frame group 132. In some examples, generating audio data per frame group 132 may increase the accuracy of the mood of the underlying scene and / or decrease the use of computing resources by breaking apart the otherw ise large inference operation into smaller ones. The generative model 126 may generate audio data using the video frames 106 of a corresponding frame group 132 as an input to the generative model 126. The audio generated by the generative model 126 may be included in the visual summarization 122.

[0036] In some examples, the frame group calculator 124 may identify a frame group 132-1, a frame group 132-2, and / or a frame group 132-3. The generative model 126 may generate first audio data using the frame group 132-1 (e.g., the video frames 106 that form the frame group 132-1) as an input to the generative model 126. The generative model 126 may generate second audio data using the frame group 132-2 (e.g., the video frames 106 that form the frame group 132-2) as an input to the generative model 126. The generative model 126 may generate third audio data using the frame group 132-3 (e.g., the video frames 106 that form the frame group 132-3) as an input to the generative model 126. The visual summarization 122 may include the first audio data, the second audio data, and / or the third audio data. The first audio data is configured to play during playback of the frame group 132-1 (e.g., the video frames 106 that form the frame group 132-1). The second audio data is configured to play during playback of the frame group 132-2 (e.g., the video frames 106 that form the frame group 132-2). The third audio data is configured to play during playback of the frame group 132-3 (e.g., the video frames 106 that form the frame group 132-3).

[0037] FIGS. 2A and 2B illustrate a streaming system 200 that includes a video snippet generator 202 for automatically generating visual summarizations 222 for media content items 208 that are hosted, streamed, or identified by a media platform 204. The video snippet generator 202 is an example of the video snippet generator 102 of FIGS. 1A to 1C and may include any of the details discussed with reference to FIGS. 1A to 1C. It will be understood that streaming system 200 is just one example implementation, and that the video snippet generator 102, 202 described herein can be implemented by any suitable system or apparatus comprising one or more processors.

[0038] The video snippet generator 202 may generate a visual summarization 222 for one or more media content items 208 (e.g., movies, programs, shows, sports events, etc.) available for playback on a display device 252 or to be published on the media platform 204. For example, some of the media content items 208 available for selection via a user interface 264 may have trailers created by their content creators. For example, a media content item208 that is published or will be published to the media platform 204 may already include a trailer. When the media content item 208 is published to the media platform 204 (thereby becoming available to be viewed and streamed), the item’s video file and trailer’s video file are stored in the media provider database 205 (or on a streaming platform 228). A user can initiate viewing of the trailer from the user interface 264 and decide whether to initiate playback of the media content item 208. However, some of the media content items 208 may not have trailers or previews.

[0039] In some examples, for a media content item 208 that does not include an existing trailer, the video snippet generator 202 may programmatically generate a visual summarization 222 (e.g.. without user interaction or minimal user interaction). In some examples, the media platform 204 (e.g., a streaming platform) may periodically determine (e.g., once a week, once a month, etc.) which media content items 208 do not have a program trailer (e.g., a visual summarization 222), and, for those media content items 208, the video snippet generator 202 may be initiated to create visual summarizations 222 so that a trailer or preview can be rendered. In some examples, the media platform 204 may transmit notifications to entities that own or manage the media content items 208 about the generated visual summarizations 222, and, upon receipt of an approval indication 223, may publish the visual summarization 222 to the media platform 204. In some examples, an entity may use a chat interface 272 to edit / change parts of the visual summarization 222. In some examples, when publishing a media content item 208 to the media platform 204, a content creator (or an entity7associated with the content creator) may use one or more user interfaces that communicate with the video snippet generator 202 to create a visual summarization 222 for the media content item 208 and / or edit / change the visual summarization 222.

[0040] When published to the media platform 204, the media platform 204 may provide information that identifies the visual summarization 222 in the user interface 264. In response to selection of the visual summarization 222, the media platform 204 may transmit (e.g., stream) the visual summarization 222 to the display device 252. At the end of the visual summarization 222, the user interface 264 may provide a selectable option to initiate playback of the corresponding media content item 208.

[0041] In some examples, an entity (e.g., program creator, producer, studio entity', post-production house entity', etc.) associated with a media content item 208 may use a display device 252a to initiate generation of a visual summarization 222 for a media content item 208 that has been published or will be published to the media platform 204. In some examples, the media platform 204 may programmatically create a visual summarization 222,and then transmit a notification to the entity that requests approval and / or permits the entity to revise the visual summarization 222.

[0042] In response to a creation request received from the display device 252a, the media platform 204 may cause the video snippet generator 202 to generate a visual summarization 222. In response to the visual summarization 222 being created, the media platform 204 may provide the visual summarization 222 for user selection on the display device 252a. If the entity approves the visual summarization 222, the display device 252 may transmit an approval indication 223 to the media platform 204. In response to the approval indication 223, the media platform 204 may publish the visual summarization 222 on the media platform 204 so that users can view and play the visual summarization 222. In some examples, the visual summarization 222 may be revised. In response to selection of one or more UI elements, the display device 252a may transmit a revision request 221. In response to the revision request 221, the video snippet generator 202 may re-generate the visual summarization 222.

[0043] Referring to FIG. 2B. the display device 252a may provide a chat interface 272 that enables the entity to provide a natural language description 274 about one or more edits to the visual summarization 222. The video snippet generator 202 may operate with a language model 278 to generate a revised visual summarization 222a according to the natural language description 274. In some examples, the chat interface 272 includes an input field that enables the user to submit a natural language description 274. A user may enter text into the input field to define the natural language description 274. In some examples, the chat interface 272 may be a user interface configured to receive text via a voice command and may display the text of the voice command in the input field. The natural language description 274 may be textual data about one or more changes to the visual summarization 222.

[0044] As shown in FIG. 2B, the display device 252a may include a prompt generator 270 configured to generate a prompt 276 with the user’s natural language description 274. In some examples, the revision request 221 includes the prompt 276. The prompt 276 may include a content identifier of the media content item 208. The content identifier may uniquely identity’ the media content item 208 on the media platform 204 and / or the streaming platform 228. In response to submission to the natural language description 274 via the chat interface 272, the prompt generator 270 may generate a prompt 276 with a request to revise and / or edit the visual summarization 222 using the natural language description 274. Prompt 276 can, in other words, include a revision request with a natural language description aboutone or more edits to the visual summarization. In some examples, the prompt generator 270 is included on the media platform 204. For example, the media platform 204 may receive, over a network 250, the natural language description 274, and the prompt generator 270 generates the prompt 276 for use as an input to the language model 278 (e.g., machinelearning (ML) model).

[0045] In some examples, the language model 278 is stored on the server computer(s) 260, which also executes the media platform 204. In some examples, the language model 278 is stored on one or more server computers that are different from the server computer(s) 260 that execute the media platform 204. In some examples, the media platform 204 receives the prompt 276 and then transmits the prompt 276 to the language model 278. In some examples, the language model 278 is stored (e.g., stored locally) on the display device 252a. In some examples, the language model 278 is included as part of the operating system.

[0046] In some examples, the streaming system 200 includes a language model 278 that receives the revision request 221 (e.g., the prompt 276) and communicates with the video snippet generator 202 to generate a revised visual summarization 222a. In some examples, the language model 278 receives and / or retrieves the frame mask 116 and modifies the frame mask 116 according to the natural language description 274. In some examples, the prompt generator 270 generates the prompt 276 with the natural language description 274 and the frame mask 116 with a request to modify the frame mask 116 according to the natural language description 274 to generate a revised frame mask. In some examples, the modified or revised frame mask 1 16 is provided to the frame selector 120 and the generative model 126 for use in generating a revised visual summarization. In some examples, in response to the prompt 276, the language model 278 communicates with the video snippet generator 202 to retrieve the frame mask 116, and then modifies the frame mask 116 according to the natural language description 274. The media platform 204 may provide the revised visual summarization 222a to the display device 252a for viewing. The user may continue to submit changes via the chat interface 272 and receive revised visual summarizations 222a.

[0047] The streaming system 200 includes a media platform 204 executable by one or more server computers 260 and a media application 256 executable by a display device 252. The media platform 204 may be a server-based television platform or streaming platform. In some examples, the media application 256 is (or is a subcomponent of) an operating system251 of the display device 252. In some examples, the media application 256 is a native application (e.g.. a standalone native application), which is preinstalled on the display device252 or downloaded to the display device 252 from a digital media store (e.g., play store,application store, etc.). The media application 256 may communicate with the media platform 204 to identify media content 206 (e.g., media content items 208) that is available for streaming to the display device 252. The media content includes a plurality of media content items 208. In some examples, the media content 206 includes media content items 208 that are stored on the media platform 204 and streamed from the media platform 204 to the media application 256. Some of the media content items 208 are associated with visual summarizations 222. and the visual summarizations 222 are stored with their media content items 208 on the media platform 204. In some examples, the media content 206 includes media content items 208 that are stored on one or more (other) streaming platforms 228 and streamed from the streaming platforms 228 to their respective streaming applications 254.

[0048] In some examples, the media application 256 is a media aggregator application that determines which providers (e.g., streaming platforms 228, associated streaming applications 254) the user has access rights to, and then identifies media content items 208, across those providers, in the user interface 264 for selection and playback. For example, the media application 256 (e.g., in conjunction with the media platform 204) may aggregate (e.g., combine, assemble, collect, etc.) information about media content 206 available for viewing (e.g., streaming) from multiple streaming platforms 228 and present the information in the user interface 264 (e.g., a single, unified user interface) so that a user can identify and / or search media content 206 across different streaming platforms (e.g., without having to search within each streaming application 254). The media content items 208 may be individual programs offered by streaming platforms 228, including the media platform 204. For example, each media content item 208 may be a program (e.g., a television show a movie, a live broadcast, etc.) from the media platform 204 or another streaming platform 228. Instead of searching for media content items 208 on a first streaming application and media content items 208 on a second streaming application, the media application 256 may combine the media content items 208 together in one interface (e.g., user interface 264) so that a user can search across multiple streaming platforms 228 at once.

[0049] In some examples, a media content item 208 may correspond to a digital video file (e.g.. video data 104 of FIGS. 1A to 1 C), which may be stored on the streaming platforms 228 (including the media platform 204) and / or the display device 252. In some examples, the media platform 204 is also considered a streaming platform 228, which may store and provide digital video files for streaming or downloading. The digital video file may include video and / or audio data that corresponds to a particular media content item 208. In some examples, the media platform 204 is configured to communicate with the streaming platforms228 to identify which media content items 208 are available on the streaming platforms 228 and may update a media provider database 205 to identify the media content items 208 offered by the streaming platforms 228.

[0050] For example, the media platform 204 may communicate, over a network 250, with the streaming platforms 228 to identify which media content items 208 are available to be streamed by display devices 252 and update a media provider database 205. The media platform 204 may identify a set or multiple sets of media content items 208 (e.g.. across the various streaming platforms 228) as recommendations to a user of the media application 256. In some examples, the media platform 204 may determine whether the user of the media application 256 has rights (e g., stored as entitlement data 212) to stream media content items 208 from one or more of the streaming platforms 228 (e.g., whether the user has subscribed to access media content items 208 from the streaming platform(s) 228), and, if so, may include those media content items 208 as candidates in a selection (e.g., ranking) mechanism to potentially be displayed in the user interface 264 of the media application 256.

[0051] The media application 256 includes a user interface 264 that identifies media content items 208 for selection and playback on the display device 252. In response to selection of a media content item 208, the media application 256 may initiate playback of the media content item 208 on a display 262 of the display device 252. In some examples, in response to selection of the media content item 208. the media platform 204 streams the media content item 208 to the media application 256, which causes the media application 256 to display the media content item 208 on the display 262. In some examples, in response to selection of the media content item 208 from the user interface 264 of the media application 256, the media application 256 causes the content’s underlying streaming application 254 to playback the media content item 208.

[0052] In some examples, selection of a media content item 208 from the user interface 264 may cause the media application 256 to launch a streaming application 254 (e.g., using a content deep link) associated with the streaming application 254. In some examples, selection of a media content item 208 from the user interface 264 causes the media application 256 to render another user interface (e.g., item’s landing page), and further selection of the media content item 208 from the item’s landing page causes the media application 256 to launch the underlying streaming application 254. In some examples, the media content item 208 may be associated with a specific provider in which the media content item 208 is streamed from a streaming platform 228 (e.g., the media platform 204itself or another streaming platform 228). In some examples, the user can control the playback of the media content item 208 from the corresponding streaming application 254.

[0053] A content deep link, corresponding to a media content item 208, may be an identifier that identifies the location of the media content item 208 in the streaming application 254. The media application 256 may transfer the content deep link to the corresponding streaming application 254. In some examples, the content deep link identifies a specific landing page (e.g.. an interface) within the streaming application 254 that corresponds to the media content item 208. In some examples, the content deep link is an operating system intent. In some examples, the content deep link is a uniform resource locator (URL). In some examples, the content deep link includes a URL format.

[0054] Streaming (or playback) of the media content item 208 may refer to the transmission of the contents of a video file (e.g., the video data 104 of FIGS. 1 A to 1C) from a streaming platform 228 or the media platform 204 to the display device 252 that displays the contents of the video file. In some examples, streaming (or playback) of the media content item 208 may refer to a continuous video stream that is transferred from one place to another place in which a received portion of the video stream is displayed while waiting for other portions of the video stream to be transferred. In some examples, after the media content item 208 is published on the media platform 204 (e.g., is live), the display device 252 may stream or download the contents of the video file.

[0055] In some examples, the user interface 264 may identify a plurality of media content items 208, which may be selected by the media platform 204 from the media provider database 205 based at least in part on information representing the user's interests and activities (e.g., the user's search queries, search results, previous watch history, purchase history, application usage history, application installation history, user actions on the network-connected display device, physical activities of the user, etc.). In some examples, the media application 256 may be associated with a user account 210, and the user account 210 may store the information representing the user’s interests and activities (e.g., user activity information 214), and the media platform 204 may use this information to select and present the media content items 208 in the user interface 264. In some examples, the media content items 208 may be organized as a plurality of clusters based on one or more categories, such as content type (e.g., “Action Movies”), viewing history (e.g., “Because You watched Movie ABC”), release time (e.g., “Trending”), and the like. In some examples, the media content items 208 provided by different streaming platforms 228 (e.g.. action movies from two different streaming platforms 228) can be recommended in the same cluster. Insome examples, the user interface 264 may include tabbed interfaces, where one of the tabbed interfaces includes personalized media content that is organized as a plurality of clusters based on one or more categories, such as release time (e.g., '‘This Week,’’ “Next week,” “Next Month,” etc.), user action and user application interaction, native app usage (e.g., items that are “From App ABC”), etc.

[0056] The media platform 204 may store user accounts 210, where each user account 210 stores information about a respective user. A user account 210 may store entitlement data 212 and / or user activity information 214. The entitlement data 212 includes information that identifies which providers (e.g., streaming platforms 228, streaming applications 254) that the user account 210 has access rights to view content. In some examples, the access rights are determined based on the user account 210 (e.g.. whether the user has subscribed to one or more streaming applications 254), which streaming applications 254 are installed on the display device 252 and / or if the user has accessed (e.g., logged-into) a user account associated with those streaming applications 254. In response to certain user activityregarding media content items 208, the media platform 204 may update the user activity information 214 with information about the activity such as a content identifier, the date / time. and / or the watch duration of the media content item 208, etc.

[0057] It is noted that a user of the media application 256 may be provided with controls allowing the user to make an election as to both if and when the streaming system 200 may enable the collection of information representing the user's interests and activities. In addition, certain data may be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user’s identity may be treated so that no personally identifiable information can be determined for the user, or a user’s geographic location may be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user of the media application 256 may have control over what information is collected about the user, how that information is used, and what information is provided to the user and / or to the server computer 260.

[0058] The display device 252 includes one or more processors, one or more memory devices, and an operating system 251 configured to execute (or assist with executing) one or more streaming applications 254. The one or more memory' devices may be a non-transitory computer-readable medium storing executable instructions that cause the one or more processors to execute operations discussed herein. The display device 252 may be any type of user device. In some examples, the display device 252 is a television device (e.g., a smarttelevision). In some examples, the display device 252 is a smartphone, a laptop computer, a desktop computer, a gaming console, and / or a wearable device such as a head-mounted display device. In some examples, the display device 252 is an augmented reality (AR) or virtual reality (VR) device. The streaming applications 254 may include a media application 256 configured to communicate, over the network, 250, with a media platform 204 executable by one or more server computers 260. In some examples, the media application 256 is a program that is part of the operating system 251. In some examples, the media application 256 is a separate standalone application that is downloaded and installed on the operating system 251. In some examples, the media application 256 may execute operation(s) discussed with reference to the operating system 251 (and / or vice versa). In some examples, the display device 252 is not a smart television, but is converted to a smart television when connected to a casting device, where the casting device is configured to connect to the network 250 and execute an operating system 251 configured to execute streaming applications 254, including the media application 256.

[0059] In some examples, the operating system 251 is a browser application. A browser application is a web browser configured to access information on the Internet and may launch one or more browser tabs in the context of one or more browser windows. In some examples, the operating system 251 is a Linux-based operating system. In some examples, the operating system 251 is a mobile operating system that is also configured to execute on smaller devices (e.g.. smartphones, tablets, wearables, etc.).

[0060] The server computer 260 may be computing devices that take the form of a number of different devices, for example a standard server, a group of such servers, or a rack server system. In some examples, the server computer 260 may be a single system sharing components such as processors and memories. The network 250 may include the Internet and / or other types of data networks, such as a local area network (LAN), a wide area network (WAN), a cellular network, satellite network, or other types of data networks. The network 250 may also include any number of computing devices (e.g., computers, servers, routers, network switches, etc.) that are configured to receive and / or transmit data within network 250. Network 250 may further include any number of hardwired and / or wireless connections.

[0061] The server computer 260 may include one or more processors formed in a substrate, an operating system (not shown) and one or more memory devices. The memory' devices may represent any kind of (or multiple kinds of) memory (e.g.. RAM, flash, cache, disk, tape, etc.). In some examples (not shown), the memory devices may include externalstorage, e.g., memory' physically remote from but accessible by the server computer 260. The server computer 260 may include one or more modules or engines representing specially programmed software.

[0062] FIG. 3 is a flowchart 300 depicting example operations of automatic trailer generation according to an aspect. The flowchart 300 may depict operations of a computer- implemented method. The flowchart 300 may depict operations of a non-transitory computer-readable medium having executable instructions that when executed by one or more processors cause the one or more processors to execute the operations of the flowchart 300. Although the flowchart 300 of FIG. 3 illustrates the operations in sequential order, it will be appreciated that this is merely an example, and that additional or alternative operations may be included. Further, operations of FIG. 3 and related operations may be executed in a different order than that shown, or in a parallel or overlapping fashion.

[0063] Operation 302 includes receiving video data associated with a media content item, the video data including a plurality of video frames. Operation 304 includes generating, by a vision transformer, tokenized video data using the video data. The tokenized data can comprise an embedding vector for each video frame. For example, generating, by a vision transformer, tokenized video data using the video data can optionally comprise generating a sequence of tokens 112 that represents the image, where the sequence of tokens 112 is referred to as an embedding vector. Operation 306 includes generating, by an encoder, a frame mask based on the tokenized video data. Operation 308 includes selecting a subset of the plurality of video frames based on the mask. Operation 310 includes generating a visual summarization of the media content item, the visual summarization including the subset. Optionally, flowchart 300 may be initiated by receiving a request to generate a visual summarization of the video data associated with the media content item. The request may be received from a user, device, system or application, for example. Optionally, the generated visual summarization may be provided to the requesting user, device, system or application.

[0064] Clause 1. A method comprising: receiving video data associated with a media content item, the video data including a plurality of video frames: generating, by a vision transformer, tokenized video data using the video data; generating, by an encoder, a frame mask based on the tokenized video data; selecting a subset of the plurality of video frames based on the frame mask; and generating a visual summarization of the media content item, the visual summarization including the subset.

[0065] Clause 2. The method of clause 1, further comprising: identifying a group of video frames within the subset using the frame mask; and generating, by a generative model.audio data using the group of video frames as an input to the generative model, the visual summarization including the audio data.

[0066] Clause 3. The method of clause 2, wherein the group of video frames is a first group of video frames, and the audio data is first audio data, the method further comprising: identifying a second group of video frames within the subset using the frame mask; and generating, by the generative model, second audio data using the second group of video frames as an input to the generative model, the first audio data configured to play during playback of the first group of video frames, the second audio data configured to play during playback of the second group of video frames.

[0067] Clause 4. The method of any of clauses 1 to 3, wherein the vision transformer includes a distributed computing system with a first inference processor and a second inference processor, the video data including a first video data portion and a second video data portion, the method further comprising: generating, by the first inference processor, a first tokenized video data portion from the first video data portion; and generating, by the second inference processor, a second tokenized video data portion from the second video data portion, wherein the second tokenized video data portion is generated at least partially in parallel with generation of the first tokenized video data portion.

[0068] Clause 5. The method of any of clauses 1 to 4, further comprising: transmitting, by a media platform, information that identifies the media content item and the visual summarization in a user interface of a media application of a computing device; receiving, by the media platform, a selection to the visual summarization; and streaming, by the media platform, the visual summarization.

[0069] Clause 6. The method of any of clauses 1 to 5, further comprising: transmitting the visual summarization to a display device: receiving a revision request with a natural language description about one or more edits to the visual summarization; generating, by a language model, a revised frame mask using the natural language description; and generating a revised visual summarization based on the revised frame mask.

[0070] Clause 7. The method of any of clauses 1 to 6, wherein the frame mask includes a sequence of values with a length that corresponds to a number of the plurality of video frames.

[0071] Clause 8. The method of clause 7, wherein a value in the sequence of values includes a first value or a second value, the first value indicating an exclusion of a respective video frame in the visual summarization, the second value indicating an inclusion of the respective video frame in the visual summarization.

[0072] Clause 9. An apparatus comprising: at least one processor; and a non- transitory computer-readable medium storing executable instructions that cause the at least one processor to execute operations, the operations comprising: receiving video data associated with a media content item, the video data including a plurality of video frames; generating, by a vision transformer, tokenized video data using the video data; generating, by an encoder, a frame mask based on the tokenized video data; selecting a subset of the plurality of video frames based on the frame mask; and generating a visual summarization of the media content item, the visual summarization including the subset.

[0073] Clause 10. The apparatus of clause 9, wherein the operations further comprise: identifying a group of video frames within the subset using the frame mask; and generating, by a generative model, audio data using the group of video frames as an input to the generative model, the visual summarization including the audio data.

[0074] Clause 11. The apparatus of clause 10, wherein the group of video frames is a first group of video frames, and the audio data is first audio data, wherein the operations further comprise: identifying a second group of video frames within the subset using the frame mask; and generating, by the generative model, second audio data using the second group of video frames as an input to the generative model, the first audio data configured to play during playback of the first group of video frames, the second audio data configured to play during playback of the second group of video frames.

[0075] Clause 12. The apparatus of any of clauses 9 to 11, wherein the vision transformer includes a distributed computing system with a first inference processor and a second inference processor, the video data including a first video data portion and a second video data portion, wherein the operations further comprise: generating, by the first inference processor, a first tokenized video data portion from the first video data portion; and generating, by the second inference processor, a second tokenized video data portion from the second video data portion, wherein the second tokenized video data portion is generated at least partially in parallel with generation of the first tokenized video data portion.

[0076] Clause 13. The apparatus of any of clauses 9 to 12, wherein the operations further comprise: transmitting, by a media platform, information that identifies the media content item and the visual summarization in a user interface of a media application of a computing device; receiving, by the media platform, a selection to the visual summarization; and streaming, by the media platform, the visual summarization.

[0077] Clause 14. The apparatus of any of clauses 9 to 13, wherein the operations further comprise: transmitting the visual summarization to a display device; receiving arevision request with a natural language description about one or more edits to the visual summarization; generating, by a language model, a revised frame mask using the natural language description; and generating a revised visual summarization based on the revised frame mask.

[0078] Clause 15. The apparatus of any of clauses 9 to 14, wherein the frame mask includes a sequence of values with a length that corresponds to a number of the plurality of video frames.

[0079] Clause 16. The apparatus of clause 15, wherein a value in the sequence of values includes a first value or a second value, the first value indicating an exclusion of a respective video frame in the visual summarization, the second value indicating an inclusion of the respective video frame in the visual summarization.

[0080] Clause 17. A non-transitory computer-readable medium storing executable instructions that cause at least one processor to execute operations, the operations comprising: receiving video data associated with a media content item, the video data including a plurality of video frames; generating, by a vision transformer, tokenized video data using the video data; generating, by an encoder, a frame mask based on the tokenized video data; selecting a subset of the plurality of video frames based on the frame mask; and generating a visual summarization of the media content item, the visual summarization including the subset.

[0081] Clause 18. The non-transitory computer-readable medium of clause 17. wherein the operations further comprise: identifying a group of video frames within the subset using the frame mask; and generating, by a generative model, audio data using the group of video frames as an input to the generative model, the visual summarization including the audio data.

[0082] Clause 19. The non-transitory computer-readable medium of clause 18, wherein the group of video frames is a first group of video frames, and the audio data is first audio data, wherein the operations further comprise: identifying a second group of video frames within the subset using the frame mask; and generating, by the generative model, second audio data using the second group of video frames as an input to the generative model, the first audio data configured to play during playback of the first group of video frames, the second audio data configured to play during playback of the second group of video frames.

[0083] Clause 20. The non-transitory computer-readable medium of any of clauses 17 to 19, wherein the vision transformer includes a distributed computing system with a firstinference processor and a second inference processor, the video data including a first video data portion and a second video data portion, wherein the operations further comprise: generating, by the first inference processor, a first tokenized video data portion from the first video data portion; and generating, by the second inference processor, a second tokenized video data portion from the second video data portion, wherein the second tokenized video data portion is generated at least partially in parallel with generation of the first tokenized video data portion.

[0084] Clause 21. A computer program product comprising computer-executable instructions which, when executed by at least one computing apparatus, cause the at least one computing apparatus to perform the method of any of clauses 1 to 8.

[0085] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0086] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms “machine-readable medium” “computer-readable medium” refers to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory. Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0087] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid cry stal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as w ell; for example, feedback provided to the user can be any form of sensory feedback(e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0088] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (“LAN”), a wide area network (“WAN”), and the Internet.

[0089] The computing system can include clients and servers. A client and server are remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship with each other.

[0090] In this specification and the appended claims, the singular forms "a," "an" and "the" do not exclude the plural reference unless the context clearly dictates otherwise. Further, conjunctions such as “and,” “or,” and “and / or” are inclusive unless the context clearly dictates otherwise. For example, “A and / or B” includes A alone, B alone, and A with B. Further, connecting lines or connectors shown in the various figures presented are intended to represent example functional relationships and / or physical or logical couplings between the various elements. Many alternative or additional functional relationships, phy sical connections or logical connections may be present in a practical device. Moreover, no item or component is essential to the practice of the implementations disclosed herein unless the element is specifically described as “essential” or “critical”.

[0091] Terms such as, but not limited to, approximately, substantially, generally, etc. are used herein to indicate that a precise value or range thereof is not required and need not be specified. As used herein, the terms discussed above will have ready and instant meaning to one of ordinary skill in the art.

[0092] Moreover, use of terms such as up, down, top, bottom, side, end, front, back, etc. herein are used with reference to a currently considered or illustrated orientation. If they are considered with respect to another orientation, it should be understood that such terms must be correspondingly modified.

[0093] Although certain example methods, apparatuses and articles of manufacture have been described herein, the scope of coverage of this patent is not limited thereto. It is to be understood that terminology employed herein is for the purpose of describing particular aspects and is not intended to be limiting. On the contrary', this patent covers all methods, apparatus and articles of manufacture fairly falling within the scope of the claims of this patent.

Claims

WHAT IS CLAIMED IS:

1. A method comprising: receiving video data associated with a media content item, the video data including a plurality7of video frames; generating, by a vision transformer, tokenized video data using the video data; generating, by an encoder, a frame mask based on the tokenized video data; selecting a subset of the plurality7of video frames based on the frame mask; and generating a visual summarization of the media content item, the visual summarization including the subset.

2. The method of claim 1 , further comprising: identifying a group of video frames within the subset using the frame mask; and generating, by a generative model, audio data using the group of video frames as an input to the generative model, the visual summarization including the audio data.

3. The method of claim 2, wherein the group of video frames is a first group of video frames, and the audio data is first audio data, the method further comprising: identifying a second group of video frames within the subset using the frame mask; and generating, by the generative model, second audio data using the second group of video frames as an input to the generative model, the first audio data configured to play during playback of the first group of video frames, the second audio data configured to play during playback of the second group of video frames.

4. The method of any of claims 1 to 3, wherein the vision transformer includes a distributed computing system with a first inference processor and a second inference processor, the video data including a first video data portion and a second video data portion, the method further comprising: generating, by the first inference processor, a first tokenized video data portion from the first video data portion; and generating, by the second inference processor, a second tokenized video data portion from the second video data portion,wherein the second tokenized video data portion is generated at least partially in parallel with generation of the first tokenized video data portion.

5. The method of any of claims 1 to 4, further comprising: transmitting, by a media platform, information that identifies the media content item and the visual summarization in a user interface of a media application of a computing device; receiving, by the media platform, a selection to the visual summarization; and streaming, by the media platform, the visual summarization.

6. The method of any of claims 1 to 5, further comprising: transmitting the visual summarization to a display device; receiving a revision request with a natural language description about one or more edits to the visual summarization; generating, by a language model, a revised frame mask using the natural language description; and generating a revised visual summarization based on the revised frame mask.

7. The method of any of claims 1 to 6, wherein the frame mask includes a sequence of values with a length that corresponds to a number of the plurality of video frames.

8. The method of claim 7, wherein a value in the sequence of values includes a first value or a second value, the first value indicating an exclusion of a respective video frame in the visual summarization, the second value indicating an inclusion of the respective video frame in the visual summarization.

9. An apparatus comprising: at least one processor; and a non-transitory computer-readable medium storing executable instructions that cause the at least one processor to execute operations, the operations comprising: receiving video data associated with a media content item, the video data including a plurality of video frames; generating, by a vision transformer, tokenized video data using the video data; generating, by an encoder, a frame mask based on the tokenized video data;selecting a subset of the plurality of video frames based on the frame mask; and generating a visual summarization of the media content item, the visual summarization including the subset.

10. The apparatus of claim 9, wherein the operations further comprise: identifying a group of video frames within the subset using the frame mask; and generating, by a generative model, audio data using the group of video frames as an input to the generative model, the visual summarization including the audio data.

11. The apparatus of claim 10. wherein the group of video frames is a first group of video frames, and the audio data is first audio data, wherein the operations further comprise: identify ing a second group of video frames within the subset using the frame mask; and generating, by the generative model, second audio data using the second group of video frames as an input to the generative model, the first audio data configured to play during playback of the first group of video frames, the second audio data configured to play during playback of the second group of video frames.

12. The apparatus of any of claims 9 to 11, wherein the vision transformer includes a distributed computing system with a first inference processor and a second inference processor, the video data including a first video data portion and a second video data portion, wherein the operations further comprise: generating, by the first inference processor, a first tokenized video data portion from the first video data portion; and generating, by the second inference processor, a second tokenized video data portion from the second video data portion, wherein the second tokenized video data portion is generated at least partially in parallel with generation of the first tokenized video data portion.

13. The apparatus of any of claims 9 to 12, wherein the operations further comprise: transmitting, by a media platform, information that identifies the media content item and the visual summarization in a user interface of a media application of a computing device;receiving, by the media platform, a selection to the visual summarization: and streaming, by the media platform, the visual summarization.

14. The apparatus of any of claims 9 to 13, wherein the operations further comprise: transmitting the visual summarization to a display device; receiving a revision request with a natural language description about one or more edits to the visual summarization; generating, by a language model, a revised frame mask using the natural language description; and generating a revised visual summarization based on the revised frame mask.

15. The apparatus of any of claims 9 to 14, wherein the frame mask includes a sequence of values with a length that corresponds to a number of the plurality of video frames.

16. The apparatus of claim 15, wherein a value in the sequence of values includes a first value or a second value, the first value indicating an exclusion of a respective video frame in the visual summarization, the second value indicating an inclusion of the respective video frame in the visual summarization.

17. A non-transitory computer-readable medium storing executable instructions that cause at least one processor to execute operations, the operations comprising: receiving video data associated with a media content item, the video data including a plurality of video frames; generating, by a vision transformer, tokenized video data using the video data; generating, by an encoder, a frame mask based on the tokenized video data; selecting a subset of the plurality of video frames based on the frame mask; and generating a visual summarization of the media content item, the visual summarization including the subset.

18. The non-transitory computer-readable medium of claim 17, wherein the operations further comprise: identifying a group of video frames within the subset using the frame mask; and generating, by a generative model, audio data using the group of video frames as an input to the generative model, the visual summarization including the audio data.

19. The non-transitory computer-readable medium of claim 18, wherein the group of video frames is a first group of video frames, and the audio data is first audio data, wherein the operations further comprise: identifying a second group of video frames within the subset using the frame mask; and generating, by the generative model, second audio data using the second group of video frames as an input to the generative model, the first audio data configured to play during playback of the first group of video frames, the second audio data configured to play during playback of the second group of video frames.

20. The non-transitory computer-readable medium of any of claims 17 to 19, wherein the vision transformer includes a distributed computing system with a first inference processor and a second inference processor, the video data including a first video data portion and a second video data portion, wherein the operations further comprise: generating, by the first inference processor, a first tokenized video data portion from the first video data portion; and generating, by the second inference processor, a second tokenized video data portion from the second video data portion, wherein the second tokenized video data portion is generated at least partially in parallel with generation of the first tokenized video data portion.

21. A computer program product comprising computer-executable instructions which, when executed by at least one computing apparatus, cause the at least one computing apparatus to perform the method of any of claims 1 to 8.

Images