Leveraging insights from real-time media streams into delayed versions

Abstract

A set of techniques utilizes the real-time version of a media stream to gather information about its content and use that information to improve or enhance the service delivery of other versions (e.g., live or video-on-demand) of the same stream. These techniques exploit the fact that the real-time version of the stream is sufficiently ahead in time than other versions to enable the gathering and use of information about the stream content.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. Patent Application No. 17 / 818,112, filed August 8, 2022, the entire contents of which are incorporated herein by reference.

[0002] This disclosure relates to media streaming, which involves the continuous transmission of audio or video files from a collection of servers to one or more clients over a packet-switched network (e.g., 1000 Mbps). [Background technology]

[0003] Media streaming is the dominant form of application traffic on the Internet. In a typical streaming service, video and / or audio data is streamed as a series of packets from a collection of servers. The stream is stored or captured on the server and then transmitted to multiple users for playback. Media streaming services typically offer multiple versions of the same stream, including real-time, live, and archived streams. In such services, archived streams are used for video-on-demand (VoD) streaming.

[0004] Real-time media streaming refers to streaming content that allows for viewer-to-viewer interaction. Real-time streams are transmitted over the Internet without first being recorded and stored. Real-time viewers watch content synchronously, fostering engagement, which enables social interaction, and increases revenue. Non-limiting examples of real-time streaming use delays of less than 0.5 seconds.

[0005] Live media streaming refers to the online streaming of media that is simultaneously recorded and broadcast as it occurs to one or more users. Live media streaming allows for rebuffering of events. Non-limiting examples of live streaming use delays ranging from 5 seconds to a maximum of 1 minute.

[0006] VoD streaming is the streaming of media stored on one or more servers to a user at any point in time, without the timing constraints of real-time or live streaming. [Brief explanation of the drawings]

[0007] [Figure 1] A diagram showing various encodings of streams such as standard definition (SD), low definition (LD), high definition (HD), and super high definition (SHD) at different physical sizes. [Figure 2] This diagram shows the implementation of key frames and predicted frames on chunks and streams. [Figure 3] A diagram illustrating adaptive bitrate streaming. [Figure 4] FIG. 1 illustrates multiple versions of a single stream, including real-time, live, and video-on-demand. [Figure 5] A diagram showing the delay between the real-time version of the stream, the live version, and the archived / video-on-demand version. [Figure 6] 1 is a flowchart illustrating a method for modifying a subsequent multimedia stream. [Figure 7] Figure 1 shows the tagging of scenes in a real-time stream, and the adaptation path for other versions of the stream. [Figure 8] FIG. 1 is a block diagram of an exemplary computing system. [Figure 9]FIG. 1 is a block diagram illustrating an example of a machine learning (ML) system according to one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0008] Different versions of the stream are targeted to different audience sizes and represent different trade-offs in the quality and definition of the media stream. From a scale perspective, real-time and live users tend to be a subset of archive / VoD users who can watch the recorded stream at any time in the future.

[0009] Common technologies used for real-time streaming include the Real-Time Messaging Protocol (RTMP), a TCP-based protocol originally developed for streaming between Flash players and servers, and Web RealTime Web Communications (WebRTC), an open, scalable industry standard that uses UDP for data transport. For live and VoD streaming, common protocols include HTTP Live Streaming (HLS), Microsoft Smooth Streaming, and Dynamic Adaptive Streaming over HTTP (DASH).

[0010] Compressed key frames consist of a sequence of key frames, sometimes called intraframes in VP8 or key frames in MPEG, and predicted or interframes in VP8, and P-frames in MPEG terminology. Key frames are decoded without reference to other frames in the sequence. Video compression improves compression efficiency by using interframe dependencies. These dependencies make the stream more susceptible to packet loss and limit random access. Successful decoding of a predicted frame typically requires successful decoding of the most recent key frame and all subsequent predicted frames. Key frames provide random access (or seek) points within the video stream.

[0011] Decoding algorithms typically cannot tolerate missing key frames because predicted frames are encoded with reference to previous frames (specifically, all previous frames up to the most recent key frame). If a frame is missing or corrupted, proper decoding requires that key frames be received correctly.

[0012] Prior art media streaming systems rely on multi-bitrate (MBR) encoding, in which source video content is encoded with alternative bitstreams at different coding rates. Multiple encoding rates 100 are shown in Figure 1. Example streams include standard definition (SD) 102, low definition (LD) 104, high definition (HD) 106, and super high definition (SHD) 108. Higher quality bitrate encodings are indicated by larger packets. Content is streamed directly or in segments / chunks. Figure 2 illustrates direct packet delivery 200 and chunk-based packet delivery 202. The quality levels corresponding to different encoding rates are based on varying network conditions. Specifically, source content is encoded at multiple bitrates and segmented into small portions, sometimes lasting a few seconds.

[0013] Figure 3 illustrates an application of adaptive bitrate encoding. A streaming client contains multiple available streams at different bitrates and processes segments within the streams (if necessary) based on a manifest file. At the start, the client subscribes to or requests data from a relatively low bitrate stream 302 (300). If the client determines that the download speed allows for a higher bitrate (encoding rate upshift), the client requests the next higher bitrate 306 (304). On the other hand, if the client determines that the download speed is lower than the stream's bitrate (308), the client subscribes to data at a lower bitrate (encoding rate downshift) (310).

[0014] Figure 4 illustrates several versions of a media or multimedia stream. Many media streaming services offer different versions 400 of the same streaming content stream, such as a real-time stream 402, a live stream 404, and an archived VoD stream 406. These services allow users to select a version or determine the stream version on their behalf. For example, if the number of real-time user subscribers is limited, additional users may be assigned to the live stream instead. In one example, all versions of stream 400 may be in use simultaneously, with dozens of users subscribed to the real-time stream 402 version, hundreds of users subscribed to the live stream 404, and thousands or even millions of users subscribed to the VoD stream 406.

[0015] Known systems treat various versions of a stream 400, such as a real-time stream 402, a live stream 404, and a VoD stream 406, as independent streams, thereby forgoing the opportunity to leverage insights across the versions of the stream. This disclosure describes techniques for leveraging live and real-time versions of a media stream to gather information about its content and then using that information to improve or enhance the serving of other versions of the same stream.

[0016] In some embodiments, the streaming platform identifies quality-sensitive scenes within a real-time or live stream to guide the service in delivering subsequent versions (i.e., the live stream 404 version and the VoD stream 406 version). While most streaming services consider all scenes in a version of a stream to be equally important in terms of quality, not all scenes are equal from the viewer's perspective. The challenge faced is distinguishing between quality-sensitive and non-quality-sensitive segments of the stream well in advance so that the delivery device can adapt to the distinction.

[0017] Content tags 408 attached to previous streams enable adaptation. There are several mechanisms for issuing content tags 408 to relevant portions of a stream to indicate that a portion is more important (e.g., rated higher) or less important (e.g., rated lower). In some embodiments, the content tags 408 indicate an increase or decrease in the priority of the stream's encoded delivery.

[0018] In some embodiments, content tags 408 are issued by a human operator. Subscribers to the real-time stream 402 or live stream 404 identify and tag important or insignificant scenes as they occur. In some embodiments, content tags 408 are crowd-sourced from viewers of the real-time stream 402 or live stream 404. Viewers interact with the stream, and their interactions (e.g., social interface control or behavioral monitoring) are collectively evaluated to generate content tags 408.

[0019] Examples of cloud interactions include issuing emojis indicating emotions (e.g., hearts, thumbs-up emojis, smiley faces, thumbs-down emojis, poop emojis), analyzing emotions or keywords in text chat, or direct ratings. In some embodiments, cloud interactions are detected by available sensors or user settings. For example, gaze detection using an available webcam identifies where viewers are focusing their attention. Additionally, stream display is ranked by operating system window priority. If a given window (e.g., a browser open to a streaming platform) is in the top layer of the operating system's display, viewers will view that scene with higher importance than users who minimize the stream or are in a lower layer of the presentation. Similarly, users who mute the stream or device's audio output indicate less interest in the content.

[0020] In some embodiments, content tags 408 are automatically issued by a machine learning model. The machine learning model evaluates the content of the stream and issues content tags. Training of the machine learning model is supervised based on observing tags issued by a human operator or a crowd. For example, the machine learning algorithm is trained in real time by tags issued by a human operator. In some embodiments, training of the machine learning model is based on training scenes associated with a subset of content (e.g., football games, rugby games, video game streamers) that are specifically provided to the model as indicative of important or less important content. High-profile scenes, such as game scores or dramatic moments, are identified through aggregated crowd-sourced sentiment data, moderated user tagging, labeled content, or continuous model training.

[0021] Content tags 408 for significant scenes are used to adapt the delivery method of subsequent versions (e.g., live stream 404 or VoD stream 406 versions) by increasing the allocated buffer size before the tagged segment, pre-buffering selected scenes, caching those scenes for a configurable amount of time, or reducing the encoding quality around the tagged content. In some embodiments, the system supports tagging scenes based on a specific maturity rating system, which is later used to filter age-inappropriate content in a slightly delayed version of the stream, including, for example, language, inappropriate attire, etc.

[0022] The techniques described herein are based on two important observations. First, a given media stream typically exists in multiple versions, including a real-time stream 402, a live stream 404, and an archive of a VoD stream 406. Second, the real-time stream 402 of a stream is sufficiently ahead in time of the other versions 404, 406 of the same stream, allowing information about its content to be gathered and used to improve or enhance the delivery of subsequent streams 404, 406. Similarly, the live stream 404 can also be used to improve or enhance the delivery of a VoD stream 406.

[0023] Figure 5 illustrates the delay 500 between the real-time version 502 of a stream and the live version 504 and archived / video-on-demand version 506 of the stream. The delay that renders the live version 504 obsolete comes from several factors, including client buffering, chunking, and polling. While real-time streaming operates on individual video frames, live protocols like HLS operate on chunks, or collections of video frames. Additionally, live streaming protocols rely on a pull-based model rather than a push-based model in which the user periodically retrieves new chunks from the server. The difference in latency between the live 504 stream and the real-time version 502 stream can vary from tens of seconds to several seconds in optimized versions of the live protocols.

[0024] FIG. 6 is a flowchart illustrating a method for modifying a subsequent multimedia stream. In step 602, a set of multimedia content begins delivery as a real-time broadcast. Most streaming services treat all scenes in a version of the stream as being equally important in terms of quality. For these services, scenes corresponding to the scoring of a goal or a period of passing in the midfield in a soccer game are equally important. From the viewer's perspective, the scoring of a goal is more important than a series of passes in the midfield. Therefore, it is more important to the quality of the content delivery that the more important scenes are most clearly presented to the viewer.

[0025] In step 604, a significant scene (a goal is scored in a soccer game) begins to occur in the real-time stream. In step 606, the streaming platform recognizes the importance of the scene. There are various embodiments for recognizing importance. Example embodiments include recognition by a managed viewer, aggregation of crowd sentiment / ratings, or machine learning models. Once a scene is identified, it is tagged / liked by the platform in step 608.

[0026] Tags are characterized by a timestamp. In some embodiments, a single timestamp defines a tag, and in other embodiments, multiple tags bind segments of the stream. In some embodiments, content tags further include metadata indicating a particular emotion or requesting particular adaptations (e.g., pre-buffering, camera cuts, introduction of multiple camera angles) for the server to implement in the subsequent stream. In step 610, the tagging is forwarded upstream to one or more servers.

[0027] In step 612, the server implements the tag to adapt the delivery of subsequent streams of the same stream (e.g., live or VoD versions) by, for example, increasing the buffer size allocated before the tagged segment, pre-buffering, temporarily lowering the resolution of previous segments to reduce the likelihood of the buffer being empty, removing content, inserting additional camera angles of the portions marked by the multiple content tags, either sequentially or in frame share, or swapping the video providing camera from a first camera of the multiple camera angle input to a second camera of the multiple camera angle input.

[0028] Modifying to insert additional camera angles is the ability to swap camera angles based on the schema implemented for a given content set when multiple cameras are included in the given content set. Inserting additional angles consecutively increases the length of the stream and can increase the delay between the subject and real-time versions. Inserting camera angles in frame share splits the display screen to show multiple camera angles simultaneously. Swapping cameras between multiple available cameras involves switching from one providing camera to another.

[0029] Another potential use of selecting / tagging important scenes in a stream is to identify highlights for replays, which in some embodiments are pre-buffered and cached.

[0030] While Figure 6 is described in terms of giving scenes a thumbs-up, some embodiments of the platform are configured to give scenes a thumbs-down. A thumbs-down scene is de-emphasized later in the stream. De-emphasizing a scene allows for buffer reduction, reduced encoding, content cutting, text overlays, audio description, or split-screen multi-camera views to be performed on the scene.

[0031] While likes and dislikes are described as binary outcomes, tiered tagging is also a valid embodiment. For example, a maturity rating system is modeled into a series of tiers that are used to modify subsequent versions. Maturity rating tags are used to skip material that is not appropriate for the viewer's age.

[0032] In operation, tagging approaches should aim to minimize errors, but when it comes to inaccurately tagging video segments as important, some implementations prioritize overtagging rather than not tagging important segments. Unnecessarily tagging segments as important can lead to unnecessary adaptations, which waste resources or misallocate resources. On the other hand, not tagging content as important can affect the quality of the experience perceived by users.

[0033] When tagging is used to identify the least disruptive moment in the stream to switch to a different view of the event (e.g., from one side of the stadium to the other), the system uses additional metadata in the content tag to indicate platform processing to automatically change the view at the tagged scene or to delay a user-requested view change until the next tag.

[0034] Figure 7 illustrates the path of tagging scenes in a real-time stream and adapting other versions of the stream. An initial server 702 delivers the real-time stream to a real-time client 704. Tagged scenes 706 are sent to a subsequent stream server 708 that implements a live stream and / or VoD. Adaptation 710 is applied to the subsequent stream 712, which is then displayed by a live or VoD client 714.

[0035] Computing Platform 8 is a block diagram illustrating an example computer system 800 according to one or more embodiments. In some embodiments, components of the example computer system 800 are used to implement the software platform described herein. At least some of the operations described herein may be implemented by the computer system 800.

[0036] Computer system 800 includes one or more central processing units ("processors") 802, main memory 806, non-volatile memory 810, network adapter 812 (e.g., network interface), video display 818, input / output devices 820, control devices 822 (e.g., keyboard or pointing device), drive unit 824 including storage medium 826, and signal generating devices 820 communicatively coupled to bus 816. Bus 816 is shown as an abstraction representing one or more physical buses and / or point-to-point connections connected by appropriate bridges, adapters, or controllers. Thus, bus 816 may include a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (I2C) bus, or an IEEE (Institute of Electrical and Electronics Engineers) standard 1394 bus (also known as "Firewire").

[0037] Computer system 800 may share a similar computer processor architecture with a desktop computer, a tablet computer, a personal digital assistant (PDA), a mobile phone, a game console, a music player, a wearable electronic device (e.g., a watch or fitness tracker), a network-connected ("smart") device (e.g., a television or home assistant device), a virtual / augmented reality system (e.g., a head-mounted display), or any other electronic device capable of executing a series of instructions (sequential or other) that specify actions to be performed by computer system 800.

[0038] Although main memory 806, non-volatile memory 810, and storage medium 826 (also referred to as "machine-readable medium") are depicted as a single medium, the terms "machine-readable medium" and "storage medium" should be interpreted to include a single medium or multiple media (e.g., centralized / distributed databases and / or associated caches and servers) that store one or more sets of instructions 828. The terms "machine-readable medium" and "storage medium" should also be interpreted to include any medium capable of storing, encoding, or carrying sets of instructions executed by computer system 800. In some embodiments, non-volatile memory 810 or storage medium 826 is a non-transitory computer-readable storage medium that stores computer instructions that can be executed by one or more central processing units ("processors") 802 to perform the functions of the embodiments disclosed herein.

[0039] Generally, the routines executed to implement embodiments of the present disclosure may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions (collectively referred to as a "computer program"). A computer program typically includes one or more instructions (e.g., instructions 804, 808, 828) that are stored at various times in various memory and storage devices of a computer device. When read and executed by one or more processors 802, the instructions cause the computer system 800 to perform operations that implement elements related to various aspects of the present disclosure.

[0040] Additionally, while embodiments are described in the context of a fully functional computing device, those skilled in the art will appreciate that various embodiments may be distributed as program products in a variety of forms, and the present disclosure applies regardless of the type of machine or computer-readable medium used to actually accomplish the distribution.

[0041] Further examples of machine-readable storage media, machine-readable media, or computer-readable media include volatile and non-volatile memory 810, floppy and other removable disks, hard disk drives, optical disks (e.g., compact disk read-only memory (CD-ROM), digital versatile disk (DVD)), and transmission-type media such as digital and analog communications links.

[0042] Network adapter 812 allows computer system 800 to broker data within network 814 to entities external to computer system 800 via any communication protocol supported by computer system 800 and the external entities. Network adapter 812 can include a network adapter card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multi-layer switch, a protocol converter, a gateway, a bridge, a bridge router, a hub, a digital media receiver, and / or a repeater.

[0043] The network adapter 812 may include a firewall that controls and / or manages access permissions to proxy data within a computer network and tracks various levels of trust between different machines and / or applications. A firewall may be any number of modules with any combination of hardware and / or software components that can enforce a predetermined set of access rights between specific machines and applications, machines and / or applications (e.g., regulate traffic flow and resource sharing between these entities). A firewall may also manage and / or access access control lists that detail the permissions, including the rights of individuals, machines, and / or applications to access and manipulate objects and the circumstances under which the permissions apply.

[0044] The techniques presented herein may be implemented using programmable circuitry (e.g., one or more microprocessors), software and / or firmware, dedicated hardwired (i.e., non-programmable) circuitry, or a combination of these forms. The dedicated circuitry may take the form of one or more application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), etc. Portions of the methods described herein may be performed using an example ML system 900, which is described in more detail with reference to FIG. 9.

[0045] Machine Learning Systems 9 is a block diagram illustrating an example ML system 900 according to one or more embodiments. The ML system 900 is implemented using components of the example computer system 800 described in more detail and with reference to FIG. 9. Similarly, embodiments of the ML system 900 may include or be coupled with different and / or additional components. The ML system 900 may also be referred to as an ML module.

[0046] The ML system 900 includes a feature extraction module 908 implemented using components of the exemplary computer system 800 shown and described in more detail with reference to FIG. 8 . In some embodiments, the feature extraction module 908 extracts a feature vector 912 from the input data 904. For example, the input data 904 can include one or more images, text sets, audio files, or video files. The feature vector 912 includes features 912a, 912b, ... 912n. The feature extraction module 908 reduces redundancy (e.g., repeated data values) in the input data 904 to convert the input data 904 into a reduced set of feature vectors 912 (features 912a, 912b, ... 912n). The feature vectors 912 include relevant information from the input data 904. Using this reduced representation, events of interest or data value thresholds can be identified by the ML model 916. In some embodiments, dimensionality reduction techniques such as principal component analysis (PCA) or autoencoders are used by the feature extraction module 908.

[0047] In an alternative embodiment, the ML model 916 performs deep learning (also known as deep structured learning or hierarchical learning) directly on the input data 904 to learn data representations, rather than using task-specific algorithms. In deep learning, no explicit feature extraction is performed. The feature vector 912 is implicitly extracted by the ML system 900. For example, the ML model 916 can use a cascade of multiple layers of nonlinear processing units for implicit feature extraction and transformation. Each subsequent layer uses the output from the previous layer as input. The ML model 916 can train in supervised (e.g., classification) and / or unsupervised (e.g., pattern analysis) modes. The ML model 916 can learn multiple levels of representations corresponding to different levels of abstraction (the different levels forming a conceptual hierarchy). In this way, the ML model 916 can be configured to distinguish features of interest from background features.

[0048] In an alternative embodiment, the ML model 916 (e.g., in the form of a CNN) generates output 924 directly from the input data 904 without the need for feature extraction. The output 924 is provided to a computing device 928. The computing device 928 may be a server, computer, tablet, smartphone, smart speaker, or the like implemented using components of the exemplary computing system 800 shown and described in more detail with reference to FIG. 8. In some embodiments, the steps performed by the ML system 900 are stored in the memory of the computing device 928 for execution. In other embodiments, the output 924 is displayed on a high-resolution monitor.

[0049] CNNs are a type of feedforward artificial neural network whose inter-neuron connection patterns are inspired by the organization of the visual cortex. Individual cortical neurons respond to stimuli within a limited spatial region known as a receptive field. The receptive fields of different neurons overlap, tiling the visual field. The response of an individual neuron to stimuli within its receptive field can be mathematically approximated by a convolution operation. CNNs are a variation of multilayer perceptrons that are based on biological processes and designed to use minimal preprocessing.

[0050] The ML model 916 can be a CNN that includes both convolutional and max-pooling layers. The architecture of the ML model 916 can be fully convolutional, meaning that it can accept variable-sized sensor data vectors as input. For all convolutional layers, the ML model 916 can specify the kernel size, the convolution stride, and the amount of zero padding applied to the layer's input. For pooling layers, the ML model 916 can specify the kernel size and the pooling stride.

[0051] In some embodiments, the ML system 900 trains an ML model 916 based on training data 920 to correlate feature vectors 912 to expected outputs in the training data 920. As part of training the ML model 916, the ML system 900 forms a training set of features and training labels by identifying a positive training set of features determined to have the desired property of interest and a negative training set of features that do not have the property of interest. The ML system 900 applies ML techniques to train the ML model 916. When applied to the feature vector 912, the ML techniques output an indication of whether the feature vector 912 has the associated desired property.

[0052] The ML system 900 can train the ML model 916 using supervised ML, with features from the training set serving as input. In some embodiments, various ML techniques are used, such as support vector machines (SVMs), regression, naive Bayes, random forests, neural networks, etc. In some embodiments, a validation set 932 is comprised of additional features beyond those in the training data 920 that have already been determined to either have or not have the characteristic of interest. The ML system 900 applies the trained ML model 916 to the features in the validation set 932 to quantify the accuracy of the ML model 916. In some embodiments, the ML system 900 iteratively retrains the ML model 916 until a stopping condition occurs, such as an accuracy measure indicating that the ML model 916 is sufficiently accurate or after performing multiple training rounds.

[0053] The description and drawings herein are illustrative and should not be construed as limiting. Numerous specific details are set forth to provide a thorough understanding of the disclosure. However, in some cases, well-known details are not described so as to avoid obscuring the description. Furthermore, various changes can be made without departing from the scope of the embodiments.

[0054] Accordingly, alternative language and synonyms may be used for one or more of the terms described herein. Furthermore, no special significance should be attached to whether a term is detailed or explained herein. Synonyms for certain terms are provided. The listing of one or more synonyms does not exclude the use of other synonyms. The use of examples, including examples of terms discussed herein, anywhere in this specification is for illustrative purposes only and does not further limit the scope and meaning of the disclosure or the scope and meaning of the exemplified term. Similarly, the present disclosure is not limited to the various embodiments set forth herein.

[0055] It should be understood that the embodiments and variations shown and described herein are merely illustrative of the principles of the invention and that various modifications can be implemented by those skilled in the art.

[0056] It should be noted that any of the above-described embodiments may be combined with each other unless otherwise stated above or unless the functions and / or structures are mutually exclusive.

[0057] While the invention has been described with reference to certain exemplary embodiments, it will be recognized that the invention is not limited to the described embodiments, but can be practiced with modification and alteration within the spirit and scope of the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Claims

1. providing a real-time multimedia stream at multiple encoding rates, including high quality encoding and low quality encoding, each portion of the real-time multimedia stream being equally ranked in the encoded delivery; receiving a plurality of content tags at designated timestamps during broadcast of the real-time multimedia stream, the plurality of content tags indicating a priority for encoding and delivery; providing a delayed multimedia stream containing content corresponding to the real-time multimedia stream, wherein delivering portions of the delayed multimedia stream corresponding to the plurality of content tags comprises: Pre-buffering or Downshifting the encoding rate of surrounding content Steps and A method for providing the above.

2. generating, by a machine learning model, a plurality of content tags at designated timestamps based on supervised training related to the subject matter of the real-time multimedia stream, the designated timestamps correlating with portions of the real-time multimedia stream that are of high interest to a viewer as determined by the supervised training; The method of claim 1 further comprising:

3. generating the plurality of content tags at the specified timestamps based on aggregate usage of social features associated with a user interface of the real-time multimedia stream by viewers of the real-time multimedia stream; The method of claim 1 further comprising:

4. The method of claim 1 , wherein the plurality of content tags is generated by an administrative user.

5. The method of claim 2 , wherein the supervised training is performed by model observation of administrative users generating content tags in past real-time multimedia streams.

6. A viewer of the real-time multimedia stream generating, the plurality of content tags for the specified timestamps: Window priority, Eye tracking, or Operating the volume or mute controls based on aggregate measured behavior of said viewers, including any combination of The method of claim 1 further comprising:

7. receiving a plurality of low-rating tags at a second set of designated timestamps during the broadcast of the real-time multimedia stream, the plurality of low-rating tags indicating a reduced priority for encoding delivery, and delivery of portions of the delayed multimedia stream corresponding to the plurality of low-rating tags: Decreasing the encoding rate for content that is down-rated based on latency, or Integrating text information overlays into said delayed multimedia stream. Steps not highlighted by any of The method of claim 1 further comprising:

8. The method of claim 1 , wherein the plurality of content tags are highly rated tags and have a high encoding priority.

9. the real-time multimedia stream includes a plurality of camera angle inputs that are exchanged with a camera providing video of the real-time multimedia stream during the broadcast; and Inserting additional camera angles, either consecutively or frame-shared, in portions marked by multiple content tags, or the plurality of content tags that switch the video providing camera from a first camera of the plurality of camera angle inputs to a second camera of the plurality of camera angle inputs; modifying the content of the delayed multimedia stream based on the specified timestamps of the plurality of content tags according to any of The method of claim 1 , comprising:

10. the delayed multimedia stream: It is video on demand, or Delayed by a specified number of seconds after the real-time multimedia stream The method of claim 1, wherein the

11. providing a first multimedia stream at multiple encoding rates, including high quality encoding and low quality encoding, the first multimedia stream including multiple camera angle inputs that are swapped into a camera providing video for the first multimedia stream during broadcast, each portion of the first multimedia stream being equally ranked in the encoded delivery; receiving a plurality of content tags at designated timestamps during the broadcast of the first multimedia stream, the plurality of content tags indicating a priority for encoding and delivery; providing a delayed multimedia stream containing content corresponding to the first multimedia stream, wherein delivering portions of the delayed multimedia stream corresponding to the plurality of content tags includes: Inserting additional camera angles, either sequentially or frame-shared, of the parts marked by content tags; switching the video providing camera from a first camera with multiple camera angle inputs to a second camera with multiple camera angle inputs; Pre-buffering, or Reduce the encoding rate of surrounding content Steps are prioritized by either A method for providing

12. generating the plurality of content tags at designated timestamps based on supervised training associated with the subject matter of the first multimedia stream, the designated timestamps correlating with portions of the first multimedia stream of high interest to a viewer as indicated by the supervised training; The method of claim 11 further comprising:

13. generating the plurality of content tags at the specified timestamps based on aggregate usage by viewers of the first multimedia stream of social features associated with a user interface of the first multimedia stream; The method of claim 11 further comprising:

14. The method of claim 11 , wherein the plurality of content tags is generated by an administrative user.

15. The method of claim 12 , wherein the supervised training is performed by model observation of administrative users generating content tags in past multimedia streams.

16. A viewer of the first multimedia stream: Window priority, Eye tracking, or Operating the volume or mute controls generating the plurality of content tags at the specified timestamp based on aggregate measured behavior of the viewer, including any combination of: The method of claim 11 further comprising:

17. 13. The method of claim 12, wherein the plurality of content tags are highly rated tags and have a high encoding priority.

18. providing a real-time multimedia stream at multiple encoding rates, including high quality encoding and low quality encoding, each portion of the real-time multimedia stream being equally ranked in the encoded delivery; receiving a plurality of low-rating tags at designated timestamps during broadcast of the real-time multimedia stream, the plurality of low-rating tags indicating a low priority for encoding and delivery; providing a delayed multimedia stream including content corresponding to the real-time multimedia stream, wherein delivering portions of the delayed multimedia stream corresponding to the plurality of low-rated tags comprises: Decrease the encoding rate for content that is down-rated based on latency, or Integrating text information overlays into delayed multimedia streams Steps not highlighted by any of A method for providing the above.

19. generating the plurality of low-rating tags at the designated timestamps based on supervised training related to the subject matter of a real-time multimedia stream, the designated timestamps correlating with portions of the real-time multimedia stream that are of high interest to viewers as determined by the supervised training; The method of claim 18 further comprising:

20. generating the plurality of dislike tags at the designated timestamps based on aggregate usage by viewers of the real-time multimedia stream of social features associated with a user interface of the real-time multimedia stream. The method of claim 18 further comprising:

21. The method of claim 18 , wherein the plurality of low-rated tags is generated by an administrative user.

22. A viewer of the real-time multimedia stream generates, the plurality of dislike tags at the specified timestamps: Window priority, Eye tracking, or Operating the volume or mute controls based on aggregately measured behavior of the viewer, including any combination of The method of claim 18 further comprising:

23. a broadcast server configured to transmit a real-time multimedia stream at multiple encoding rates, including high-quality encoding and low-quality encoding, wherein each portion of the real-time multimedia stream is equally ranked in the encoding delivery; a feedback platform configured to receive a plurality of content tags at designated timestamps during the broadcast of the real-time multimedia stream, the plurality of content tags indicating a priority for encoding and delivery; a memory for storing a delayed multimedia stream containing content corresponding to the real-time multimedia stream, wherein the delivery of portions of the delayed multimedia stream corresponding to the plurality of content tags comprises: Pre-buffering, or Downshift the encoding rate of the surrounding content Priority is given to memory and A system comprising:

24. a machine learning model stored in the memory configured to generate the plurality of content tags at the specified timestamps based on supervised training related to the subject matter of the real-time multimedia stream, the specified timestamps correlating with portions of the real-time multimedia stream that are of high interest to a viewer as indicated by the supervised training; 24. The system of claim 23, further comprising:

25. a user display interface configured to generate the plurality of content tags at the specified timestamps based on a viewer of the real-time multimedia stream's aggregate usage of social features of the user display interface; 24. The system of claim 23, further comprising:

26. 24. The system of claim 23, wherein the plurality of content tags are generated by an administrative user.

27. 25. The system of claim 24, wherein the supervised training is performed by model observation of administrative users generating content tags in past real-time multimedia streams.

28. a user display interface configured to allow a viewer of the real-time multimedia stream to generate the plurality of content tags, the specified timestamps comprising: window priority, Eye tracking, or Operating the volume or mute controls a display interface based on aggregate measured behavior of said viewer, including any combination of 24. The system of claim 23, further comprising:

29. 24. The system of claim 23, wherein the plurality of content tags are highly rated tags and have a high encoding priority.

Images