Dynamic conditional ad insertion

Abstract

In one example, an apparatus for acquiring media data includes: a memory configured to store media data including advertising content and main media data; and one or more processors implemented in a circuit and configured to: send advertising information to an advertising server device; receive advertising content from an advertising server in response to sending the advertising information; acquire the main media data; and set the advertising content into the main media data. The apparatus may further include: an encoded image buffer (CPB) and a video decoder for decoding encoded video data obtained from the CPB. One or more processors can set the advertising content into the main media data by sending both the advertising content and the main media data to the CPB.

Description

[0001] This application claims the benefits of U.S. Provisional Application No. 62 / 624,603, filed January 31, 2018; U.S. Provisional Application No. 62 / 633,472, filed February 21, 2018; and U.S. Application No. 16 / 262,273, filed January 30, 2019, the entire contents of each of these applications are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the storage and transmission of encoded media data. Background Technology

[0003] Digital video capabilities can be incorporated into a wide variety of devices, including digital televisions, digital live broadcast systems, wireless broadcasting systems, personal digital assistants (PDAs), laptops or desktop computers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radio phones, video conferencing equipment, and more. Digital video devices implement video compression technologies (such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263 or ITU-T H.264 / MPEG-4 (Part 10, Advanced Video Coding (AVC)), ITU-T H.265 (also known as High Efficiency Video Coding (HEVC)), and extensions of such standards) to transmit and receive digital video information more efficiently.

[0004] After video data has been encoded, it can be grouped for transmission or storage. Video data can be assembled into video files that conform to any of various standards, such as the International Organization for Standardization (ISO) base media file formats and their extensions (such as AVC). Summary of the Invention

[0005] In summary, this disclosure describes techniques for a client device to provide an ad server with information representing ad content relevant to the client device. For example, this relevance may correspond to user demographics and / or ad quality data based on the client device's encoding and / or rendering capabilities. For instance, this information may indicate the quality and / or characteristics of the main content being rendered by the client device, into which the ad content will be inserted. In this way, the ad server can send ad content that matches the quality of the main content being rendered and satisfies the client device's capabilities.

[0006] In one example, a method for obtaining media data includes: sending advertising information to an advertising server device by one or more processors implemented in the circuitry of a client device; receiving advertising content from the advertising server by the one or more processors in response to sending the advertising information; obtaining master media data by the one or more processors; and provisioning the advertising content into the master media data by the one or more processors.

[0007] In another example, an apparatus for acquiring media data includes: a memory configured to store media data including advertising content and main media data; and one or more processors implemented in a circuit and configured to: send advertising information to an advertising server device; receive the advertising content from the advertising server in response to sending the advertising information; acquire the main media data; and set the advertising content into the main media data.

[0008] In another example, a computer-readable storage medium having instructions stored thereon, which, when executed, cause a processor of a client device to: send advertising information to an advertising server device; receive advertising content from the advertising server in response to sending the advertising information; obtain master media data; and set the advertising content into the master media data.

[0009] In another example, an apparatus for acquiring media data includes: a unit for sending advertising information to an advertising server device; a unit for receiving advertising content from the advertising server in response to sending the advertising information; a unit for receiving main media data; and a unit for setting the advertising content into the main media data.

[0010] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will be apparent from the specification, the drawings, and the claims. Attached Figure Description

[0011] Figure 1 This is a block diagram illustrating an example system for implementing a technology for streaming media data over a network.

[0012] Figure 2 To show in more detail Figure 1 A block diagram of the example component collection for obtaining the unit.

[0013] Figure 3 This is a conceptual diagram illustrating the elements of example multimedia content.

[0014] Figure 4This is a block diagram showing the elements of an example video file that can correspond to the segments represented.

[0015] Figure 5 This is an example of various timelines in DASH and a conceptual diagram showing the relationships between the timelines.

[0016] Figure 6 This is a conceptual diagram illustrating an example system in which advertisements will be inserted into the main content.

[0017] Figure 7 This is a conceptual diagram illustrating an example system in which advertisements will be inserted into the main content.

[0018] Figure 8 This is a flowchart illustrating an example method through which the techniques disclosed herein can be performed.

[0019] Figure 9 This is a flowchart illustrating an example method for obtaining media data according to the technology described in this disclosure. Detailed Implementation

[0020] Generally, this disclosure describes techniques for client devices to send information to an advertising (ad) server representing various types of advertisements and / or ad content relevant to the client device. For example, this information may indicate that certain advertisements are more relevant to the user of the client device based on one or more demographic characteristics of the user; and the format of the ad content (such as indications of the client device's encoding and / or rendering capabilities).

[0021] The technology disclosed herein can be applied to media (e.g., video) files that conform to video data encapsulated according to any of the following: ISO-based media file format, Scalable Video Coding (SVC) file format, Advanced Video Coding (AVC) file format, 3rd Generation Partnership Project (3GPP) file format, and / or Multi-View Video Coding (MVC) file format, or other similar video file formats.

[0022] In HTTP streaming, frequently used operations include HEAD, GET, and partial GET. The HEAD operation retrieves the header of a file associated with a given Uniform Resource Locator (URL) or Uniform Resource Name (URN), but not the payload associated with the URL or URN. The GET operation retrieves the entire file associated with a given URL or URN. The partial GET operation receives a range of bytes as input and retrieves a consecutive number of bytes from the file, where the number of bytes corresponds to the received range. Therefore, movie fragments can be provided for HTTP streaming because a partial GET operation can retrieve one or more individual movie fragments. Within a movie fragment, there can be several track segments from different tracks. In HTTP streaming, media presentation can be a structured collection of data accessible to the client. The client can request and download media data information to present the streaming service to the user.

[0023] In examples of using HTTP streaming to stream 3GPP data, multiple representations can exist for video and / or audio data of multimedia content. As explained below, different representations can correspond to different encoding characteristics (e.g., different profiles or levels of the video coding standard), different coding standards or extensions of coding standards (e.g., multi-view and / or scalable extensions), or different bitrates. Such a list of representations can be defined in a Media Presentation Description (MPD) data structure. A media presentation can correspond to a structured set of data accessible to the HTTP streaming client device. The HTTP streaming client device can request and download media data information to present the streaming service to the user of the client device. The media presentation can be described in an MPD data structure, which can include updates to the MPD.

[0024] A media presentation may consist of a sequence of one or more time segments. Each time segment may extend until the beginning of the next time segment, or until the end of the media presentation (in the case of the last time segment). Each time segment may contain one or more representations of the same media content. A representation may be one of several alternative encoded versions of audio, video, time-series text, or other such data. Representations may differ in terms of encoding type (e.g., bitrate, resolution, and / or codec for video data, and bitrate, language, and / or codec for audio data). The term "representation" can be used to refer to a portion of encoded audio or video data that corresponds to a specific time segment of multimedia content and is encoded in a specific manner.

[0025] Representations for a specific time period can be assigned to groups indicated by attributes in the MPD that specify the adaptive set to which these representations belong. Representations within the same adaptive set are generally considered interchangeable, as client devices can dynamically and seamlessly switch between these representations, for example, to perform bandwidth adaptation. For instance, each representation of video data for a specific time period can be assigned to the same adaptive set, allowing the selection of any of these representations for decoding to render multimedia content for the corresponding time period, such as video or audio data. In some examples, the media content within a time period can be represented by a single representation from group 0 (if present) or a combination of at most one representation from each non-zero group. The timing data for each representation of a time period can be expressed relative to the start time of that time period.

[0026] A representation may include one or more segments. Each representation may include an initialization segment, or each segment of a representation may be self-initialized. When present, the initialization segment may contain initialization information for accessing the representation. Typically, the initialization segment does not contain media data. Segments may be uniquely referenced by identifiers such as Uniform Resource Locators (URLs), Uniform Resource Names (URNs), or Uniform Resource Identifiers (URIs). The MPD may provide an identifier for each segment. In some examples, the MPD may also provide a byte range in the form of a range attribute, which may correspond to the data used for segments accessible within the file via URL, URN, or URI.

[0027] Different representations can be selected to acquire different types of media data substantially simultaneously. For example, a client device can choose to acquire segmented audio representations, video representations, and time-series text representations from it. In some examples, the client device can select a specific adaptive set to perform bandwidth adaptation. That is, the client device can select an adaptive set that includes video representations, an adaptive set that includes audio representations, and / or an adaptive set that includes time-series text. Alternatively, the client device can select an adaptive set for some types of media (e.g., video) and directly select a representation for other types of media (e.g., audio and / or time-series text).

[0028] Common Media Application Format (CMAF) is typically limited to CMAF presentations that cover consistent content offering. However, in many cases, it is necessary to serve content as a series of CMAF presentations. Examples of such serving include:

[0029] ● Enables content splicing, such as inserting ads.

[0030] ● Enables the splicing of content with different source formats, such as switching from 5.1 to stereo, changing the frame rate, etc.

[0031] ● Enables the splicing of different types of content.

[0032] ● Provide synchronization during segment numbering, for example, to compensate for non-constant segment durations.

[0033] ●Remove or add certain representations / CMAF tracks in the adaptive set / switching set.

[0034] ●Remove or add certain adaptive sets / toggle sets

[0035] ●Removing or adding content providers on certain CDNs

[0036] ● Implement shorter signaling segments (if generated by the encoder).

[0037] ●For the sake of prudence

[0038] In some cases, such interruptions are relatively "difficult," but in others, the content is a continuation of previous content. Continuity can be addressed on two levels:

[0039] - Semantic continuity (i.e., content continuity)

[0040] - To allow for proper playback continuity (i.e., limited changes across such splicing points to achieve good perceptual output).

[0041] Practical examples of this type of problem (from heuristic testing observations) include:

[0042] - Replay highlights from Bundesliga matches. Each match is presented separately, but during automatic program continuation and when replaying via HDMI, the HDMI connection is reset and the screen goes black for a few seconds.

[0043] - A pre-roll ad when joining a live service. After the pre-roll ad completes, there's a black screen for about 1-2 seconds until the show starts—and more examples.

[0044] Figure 1This is a block diagram illustrating an example system 10 for implementing techniques for streaming media data over a network. In this example, system 10 includes a content preparation device 20, a server device 60, and a client device 40. Client device 40 and server device 60 are communicatively coupled via a network 74, which may include the Internet. In some examples, content preparation device 20 and server device 60 may also be coupled via network 74 or another network, or they may be directly communicatively coupled. In some examples, content preparation device 20 and server device 60 may include the same device.

[0045] exist Figure 1 In the example, content preparation device 20 includes an audio source 22 and a video source 24. Audio source 22 may include, for example, a microphone that generates electrical signals representing captured audio data to be encoded by audio encoder 26. Alternatively, audio source 22 may include a storage medium storing previously recorded audio data, an audio data generator (such as a computer synthesizer), or any other audio data source. Video source 24 may include a camera generating video data to be encoded by video encoder 28, a storage medium encoding using previously recorded video data, a video data generation unit (such as a computer graphics source), or any other video data source. In all examples, content preparation device 20 is not necessarily communicatively coupled to server device 60, but rather multimedia content may be stored in a separate medium that is read by server device 60.

[0046] The raw audio and video data may include analog or digital data. Analog data may be digitized before being encoded by audio encoder 26 and / or video encoder 28. Audio source 22 may acquire audio data from the speaker while the speaker is speaking, and video source 24 may acquire video data of the speaker simultaneously. In other examples, audio source 22 may include a computer-readable storage medium containing stored audio data, and video source 24 may include a computer-readable storage medium containing stored video data. In this way, the techniques described in this disclosure can be applied to live, streaming, real-time audio and video data or archived, pre-recorded audio and video data.

[0047] An audio frame corresponding to a video frame is typically an audio frame containing audio data, which is captured (or generated) simultaneously by audio source 22 and video source 24, which is included within the video frame. For example, when a speaker typically generates audio data by speaking, audio source 22 captures the audio data, while video source 24 simultaneously (i.e., when audio source 22 captures the audio data) captures the speaker's video data. Therefore, an audio frame may temporally correspond to one or more specific video frames. Accordingly, an audio frame corresponding to a video frame typically corresponds to the situation where audio and video data are captured simultaneously, and the audio frame and video frame respectively include the simultaneously captured audio and video data.

[0048] In some examples, audio encoder 26 may encode a timestamp indicating the time when audio data for each encoded audio frame was recorded into the encoded audio frame, and similarly, video encoder 28 may encode a timestamp indicating the time when video data for each encoded video frame was recorded into the encoded video frame. In such examples, an audio frame corresponding to a video frame may include both an audio frame including a timestamp and a video frame including the same timestamp. Content preparation device 20 may include an internal clock, which audio encoder 26 and / or video encoder 28 may use to generate timestamps, or audio source 22 and video source 24 may use the internal clock to associate audio data and video data with timestamps, respectively.

[0049] In some examples, audio source 22 may send data to audio encoder 26 corresponding to the time the audio data was recorded, while video source 24 may send data to video encoder 28 corresponding to the time the video data was recorded. In some examples, audio encoder 26 may encode sequence identifiers into the encoded audio data to indicate the relative temporal order of the encoded audio data, but not necessarily the absolute time the audio data was recorded; similarly, video encoder 28 may use sequence identifiers to indicate the relative temporal order of the encoded video data. Similarly, in some examples, sequence identifiers may be mapped or otherwise associated with timestamps.

[0050] Audio encoder 26 typically produces a stream of encoded audio data, while video encoder 28 produces a stream of encoded video data. Each individual data stream (whether audio or video) can be referred to as an elementary stream. An elementary stream is a single digitally encoded (and possibly compressed) component. For example, an encoded video or audio portion can be an elementary stream. Before encapsulating an elementary stream within a video file, it can be converted to a Packed Elementary Stream (PES). Within the same representation, a stream ID can be used to distinguish PES packets belonging to one elementary stream from those belonging to another. The fundamental data unit of an elementary stream is the Packed Elementary Stream (PES) packet. Therefore, encoded video data typically corresponds to an elementary video stream. Similarly, audio data corresponds to one or more corresponding elementary streams.

[0051] Many video coding standards (such as ITU-T H.264 / AVC and the upcoming High Efficiency Video Coding (HEVC) standard) define the syntax, semantics, and decoding processes for error-free bitstreams, each conforming to a profile or level. Video coding standards typically do not specify the encoder, but the encoder's task is to ensure that the generated bitstream conforms to the standard for the decoder. In the context of video coding standards, a "profile" corresponds to a subset of the algorithms, features, or tools and constraints applicable to them. For example, as defined by the H.264 standard, a "profile" is a subset of the entire bitstream syntax specified by the H.264 standard. A "level" corresponds to limitations on decoder resource consumption related to image resolution, bit rate, and block processing rate, such as decoder memory and computation. Profiles can be signaled using the `profile_idc` (profile indicator) value, while levels can be signaled using the `level_idc` (level indicator) value.

[0052] For example, the H.264 standard acknowledges that, within the limits imposed by the syntax of a given profile, large variations in encoder and decoder performance may still be required, depending on the values ​​of the syntax elements in the bitstream, such as the specified size of the decoded images. The H.264 standard further acknowledges that, in many applications, implementing a decoder capable of handling all hypothetical uses of the syntax within a particular profile is impractical and uneconomical. Therefore, the H.264 standard defines a “level” as a specified set of constraints imposed on the values ​​of syntax elements in the bitstream. These constraints can be simple restrictions on the values. Alternatively, these constraints can take the form of constraints on arithmetic combinations of values ​​(e.g., image width multiplied by image height multiplied by the number of images decoded per second). The H.264 standard also specifies that various implementations can support different levels for each supported profile.

[0053] A profile-compliant decoder typically supports all features defined in the profile. For example, B-picture coding, as an encoding feature, is not supported in the baseline H.264 / AVC profile but is supported in other H.264 / AVC profiles. A level-compliant decoder should be able to decode any bitstream that does not require exceeding the limits defined in that level. The definition of the profile and level can contribute to interpretability. For example, during video transmission, a pair of profile and level definitions can be negotiated and agreed upon for the entire transmission session. More specifically, in H.264 / AVC, a level can define limitations on the following: the number of macroblocks to be processed, the size of the decoded picture buffer (DPB), the size of the encoded picture buffer (CPB), the vertical motion vector range, the maximum number of motion vectors per two consecutive MB, and whether a B-block can have sub-macroblock partitions of less than 8x8 pixels. In this way, the decoder can determine whether it can correctly decode the bitstream.

[0054] exist Figure 1 In one example, the encapsulation unit 30 of the content preparation device 20 receives a base stream comprising encoded video data from the video encoder 28 and a base stream comprising encoded audio data from the audio encoder 26. In some examples, the video encoder 28 and the audio encoder 26 may each include a packetizer for forming PES packets from the encoded data. In other examples, the video encoder 28 and the audio encoder 26 may each interface with a corresponding packetizer for forming PES packets from the encoded data. In still other examples, the encapsulation unit 30 may include packetizers for forming PES packets from the encoded audio and video data.

[0055] Video encoder 28 can encode video data of multimedia content in various ways to produce different representations of the multimedia content at various bit rates and with various characteristics (e.g., pixel resolution, frame rate, conformance to various encoding standards, conformance to various profiles and / or profile levels for various encoding standards, representation with one or more views (e.g., for two-dimensional or three-dimensional playback) or other such characteristics). Representations used in this disclosure may include audio data, video data, text data (e.g., for closed captions), or other such data. Representations may include elementary streams, such as audio elementary streams or video elementary streams. Each PES packet may include a stream_id identifying the elementary stream to which the PES packet belongs. Encapsulation unit 30 is responsible for assembling the elementary streams into video files (e.g., segments) of the various representations.

[0056] Encapsulation unit 30 receives PES packets for representing the basic stream from audio encoder 26 and video encoder 28, and forms corresponding Network Abstraction Layer (NAL) units from the PES packets. Encoded video segments can be organized into NAL units, which provide a “network-friendly” video representation, addressing applications such as video telephony, storage, broadcasting, or streaming. NAL units can be categorized as Video Coding Layer (VCL) NAL units and non-VCL NAL units. VCL units may contain the core compression engine and may include block, macroblock, and / or slice-level data. Other NAL units may be non-VCL NAL units. In some examples, the encoded image typically presented as the basic encoded image in a temporal instance may be included in an access unit, which may include one or more NAL units.

[0057] In addition to the above, non-VCL NAL units can also include parameter set NAL units and SEI NAL units. Parameter sets can contain sequence-level header information (in the sequence parameter set (SPS)) and infrequently changing image-level header information (in the image parameter set (PPS)). Using parameter sets (e.g., PPS and SPS), it is not necessary to repeat infrequently changing information for each sequence or image, thus improving coding efficiency. Furthermore, using parameter sets enables out-of-band transmission of important header information, thereby avoiding the need for redundant transmissions for error recovery. In an out-of-band transmission example, parameter set NAL units can be transmitted on a different channel than other NAL units (such as SEI NAL units).

[0058] Supplemental Enhancement Information (SEI) may contain information unnecessary for decoding encoded image samples from VCL NAL units, but may be helpful for processes related to decoding, display, error recovery, and other purposes. SEI messages can be included in non-VCL NAL units. SEI messages are a specification of some standards and are therefore not always mandatory for standards-compliant decoder implementations. SEI messages can be sequence-level or image-level. Some sequence-level information can be included in SEI messages, such as the scalability information SEI message in the SVC example and the view scalability information SEI message in MVC. These example SEI messages can convey information about, for example, the extraction and characteristics of operation points. Additionally, the encapsulation unit 30 can form a manifest file, such as a Media Rendering Descriptor (MPD) describing the characteristics of the representation. The encapsulation unit 30 can format the MPD according to Extensible Markup Language (XML).

[0059] The encapsulation unit 30 can provide data for one or more representations of multimedia content, along with a manifest file (e.g., MPD), to the output interface 32. The output interface 32 may include a network interface or an interface for writing to storage media (e.g., a Universal Serial Bus (USB) interface, a CD or DVD burner, an interface with magnetic or flash storage media, or other interfaces for storing or transmitting media data). The encapsulation unit 30 can provide data for each representation of the multimedia content to the output interface 32, which can then transmit the data to the server device 60 via network transmission or storage media. Figure 1 In the example, server device 60 includes storage medium 62 for storing various multimedia content 64, each multimedia content including a corresponding manifest file 66 and one or more representations 68A-68N (representations 68). In some examples, output interface 32 can also send data directly to network 74.

[0060] In some examples, representation 68 can be divided into adaptive sets. That is, each subset of representation 68 may include a corresponding set of common characteristics, such as codec, profile and level, resolution, number of views, file format used for segmentation, text type information that can identify the language or other characteristics of the text to be displayed along with the representation and / or audio data to be decoded and presented by, for example, a speaker, camera angle information that can describe the camera angle or real-world perspective of the scene for the representation in the adaptive set, rating information describing the suitability of the content for a particular audience, etc.

[0061] Manifest file 66 may include data indicating a subset of representations 68 corresponding to a particular adaptive set, as well as data on common characteristics used for the adaptive set. Manifest file 66 may also include data representing individual characteristics of the individual representations used in the adaptive set, such as bit rate. In this way, the adaptive set can provide simplified network bandwidth adaptation. Sub-elements in the adaptive set elements of manifest file 66 can be used to indicate the representation in the adaptive set.

[0062] Server device 60 includes a request processing unit 70 and a network interface 72. In some examples, server device 60 may include multiple network interfaces. Furthermore, any or all features of server device 60 may be implemented on other devices in the content delivery network, such as routers, bridges, proxy devices, switches, or other devices. In some examples, intermediate devices in the content delivery network may cache data for multimedia content 64 and include components substantially consistent with those of server device 60. Typically, network interface 72 is configured to send and receive data via network 74.

[0063] Request processing unit 70 is configured to receive network requests for data on storage medium 62 from a client device, such as client device 40. For example, request processing unit 70 may implement Hypertext Transfer Protocol (HTTP) version 1.1 as described in RFC 2616 (June 1999, IETF, Networking Working Group, R. Fielding et al., “Hypertext Transfer Protocol – HTTP / 1.1”). That is, request processing unit 70 may be configured to receive HTTP GET or partial GET requests and, in response to said requests, provide data for multimedia content 64. The request may specify a segment of a representation in representation 68 (e.g., a URL using that segment). In some examples, the request may also specify one or more byte ranges of the segment, thereby including partial GET requests. Request processing unit 70 may also be configured to service HTTP HEAD requests to provide header data for a segment of a representation in representation 68. In any case, request processing unit 70 may be configured to process these requests to provide the requested data to the requesting device, such as client device 40.

[0064] Alternatively, request processing unit 70 may be configured to deliver media data via a broadcast or multicast protocol such as eMBMS. Content preparation device 20 may create DASH segments and / or sub-segments in substantially the same manner as described, but server device 60 may use eMBMS or another broadcast or multicast network transport protocol to deliver these segments or sub-segments. For example, request processing unit 70 may be configured to receive multicast group join requests from client device 40. That is, server device 60 may advertise to client devices including client device 40 the Internet Protocol (IP) address associated with a multicast group that is associated with specific media content (e.g., a broadcast of a live event). Client device 40 may then submit a request to join the multicast group. This request may be propagated throughout network 74 (e.g., routers comprising network 74), causing routers to direct traffic to the IP address associated with the multicast group to the customized client device (e.g., client device 40).

[0065] As in Figure 1 As shown in the example, multimedia content 64 includes a manifest file 66, which may correspond to a Media Presentation Description (MPD). The manifest file 66 may contain descriptions of different alternative representations 68 (e.g., video services with different qualities), and this description may include, for example, codec information, profile values, level values, bitrate, and other descriptive characteristics of representation 68. Client device 40 may obtain the MPD of the media presentation to determine how to access segments of representation 68.

[0066] Specifically, the acquisition unit 52 can acquire configuration data (not shown) from the client device 40 to determine the decoding capabilities of the video decoder 48 and the rendering capabilities of the video output 44. The configuration data may also include any or all of the following: language preferences selected by the user of the client device 40, one or more camera angles corresponding to depth preferences set by the user of the client device 40, and / or rating preferences selected by the user of the client device 40. The acquisition unit 52 may include, for example, a web browser or media client configured to submit HTTP GET and partial GET requests. The acquisition unit 52 may correspond to software instructions executed by one or more processors or processing units (not shown) of the client device 40. In some examples, all or part of the functions described with respect to the acquisition unit 52 may be implemented in hardware, or a combination of hardware, software, and / or firmware, wherein the necessary hardware may be provided to execute instructions for the software or firmware.

[0067] The acquisition unit 52 can compare the decoding and rendering capabilities of the client device 40 with the characteristics of the representation 68 indicated by the information in the manifest file 66. The acquisition unit 52 can initially acquire at least a portion of the manifest file 66 to determine the characteristics of the representation 68. For example, the acquisition unit 52 can request a portion of the manifest file 66 describing the characteristics of one or more adaptive sets. The acquisition unit 52 can select a subset (e.g., an adaptive set) of representations 68 having characteristics that can be satisfied by the encoding and rendering capabilities of the client device 40. The acquisition unit 52 can then determine the bit rate for the representation in the adaptive set, determine the amount of currently available network bandwidth, and acquire a segment from one of the representations with a bit rate that the network bandwidth can satisfy.

[0068] Generally, a higher bitrate representation can produce higher quality video playback, while a lower bitrate representation can provide sufficient quality video playback when available network bandwidth is reduced. Accordingly, when available network bandwidth is relatively high, the acquisition unit 52 can acquire data from a relatively high bitrate representation, and when available network bandwidth is low, the acquisition unit 52 can acquire data from a relatively low bitrate representation. In this way, the client device 40 can stream multimedia data on network 74 while adapting to varying network bandwidth availability on network 74.

[0069] Alternatively, the acquisition unit 52 can be configured to receive data according to a broadcast or multicast network protocol such as eMBMS or IP multicast. In such an example, the acquisition unit 52 can submit a request to join a multicast network group associated with specific media content. After joining the multicast group, the acquisition unit 52 can receive data from that multicast group without issuing additional requests to the server device 60 or the content preparation device 20. When the multicast group's data is no longer needed, the acquisition unit 52 can submit a request to leave the multicast group, for example, to stop playback or change the channel to a different multicast group.

[0070] Network interface 54 can receive the selected segmented data and provide it to acquisition unit 52, which in turn can provide these segments to decapsulation unit 50. Decapsulation unit 50 can decapsulate the elements of the video file into a composed PES stream, degroup the PES stream to obtain encoded data, and send the encoded data to audio decoder 46 or video decoder 48, depending on whether the encoded data is part of an audio stream or a video stream (e.g., as indicated by the PES packet header of that stream).

[0071] Despite Figure 1 While not explicitly shown, it should be understood that an encoded image buffer (CPB) (e.g., a memory) may be placed between the decapsulation unit 50 and the video decoder 48 to receive encoded video data (i.e., encoded images) to be decoded by the video decoder 48. In some examples, as explained in more detail below, the decapsulation unit 50 may provide both main media content and advertising content received from different servers to the video decoder 48 via the same CPB. That is, the acquisition unit 52 may set the advertising content into the main media content and enable the video decoder 48 to decode both the main media content and the advertising content. The audio decoder 46 decodes the encoded audio data and sends the decoded audio data to the audio output 42, while the video decoder 48 decodes the encoded video data and sends the decoded video data (which may include multiple views of the stream) to the video output 44.

[0072] The video encoder 28, video decoder 48, audio encoder 26, audio decoder 46, packaging unit 30, acquisition unit 52, and depackaging unit 50 can all be implemented as any of a variety of suitable processing circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic circuits, software, hardware, firmware, or any combination thereof. Each of the video encoder 28 and video decoder 48 can be included in one or more encoders or decoders, wherein either can be integrated as part of a combined video encoder / decoder (CODEC). Similarly, each of the audio encoder 26 and audio decoder 46 can be included in one or more encoders or decoders, wherein either can be integrated as part of a combined CODEC. The apparatus including the video encoder 28, video decoder 48, audio encoder 26, audio decoder 46, packaging unit 30, acquisition unit 52, and / or depackaging unit 50 can include integrated circuits, microprocessors, and / or wireless communication devices (such as cellular phones).

[0073] Client device 40, server device 60, and / or content preparation device 20 may be configured to operate according to the techniques of this disclosure. For illustrative purposes, this disclosure describes these techniques with respect to client device 40 and server device 60. However, it should be understood that content preparation device 20 may be configured to perform these techniques in place of (or in addition to) server device 60.

[0074] Encapsulation unit 30 can form NAL units, which include a header identifying the program to which the NAL unit belongs and a payload (e.g., audio data, video data, or data describing the transmission or program stream corresponding to the NAL unit). For example, in H.264 / AVC, a NAL unit includes a 1-byte header and a payload of variable size. NAL units that include video data in their payload can include video data at various granularities. For example, a NAL unit can include blocks of video data, multiple blocks, slices of video data, or the entire image of video data. Encapsulation unit 30 can receive encoded video data from video encoder 28 in the form of PES packets of the elementary stream. Encapsulation unit 30 can associate each elementary stream with the corresponding program.

[0075] Encapsulation unit 30 can also assemble access units from multiple NAL units. Typically, an access unit may include one or more NAL units representing frames of video data and corresponding audio data (when such audio data is available). Access units typically include all NAL units for an output time instance, such as all audio and video data for a time instance. For example, if each view has a frame rate of 20 frames per second (fps), each time instance may correspond to a time interval of 0.05 seconds. During this time interval, specific frames for all views of the same access unit (same time instance) can be rendered simultaneously. In one example, an access unit may include an encoded image within a time instance, which can be rendered as a substantially encoded image.

[0076] Accordingly, an access unit may include all audio and video frames of a common time instance, for example, all views corresponding to time X. This disclosure also refers to the encoded image of a particular view as a "view component." That is, a view component may include the encoded image (or frame) used for a particular view at a particular time. Accordingly, an access unit may be defined as including all view components of a common time instance. The decoding order of the access units does not necessarily need to be the same as the output or display order.

[0077] Media presentations may include a Media Presentation Description (MPD), which may contain descriptions of different alternative representations (e.g., video services with different qualities), and this description may include, for example, codec information, profile values, and level values. An MPD is an example of a manifest file (e.g., manifest file 66). Client device 40 may obtain the MPD of the media presentation to determine how to access the movie clips of each presentation. Movie clips may reside within a movie clip box (moof box) of the video file.

[0078] The manifest file 66 (which may include, for example, an MPD) can announce the availability of segments representing 68. That is, the MPD may include information indicating the clock time at which a first segment of a representation in 68 becomes available, and information indicating the duration of segments within representation 68. In this way, the acquisition unit 52 of the client device 40 can determine when each segment is available based on the start time and duration of segments preceding a particular segment.

[0079] After the encapsulation unit 30 has assembled the NAL units and / or access units into a video file based on the received data, the encapsulation unit 30 passes the video file to the output interface 32 for output. In some examples, the encapsulation unit 30 may store the video file locally or send the video file to a remote server via the output interface 32, instead of sending the video file directly to the client device 40. The output interface 32 may include, for example, a transmitter, transceiver, a device for writing data to a computer-readable medium (e.g., an optical driver, a magnetic media driver (e.g., a floppy disk drive)), a universal serial bus (USB) port, a network interface, or other output interface. The output interface 32 outputs the video file to a computer-readable medium, such as a transmission signal, magnetic media, optical media, memory, flash memory, or other computer-readable media.

[0080] Network interface 54 can receive NAL units or access units via network 74 and provide NAL units or access units to decapsulation unit 50 via acquisition unit 52. Decapsulation unit 50 can decapsulate the elements of the video file into a PES stream, degroup the PES stream to obtain encoded data, and send the encoded data to audio decoder 46 or video decoder 48 (depending on whether the encoded data is part of an audio stream or a video stream, as indicated by the PES packet header of the PES stream). Audio decoder 46 decodes the encoded audio data and sends the decoded audio data to audio output 42, while video decoder 48 decodes the encoded video data and sends the decoded video data (which may include multiple views of the stream) to video output 44.

[0081] Figure 2 To show in more detail Figure 1 A block diagram of an example set of components for the acquisition unit 52. In this example, the acquisition unit 52 includes an eMBMS middleware unit 100, a DASH client 110, and a media application 112.

[0082] In this example, the eMBMS middleware unit 100 also includes an eMBMS receiving unit 106, a cache 104, and a proxy server unit 102. In this example, the eMBMS receiving unit 106 is configured to receive data via eMBMS (e.g., according to File Delivery over Unidirectional Transport (FLUTE)), which is described in "FLUTE—File Delivery over Unidirectional Transport" by T. Paila et al., November 2012, RFC 6726, Network Working Group, and available at http: / / tools.ietf.org / html / rfc6726. That is, the eMBMS receiving unit 106 can receive files via broadcast from, for example, a server device 60, which can act as a broadcast / multicast service center (BM-SC).

[0083] As the eMBMS middleware unit 100 receives data for a file, it can store the received data in cache 104. Cache 104 may include computer-readable storage media, such as flash memory, hard disk drive, RAM, or any other suitable storage media.

[0084] Proxy server unit 102 can act as a server for DASH client 110. For example, proxy server unit 102 can provide DASH client 110 with an MPD file or other manifest file. Proxy server unit 102 can advertise the availability time of segments and hyperlinks from which segments can be obtained in the MPD file. These hyperlinks may include a local host address prefix corresponding to client device 40 (e.g., 127.0.0.1 for IPv4). In this way, DASH client 110 can request segments from proxy server unit 102 using an HTTP GET or partial GET request. For example, for a segment obtainable from the link http: / / 127.0.0.1 / rep1 / seg3, DASH client 110 can construct an HTTP GET request including a request for http: / / 127.0.0.1 / rep1 / seg3 and submit the request to proxy server unit 102. The proxy server unit 102 can respond to such a request by retrieving the requested data from the cache 104 and providing the data to the DASH client 110.

[0085] Figure 3 This is a conceptual diagram illustrating the elements of example multimedia content 120. Multimedia content 120 may correspond to multimedia content 64 ( Figure 1 (or another multimedia content stored in storage medium 62.) Figure 3 In the example, multimedia content 120 includes a Media Presentation Description (MPD) 122 and multiple representations 124A-124N (representations 124). Representation 124A includes optional header data 126 and segments 128A-128N (segments 128), while representation 124N includes optional header data 130 and segments 132A-132N (segments 132). For convenience, the letter N is used to specify the last movie segment in each representation 124. In some examples, a different number of movie segments may exist between representations 124.

[0086] MPD 122 may include a separate data structure from the representation 124. MPD 122 may correspond to Figure 1 The manifest file is 66. Similarly, it indicates that 124 can correspond to... Figure 1 The representation 68. Typically, MPD 122 may include data that typically describes the characteristics of representation 124, such as encoding and rendering characteristics, adaptive sets, a profile corresponding to MPD 122, text type information, camera angle information, rating information, track mode information (e.g., information indicating a representation including a time subsequence), and / or information for obtaining remote time periods (e.g., information for inserting targeted advertisements into media content during playback).

[0087] Header data 126 (if present) may describe characteristics of segment 128, such as the time position of the Random Access Point (RAP, also known as a Streaming Access Point (SAP)), which segments of 128 include RAPs, the byte offset from RAPs within segment 128, the Uniform Resource Locator (URL) of segment 128, or other aspects of segment 128. Header data 130 (if present) may describe similar characteristics of segment 132. Alternatively, such characteristics may be fully included in MPD 122.

[0088] Segments 128 and 132 include one or more encoded video samples, each of which may comprise a frame or slice of video data. Each of the encoded video samples in segment 128 may have similar characteristics, such as height, width, and bandwidth requirements. Such characteristics can be described by the data in MPD 122, but... Figure 3 Such data is not shown in the example. MPD 122 may include features described by 3GPP specifications, in which any or all of the information transmitted by signaling as described in this disclosure is added.

[0089] Each segment in segments 128 and 132 can be associated with a unique Uniform Resource Locator (URL). Therefore, each segment in segments 128 and 132 can be obtained independently using a streaming network protocol such as DASH. In this way, a destination device, such as client device 40, can use an HTTP GET request to obtain segment 128 or 132. In some examples, client device 40 can use a partial HTTP GET request to obtain a specific byte range of segment 128 or 132.

[0090] Figure 4 This is a block diagram illustrating the elements of an example video file 150, which may correspond to segments such as... Figure 3 One of segments 128 and 132. Each segment in segment 128 and 132 may include segments that substantially conform to... Figure 4 The example illustrates the arrangement of data. Video file 150 can be considered as encapsulated segments. As mentioned above, video files, based on the ISO-based media file format and its extensions, store data in a series of objects called "boxes." Figure 4 In the example, video file 150 includes a file type (FTYP) box 152, a movie (MOOV) box 154, a segment index (sidx) box 162, a movie clip (MOOF) box 164, and a movie clip random access (MFRA) box 166. Although Figure 4 This represents an example of a video file, but it should be understood that other media files may include other types of media data (e.g., audio data, time-series text data, etc.) that are constructed similarly to the data in video file 150 according to the ISO base media file format and its extensions.

[0091] The File Type (FTYP) box 152 typically describes the file type used for the video file 150. The File Type box 152 may include data identifying specifications describing the best use of the video file 150. The File Type box 152 may alternatively be placed before the MOOV box 154, the Movie Clip box 164, and / or the MFRA box 166.

[0092] In some examples, a segment such as video file 150 may include an MPD update box (not shown) before the FTYP box 152. The MPD update box may include information indicating that the MPD corresponding to the representation including video file 150 will be updated, as well as information for updating the MPD. For example, the MPD update box may provide a URI or URL for the resource used to update the MPD. As another example, the MPD update box may include data for updating the MPD. In some examples, the MPD update box may immediately follow the segment type (STYP) box (not shown) of video file 150, where the STYP box may define the segment type for video file 150.

[0093] exist Figure 4 In the example, MOOV box 154 includes a Movie Header (MVHD) box 156, a Track (TRAK) box 158, and one or more Movie Extension (MVEX) boxes 160. Typically, the MVHD box 156 can describe general characteristics of the video file 150. For example, the MVHD box 156 may include data describing when the video file 150 was initially created, when the video file 150 was most recently modified, the time scale used for the video file 150, the duration of playback of the video file 150, or other data that generally describes the video file 150.

[0094] TRAK box 158 may include data for a track in video file 150. TRAK box 158 may include a Track Header (TKHD) box describing the characteristics of the track corresponding to TRAK box 158. In some examples, TRAK box 158 may include an encoded video image, while in other examples, the encoded video image of the track may be included in movie clip 164, which may be referenced by data from TRAK box 158 and / or sidx box 162.

[0095] In some examples, video file 150 may include more than one track. Therefore, MOOV box 154 may include a number of TRAK boxes, the number of which equals the number of tracks in video file 150. TRAK boxes 158 can describe the characteristics of a corresponding track in video file 150. For example, TRAK box 158 can describe the temporal and / or spatial information used for the corresponding track. When encapsulation unit 30 ( Figure 3 When a parameter set track is included in a video file such as video file 150, a TRAK box, similar to MOOV box 154 and TRAK box 158, can describe the characteristics of the parameter set track. Encapsulation unit 30 can use signals to send the presence of sequence-level SEI messages describing the parameter set track within the TRAK box.

[0096] The MVEX box 160 can describe the characteristics of the corresponding movie clip 164, for example, so that the video file 150 also includes the movie clip 164 in addition to the video data (if any) included in the MOOV box 154, which can be transmitted by signal. In the case of streaming video data, the encoded video images can be included in the movie clip 164 instead of in the MOOV box 154. Therefore, all encoded video samples can be included in the movie clip 164 instead of in the MOOV box 154.

[0097] MOOV box 154 may include a number of MVEX boxes 160, the number of MVEX boxes 160 being equal to the number of movie segments 164 in video file 150. Each of the MVEX boxes 160 may describe the characteristics of a corresponding segment within the movie segment 164. For example, each MVEX box may include a Movie Extended Header Box (MEHD) that describes the duration of the corresponding segment within the movie segment 164.

[0098] As described above, encapsulation unit 30 can store the sequence dataset in a video sample that does not include the actual encoded video data. The video sample typically corresponds to an access unit, which is a representation of an encoded image at a specific time instance. In the case of AVC, the encoded image includes one or more VCL NAL units containing information about all pixels to construct the access unit, and other associated non-VCL NAL units (e.g., SEI messages). Accordingly, encapsulation unit 30 can include the sequence dataset in one of the movie clips 164, the sequence dataset including a sequence-level SEI message. Encapsulation unit 30 can also signal the presence of the sequence dataset and / or sequence-level SEI message in the MVEX box corresponding to one of the movie clips 164 within the MVEX box 160, indicating its presence in that movie clip 164.

[0099] SIDX box 162 is an optional element of video file 150. That is, video files conforming to 3GPP file formats or other such file formats do not necessarily include SIDX box 162. According to examples of 3GPP file formats, SIDX boxes can be used to identify sub-segments of a segment (e.g., a segment contained within video file 150). The 3GPP file format defines a sub-segment as "a self-contained set of one or more consecutive movie clip boxes with corresponding media data boxes, and a media data box containing data referenced by a movie clip box must follow that movie clip box and precede the next movie clip box containing information about the same track." The 3GPP file format also instructs that the SIDX box "contains a sequence of references to the sub-segments of the (sub)segment recorded in that box. The referenced sub-segments are sequential in presentation time. Similarly, bytes referenced by a segment index box are always sequential within a segment. The size of the reference gives a count of the number of bytes in the referenced material."

[0100] SIDX box 162 typically provides information representing one or more sub-segments of a segment included in video file 150. Such information may include, for example, the playback times at the start and / or end of the sub-segment, the byte offset for the sub-segment, whether the sub-segment includes a Stream Access Point (SAP) (e.g., from which it begins), the type of SAP used (e.g., whether the SAP is an Instant Decoder Refresh (IDR) image, a Clear Random Access (CRA) image, a Broken Link Access (BLA) image, etc.), the position of the SAP within the sub-segment (based on the playback time and / or byte offset), etc.

[0101] Movie clip 164 may include one or more encoded video images. In some examples, movie clip 164 may include one or more groups of images (GOPs), each group of images may include multiple encoded video images, such as frames or images. Additionally, as mentioned above, in some examples, movie clip 164 may include a sequence dataset. Each movie clip in movie clip 164 may include a Movie Clip Header Box (MFHD, in... Figure 4 (Not shown in the image). The MFHD box can describe the characteristics of the corresponding movie clip, such as the sequence number used for that movie clip. Movie clips 164 can be included in video file 150 in the order of their sequence numbers.

[0102] MFRA box 166 can describe random access points within movie segments 164 of video file 150. This can assist in performing track patterns, such as performing searches for specific time positions (i.e., playback times) within segments encapsulated by video file 150. MFRA box 166 is typically optional and, in some examples, does not need to be included in the video file. Similarly, a client device (e.g., client device 40) does not necessarily need to reference MFRA box 166 to correctly decode and display the video data of video file 150. MFRA box 166 can include a number of Track Segment Random Access (TFRA) boxes (not shown), the number of TFRA boxes being equal to the number of tracks in video file 150, or, in some examples, equal to the number of media tracks (e.g., non-cue tracks) in video file 150.

[0103] In some examples, movie clip 164 may include one or more Stream Access Points (SAPs), such as IDR images. Similarly, MFRA box 166 can provide an indication of the location of the SAP within video file 150. Accordingly, a temporal subsequence of video file 150 can be formed from the SAPs of video file 150. The temporal subsequence may also include other images, such as SAP-dependent P-frames and / or B-frames. Frames and / or slices of the temporal subsequence can be arranged within segments such that frames / slices of other frames / slices dependent on that subsequence can be correctly decoded. For example, in a hierarchical arrangement of data, data used for prediction against other data may also be included in the temporal subsequence.

[0104] Figure 5 This is an example of various timelines in DASH and a conceptual diagram showing the relationships between the timelines. Figure 5 Three exemplary time periods are depicted, each containing multiple representations. For the purpose of discussing the techniques of this disclosure, it is irrelevant whether these representations are included in the same or different adaptive sets.

[0105] The following information can be obtained from timing-related MPDs (or other manifest files):

[0106] ●MPD@availabilityStartTime: The start time is the anchor point for MPD in clock time. This value is in... Figure 5 In this context, it is represented as "AST" (for "Availability Start Time").

[0107] ●Period@start: The start time of the period relative to the start time of MPD availability. This value is in... Figure 5 In this context, it is represented as "PST" (used for "time period start time").

[0108] ●Representation@presentationTimeOffset: Represents the presentation time offset within a time period; that is, it provides the media time at which the representation should be rendered at the beginning of the time period. Typically, this time is the earliest presentation time of the first segment, or a slightly larger value, to ensure synchronization of different media components. If it is larger, the representation is rendered with a short delay relative to the start of the time period. This value... Figure 5 This is represented as "PTO" (used for "Presentation Time Offset").

[0109] Furthermore, the start time of MPD for each segment and the duration of MPD for each segment can be derived using Representation@duration or Representation.SegmentTimeline. For details, refer to ISO / IEC 23009-1 and the discussion below.

[0110] according to Figure 5 AST stands for Clock Time. In this example, AST provides the anchor for all clock time calculations in MPD. The sum of Period@start for the first period and AST provides the period start time PST1 value for the first period in clock time.

[0111] The media timeline origin for tracks in ISO-based media format files is defined as zero. A presentation time offset is assigned to each representation via the value of the `Representation@presentationTimeOffset` attribute, or it is set to 0 by default. The value of this attribute is represented as "PTO". Typically, for a complete file ordered by time slot, this value is 0. For "partial" files or live streams, `@presentationTimeOffset` indicates the media composition / presentation time of a sample synchronized with the start of the time slot. The `@presentationTimeOffset` for live streams will typically be non-zero because the encoder is usually started before presentation availability, so the media timestamp on the first available segment will increment since the encoder started. The encoding timestamp can be set to Coordinated Universal Time (UTC) (like starting the encoder at midnight on January 1, 1970). As long as media is continuously encoded, representations within a time slot of live content will typically have the same `@presentationTimeOffset` because UTC time and media time increment at the same rate and maintain the same offset.

[0112] Referring to the DASH-IF IOP Guidelines and MPEG-DASH Amendment 3 (Amd. 3), some considerations discussed below relate to DASH. In some cases, media presentation is supplied so that the next segment is a continuation of content from the previous segment, either in the immediately following segment or a later segment (e.g., after an advertising segment has been inserted), especially continuing certain media components.

[0113] Content providers (e.g., Figure 1 The content preparation device 20) can express that the media components contained in the two adaptive sets in two different time periods are associated by assigning equal asset identifiers to the two time periods and identifying the two adaptive sets by using the same value for attribute@id. The association expresses the logical continuation of the media components in the next time period and can be used, for example, by the client to continue playing the associated adaptive set in the new time period.

[0114] Additionally, DASH defines two adaptive sets in an MPD as time-continuous if all of the following conditions are met:

[0115] ● These adaptive sets are correlated.

[0116] ● The sum of the @presentationTimeOffset values ​​and presentation durations for all representations in an adaptive set is the same as the @presentationTimeOffset value for the adaptive set associated with the next time period.

[0117] ● If representations in two adaptive sets have the same value for @id, they should have functionally equivalent initialization segments, i.e., initialization segments can be used to continue playing representations. The initialization segment of the first time period (if it exists), concatenated with all consecutive media segments in the representation of the first time period, and subsequently concatenated with all consecutive media segments in the representation of the second time period, should represent a consistent sequence of segments conforming to the media type specified in the @mimeType attribute for the representation of the first time period, as defined in 4.5.4. Furthermore, the @mimeType attribute used for the representation in the next time period should be the same as that for the first time period.

[0118] Media presentation should use an adaptive set with continuous signal transmission periods at the adaptive set level, using a supplementary descriptor (where @schemeIdUri is set to "urn:mpeg:dash:period-continuity:2015"), where:

[0119] ● The @value of the descriptor matches the @id value of the time period included in the MPD.

[0120] ● The value of AdaptationSet@id is the same in both time periods.

[0121] If the MPD contains time periods with the same asset identifier, then the MPD should use an adaptive set of signaling time periods that are consecutive.

[0122] There exists a special case where the media in an adaptive set is a continuation of a previous adaptive set, but the timestamps are not consecutive. Examples include timestamp wraparound, encoder reset, splicing, or other aspects. If all the conditions for time-series continuity above hold, then for each DASH, two adaptive sets in an MPD are time-connected, except that timestamps across time-series boundaries may be discontinuous, but this can be adjusted at the time-series boundaries via the value of @presentationTimeOffset. However, for example, initial segmentation can be equivalent within two adaptive sets. Media presentation should be signaled with time-connected adaptive sets using a supplementary descriptor at the adaptive set level (where @schemeIdUri is set to "urn:mpeg:dash:period-connectivity:2015").

[0123] Continuity of time periods can imply connectivity of time periods.

[0124] If content follows these rules, content authors in DASH are encouraged to provision their content as time-continuous or time-connected signaling. Clients should utilize such signals for a seamless user experience across time-bound boundaries.

[0125] DASH does not address the case of bitstream switching / single initialization segments. However, if initialization segments have the same time-bound boundary, reinitialization is clearly unnecessary. Identical initialization segments can be easily detected by having the same URL.

[0126] DASH also does not provide any cross-time guarantees or promises regarding time-period connectivity and continuity, as it is always assumed that content can be replayed regardless of the characteristics, possibly only with some reduction in quality.

[0127] Due to discussions surrounding ad insertion, time-based continuity and connectivity were initially developed for DVB DASH. Its application has been found in DASH-IF and DVB, and testing is available.

[0128] Single initial segmentation, time-segment continuity, and time-segment connectivity can be used to simplify client playback. In typical DASH mode, at time-segment boundaries, the DASH client will typically select an audio adaptive set and a video adaptive set, and may also select a subtitle track. At time-segment boundaries, a complete reinitialization may occur as follows:

[0129] 1) The DASH client removes the source buffer of the old time period.

[0130] 2) DASH clients use the same MIME type to create new source buffers.

[0131] 3) DASH client attaches initialization segment

[0132] 4) DASH client attaches the first media segment

[0133] Reinitializing the source buffer can cause significant problems. However, if the content is generated with time-series continuity and connectivity, tearing down the source buffer is not necessary. Instead, DASH clients can use the same source buffer as follows:

[0134] 1) If time-duration continuity or connectivity is sent by signaling, the source buffer is maintained.

[0135] 2) If the initial segment span is the same for the two time periods of the segment to be continued, then the initial segment is not added to the source buffer.

[0136] 3) If the time period continuity is sent by signaling, the first media segment of the new time period is appended to the source buffer.

[0137] 4) If time-segment connectivity is sent using a signal, the first media segment of the new time segment is added to the source buffer, and the playback timing of the source buffer is adjusted using a presentation time offset to ensure continuous playback.

[0138] In DASH, adaptive sets are continuous, meaning the problem is handled on a media component basis. This also means that any object that allows seamless switching within an adaptive set across representations is identical at time-segment boundaries. Therefore, constraints across time-segments are directly inherited.

[0139] In summary, following the discussion in CTA WAVE (Consumer Technology Association Web Application Video Ecosystem) as an example, CMAF will benefit from similar concepts, and this will also be beneficial for the mapping of CMAF content to DASH. Two aspects can be addressed:

[0140] 1) Track and switch set continuity / connectivity across CMAF presentation.

[0141] 2) A CMAF presentation sequence profile used to meet certain requirements at the boundary.

[0142] The latter aspect can be postponed to later, or it can be handled by the application of CMAF. The focus of CMAF should be on the first aspect mentioned above.

[0143] Regarding track and switch set continuity / connectivity across CMAF presentation, three example options include:

[0144] 1) Each media profile defines its own continuity / connectivity requirements.

[0145] 2) The concept of continuity / connectivity is directly inherited from the definition through well-defined media profiles and switch set constraints.

[0146] 3) Inheritance is performed by default, but media profiles can further restrict the connectivity of switching sets.

[0147] Using inheritance but allowing media profiles to limit switch set connectivity may be appropriate, as it results in minimal changes to the existing CMAF specification, but leaves room and opportunity for further consideration for each media profile.

[0148] This disclosure recognizes a potential inherent problem in the CMAF specification: all CMAF tracks in a CMAF presentation containing a video switch set should begin aligned with CMAF presentation time zero, which is equal to the presentation start time of the earliest video media sample in the earliest CMAF segment (see 6.6.8 CMAF Presentation and Timing Model). This means that subsequent CMAF presentations cannot have presentation time zero, as this would result in discontinuous presentation timing.

[0149] This is an excessive restriction in CMAF. DASH allows segments to start with a presentation time other than 0 (and can be compensated for using presentationTimeOffset), but CMAF prohibits non-zero start times. Therefore, it is impossible to construct a baseline WAVE program. The techniques disclosed herein can be used to solve this problem. Specifically, the following clause can be added to CMAF as an update:

[0150] 7.3.7 Rendering of switch sets across rendering boundaries

[0151] In many cases, it is necessary to provision content as a CMAF rendering sequence. An example of such provisioning is:

[0152] ● Enables content splicing, such as for inserting advertisements.

[0153] ● Enables the splicing of content with different source formats, such as switching from 5.1 to stereo, changing the frame rate, etc.

[0154] ●Remove or add certain representations / CMAF tracks in the adaptive set / switching set.

[0155] ●Remove or add certain adaptive sets / toggle sets

[0156] ●For the sake of prudence

[0157] While it may be difficult in general to present a particular media type consecutively across CMAF presentation boundaries, with proper content preparation, it is possible to generate playback of media types within a switch set in a CMAF presentation, allowing it to be played back seamlessly across such boundaries as a continuation of a switch set in another / previous CMAF presentation.

[0158] Content providers can express that media components included in a switchset in a CMAF presentation are already connected to a switchset in a previous / another CMAF presentation. CMAF switchset connectivity can be signaled if and only if the switchset in the following CMAF presentation has the following properties:

[0159] ● All CMAF tracks in the switch set of a subsequent CMAF presentation should have a CMAF header. The CMAF header may also exist in the switch set of a previous CMAF presentation.

[0160] ● All encoding parameters of the CMAF tracks in the switch set of the following CMAF presentation should satisfy the switch set constraints of the switch set in the previous CMAF presentation.

[0161] ● If the media profile defines further restrictions, then it satisfies the media profile constraints for the connected switch set.

[0162] For this purpose, no explicit signaling is defined in CMAF; instead, it is left to the manifest to instruct the player to support this approach.

[0163] Content providers can express that the media components contained in a switchset in a previous / another CMAF presentation are continuous with the switchset in the previous CMAF presentation. Switchset continuity can be signaled if the switchset in the following CMAF presentation has the following properties:

[0164] ● Switch sets should be connected.

[0165] ●The first presentation time of a CMAF track is the same as the sum of the CMAF track durations of all tracks in the previous CMAF switch set, and is the same as the first presentation time of the CMAF track in the following CMAF presentation.

[0166] Under certain circumstances, two switch sets may also share a common CMAF header.

[0167] In a hypothetical playback model spanning CMAF rendering boundaries, the player typically selects one audio CMAF switch set and one video CMAF switch set, and may also select one for subtitles. At the rendering boundary, a complete reinitialization will occur as follows:

[0168] 1) The player removes the source buffer for each media type used in the previous CMAF rendering.

[0169] 2) The player creates a new source buffer for each switchset rendered by the new CMAF.

[0170] 3) For each media type,

[0171] a. Add CMAF header to the player

[0172] b. The player attaches the first CMAF segment and may set the timing to the presentation time of the first presentation time sent by the signal.

[0173] Source buffer reinitialization can cause significant problems in continuous playback perception because the media pipeline may be reinitialized.

[0174] Single initialization segmentation, CMAF switchset continuity, and CMAF switchset connectivity can be used to simplify client playback. In a hypothetical playback model across CMAF presentation boundaries, the application client can use signaling as follows:

[0175] 1) If a switch set continuity or connectivity is signaled, the source buffer is maintained.

[0176] 2) If the two CMAF headers spanning the two tracks to be played are identical, then the CMAF headers are not added to the source buffer.

[0177] 3) If switch set continuity is signaled, the first CMAF fragment of the newly presented segment is appended to the source buffer.

[0178] 4) If the switching set connectivity is signaled, the first CMAF fragment of the newly presented set is appended to the source buffer.

[0179] Furthermore, a time offset is used to adjust the playback timing of the source buffer to ensure continuous playback.

[0180] When ads are inserted on client device 40, server device 60, or other such devices, several activities can address the issue of seamless content playback. Examples include:

[0181] ●CTA WAVE resolves program and device constraints related to ad insertion.

[0182] ●MPEG CMAF Core Enabler

[0183] ●DASH-IF Ad Insertion Task Force for DASH-IF IOP

[0184] ●DVB targeted advertising insertion has developed business demand.

[0185] Figure 6 This is a conceptual diagram illustrating an example system 250 in which advertisements will be inserted into the main content. System 250 includes an advertisement (“ad”) server 254, a main content server 252, and an inventory generator 256. In some cases, the functionality assigned to the ad server 254, the main content server 252, and the inventory generator 256 can be combined into a single device. However, in... Figure 6 In the example, a separate server device, as shown in the figure, performs this function. Specifically, ad server 254 provides ad insertion segments 260 to constrained device 258, main content server 252 provides segments of main content 262 to constrained device 258, and manifest generator 256 provides manifest file 264 (e.g., MPD) to constrained device 258.

[0186] In many cases, ad content is delivered on a separate library and independently of the main content. While ads may generally be considered "annoying" to users, there are significant benefits if the quality of the ads matches some or all of the following:

[0187] 1. The quality of an advertisement is matched to the capabilities of the client device (e.g., decoding and / or rendering capabilities).

[0188] 2. The characteristics of the advertisement, such as whether it supports interactivity.

[0189] 3. The quality of the advertisement should match the quality of the currently playing main content. For example, avoid any HDMI resets, source buffer reinitializations, or unnecessary black frames caused by content splicing, mismatched encryption and protected content, different audio codec configurations, etc. This can be further improved by combining the content being played with the device's capabilities; for example, for certain capabilities, advertisement performance can be improved.

[0190] 4. Encoding requirements (timing, splicing) for advertisements at the ad splicing points.

[0191] 5. Personalized advertising for users, such as based on user ID, geolocation, etc.

[0192] 6. Duration of the advertising slot.

[0193] 7. Allow other questions that are targeted at advertising.

[0194] Traditional technologies have not, to this day, placed particular emphasis on the third issue of matching the quality of ads to the quality of the main content currently playing, because many workflows are quite vertical, and content providers also tune their ads. However, with the increasing variety of content formats, device capabilities, and formats, there are benefits to combining the aforementioned content in a consistent manner and providing ad servers with as much information as possible to deliver “good” spliced ​​ads.

[0195] Advertisements may also include data that provides instructions on the conditions for the advertisement in order to optimize the user experience through the use of appropriate playback instructions.

[0196] On the other hand, there is the question of whether the receiver can add two SourceBuffers (media decoding elements) (one for ad playback and the other for the main content).

[0197] Figure 7 This is a conceptual diagram illustrating an example system 280 in which advertisements will be inserted into the main content. System 280 includes an advertisement (“ad”) server 284, a main content server 282, and an inventory generator 286. In some cases, the functionality assigned to the advertisement server 284, the main content server 282, and the inventory generator 286 can be combined into a single device. However, in... Figure 7 In the example, a separate server device, as shown in the figure, performs this function. Specifically, ad server 284 provides ad insertion segments 298 to constrained device 288, main content server 282 provides segments of main content 290 to constrained device 288, and manifest generator 286 provides manifest file 292 (e.g., MPD) to constrained device 288.

[0198] According to the technology of this disclosure, the constrained device 288 (which can represent, for example) Figure 1A client device (such as client device 40) sends a conditional request message 294 to an ad server 284. The ad server 284 uses this message to provide ad content to the constrained device 288. Specifically, the ad server 284 can play the ad content in an appropriate and tailored manner. This includes the constrained device 288 setting the ad content into the main content so that it can be displayed in a source buffer (e.g., an encoded image buffer (CPB) of a video decoder). Specifically, the ad content provided by the ad server 284 includes a target sub-manifest file 296 and an ad insertion segment 298. An important consideration is that the ad server 284 can have sufficient information about the capabilities of the constrained device 288 (e.g., decoding and / or rendering capabilities) and about the content currently being played by the constrained device 288. Based on this information, the ad server 284 can appropriately prepare and serve content so that playback is seamless.

[0199] As an example, the constrained device 288 may provide any of the following information to the advertising server 284, either individually or in any combination:

[0200] 1) The capabilities of the constrained device 288, for example, how to handle ad insertion. Some example options include:

[0201] a. A single source buffer with static initialization.

[0202] b. HDMI properties

[0203] c. Multiple source buffers

[0204] d. Standard Media Source Extension (MSE) Operations

[0205] 2) Ad insertion slots. This allows the ad server 284 to accurately select ads and ad boundaries.

[0206] 3) The nature of the content being played, such as the CMAF header of the content being played for each media component. This allows the ad server 284 to select the appropriate content to match the playback.

[0207] 4) The most recently played sequence of the main content for each media component. This allows the ad server 284 to correctly stitch the content, for example, to avoid overlap or to precisely stitch the ads with the main content.

[0208] According to the technology of this disclosure, the constrained device 288 (e.g., Figure 1 Client device 40) and advertising server 284 (e.g., Figure 1 Server device 60) can exchange any or all of the following information:

[0209] ● The constrained device 288 may signal data representing relevant information, including content nature, playback status, device capabilities, user information, typical other ad insertion information, and / or any one or all of the ad insertion slots. The ad server 284 may use this information to customize ad content for seamless and appropriate playback.

[0210] ● The ad server 284 (or manifest generator 286) can send a portion of the relevant information (as described above) in the manifest file of the main content (e.g., the MPD of DASH).

[0211] For example, the ad server 284 can use signals to send ad insertion slots and static content properties.

[0212] ● The constrained device 288 can send the above relevant information to the advertising server 284 via a signal in the query parameters.

[0213] ● The constrained device 288 can send the above relevant information to the advertising server 284 in the HTTP header using a signal.

[0214] ●The information discussed above may include any one or all of the following:

[0215] ○ Content nature, playback status, device capabilities, user information, other typical ad insertion information, and / or ad insertion time slot.

[0216] ●The nature of the content may include, but is not limited to:

[0217] ○ For each type of video media

[0218] ■Codecs, Profiles, and Levels

[0219] ■ Frame Rate

[0220] ■ Encryption Scheme

[0221] ■Spatial resolution

[0222] ■Transmission characteristics

[0223] ■Color Scheme

[0224] ■ etc.

[0225] ○ For each type of audio media

[0226] ■Codecs, Profiles, and Levels

[0227] ■Sampling frequency

[0228] ■ Output Channel Configuration

[0229] ■ etc.

[0230] ●The nature of the content can be expressed as any one or all of the following:

[0231] ○ Movie header, CMAF header, MIME type and sub-MIME parameters, DASH MPD signaling, etc.

[0232] ●Playback status can be expressed as:

[0233] ○ The most recent time played within the media decoding time and presentation time

[0234] ● Equipment capabilities are expressed as

[0235] ○MSE version

[0236] ○CTA WAVE capabilities

[0237] ○ Number of source buffers

[0238] ○ etc.

[0239] ●The advertising content is expressed as:

[0240] ○Time period in MPD

[0241] ○Author MPD

[0242] ○ High-Level Signaling (HLS) List

[0243] ○ISO BMFF file

[0244] The constrained device 288 can send advertising information to the advertising server 284 via signals in various ways. For example, the constrained device 288 can add a query string to a sublist request that has one or more of the following properties: content nature, device capability, and / or timing. Alternatively, the constrained device 288 can add a special HTTP extension header to the request for the list or sublist. Alternatively, the constrained device 288 can add a URL corresponding to a network location to the request for the list or sublist from which the advertising server 284 can obtain advertising information for the constrained device 288.

[0245] Figure 8 This is a flowchart illustrating an example method through which the techniques of this disclosure can be performed. In this example, Figure 8 The methods include, for example, by Figure 7 The main content server 282, advertising server 284, manifest generator 286, and constrained device 288 perform the actions. It should be understood that in other examples, other devices (e.g., server device 60, content preparation device 20, and client device 40) may also perform the actions. Figure 1 ), and in Figure 1A separate advertising server device (not shown) may be configured to perform this method or similar method according to the technology disclosed herein.

[0246] Initially, the main content server 282 sends manifest content (300) to the manifest generator 286. For example, the main content server 282 may send data representing audio and video datasets (e.g., format, codec, language, bitrate, etc.) used for the main content. The manifest generator 286 may then generate a manifest file (such as a Media Presentation Description (MPD)) and provide the manifest file (302) to the constrained device 288.

[0247] The constrained device 288 can then analyze the manifest file and select appropriate content, and then request the main content segment (304) of the selected content. For example, the constrained device 288 can select content based on device capabilities (e.g., decoding and rendering capabilities), available network bandwidth, and user preferences (e.g., language preferences). Additional parameters may include, for example, the codecs supported by the decoder of the constrained device 288, the spatial and temporal resolution of the display, high dynamic range (HDR) capabilities, audio codecs, and rendering capabilities. To request a segment, the constrained device 288 can submit an HTTP GET or partial GET request to the URL provided in the manifest file for the segment corresponding to the selected content.

[0248] At a certain point in time, the main content server 282 provides the manifest generator 286 with an ad insertion opportunity (306). The ad server 284 may also provide the manifest generator 286 with a link to the ad content (308). The main content server 282 may also add a signal to the content to cause the manifest file previously received by the constrained device 288 to expire (310), prompting the streaming client of the constrained device 288 to request an updated manifest file including the ad opportunity (312). In response to this request, the manifest generator 286 may send the updated manifest for the ad content to the constrained device 288 (314). The updated manifest may include a sub-manifest that directs the media player of the constrained device 288 to the ad server 284. The constrained device 288 may generate a request for the updated manifest file to include additional parameters, so that the ad server 284 is aware of the currently playing content, personalization information, and the capabilities of the media player of the constrained device 288.

[0249] The constrained device 288 can use various techniques to indicate parameters of the currently playing main content to the list generator 286 and / or the advertising server 284. For example, the constrained device 288 can send the current movie header / initial segment / CMAF header for each track currently playing to the list generator 286 and / or the advertising server 284. Such header data typically provides a good overview of the codec, format, etc. In another example, the constrained device 288 can send MIME type data with codec, sub-parameters, etc. In some examples, the constrained device 288 can additionally send the rendering time of the most recently rendered sample to provide continuous rendering.

[0250] Using the updated manifest, the constrained device 288 can request, from the ad server 284, a segment of ad content with parameters of the previously acquired main content (e.g., the same codec, frame rate, spatial resolution, language, etc.) (316). That is, the constrained device 288 can send ad information to the ad server 284. Similarly, such a request for the ad content segment can be an HTTP GET or partial GET request. These requests can specify ad information. In response to these requests, the ad server 284 can provide the target ad content to the constrained device 288 (318). The constrained device 288 can also request another updated manifest file (320) with parameters of the ad content from the manifest generator 286, which can provide the constrained device 288 with an updated manifest file for the main content (322). The constrained device 288 can then resume requesting the main content segment, as discussed above regarding step 304. The constrained device 288 can also play the received media data, i.e., the main media content and the ad content. In this way, the constrained device 288 can set the advertising content to the main media content (e.g., decode the advertising content together with the main media content).

[0251] One commonly observed problem is the issue of content splicing (in which case the two media types (typically audio and video) have different "end times" or different start times before the splice point). In this case, the conversion for each media type can be done carefully and separately. For example, this might involve: one of the media types having gaps, or the content potentially overlapping. Because the splice point is ambiguous, overlapping within a single source buffer can be difficult. Gaps can cause problems because playback is paused or interrupted.

[0252] In this way, Figure 8The method can represent an example of a method for receiving media data, the method comprising: sending advertising information to an advertising server device by one or more processors implemented in the circuitry of a client device; receiving advertising content from the advertising server by one or more processors in response to sending the advertising information; receiving main media data by one or more processors; and setting the advertising content into the main media data by one or more processors.

[0253] Figure 9 This is a flowchart illustrating an example method for obtaining media data according to the technology described in this disclosure. For illustrative purposes, Figure 9 The method is about Figure 1 This is explained using client device 40. However, it should be understood that other devices can be configured to execute this method or similar methods, for example... Figure 7 288. Constrained equipment.

[0254] First, the acquisition unit 52 of the client device 40 acquires the manifest file (350) for the main content. For example, a user may request to watch a specific movie, and the acquisition unit 52 may determine the manifest file associated with the main content corresponding to the movie selected by the user. The acquisition unit 52 may then acquire the manifest file, for example, by issuing an HTTPGet request to the URL associated with the manifest file. As discussed above, the manifest file may be a DASH MPD.

[0255] Then, the acquisition unit 52 can determine the main content to be acquired (352). For example, the acquisition unit 52 can determine the encoding and rendering capabilities of the video decoder 48, audio decoder 46, video output 44, and audio output 42, as well as user preferences (such as language preferences), and then select a suitable adaptive set based on these capabilities and preferences. Then, the acquisition unit 52 can select a representation within the adaptive set based on the available network bandwidth. Then, the acquisition unit 52 can obtain segments of the selected representation, for example, by issuing an HTTP Get or partial Get request, thereby requesting segments of the main content (354).

[0256] At some point, content preparation device 20 or server device 60 can send data indicating that the manifest file has expired. Content preparation device 20 can send this data in response to determining that an advertisement will be inserted into the main content. Similarly, client device 40 can receive data (356) indicating that the manifest file has expired. This data can be included in the band along with the acquired main content or as supplementary information. In response, acquisition unit 52 can acquire an updated manifest file (358), which signals the availability of the advertisement content.

[0257] In response, the acquisition unit 52 can send advertising information (360) to the advertising server device. The advertising information may represent the selected main content (e.g., codec, frame rate, resolution, etc.), information indicating the profile for the user of the client device 40 to ensure the advertising content is relevant and of interest to the user, etc. In response, the acquisition unit 52 can receive advertising content from the advertising server (362). After receiving the advertising content, the acquisition unit 52 can again obtain the updated manifest file for the main content (364), and use the updated manifest file to obtain the main content (366). The client device 60 can then set the advertising content into the main content (368). That is, the acquisition unit 52 can provide the main content and advertising content together for the client device 60 to decode, process, and present.

[0258] In this way, Figure 9 The method can represent an example of a method for receiving media data, the method comprising: sending advertising information to an advertising server device by one or more processors implemented in the circuitry of a client device; receiving advertising content from the advertising server by one or more processors in response to sending the advertising information; receiving main media data by one or more processors; and setting the advertising content into the main media data by one or more processors.

[0259] In one or more examples, the described functionality can be implemented using hardware, software, firmware, or any combination thereof. If implemented in software, the functionality can be stored or transmitted as one or more instructions or code on or through a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium can include a computer-readable storage medium, which corresponds to a tangible medium such as a data storage medium or a communication medium, including, for example, any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium can generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. A data storage medium can be any available medium that can be accessed by one or more computers or one or more processors to obtain instructions, code, and / or data structures for implementing the techniques described in this disclosure. Computer program products can include computer-readable media.

[0260] By way of example, and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, flash memory, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technology (e.g., infrared, radio, and microwave), then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technology (e.g., infrared, radio, and microwave) is included in the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but instead refer to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically magnetically copy data, while optical discs utilize lasers to optically copy data. Combinations of the above items should also be included within the scope of computer-readable media.

[0261] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Furthermore, the techniques can be implemented entirely within one or more circuit or logic elements.

[0262] The technologies disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or a set of ICs (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed technologies, but they do not necessarily need to be implemented through different hardware units. Specifically, as described above, the various units can be combined in a codec hardware unit, or provided by a collection of interoperable hardware units (including one or more processors as described above) combined with appropriate software and / or firmware.

[0263] Various examples have been described. These and other examples are within the scope of the appended claims.

Claims

1. A method for acquiring media data, the method comprising: One or more processors implemented in the circuitry of the client device obtain the main media data from the main media server device according to a manifest file of the main media data, the main media server device being separate from the advertising server device; The one or more processors determine the ad insertion slots of the main media data from the manifest file of the main media data, the ad insertion slots representing defined intervals in the main media data at presentation times; The one or more processors send advertising information to the advertising server device, the advertising information representing the characteristics of advertising content to be received to fill the defined gaps in the main media data, the characteristics including parameters representing the main media data and / or one or more data in the advertising insertion slots; In response to sending the advertising information, the one or more processors receive advertising content conforming to the characteristics from the advertising server device, the advertising content being formatted to fill the defined gap at the presentation time; The one or more processors set the advertising content into the main media data, wherein setting the advertising content into the main media data includes: the one or more processors inserting the advertising content into the main media data at the advertising insertion time slot in the main media data to fill the defined gap at the presentation time; and The one or more processors send the main media data, including the advertising content at the time slot where the advertisement is inserted, to the video decoder of the client device.

2. The method according to claim 1, wherein, The settings include: The advertisement is set into the main media data so that it can be displayed in a source buffer on the client device.

3. The method according to claim 1, wherein, Sending the advertising information includes sending data representing one or more of the following: the quality of media data that can be decoded or rendered by the client device, one or more features of the media data supported by the client device, the matching quality of the main media data, the requirements of the advertising content at the ad splicing point, personalized information for the user of the client device, or the duration of the ad slot.

4. The method according to claim 3, wherein, The matching quality includes one or more of the following: HDMI reset, source buffer reinitialization, avoidance of unwanted black frames, encryption and protected content mismatch information, audio codec configuration, or the capabilities of the client device.

5. The method according to claim 1, wherein, Sending the advertising information includes sending data representing one or more of the following: the capabilities of the client device, the advertising insertion slot, the nature of the main media data, or the most recent playback sequence of the main media data for a media component.

6. The method according to claim 5, wherein, The data representing the one or more capabilities of the client device includes data representing how the client device is capable of handling ad insertion.

7. The method according to claim 5, wherein, The data representing the one or more capabilities of the client device includes data indicating whether the client device supports one or more of the following: a single source buffer with static initialization, HDMI properties, multiple source buffers, or conventional Media Source Extension (MSE) operation.

8. The method according to claim 5, wherein, The data representing the ad insertion slot includes: data representing the appropriate ad and the ad boundary for the ad insertion slot.

9. The method according to claim 5, wherein, The data representing the nature of the main media data includes: a Common Media Application Format (CMAF) header for one or more media components of the main media data.

10. The method according to claim 5, wherein, The data representing the most recent playback sequence includes data indicating how the advertising content can be spliced ​​with the main media data.

11. The method according to claim 1, wherein, Sending the advertising information includes sending one or more of the following: the content nature of the main media data, the playback status of the main media data, the capabilities of the client device, user information of the user of the client device, or advertising insertion time slot information.

12. The method of claim 11, further comprising: Receive the manifest file used for the main media data; as well as The advertising information to be sent to the advertising server device is determined based on the data in the manifest file.

13. The method according to claim 12, wherein, Determining the advertising information based on the data in the manifest file includes determining one or more of the advertising insertion slots or static content properties based on the data in the manifest file.

14. The method according to claim 11, wherein, For the video media of the main media data, the content properties include one or more of the following: video codec, profile indication, level indication, frame rate, encryption scheme, transmission characteristics, or color scheme.

15. The method according to claim 11, wherein, For the audio media of the main media data, the content properties include one or more of the following: audio codec, profile indication, level indication, sampling frequency, or output channel configuration.

16. The method according to claim 11, wherein, Sending the advertising information includes sending the content nature expressed as one or more of the following: movie header, CMAF header, MIME type, sub-MIME type, or DASHMPD signaling.

17. The method according to claim 11, wherein, Sending the advertising information includes sending the capabilities of the client device, which are expressed as one or more of the following: a Media Source Extension (MSE) version, one or more Consumer Technology Association Web Application Video Ecosystem (CTA WAVE) capabilities, or the number of source buffers supported by the client device.

18. The method according to claim 1, wherein, Sending the advertising information includes sending at least a portion of the advertising information in one or more query parameters.

19. The method according to claim 1, wherein, Sending the advertising information includes sending at least a portion of the advertising information in one or more HTTP headers.

20. The method according to claim 1, further comprising: Receive descriptive data for the advertising content, the descriptive data being expressed as one or more of the following: a time period in the manifest file used for the master media data, a manifest file specific to the advertising content and separate from the manifest file used for the master media data, an Advanced Signaling (HLS) manifest, or data in an ISO Base Media File Format (BMFF) file.

21. An apparatus for acquiring media data, the apparatus comprising: A video decoder implemented in a circuit; The memory is configured to store media data, including advertising content and main media data; as well as One or more processors implemented in a circuit and configured to perform the following operations: The main media data is obtained from the main media server device according to the list file of the main media data, and the main media server device is separate from the advertising server device; The ad insertion time slots of the main media data are determined from the manifest file of the main media data, and the ad insertion time slots represent the defined intervals of the main media data at the presentation time. Send advertising information to the advertising server device, the advertising information representing the features of advertising content to be received to fill the defined gaps in the main media data, the features including parameters representing the main media data and / or one or more data in the advertising insertion slots; In response to sending the advertising information, the advertising content conforming to the characteristics is received from the advertising server device, and the advertising content is formatted to fill the defined gap at the presentation time; The advertising content is set into the main media data, wherein, in order to set the advertising content into the main media data, the one or more processors are configured to insert the advertising content into the main media data at the advertising insertion slot of the main media data to fill the defined gap at the presentation time; as well as The main media data, including the advertisement content at the advertisement insertion time slot, is sent to the video decoder.

22. The apparatus of claim 21, further comprising: An encoded image buffer, wherein the main media data and the advertising content include encoded video data, wherein, in order to send the encoded video data to the video decoder, the one or more processors are configured to store the encoded video data in the encoded image buffer, and wherein the video decoder is configured to obtain the encoded video data to be decoded from the encoded image buffer.

23. The device according to claim 21, wherein, The advertising information includes one or more of the following: the quality of media data that can be decoded or rendered by the device, one or more features of the media data supported by the device, the matching quality of the main media data, the requirements of the advertising content at the ad splicing point, the personalization information of the user of the device, or the duration of the ad slot.

24. The device according to claim 21, wherein, The advertising information includes one or more of the following: the capabilities of the client device, the advertising insertion time slot, the nature of the main media data, or the most recent playback sequence of the main media data for a media component.

25. The device according to claim 21, wherein, The advertising information includes one or more of the following: the content nature of the main media data, the playback status of the main media data, the decoding or rendering capabilities of the device, user information of the user on the client device, or advertising insertion time slot information.

26. The device according to claim 21, wherein, The one or more processors are configured to send at least a portion of the advertising information in one or more query parameters.

27. The device according to claim 21, wherein, The one or more processors are configured to send at least a portion of the advertising information in one or more HTTP headers.

28. The device according to claim 21, wherein, The one or more processors are configured to receive descriptive data for the advertising content, the descriptive data being expressed as one or more of the following: a time period in a manifest file for the master media data, a manifest file specific to the advertising content and separate from the manifest file for the master media data, an Advanced Signaling (HLS) manifest, or data in an ISO Base Media File Format (BMFF) file.

29. A computer-readable storage medium having instructions stored thereon, which, when executed, cause a processor of a client device to perform the following operations: The main media data is obtained from the main media server device according to the list file of the main media data, and the main media server device is separate from the advertising server device; The ad insertion time slots of the main media data are determined from the manifest file of the main media data, and the ad insertion time slots represent the defined intervals of the main media data at the presentation time. Send advertising information to the advertising server device, the advertising information representing the features of advertising content to be received to fill the defined gaps in the main media data, the features including parameters representing the main media data and / or one or more data in the advertising insertion slots; In response to sending the advertising information, the advertising content conforming to the characteristics is received from the advertising server device, and the advertising content is formatted to fill the defined gap at the presentation time; Setting the advertising content into the main media data, wherein the instruction for the processor to set the advertising content into the main media data includes an instruction for the processor to perform the following operation: inserting the advertising content into the main media data at the advertising insertion time slot in the main media data to fill the defined gap at the presentation time; as well as The main media data, including the advertising content at the time slot where the advertisement is inserted, is sent to the video decoder of the client device.

30. An apparatus for receiving media data, the apparatus comprising: A unit for receiving the main media data from the main media server device according to the manifest file of the main media data, wherein the main media server device is separate from the advertising server device; A unit for determining the ad insertion slots of the main media data from the manifest file of the main media data, wherein the ad insertion slots represent defined intervals in the main media data at the presentation time; A unit for sending advertising information to the advertising server device, the advertising information representing the characteristics of advertising content to be received to fill the defined gaps in the main media data, the characteristics including parameters representing the main media data and / or one or more data in the advertising insertion time slot; A unit for receiving, in response to sending the advertising information, advertising content conforming to the characteristics from the advertising server device, the advertising content being formatted to fill the defined gap at the presentation time; The unit for setting the advertising content into the main media data includes a unit for performing the following operations: inserting the advertising content into the main media data at the advertising insertion time slot in the main media data to fill the defined gap at the presentation time; as well as A unit for sending the main media data, including the advertising content at the time slot of the advertising insertion, to the video decoder of the client device.

Images