Method performed by wireless transmitting / receiving unit and wireless transmitting / receiving unit

By receiving and using video segmented quality information on the client, dynamically adjusting the bit rate and mass stream, the problems of low bandwidth usage efficiency and poor user experience in the MPEG/3GPP DASH standard are solved, and efficient video streaming is achieved on wireless In the environment.

CN114422833BActive Publication Date: 2024-11-19INTERDIGITAL VC HOLDINGS INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111520871.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2013-06-14
Filing Date
2013-07-10
Publication Date
2024-11-19
Estimated Expiration
2033-07-10

AI Technical Summary

Technical Problem

The MPEG/3GPP DASH standard is inefficient in the use of network bandwidth resources, resulting in poor user experience and lack of adaptation mechanisms for the complexity and quality of video content.

Method used

By receiving and leveraging quality information related to the encoded segmentation on the client, enabling quality-based switching, selecting the appropriate bit rate and mass stream to save bandwidth and maintain video quality.

Benefits of technology

It realizes efficient bandwidth usage of video streaming in wireless environments, improves user experience, and adapts to network conditions and the complexity of video content by dynamically adjusting bit rate and quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114422833B_ABST
    Figure CN114422833B_ABST
Patent Text Reader

Abstract

The present application discloses a method performed by a wireless transmit / receive unit and a wireless transmit / receive unit, wherein the method comprises: receiving a manifest file, the manifest file comprising a description of a plurality of segments of media content, wherein each segment represents a time-based portion of the media content, and wherein the manifest file specifies a bit rate and an absolute quality level for each of the plurality of segments; selecting a segment of the plurality of segments based on the bit rate and the absolute quality level associated with the segment; sending a request for the segment; and receiving the segment.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the Chinese invention patent application with the application date of July 10, 2013, application number 2013800371113, and name “Quality Driven Streaming” (application number 201811202066.3, and name “Method Executed by a Wireless Transmit / Receive Unit and a Wireless Transmit / Receive Unit”).

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Patent Application No. 61 / 669,983, filed on July 10, 2012, and U.S. Provisional Patent Application No. 61 / 835,105, filed on June 14, 2013, the contents of which are incorporated herein by reference in their entirety. Background Art

[0004] The MPEG / 3GPP Dynamic Adaptive HTTP Streaming (DASH) standard can define a framework for the design of bandwidth adaptive delivery of streaming content over wireless and wired networks. However, the MPEG / 3GPP DASH standard does not provide a mechanism for sensing and adapting to the complexity and quality of the encoded video content. This leads to certain inefficiencies in the use of network bandwidth resources and results in a suboptimal user experience. Summary of the invention

[0005] Disclosed are systems, methods, and means for enabling quality-based optimization of the delivery process for streaming content. The optimization may take the form of quality-based switching (e.g., may also be referred to as quality-driven or quality-aware, etc.). Quality-based switching may be enabled in a framework based on adaptive HTTP streaming. If the client has information about the quality of each encoded segment it receives, then the client may enable quality-based switching. There are different ways to communicate information about segment quality to the client. This communication may enable quality-based adaptation in the client.

[0006] In order to enable quality-based decisions in the streaming client, the client can access quality information related to each encoded segment. Quality-related information may include one or more quality metrics related to the encoded segment and / or the encoded sub-segment. Adding quality-related information can be done by including quality-related information in a manifest file (e.g., an mdp file). For example, quality-related information may be included in a segment index stored in a segment index file (e.g., an MP4 or M4S file) and / or an additional file with quality-related segment information is provided such as by providing a link to information from a manifest file. When quality-related information is received, the client can request and / or receive a stream with a lower bit rate, thereby saving bandwidth while maintaining the quality of the streaming content. For example, the client can request and / or receive a stream with a lower bit rate, which has a quality acceptable to the client for the stream.

[0007] A method for content switching in a wireless transmit / receive unit (WTRU), the method comprising receiving quality information associated with a content segment encoded as a plurality of streams. The content segment may form part of a content period. A stream of the content segment may be selected based on respective bit rates and quality information associated with the stream. The WTRU may request and / or receive the selected stream.

[0008] A method for content switching in a wireless transmit / receive unit (WTRU), the method comprising receiving quality information associated with a content segment encoded as a plurality of streams. The content sub-segment may form a portion of a content segment, the content segment forming a portion of a content time segment. A stream of the content segment may be selected based on respective bit rates and quality information associated with the stream. The WTRU may request and / or receive the selected stream.

[0009] A method for quality driven handover in a wireless transmit / receive unit (WTRU), the method comprising receiving a first stream of content at a first bit rate. The first stream of content has at least a quality threshold level. Quality information associated with a segment of a time period of the first stream of content is received. A second stream of content at a second bit rate is determined based on the received quality information. The second bit rate is lower than the first bit rate, and the second stream of content may have at least the quality threshold level. The second stream of content at the second bit rate may be requested and / or received. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1A is a system diagram of an example communication system in which one or more disclosed embodiments may be implemented.

[0011] Figure 1Bis a system diagram of an example wireless transmit / receive unit (WTRU), wherein the WTRU may be configured as follows: Figure 1A Used in the communication system shown.

[0012] Figure 1C is a system diagram of an example radio access network and an example core network, wherein the example core network may be Figure 1A Used in the communication system shown.

[0013] Figure 1D is a system diagram of an example radio access network and another example core network, wherein the example core network may be Figure 1A Used in the communication system shown.

[0014] Figure 1E is a system diagram of an example radio access network and another example core network, wherein the example core network may be Figure 1A Used in the communication system shown.

[0015] Figure 2 is a diagram illustrating an example of MOS fluctuations through bitrate-based encoding and quality-based encoding.

[0016] Figure 3 Graph depicting example distributions of MOS scores for bitrate-based encoding and quality-based encoding.

[0017] Figure 4 A diagram illustrating an example architecture of an adaptive HTTP streaming system.

[0018] Figure 5 A diagram illustrating an example of a possibility for reducing the bit rate by using a quality-based adaptive approach.

[0019] Figure 6 An example diagram depicting the fullness state of the client cache.

[0020] Figure 7 A diagram illustrating a model representation of the operation of a DASH streaming client when quality information is not provided.

[0021] Figure 8 An illustration of a model representation describing the operation of a DASH streaming client using quality information.

[0022] Fig. 9 Example legend describing the quality information track added to a DASH presentation.

[0023] Fig.10An example diagram describing the quality information stored in the mdat box within a DASH segment range.

[0024] Fig.11 An example diagram depicting quality information stored in a free or skip box within a DASH segment.

[0025] Fig.12 An example diagram depicting the high-level architecture of a DASH system.

[0026] Fig.13 An example diagram describing the logical components of the DASH client model.

[0027] Fig.14 An example diagram presenting a high-level data model for describing DSAH media.

[0028] Fig.15 is a diagram illustrating an example of an encoded video stream having three different frame types.

[0029] Fig.16 Figure 2 is an example diagram of six different DASH profiles.

[0030] Fig.17 is a block diagram illustrating an example of a block-based video encoder.

[0031] Fig.18 is a block diagram illustrating an example of a block-based video decoder. DETAILED DESCRIPTION

[0032] The following is a detailed description of exemplary embodiments with reference to various drawings. Although the present invention provides specific possible embodiments, it should be understood that these details are intended to be illustrative and not to limit the scope of the invention.

[0033] Figure 1A 1 is an illustration of an example communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content such as voice, data, video, messaging, broadcast, etc. to multiple wireless users. The communication system 100 may enable multiple wireless users to access the content through sharing of system resources (including wireless bandwidth). For example, the communication system 100 may use one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single carrier FDMA (SC-FDMA), etc.

[0034] like Figure 1AAs shown, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, and / or 102d (generally or collectively referred to as WTRUs 102), radio access networks (RANs) 103 / 104 / 105, core networks 106 / 107 / 109, public switched telephone networks (PSTNs) 108, the Internet 110, and other networks 112, but it will be appreciated that the disclosed embodiments encompass any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in wireless communications. As examples, the WTRUs 102a, 102b, 102c, 102d may be configured to transmit and / or receive wireless signals and may include user equipment (UE), mobile stations, users, subscriber stations, advanced mobile stations (AMS), fixed or mobile subscriber units, pagers, cellular phones, personal digital assistants (PDAs), smart phones, laptops, netbooks, personal computers, wireless sensors, consumer electronics, and the like.

[0035] The communication system 100 may also include a base station 114a and a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks (e.g., the core network 106 / 107 / 109, the Internet 110, and / or the network 112). For example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node B, an eNode B, a Home Node B, a Home eNode B, a site controller, an access point (AP), a wireless router, and the like. Although the base stations 114a, 114b are each described as a single element, it is to be understood that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0036] The base station 114a may be part of the RAN 103 / 104 / 105, which may also include other base stations and / or network elements (not shown) such as a site controller (BSC), a radio network controller (RNC), a relay node, etc. The base station 114a and / or the base station 114b may be configured to transmit and / or receive wireless signals within a specific geographic area, which may be referred to as a cell (not shown). The cell may also be divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Therefore, in one embodiment, the base station 114a may include three transceivers, for example, one transceiver for each sector of the cell. In another embodiment, the base station 114a may use multiple-input multiple-output (MIMO) technology, and thus multiple transceivers for each sector of the cell may be used.

[0037] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 115 / 116 / 117, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 115 / 116 / 117 may be established using any suitable radio access technology (RAT).

[0038] More specifically, as previously described, the communication system 100 may be a multiple access system and may use one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a and the WTRUs 102a, 102b, 102c in the RAN 103 / 104 / 105 may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may use Wideband CDMA (WCDMA) to establish the air interface 115 / 116 / 117. WCDMA may include communication protocols such as High Speed ​​Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High Speed ​​Downlink Packet Access (HSDPA) and / or High Speed ​​Uplink Packet Access (HSUPA).

[0039] In another embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 115 / 116 / 117 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A).

[0040] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as IEEE 802.16 (e.g., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1x, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), etc.

[0041] For example, Figure 1A The base station 114b in the example may be a wireless router, a Home NodeB, a Home eNodeB, or an access point, and may use any suitable RAT to facilitate wireless connectivity in a local area, such as a company, a home, a vehicle, a campus, or the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In another embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may use a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, etc.) to establish a picocell and a femtocell. Figure 1A As shown, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b does not have to access the Internet 110 via the core network 106 / 107 / 109.

[0042] The RAN 103 / 104 / 105 may be in communication with a core network 106 / 107 / 109, which may be any type of network configured to provide voice, data, applications, and / or voice over Internet Protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. For example, the core network 106 / 107 / 109 may provide call control, billing services, mobile location-based services, pre-paid calling, internet connectivity, video distribution, etc., and / or perform high-level security functions, such as user authentication. Although Figure 1AIt is not shown in the figure, and it should be understood that the RAN 103 / 104 / 105 and / or the core network 106 / 107 / 109 may be in direct or indirect communication with other RANs, which may use the same RAT as the RAN 103 / 104 / 105 or a different RAT. For example, in addition to being connected to the RAN 103 / 104 / 105, which may adopt the E-UTRA radio technology, the core network 106 / 107 / 109 may also be in communication with other RANs (not shown) using the GSM radio technology.

[0043] The core network 106 / 107 / 109 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or other networks 112. The PSTN 108 may include a circuit-switched telephone network that provides plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as TCP, User Datagram Protocol (UDP), and IP in the Transmission Control Protocol (TCP) / Internet Protocol (IP) Internet protocol suite. The networks 112 may include wireless or wired communication networks owned and / or operated by other service providers. For example, the networks 112 may include another core network connected to one or more RANs, which may use the same RAT as the RAN 103 / 104 / 105 or a different RAT.

[0044] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communication system 100 may include multi-mode capabilities. For example, the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks via multiple communication links. Figure 1A The WTRU 102c shown in FIG. 1 may be configured to communicate with the base station 114a employing a cellular-based radio technology, and with the base station 114b employing an IEEE 802 radio technology.

[0045] Figure 1B is a system block diagram of an example WTRU 102. Figure 1BAs shown, the WTRU 102 may include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keyboard 126, a display / touchpad 128, a non-removable memory 130, a removable memory 132, a power supply 134, a global positioning system (GPS) chipset 136, and other peripherals 138. It should be understood that the WTRU 102 may include any subset of the above elements while remaining consistent with an embodiment. In addition, embodiments contemplate that the base stations 114a and 114b, and / or nodes represented by the base stations 114a and 114b (such as but not limited to transceiver stations (BTS), node Bs, site controllers, access points (APs), home node Bs, evolved home node Bs (eNodeBs), home evolved node Bs (HeNBs), home evolved node B gateways and proxy nodes, etc.) may include Figure 1B Some or all of the elements described in and described herein.

[0046] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. Although Figure 1B 1. The processor 118 and the transceiver 120 are described as separate components, but it is understood that the processor 118 and the transceiver 120 may be integrated together into an electronic package or chip.

[0047] The transmit / receive element 122 may be configured to send signals to a base station (e.g., base station 114a) or receive signals from a base station (e.g., base station 114a) via an air interface 115 / 116 / 117. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In another embodiment, the transmit / receive element 122 may be a transmitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and receive both RF signals and optical signals. It should be understood that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0048] Furthermore, although the transmit / receive element 122 is Figure 1B Although depicted as a single element in the WTRU 102, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 115 / 116 / 117.

[0049] The transceiver 120 may be configured to modulate signals to be transmitted by the transmit / receive element 122 and to demodulate signals received by the transmit / receive element 122. As described above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as UTRA and IEEE 802.11.

[0050] The processor 118 of the WTRU 102 may be coupled to a speaker / microphone 124, a keyboard 126, and / or a display screen / touch pad 128 (e.g., a liquid crystal display (LCD) unit or an organic light emitting diode (OLED) display unit), and may receive user input data from the above devices. The processor 118 may also output data to the speaker / microphone 124, the keyboard 126, and / or the display screen / touch pad 128. In addition, the processor 118 may access information from and store data in any type of suitable memory, such as a non-removable memory 130 and / or a removable memory 132. The non-removable memory 130 may include a random access memory (RAM), a read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, or the like. In other embodiments, the processor 118 may access data from and store data in a memory that is not physically located on the WTRU 102 but is located on a server or a home computer (not shown).

[0051] The processor 118 may receive power from the power source 134 and may be configured to distribute and / or control power to other components in the WTRU 102. The power source 134 may be any device suitable for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (nickel cadmium (NiCd), nickel zinc (NiZn), nickel metal hydride (NiMH), lithium ion (Li-ion), etc.), solar cells, fuel cells, etc.

[0052] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to or in lieu of the information from the GPS chipset 136, the WTRU may receive location information from a base station (e.g., base stations 114a, 114b) over the air interface 115 / 116 / 117 and / or determine its location based on the timing of signals received from two or more neighboring base stations. It will be appreciated that the WTRU 102 may acquire location information by any suitable location-determination method while remaining consistent with an embodiment.

[0053] The processor 118 may also be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality, and / or wireless or wired connectivity. For example, the peripherals 138 may include an accelerometer, an electronic compass (e-compass), a satellite transceiver, a digital camera (for photos or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, modules, FM radio units, digital music players, media players, video game player modules, Internet browsers, and more.

[0054] Figure 1C 1 is a system block diagram of the RAN 103 and the core network 106 according to one embodiment. As described above, the RAN 103 may communicate with the WTRUs 102a, 102b, 102c over the air interface 115 using UTRA radio technology. The RAN 103 may also communicate with the core network 106. Figure 1C As shown, the RAN 103 may include Node-Bs 140a, 140b, 140c, which may each include one or more transceivers that communicate with the WTRUs 102a, 102b, 102c over the air interface 115. Each of the Node-Bs 140a, 140b, 140c may be associated with a specific unit (not shown) within the RAN 103. The RAN 103 may also include RNCs 142a, 142b. It will be appreciated that the RAN 103 may include any number of Node-Bs and RNCs while still being consistent with an embodiment.

[0055] like Figure 1CAs shown, Node B 140a, 140b can communicate with RNC 142a. In addition, Node B 140c can communicate with RNC 142b. Node B 140a, 140b, 140c can communicate with corresponding RNC 142a, 142b via Iub interface. RNC 142a, 142b can communicate with each other via Iur interface. RNC 142a, 142b can be respectively configured to control the corresponding Node B 140a, 140b, 140c connected thereto. In addition, RNC 142a, 142b can be respectively configured to implement or support other functions, such as outer loop power control, load control, admission control, packet scheduling, handover control, macro diversity, security functions, data encryption, etc.

[0056] Figure 1C The core network 106 shown in FIG. 1 may include a media gateway (MGW) 144, a mobile switching center (MSC) 146, a serving GPRS support node (SGSN) 148, and / or a gateway GPRS support node (GGSN) 150. Although each of the above elements is described as part of the core network 106, it should be understood that any of these elements may be owned and / or operated by an entity other than the core network operator.

[0057] The RNC 142a in the RAN 103 may be connected to the MSC 146 in the core network 106 via an IuCS interface. The MSC 146 may be connected to the MGW 144. The MSC 146 and the MGW 144 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices.

[0058] The RNC 142a in the RAN 103 may also be connected to the SGSN 148 in the core network 106 via an IuPS interface. The SGSN 148 may be connected to the GGSN 150. The SGSN 148 and the GGSN 150 may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, thereby facilitating communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0059] As described above, the core network 106 may also be connected to other networks 112, which may include other wired or wireless networks owned and / or operated by other service providers.

[0060] Figure 1D1 is a system diagram of the RAN 104 and the core network 107 according to one embodiment. As described above, the RAN 104 may employ an E-UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also be in communication with the core network 107.

[0061] The RAN 104 may include eNode-Bs 160a, 160b, 160c, it being appreciated that the RAN 104 may include any number of eNode-Bs while remaining consistent with an embodiment. The eNode-Bs 160a, 160b, 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the eNode-Bs 160a, 160b, 160c may implement MIMO technology. Thus, for example, the eNode-B 160a may use multiple antennas to transmit wireless signals to, and receive wireless signals from, the WTRU 102a.

[0062] Each of the eNodeBs 160a, 160b, 160c may be associated with a specific unit (not shown) and may be configured to handle radio resource management decisions, handover decisions, user scheduling, etc. in the uplink and / or downlink. Figure 1D As shown in FIG. , the eNode-Bs 160a, 160b, 160c may communicate with one another via an X2 interface.

[0063] Figure 1D The core network 107 shown in FIG. 1 may include a mobility management gateway (MME) 162, a serving gateway 164, and a packet data network (PDN) gateway 166. Although each of the above elements is described as part of the core network 107, it should be understood that any of these elements may be owned and / or operated by an entity other than the core network operator.

[0064] The MME 162 may be connected to each of the eNode-Bs 160a, 160b, 160c in the RAN 104 via an S1 interface and may serve as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, bearer activation / deactivation, selecting a particular serving gateway during an initial attach of the WTRUs 102a, 102b, 102c, and the like. The MME 162 may also provide control plane functions for switching between the RAN 104 and RANs (not shown) that employ other radio technologies, such as GSM or WCDMA.

[0065] The serving gateway 164 may be connected to each of the eNode-Bs 160a, 160b, 160c in the RAN 104 via an S1 interface. The serving gateway 164 may generally route and forward user data packets to / from the WTRUs 102a, 102b, 102c. The serving gateway 164 may also perform other functions, such as anchoring a user plane during inter-eNode-B handovers, triggering paging when downlink data is available for the WTRUs 102a, 102b, 102c, managing and storing contexts for the WTRUs 102a, 102b, 102c, and the like.

[0066] The serving gateway 164 may also be connected to the PDN gateway 166, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, thereby facilitating communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0067] The core network 107 may facilitate communications with other networks. For example, the core network 107 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. For example, the core network 107 may include, or may be in communication with, an IP gateway (e.g., an IP Multimedia Subsystem (IMS) service) that serves as an interface between the core network 107 and the PSTN 108. In addition, the core network 107 may provide the WTRUs 102a, 102b, 102c with access to the networks 112, which may include other wired or wireless networks that are owned and / or operated by other service providers.

[0068] Figure 1E 1 is a system diagram of the RAN 105 and the core network 109 according to one embodiment. The RAN 105 may employ IEEE 802.16 radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 117. As will be discussed below, the communication links between the different functional entities of the WTRUs 102a, 102b, 102c, the RAN 105, and the core network 109 may be defined as reference points.

[0069] like Figure 1EAs shown, the RAN 105 may include base stations 180a, 180b, 180c and an ASN gateway 182, although it should be understood that the RAN 105 may include any number of base stations and ASN gateways while remaining consistent with an embodiment. The base stations 180a, 180b, 180c are each associated with a particular cell (not shown) in the RAN 105 and may each include one or more transceivers that communicate with the WTRUs 102a, 102b, 102c over the air interface 117. In one embodiment, the base stations 180a, 180b, 180c may implement MIMO technology. Thus, for example, the base station 180a may use multiple antennas to transmit wireless signals to the WTRU 102a and receive wireless signals from the WTRU 102a. The base stations 180a, 180b, 180c may also provide mobility management functions, such as handover triggering, tunnel establishment, radio resource management, service classification, quality of service (QoS) policy enforcement, and the like. The ASN gateway 182 may serve as a service aggregation point and may be responsible for paging, caching, routing of user profiles to the core network 109, and the like.

[0070] The air interface 117 between the WTRUs 102a, 102b, 102c and the RAN 105 may be defined as an R1 reference point that implements the IEEE 802.16 specification. In addition, each of the WTRUs 102a, 102b, 102c may establish a logical interface (not shown) with the core network 109. The logical interface between the WTRUs 102a, 102b, 102c and the core network 109 may be defined as an R2 reference point, which may be used for authentication, authorization, IP host configuration management, and / or mobility management.

[0071] The communication link between each of the base stations 180a, 180b, 180c may be defined as an R8 reference point that includes protocols for facilitating WTRU handovers and data transfer between base stations. The communication link between the base stations 180a, 180b, 180c and the ASN gateway 182 may be defined as an R6 reference point. The R6 reference point may include protocols for facilitating mobility management based on mobility events associated with each WTRU 102a, 102b, 102c.

[0072] like Figure 1EAs shown, the RAN 105 may be connected to a core network 109. The communication link between the RAN 105 and the core network 109 may be defined, for example, as an R3 reference point including protocols for facilitating data transfer and mobility management capabilities. The core network 109 may include a mobile IP home agent (MIP-HA) 184, an authentication, authorization, accounting (AAA) service 186, and a gateway 188. Although each of the above elements is described as part of the core network 109, it should be understood that any of these elements may be owned and / or operated by an entity other than the core network operator.

[0073] The MIP-HA may be responsible for IP address management and may enable the WTRUs 102a, 102b, 102c to roam between different ASNs and / or different core networks. The MIP-HA 184 may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices. The AAA server 186 may be responsible for user authentication and supporting user services. The gateway 188 may facilitate interworking with other networks. For example, the gateway 188 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. In addition, the gateway 188 may provide the WTRUs 102a, 102b, 102c with access to the networks 112, which may include other wired or wireless networks that are owned and / or operated by other service providers.

[0074] Although in Figure 1E 105 may be connected to other ASNs and the core network 109 may be connected to other core networks. The communication link between the RAN 105 and the other ASNs may be defined as an R4 reference point, which may include protocols for coordinating the mobility of the WTRUs 102a, 102b, 102c between the RAN 105 and the other ASNs. The communication link between the core network 109 and the other core networks may be defined as an R5 reference point, which may include protocols for facilitating interworking between a local core network and a visited core network.

[0075] The techniques discussed herein may be performed partially or in full by the WTRUs 102a, 102b, 102c, 102d, the RAN 104, the core network 106, the Internet 110, and / or other networks 112. For example, video streaming performed by the WTRUs 102a, 102b, 102c, 102d may engage in various processes as discussed herein.

[0076] The techniques discussed herein may be performed partially or in full by the WTRUs 102a, 102b, 102c, 102d, the RAN 104, the core network 106, the Internet 110, and / or other networks 112. For example, video streaming performed by the WTRUs 102a, 102b, 102c, 102d may engage in various processes as discussed herein.

[0077] Systems, methods, and means for enabling quality-based optimization for video delivery are disclosed. The disclosed techniques are described with reference to the MPEG / 3GPP DASH standard, but are not limited thereto. For example, the manner in which information about segment quality is communicated to a DASH client is described herein. Such communication can enable quality-based adaptation in the client. The techniques described herein can be implemented as extensions to the MPEG-DASH standard.

[0078] The effectiveness of image compression can be evaluated using, for example, the bit rate and / or bandwidth of the transmitted signal, and / or the quality of the reconstructed image. If the time and / or sequence of images / frames or videos is considered, there are multiple bit rate and / or quality characteristics that can be achieved by the video codec for each frame.

[0079] To report the rate and / or quality parameters (e.g., PSNR, SSIM, and / or MOS) for a sequence, an average of the bitrate and / or quality parameters across one or more frames may be used. The average is not reliable, such as because there are different ways to achieve the same average score, which, for example, have different effects on the overall quality of experience. For example, an encoder may use different strategies to balance the momentary tradeoff between quality and rate for a single frame in a video sequence. The bitrate may be maintained as close to a given target as possible while achieving the best possible image / frame quality given the rate. This strategy may be referred to as constant bitrate (CBR) encoding. The quality may be maintained close to a given target while using the minimum number of bits required to achieve the quality for each frame. This strategy may be referred to as constant quality encoding.

[0080] In order to adapt to the complexity of changing video content, the encoder can implement a version of constant rate and / or constant quality encoding, where the encoder allows the rate and / or quality to fluctuate from frame to frame. For example, the encoder can attempt to reduce or minimize the fluctuation. The encoding strategies can be referred to as bit rate-based and quality-based encoding, respectively.

[0081] Figure 2 An example graph illustrating MOS fluctuations over bitrate-based encoding and quality-based encoding of the same sequence. Figure 2 An example of MOS scores resulting from bitrate-based encoding and quality-based encoding of a video sequence is described. In this example, quality-based encoding can utilize a higher peak bitrate (2.7 Mbps), but can achieve approximately the same average bitrate (1.8 Mbps) as bitrate-based encoding. Quality-based encoding is characterized by more consistent quality. When quality drops below a certain acceptable level (e.g., fair), quality-based encoding has fewer frame / segment instances.

[0082] Figure 3 To illustrate examples of MOS score distributions for bitrate-based encoding and quality-based encoding. Quality loss falling below an acceptable level results in a better overall quality of experience. This can be used for a variety of different types of visual content and reproduction settings.

[0083] Figure 4 An example diagram is provided to describe the operation of an adaptive HTTP streaming system 400. The streaming system may use rate-based encoding of video, such as by preparing multiple streams encoded at different target rates. A client application may be used to dynamically select between one or more encoded streams. The stream switching implemented by the client may have a specific granularity, such as about 2-10 seconds in practice. The point at which the client switches between encoded streams may be referred to as a switching point. The portion of the encoded content between the encoded streams may be referred to as a segment.

[0084] During a streaming session, a streaming client can calculate the delivery rate of one or more segments. The delivery rate of a segment can be used to estimate the available network bandwidth for the client to receive the next segment. Based on this evaluation, the client can decide which next encoding / rate to use for the next segment. This allows the client to adapt to changing network conditions. For example, high-level information about each encoded stream, including but not limited to its rate, can be stored in a manifest file or a multimedia presentation description (MPD) file. For example, offset and timing information for the encoded segments within a stream range can be stored in one or more segment index files.

[0085] The manifest file and / or index file may not include quality information about each encoded segment. The streaming client may not have knowledge about the quality of each segment. Without this knowledge, the streaming client cannot implement quality-driven adaptation. This will produce certain inefficiencies in the streaming delivery system. For example, there are cases when the content is difficult to encode, which will produce average quality at the current rate. For example, there are cases when the content is easy to encode and it makes sense to reduce the rate without affecting the quality. Figure 5 An example of this is described in .

[0086] Figure 5 FIG. 1 is a diagram illustrating an example of a possibility for reducing the bit rate by using a quality-based adaptive approach. Figure 5 As shown in , a video may include one or more segments that are easy to encode. In this case, switching to a lower rate can save bandwidth while maintaining quality at a specific cap level. Here, how to enable quality-aware (e.g., quality-driven) switching in a framework based on adaptive HTTP streaming, and in particular the MPEG / 3GPP DASH standard, is described.

[0087] Such as Figure 5 As shown in , the streaming client can utilize some quality caps. The client can know one or more quality values ​​(e.g., PSNR) achieved for the segment. For example, if the client determines that the next segment achieves one or more quality values ​​equal to or greater than a given quality target (e.g., 48dB instead of the target 38dB), the client can choose to encode at a lower rate sufficient to achieve the given quality target. The client can save bandwidth and / or improve the consistency of the quality of the visual content being delivered.

[0088] The quality information can be used by DASH streaming clients to improve video delivery. A client caching model can be described. Figure 6 An example legend for describing the number of client cache full states. The client can operate on a pre-roll buffer of a specific length (e.g., 10-30 seconds). The client can use one or more cache full thresholds, such as low_buf 602 and high_buf 604. Low_buf 602 can be referred to as a threshold when the client is at risk of a re-caching situation. When at low_buf threshold 602, the client can perform a measurement to replenish the buffer. High_buf 604 can be referred to as a threshold when the client has accumulated a large amount of information to consider switching up and / or throttling (e.g., if the maximum rate has been reached).

[0089] Figure 7FIG. 7 is a diagram of a model representation 700 for describing the operation of a DASH streaming client when quality information is not provided. A DASH client operation model is described. Figure 7 As shown at 702, when the cache level Buf is below the low_cache threshold, the client can select the lowest rate representation to avoid re-caching. As shown at 704, when the cache level Buf is between the low_cache and high_cache thresholds, the client can, for example, select the highest rate representation below the available bandwidth. As shown at 706, when the cache level Buf is above the buffer_high (buf_high) threshold, the client can check whether the highest rate representation has been reached, and if so, the client can introduce one or more pauses between segment loading (e.g., throttling) because it already has enough data for real-time playback.

[0090] A client adaptation model using per-segment quality and rate information is provided. Figure 8 800 is a diagram of a model representation of the operation of a DASH streaming client using quality information, wherein the model representation 800 may prevent selection of streams / segments whose quality is above a threshold value, such as quality_cap, regardless of bitrate. This selection may result in more conservative bandwidth usage and / or a more consistent quality of experience for the end user. This quality information may be used, such as in addition to buffer fullness information. As in Figure 8 As shown at 802 in, when the cache level Buf is below the low_cache threshold, the client can select the lowest rate representation to avoid re-caching. As shown at 804, when the cache level Buf is between the low_cache and high_cache thresholds, the client can, for example, select the highest rate representation that is lower than the available bandwidth and has a quality that does not exceed the quality threshold quality_cap. As shown at 806, when the cache level Buf is above the cache_high threshold, the client can check whether the highest rate representation is reached. If so, the client can introduce one or more pauses between segment loads (e.g., adjustments) because it may already have enough data for real-time playback.

[0091] To enable quality-based decisions in a streaming client, the client may have access to quality information about one or more encoded segments. There are multiple places and multiple ways on how this information is introduced in the MPEG-DASH standard. For example, one or more of PSNR, SSIM, VQM, VIF, J.341, MOS (e.g., in any allowed combination) and / or another objective and / or subjective measurement may be used as an added quality metric.

[0092] The quality metric may utilize a dB scale such as PSNR. The quality metric may be mapped to an interval, for example, an interval [1 ... 5] associated with a 5-level MOS ratio. The signaling for the quality value may be flexible, such as by allowing additions and extensions. The signaling for the quality value may allow communication of a metric scaled to a range of values. For example, a metric may be scaled to a range of values ​​for a MOS score, such as 1 (lowest) to 5 (highest). The signaling for the quality value may allow communication of the PSNR.

[0093] The quality information may be granular. For example, the quality information may be defined at a segment and / or sub-segment level, which allows the DASH client to make decisions. The quality information may be accessible. For example, the quality information may be accessible for segments and / or representations such as in an adaptation set, whereby the DASH client may extract (e.g., separately) the quality information in advance and before loading the actual data segments. The quality information may be concise. For example, the quality information may be concise whereby loading the quality information may not incur a major overhead in terms of data loaded by the streaming client.

[0094] One way to add information about the quality of the encoded segments can be by using a tag (e.g., an added tag) in the segment list portion of the MPD file. An adaptation set may include a tag indicating the presence of a quality value in the segment list. An example of this declaration is listed below:

[0095] <AdaptationSet segmentAlignment="true"bitstreamSwitching="true"subsegmentAlignment="true"qualityInfo=”true”qualityMetric=”PSNR”>

[0096] <ContentComponent id="101"contentType="video" / >

[0097] <ContentComponent id="102"contentType="audio" / >

[0098] <Representation id="1"mimeType="video / mp4"codecs="avc1.42c00d"width="320"height="180"sampleRate="44100"numChannels="1"startWithRAP="1"bandwidth="134829">

[0099] <SegmentList timescale="1000"duration="10000">

[0100] <representationindex> example_320x180_48kbps_av_index.didx< / representationindex>

[0101] <SegmentURL media="example_320x180_48kbps_av0.m4s"quality=”33.1” / >

[0102] <SegmentURL media="example_320x180_48kbps_av1.m4s"quality=”33.3” / >

[0103] <SegmentURL media="example_320x180_48kbps_av2.m4s"quality=”33.8” / >

[0104] <SegmentURL media="example_320x180_48kbps_av3.m4s"quality=”32.6” / >

[0105] A separate quality sequence descriptor is described herein. Quality information may be included as a separate descriptor, such as at a representation layer. The descriptor may define a quality metric, may define the precision of the quality representation (e.g., a coarse representation may be used to make it more concise), and / or define a sequence of quality values ​​associated with segments in a representation.

[0106] The quality information descriptor at the presentation layer may include information about the actual compressed length of each segment, such as when the MPD may use segment templates and / or URLs to individual segment files. The quality information at the presentation layer may be defined as a supplementary attribute descriptor, and may include a scheme uniform resource name (URN), which may provide a unique identifier associated with the quality signaling information.

[0107] An example implementation of a quality sequence descriptor is provided in Table 1.

[0108] Table 1

[0109]

[0110] One way to add information about the quality of the encoded segments is to use a segment index (or index segment) file. The segment index file may include a file (e.g., a specific file), may include segment index information, and / or may be stored on an HTTP server together with the .mpd and segment files. The segment index file may include a version of the .mp4 file (e.g., a specific version). The segment index file may include a STYP, SIDX, and / or SSIX type box, such as with segment index information for the encoded representation. The segment index may be embedded in the encoded media file (e.g., a complete .mp4 file), where the index-related box may be located at the beginning of the file.

[0111] The segment index file may be referenced from the MPD file such as by the presence of a RepresentationIndex element that provides a segment index for a representation. The segment index file may be referenced from the MPD file such as by the presence of at least one of the two attributes @index or @indexRange in the SegmentList.SegmentURL element. The segment index file may be referenced from the MPD file such as by the presence of a SegmentTemplate@index attribute.

[0112] The @indexRange attribute can be used to provide a byte range for indexing within a media segment. This can be done when permitted by the media segment format. For example, the @index attribute may not be present, and the specified range may lie partially or entirely within the byte range specified for the media segment.

[0113] The segment index file used for quality signaling may include an indication that quality information is present in the index file (e.g., by means of a descriptor or attribute in the MPD file), a specification of a quality metric type (e.g., included in the MPD and / or within the segment index box), and / or a list of quality values ​​for segments and / or subsegments in the representation. The list of quality values ​​may be compressed, such as by means of allowed length encoding.

[0114] The segment index file for quality signaling may include multiple descriptive elements. For example, the segment index file may indicate that quality information is present in the index file, such as using a descriptor or attribute in the MPD file. The segment index file may include specifications for quality metric types, such as in the MPD file or within a segment index box. The segment index file may include a list of quality values ​​for one or more segments or subsegments in a representation. The list may be compressed, such as using run-length encoding.

[0115] In order to store quality parameters in an ISOBMFF-based file container, a box may be defined. For example, Table 2 listed below may be added at the end of Table 1 of ISO / IEC 14496-12.

[0116] Table 2

[0117] qtyp < <ref> > < / ref> Quality metric types sqls < <ref> > < / ref> Segment Quality List ssql < <ref> > < / ref> Subsegment Quality List

[0118] The following definitions may apply. qtyp may represent a box describing a quality metric type, such as but not limited to PSNR, SSIM, MS-SSIM, and / or VQM. sqls may represent a box including a list of quality metrics for segments in a representation. An example syntax is described below.

[0119]

[0120] ssql may represent a box that includes a list of quality metrics for subsegments in a segment. An example syntax is described below.

[0121]

[0122] In order to indicate to the client that quality information is present in the index file, the MPD file syntax may be extended to allow an additional qualityInfoPresentInIndexFiles flag. An example of the use of the file is shown below.

[0123] <AdaptationSet segmentAlignment="true"bitstreamSwitching="true"subsegmentAlignment="true"qualityInfoPresentInIndexFiles=”true”>

[0124] <ContentComponent id="101"contentType="video" / >

[0125] <ContentComponent id="102"contentType="audio" / >

[0126] <Representation id="1"mimeType="video / mp4"codecs="avc1.42c00d"width="320"height="180"sampleRate="44100"numChannels="1"startWithRAP="1"bandwidth="134829">

[0127] <SegmentList timescale="1000"duration="10000">

[0128] <representationindex>example_320x180_48kbps_

[0129] av_index.didx< / representationindex>

[0130] The quality information about the encoded segments can be added by using and / or storing a separate file with its own syntax. This file can be associated with the corresponding adaptation set in the MPD file, such as by means of a descriptor. An example of a descriptor is shown below.

[0131] <QualityInfo

[0132] schemeIdUri="urn:mpeg:dash:qual-dsc:2012">

[0133] <component>=”video”

[0134] <qualitymetric>=”PSNR”

[0135] QualityUrl="http: / / example.com / ... / segments_psnr_t328.xyz" / >

[0136]

[0137] This may provide a path to systems (eg, existing systems) and deployment of encoded content.

[0138] Fig. 9 1 is an example diagram illustrating a quality information track 900 added to a DASH presentation. The quality information may be embedded in a media data container (mdat) box 902. The quality information may be stored as a quality information track 900 within the media data container (mdat) box 902 and a video track 904 within the presentation. To add the quality information track 900, the DASH initialization segment may list a video track such as Fig. 9 The added tracks are shown in .

[0139] Within the container 906 for media information in a track (trak box), a container 908 for media information in a track (e.g., mdia box) may include a media information container 910 (e.g., minf box). For the quality information track 900, the media information container 910 (e.g., minf box) may utilize a null media header (e.g., nmhd box 912) type. In the nmhd box 912, quality information may be provided and include, for example, a quality metric type (e.g., PSNR, SSIM, etc.). The following provides a syntax example for the nmhd box 912 including quality information.

[0140] aligned(8)class NullMediaHeaderWithQualityInfo extends FullBox('nmhd,version,0){

[0141] unsigned int(16)reserved=0;

[0142] unsigned int(8)quality_metric_type;

[0143] }

[0144] quality_metric_type is an enumeration. For example, 0 represents PSNR, 1 represents SSIM, etc. The presence of the quality information track 900 can be signaled in the description file, for example, as provided below:

[0145] <AdaptationSet segmentAlignment="true"bitstreamSwitching="true"subsegmentAlignment="true"qualityInfoTrackPresent=”true”>

[0146] <ContentComponent id="100"contentType="quality" / >

[0147] <ContentComponent id="101"contentType="video" / >

[0148] <ContentComponent id="102"contentType="audio" / >

[0149] <Representation id="1"mimeType="video / mp4"codecs="avc1.42c00d"width="320"height="180"sampleRate="44100"numChannels="1"startWithRAP="1"bandwidth="134829">

[0150] <SegmentList timescale="1000"duration="10000">

[0151] As in Fig.10 As shown in FIG. 1 , a sidx (segment index) box 1002 may include a list of pointers to pairing a moof box 1004 and a madat box 1006. For example, Fig.10 As shown in , each pair represents a sub-segment. Moof box 1004 can list the tracks present in mdat box 1006.

[0152] Fig.10 An example diagram describing quality information stored in an mdat box within a DASH segment 1008. Quality information may be added at a sub-segment level. For example, a quality track 1010 may be added in an mdat box 1006, such as the mdat box 1006 which may summarize quality information of a video track at a sub-segment level. The quality information may appear at the beginning of an mdat box 1006, such as where it may be easily extracted by a client. The moof box 1004 may include a list 1012 indicating the presence of a quality track 1010. Example syntax for quality information in an mdat box 1006 is provided below.

[0153] aligned(8)class QualityTrack extends{

[0154] unsigned int(16)sub_segment_quality;

[0155] }

[0156] Quality information may be provided in a free space (e.g., free or skip) box. Quality information may be stored in a free space (e.g., free or skip) box within the MPEG DASH segment. For example, the contents of the free space box are irrelevant and / or may be ignored without affecting the description. The free or skip box may be a top-level box (e.g., a peer of a movie box mbox box and / or a media data mdat box). Fig.11 As shown in , a free or skip box may be placed after a segment index sidx box 1100 in an MPEG DASH segment 1102, such as near the beginning of the segment. A pair of a moof box 1104 and / or an mdat box 1106 may represent a sub-segment and / or correspond to a moov and / or mdat pair in an MP4 file. A free and / or skip box 1108 may be added, such as near the beginning of a segment, so that it may be accessed before the entire segment is extracted.

[0157] The presence of free and / or skip boxes 1108 may be signaled in the description file, for example, as provided below:

[0158] <AdaptationSet segmentAlignment="true"bitstreamSwitching="true"subsegmentAlignment="true"qualityInfoPresentInFreeBoxes=”true”>

[0159] <ContentComponent id="101"contentType="video" / >

[0160] <ContentComponent id="102"contentType="audio" / >

[0161] <Representation id="1"mimeType="video / mp4"codecs="avc1.42c00d"width="320"height="180"sampleRate="44100"numChannels="1"startWithRAP="1"bandwidth="134829">

[0162] <SegmentList timescale="1000"duration="10000">

[0163] Fig.11 An example diagram depicting quality information stored in a free or skip box 1108 within a DASH segment 1102. For example, a free and / or skip box 1108 may be added at the time the segment is generated. If a free and / or skip box 1108 is added after the segment is generated (e.g., in previously generated DASH content), the offset used in the sample table of sidx may be recalculated.

[0164] The format of the free and / or skip boxes 1108 is defined. For example, a syntax similar to that proposed for use with segment index files may be utilized. An example syntax is provided below:

[0165]

[0166]

[0167] The features and elements described below can be applied to HTTP Live Streaming (HLS) and / or other streaming systems. Quality information signaling for segments can be added in advance. The client can use this information when selecting a stream to request and / or subscribe to.

[0168] Adding quality-related information may be achieved by including quality-related information in a manifest file (e.g., a .mdp file), including quality-related information in a segment index stored in a segment index file (e.g., an MP4 or M4S file), and / or providing an additional file with quality / segment information and providing a link to the quality / segment information from the MPD file.

[0169] Dynamic Adaptive HTTP Streaming (DASH) is a standard that consolidates some of the methods of HTTP streaming. MPEG DASH is an extension of "3GP-DASH". DASH can be used to handle variable bandwidth in wireless and wired networks and is supported by content providers and devices. DASH can enable multimedia streaming services over any access network to any device.

[0170] Fig.12 An example diagram depicting a high-level architecture of a DASH system 1200 is shown. A DASH system can be deployed as a set of HTTP servers 1202 that can distribute live and / or on-demand content that has been prepared in a suitable format. Clients can access content directly from these HTTP servers and / or from a content delivery network (CDN) or CDN 1204. CDNs can be used for deployments when a large number of clients are expected because they can cache content and are located close to clients at the edge of the network.

[0171] In DASH, a streaming session can be controlled by the client 1206 by using HTTP to request segments and splice them together as they are received from the content provider and / or CDN 1204. Clients can monitor, e.g., continuously monitor, and adjust the media rate based on network conditions (e.g., packet error rate, delay jitter) and their own status (e.g., buffer full, user behavior, preferences), effectively moving intelligence from the network to the client.

[0172] The DASH standard may be similar to a client-side model of information. Fig.13 is an example of the logical components of the conceptual DASH client model 1300. The DASH access engine can receive a media presentation description file (MPD). The DASH access engine 1302 can construct and issue requests and receive segments or partial segments. The output of the DASH access engine 1302 can include media in an MPEG container format (e.g., MP4 file format and / or MPEG-2 transport stream) and / or timing information that maps the internal timing of the media to the timeline of the presentation. The combination of the encoded media blocks and / or timing information is sufficient for the correct rendering of the content.

[0173] Some of the restrictions that DASH imposes on the encoded media segments may be based on the assumption that decoding, post-processing, and / or playback is performed by the media engine 1304 without information about the properties of the encoded media segments and / or how the encoded media segments are delivered. The media engine 1304 may decode and play a persistent media file that is provided in chunks by the DASH access engine.

[0174] For example, the DASH access engine 1302 may use JavaScript, and the media engine 1304 may be provided by a browser, a browser plug-in (such as but not limited to Flash or Silverlight), and / or an operating system.

[0175] Fig.14 1400 is an example block diagram describing a DASH media presentation high-level data model 1400. In DASH, the organization of multimedia presentations 1402 can be based on a hierarchical data model. A media presentation description (MPD) can describe a sequence of time segments 1404 that make up a DASH media presentation (e.g., multimedia content). Time segments 1404 can represent a media content time segment during which a set of encoded versions of the media content is available. For example, the set of available bit rates, languages, and / or subtitles does not change during time segment 1404.

[0176] Adaptation set 1406 may represent a collection of interoperable encoded versions of one or more media content components, e.g., there may be adaptation sets 1406 for video, one for primary video, one for secondary video, and / or one for subtitles. Adaptation sets 1406 may be multiplexed, in which case the multiplexed interoperable versions are described as a single adaptation set 1406. For example, adaptation set 1406 may include video and / or audio for time period 1404.

[0177] Representation 1408 can describe the transmissible encoded version of one or more multimedia content components. Representation 1408 can include one or more media streams (e.g., one for each media content component in the multiplexing). A single representation 1408 in the adaptation set 1406 is sufficient for rendering the included media content components. The client can switch from one representation 1408 to another representation 1408 within the adaptation set 1406 to adapt to network conditions or other factors. The client can ignore the representation 1408 that uses a codec, profile and / or parameter that the client does not support. The content in the representation 1408 can be divided into segments 1410 of fixed or variable length in time. URLs can be provided for each segment 1410. Segment 1410 can be the largest data unit obtained in a single HTTP request.

[0178] A media presentation description (MPD) may be an XML document that may include metadata used by a DASH client to construct appropriate HTTP-URLs to access segments and / or provide streaming services to users. The base URLs in the MPD may be used by clients to generate HTTP get (GET) requests for segments and other resources in a media presentation. HTTP partial get requests may be used to access limited portions of a segment by using byte ranges (e.g., "Range" in an HTTP header). Alternative base URLs may be specified to allow access to a presentation in the event that a location is unavailable, thereby providing redundancy to multimedia stream delivery to allow client-side load balancing, and / or parallel downloading.

[0179] MPDs can be static or dynamic in type. Static MPD types may or may not change during multimedia presentation and may be used for on-demand presentation. Dynamic MPD types may be updated during media presentation and may be used for live presentation. MPDs may be updated to extend the segment list for each representation, introduce new time periods, and / or terminate media presentation.

[0180] In DASH, encoded versions of different media content components (e.g., video, audio) can share a common timeline. The presentation time of an access unit within the media content can be mapped to a global common presentation timeline, which can be called a media presentation timeline. This mapping can allow synchronization of different media components and can facilitate seamless switching of different encoded versions (e.g., representations) of the same media component.

[0181] Segments may include the actual segmented media streams. They may include information about how to map the media streams to a media presentation timeline for switching and / or synchronizing presentation with other representations.

[0182] The segment availability timeline can be used to inform the client of the available time of the segment at a specific HTTP URL. For example, these times can be provided in wall clock time. Before accessing the segment at a specific HTTP URL, the client can compare the wall clock time with the segment availability time. For on-demand content, the segment availability time can be the same. Once the segment is available, the segment of the media presentation is available on the server. The MPD can be a static document.

[0183] For live content, the available time of a segment can depend on the location of the segment in the media presentation timeline. Segments can be available over time as content is generated. The MPD is periodically updated to reflect changes in presentation over time. For example, segment URLs for new segments can be added to the MPD. Old segments that are no longer available can be removed from the MPD. If a template is used to describe the segment URLs, updating the MPD may be omitted.

[0184] The duration of a segment may represent the duration of the media included in the segment at normal speed. The segments in a representation may have the same or approximately the same duration. The segment duration may vary from representation to representation. DASH representations may be constructed using relatively short segments (e.g., a few seconds), or longer segments, including a single segment for the entire representation.

[0185] Short segments are suitable for live content (e.g., by reducing end-to-end latency) and allow high switching granularity at the segment level. Small segments can increase the number of files in a presentation. Long segments can improve cache performance by reducing the number of files in a presentation. Long segments can enable clients to make flexible request sizes (e.g., by using byte range requests). Using long segments can include using segment indexes and is less suitable for live events. Segments may or may not expand over time. Segments can be complete and discrete units that are available as a whole.

[0186] Segments may be further subdivided into subsegments. Each subsegment may include the full number of complete access units. An access unit may be a media stream with an assigned media presentation time. If segments are divided into subsegments, these may be described by a segment index. The segment index may provide the presentation time range in the representation and the corresponding byte range in the segment occupied by each subsegment. The client may download this index in advance and issue requests for individual subsegments using, for example, an HTTP partial GET request. The segment index may be included in the media segment, for example at the start of the file. Segment index information may be provided in a separate index segment.

[0187] DASH, for example, may define many segment types, including but not limited to initialization segments, media segments, index segments, and / or bitstream switch segments. The initialization segment includes initialization information for accessing a representation. The initialization segment may or may not include media data with an assigned presentation time. Conceptually, the initialization segment may be processed by a client to initialize a media engine for starting playback of media segments of the included representation.

[0188] A media segment may include and may encapsulate media streams described in the media segment and / or described by an initialization segment of a representation. A media segment may include a number of complete access units and may include a stream access point (SAP) for each included media stream.

[0189] Index segments may include information related to media segments. Index segments may include index information for media segments. Index segments may provide information for one or more media segments. Index segments may be media format specific, and details may be defined for each media format that supports index segments.

[0190] A bitstream switch segment may include data for switching to its assigned representation. A bitstream switch segment may be media format specific and the details may be defined for media formats that allow bitstream switch segments. One bitstream switch segment may be defined for each representation.

[0191] The client can switch from representation to representation within an adaptation set at any point in the media. Switching at arbitrary locations can be complex due to coding dependencies within representations and other factors. Downloading overlapping data (e.g., media from the same time period from multiple representations) can be avoided. Switching is simpler at random access points in a new stream. DASH can define the codec-independent concept of a stream access point (SAP) and identify various types of media access points. The stream access point type can be communicated as one of the characteristics of the adaptation set (e.g., assuming that all segments in an adaptation set have the same SAP type).

[0192] A stream access point (SAP) may enable random access to a file container of one or more media streams. A SAP may be a location in a container where playback of an identified video to be started can be performed using information included in the container from a previous location and / or possible initialization data from another portion or other portions of the container. The initialization data may be externally available.

[0193] Several file container properties can be defined. SAP It can be a presentation time, for example, the earliest presentation time of an access unit of a media stream, whereby the presentation time is greater than or equal to T SAP All access units of the media stream can be used in I SAP The data in the bit stream at the beginning and no I SAP The previous data was decoded correctly.

[0194] I SAP can be a position in the bitstream whereby the presentation time is greater than or equal to T SAP All access units of the media stream can be used in I SAP The bit stream data starts at I SAP The data that started before is decoded correctly.

[0195] I SAU It may be the starting position in the bitstream of the most recently accessed unit in decoding order within the media stream, such that the presentation time is greater than or equal to T SAP All access units of the media stream can be correctly decoded using this latest access unit and the access units that follow in decoding order and without access units earlier in decoding order.

[0196] T DEC It can be the earliest presentation time of an access unit of the media stream, and the access unit can be presented using the I SAU The data at the beginning and with or without I SAU The data started before is decoded correctly. EPT It can be in the bit stream. SAU The earliest presentation time of the access unit of the starting media stream. PTF It can be in the bit stream. SAU The presentation time of the first access unit of the media stream in decoding order.

[0197] Fig.15 is an example of a stream access point with parameters. Fig.15 A coded video stream 1500 is described with three different types of frames: I, P, and B. P frames can utilize (or only utilize) a previous I or P frame to be decoded, while B frames can utilize previous and subsequent I and / or P frames. I, P, and / or B frames differ in transmission, decoding, and / or presentation order.

[0198] The type of SAP may depend on which access units are correctly decodable and / or their arrangement in presentation order. Examples of six SAP types are described herein. EPT =T DEC =T SAP =T PTF , which may correspond to a so-called "closed GoP random access point". SAP The first access unit (in decoding order) may be correctly decoded. The result is a duration sequence of correctly decoded access units without gaps. The first access unit in decoding order may be the first access unit in presentation order.

[0199] In another SAP type, T EPT =T DEC =T SAP <T PTF This SAP type corresponds to a so-called "closed GoP random access point", for which SAU The first access unit in decoding order in the starting media stream is not the first access unit in presentation order. For example, the first two frames may be backward-predicted P frames (which are semantically encoded as forward-only B frames in H.264 and some other codecs), and / or they may or may not require a third frame to be decoded.

[0200] In another SAP type, T EPT <T DEC =T SAP <=T PTF This SAP type corresponds to a so-called "open GoP random access point" where there are random access points in decoding order. SAU Some subsequent access units were not correctly decoded and / or presented for less than T SAP .

[0201] In another SAP type, T EPT <=T PFT <T DEC =T SAP This SAP type corresponds to a random access called "gradual decoding update (GDR) random access point" or "dirty", in which there are random accesses in decoding order from I SAU Start and in I SAU Some subsequent access units were not correctly decoded and / or presented for less than T SAP An example case of GDR is an intra refresh process, where the intra refresh process can be extended over N frames with a portion of the frame coded using intra MB. The non-overlapping portion can be intra-coded over N frames. The process can be repeated until the entire frame is refreshed.

[0202] In another SAP type, T EPT =T DEC <T SAP This SAP type corresponds to when there is a SAP At least one access unit at the beginning is not decoded correctly and has a value greater than T DEC Large presentation time, and / or when T DEC For I SAU The earliest presentation time of the access unit that starts.

[0203] In another SAP type, T EPT <T DEC <T SAP This SAP type corresponds to the case where there is at least one access unit starting from the ISAP in decoding order that is not correctly decoded and has a value greater than T DEC Large presentation time, and / or when T DEC For I SAU The earliest presentation time of the access unit that starts.

[0204] DASH profiles can be defined to enable interoperability and signaling of feature usage. Profiles can impose a set of restrictions. These restrictions can be features and / or segment formats about a media presentation description (MPD) document. The restrictions can be about content delivered within the segment range, such as but not limited to media content type, media format, codec, and / or protection format, and / or about quantifiable measurements, such as but not limited to bit rate, segment duration and size, and / or horizontal and vertical visual presentation size.

[0205] For example, DASH can be defined in Fig.16 . The profiles are organized into two categories according to the file container type used for segmentation. These profiles 1600, 1602, 1604 can use ISO base media file containers, two profiles 1606, 1608 can use file containers based on MPEG-2 transport streams (TS), and one profile 1610 can support two file container types. Each container type can be codec independent.

[0206] The ISO base media file format of the on-demand profile 1602 can provide basic support for on-demand content. The restrictions of the on-demand profile 1602 can be that each representation can be provided as a single segment, sub-segments can be aligned between representations of the adaptation set range, and / or sub-segments can start at a stream access point. The on-demand profile 1602 can be used to support a large video on demand (VoD) library with relatively little content management. The on-demand profile 1602 can allow scalable and efficient use of HTTP servers and simplify seamless switching.

[0207] The ISO base media file format live profile 1604 can be optimized for live encoding and / or low-latency delivery of segments consisting of a single movie fragment in ISO file format with a relatively short duration. Each movie fragment can be requested when needed. This can be done, for example, using a template-generated URL. Requests for MPD updates can be ignored for some segment requests. Segments can be restricted so that the segments can be connected on segment boundaries and decrypted without gaps and / or overlaps in the media data. This can be independent of the adaptive switching of representations in the adaptive set. The profile 1604 can be used to distribute non-live content. For example, when the live media presentation has been terminated but is kept available as a on-demand service, the live profile 1604 can be used. The ISO base media file format main profile 1600 can be a superset of the ISO base media file format on-demand profile 1602 and the live profile 1604.

[0208] The MPEG-2TS main profile 1606 can impose a small number of restrictions on the media segmentation format for MPEG-2 transport stream (TS) content. For example, representations can be multiplexed, thereby not requiring the media streams (audio, video) to be bound at the client. For example, a segment can include an integer number of MPEG-2 TS packets. For example, indexing and segment arrangement can be recommended. HTTP Live Streaming (HLS) content can be integrated with the profile 1606 by converting the HLS media presentation description (.m3u8) into a DASH MPD.

[0209] MPEG-2 TS brief profile 1608 may be a subset of MPEG-2 TS main profile 1606. This may impose more restrictions on content encoding and multiplexing in order to allow for simple implementation of seamless switching. Seamless switching may be achieved by ensuring that a media engine compliant with ISO / IEC 13818-1 (MPEG-2 Systems) may play any bitstream generated by concatenating consecutive segments from any representation within the same adaptation set. Full profile 1610 may be a superset of ISO base media file format main profile 1600 and MPEG-2 TS main profile 1606.

[0210] Fig.17 An example block diagram for describing a block-based video encoder such as a hybrid video coding system. An input video signal 1702 may be processed block by block. A video block unit may include 16x16 pixels. The block unit may be referred to as a macroblock (MB). In high efficiency video coding (HEVC), an extended block size (e.g., may be referred to as a "coding unit" or CU) may be used to efficiently compress high resolution (e.g., 1080p and below) video signals. In HEVC, a CU may be up to 64x64 pixels. A CU may be partitioned into prediction units (PUs), to which independent prediction methods may be applied.

[0211] For an input video block (e.g., MB or CU), spatial prediction 1760 and / or temporal prediction 1762 may be performed. Spatial prediction (e.g., "intra-prediction") may use pixels from already encoded neighboring blocks in the same video picture / slice to predict the current video block. Spatial prediction may reduce spatial redundancy inherent in a video signal. Temporal prediction (e.g., "inter-prediction" or "motion compensated prediction") may use pixels from already encoded video pictures (e.g., which may be referred to as "reference pictures") to predict the current video block. Temporal prediction may reduce temporal redundancy inherent in a video signal. The temporal prediction signal for a video block may be represented by one or more motion vectors, which may indicate the amount and / or direction of motion between the current block and its prediction block in a reference picture. If multiple reference pictures are supported (e.g., for H.264 / AVC and / or HEVC), then for each video block, its reference picture index may be additionally sent. The reference index may be used to identify which reference picture in a reference picture store 1764 (eg, may be referred to as a “decoded picture buffer” or DPB) the temporal prediction signal comes from.

[0212] After spatial and / or temporal prediction, a prediction mode may be selected by a mode decision block 1780 in the encoder. The prediction block may be subtracted from the current video block 1716. The prediction residual may be transformed 1704 and / or quantized 1706. The quantized residual coefficients may be inverse quantized 1710 and / or inverse transformed 1712 to form a reconstructed residual, which may be added back to the prediction block 1726 to form a reconstructed video block.

[0213] In-loop filtering, such as but not limited to a deblocking filter, sample adaptive offset, and / or an adaptive loop filter may be applied to the reconstructed video block 1766 before it is placed in a reference picture store 1764 and / or used to encode future video blocks. To form an output video bitstream 1720, the coding mode (e.g., inter-prediction mode or intra-prediction mode), prediction mode information, motion information, and / or quantized residual coefficients may be sent to an entropy coding unit 1708 to be compressed and / or packed to form a bitstream.

[0214] Fig.18 is an example block diagram describing a block-based video decoder. The video bitstream 1802 may be unpacked and / or entropy decoded at an entropy decoding unit 1808. The coding mode and / or prediction information may be sent to a spatial prediction unit 1860 (e.g., if intra-coded) and / or a temporal prediction unit 1862 (e.g., if inter-coded) to form a prediction block. If inter-coded, the prediction information may include a prediction block size, one or more motion vectors (e.g., which may indicate a direction and amount of motion), and / or one or more indexes (e.g., which may indicate from which reference picture the prediction signal is to be obtained).

[0215] Motion compensated prediction may be applied by a temporal prediction unit 1862 to form a temporal prediction block. The residual transform coefficients may be sent to an inverse quantization unit 1810 and an inverse transform unit 1812 to reconstruct the residual block. The prediction block and the residual block may be added together at 1826. The reconstructed block may be filtered in-loop before being stored in a reference picture store 1864. The video reconstructed in the reference picture store 1864 may be used to drive a video device and / or be used to predict future video blocks.

[0216] A single-layer video encoder can take a single video sequence input and generate a single compressed bitstream that is transmitted to a single-layer decoder. Video codecs can be designed for digital video services (e.g., such as, but not limited to, TV signals transmitted via satellite, optical cable, and terrestrial transmission channels). Multi-layer video coding techniques can be deployed as extensions of video coding standards to enable a variety of applications using video-centric applications deployed in heterogeneous environments. For example, scalable video coding techniques can be designed to process more than one video layer, where each layer can be decoded to reconstruct a video signal of a specific spatial resolution, temporal resolution, fidelity, and / or view. Any of the concepts described herein can be modified by, for example, those referenced herein. Fig.17 and Fig.18 Furthermore, although reference is made to the encoder and / or decoder described herein, Fig.17 and Fig.18 Single-layer encoders and decoders are described, but the concepts described herein may be utilized with multi-layer encoders and decoders, such as for multi-layer or scalable coding techniques.

[0217] The process described above may be implemented in a computer program, software and / or firmware executed by a computer or processor, wherein the computer program, software and / or firmware is contained in a computer-readable storage medium. Examples of computer-readable media include, but are not limited to, electronic signals (transmitted via wired and / or wireless connections) and / or computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, buffer memory, semiconductor storage devices, magnetic media (such as, but not limited to, internal hard disks or removable disks), magneto-optical media and / or optical media such as CD-ROM disks and / or digital versatile disks (DVDs). A processor associated with the software may be used to implement a radio frequency transceiver used in a WTRU, a user equipment (UE), a terminal, a base station, an RNC and / or any host computer.< / qualitymetric> < / component>

Claims

1. A method performed by a wireless transmit / receive unit WTRU, the method comprising: receiving a manifest file comprising a description of a plurality of segments of media content, wherein each segment represents a time-based portion of the media content, wherein the manifest file specifies a bitrate and a quality value for each of the plurality of segments, and wherein the quality value is a measure of a quality of each of the plurality of segments being encoded; selecting a first segment from the plurality of segments based on a first bit rate and a first quality value, wherein the first bit rate and the first quality value are associated with the first segment; sending a request for the first segment; receiving the first segment; selecting a second segment from the plurality of segments based on a second bit rate and a second quality value, wherein the second bit rate and the second quality value are associated with the second segment, wherein the second bit rate is lower than the first bit rate, wherein the second segment is selected based on the second quality value being the same as or greater than a quality threshold level; sending a request for the second segment; and The second segment is received.

2. The method of claim 1, wherein the quality value comprises at least one of: peak signal-to-noise ratio, structural similarity, video quality metric, visual information fidelity, J.341, mean opinion score, or multi-metric structural similarity. The method of claim 1 , wherein the manifest file comprises a media presentation description file. 4 . The method of claim 1 , wherein the manifest file further comprises a description of a plurality of representations of the media content, and wherein each representation comprises at least one segment of the media content. 5 . The method of claim 1 , wherein the manifest file specifies a same duration of the media content for each of the plurality of segments. The method of claim 1 , wherein the manifest file comprises a segment index file.

7. The method of claim 6, wherein the segment index file comprises at least one of an MP4 file and an M4S file.

8. The method of claim 6, wherein the segment index file comprises an ISOBMFF-based file container, the ISOBMFF-based file container comprising at least one box, and wherein the segment quality parameter is included within the at least one box of the ISOBMFF-based file container.

9. The method according to claim 1, further comprising: Based on a flag in the manifest file, the presence of the quality value is identified.

10. A wireless transmit / receive unit WTRU, comprising: A processor configured to: receiving a manifest file, wherein the manifest file comprises a description of a plurality of segments of media content, wherein each segment represents a time-based portion of the media content, wherein the manifest file specifies a bitrate and a quality value for each of the plurality of segments, and wherein the quality value is a measure of a quality of each of the plurality of segments being encoded; selecting a first segment from the plurality of segments based on a first bit rate and a first quality value, wherein the first bit rate and the first quality value are associated with the first segment; sending a request for the first segment; receiving the first segment; selecting a second segment from the plurality of segments based on a second bit rate and a second quality value, wherein the second bit rate and the second quality value are associated with the second segment, wherein the second bit rate is lower than the first bit rate, wherein the second segment is selected based on the second quality value being the same as or greater than a quality threshold level; sending a request for the second segment; and The second segment is received.

11. The WTRU of claim 10, wherein the quality value comprises at least one of: peak signal-to-noise ratio, structural similarity, video quality metric, visual information fidelity, J.341, mean opinion score, or multi-metric structural similarity.

12. The WTRU of claim 10, wherein the manifest file comprises a media presentation description file.

13. The WTRU of claim 10, wherein the manifest file further comprises a description of a plurality of representations of the media content, and wherein each representation comprises at least one segment of the media content.

14. The WTRU of claim 10, wherein the manifest file specifies the same duration of the media content for each of the plurality of segments.

15. The WTRU of claim 10, wherein the manifest file comprises a segment index file.

16. The WTRU of claim 15, wherein the segment index file comprises at least one of an MP4 file and an M4S file.

17. The WTRU of claim 15, wherein the segment index file comprises an ISOBMFF-based file container, the ISOBMFF-based file container comprising at least one box, and wherein the segment quality parameter is included within the at least one box of the ISOBMFF-based file container.

18. The WTRU of claim 10 wherein the processor is further configured to: Based on a flag in the manifest file, the presence of the quality value is identified.