Transmission of time information data for measuring qoe metrics

Abstract

Systems, methods, and instrumentalities may be configured for transmitting / receiving QoE timing information for measuring quality of experience (QoE) metrics. A device (e.g., a wireless / transmit receive unit (WTRU) and / or a media capabilities for Augmented Reality (MeCAR) device) may receive, from a split- rendering server (SRS), a real-time control protocol (RTCP) message comprising a QoE timing information extended report (XR) block. The QoE timing information XR block may indicate a block type for QoE timing information. The QoE timing information (e.g., recorded at the SRS) may include an estimated at time, a start to render at time, a split-rendering server output time, and a scene update time. The device may calculate at least one QoE metric based on the QoE timing information.

Description

TRANSMISSION OF TIME INFORMATION DATA FOR MEASURING QOE METRICSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of provisional EP patent application No. 23306369.2, filed August 11 , 2023, and EP patent application No. 23306648.9, filed September 29, 2023, the disclosure of which is incorporated herein by reference in its entirety.BACKGROUND

[0002] Video coding systems may be used to compress digital video signals, for example, to reduce the storage and / or transmission bandwidth needed for such signals. Video coding systems may include, for example, wavelet-based systems, object-based systems, and / or block-based systems, such as a blockbased hybrid video coding system. In extended reality applications, the mechanisms for providing control of degree of freedom boundaries may not be adequate.SUMMARY

[0003] Systems, methods, and instrumentalities may be configured for transmission of time information data for measuring quality of experience (QoE) metrics. A device (e.g., a wireless / transmit receive unit (WTRU) and / or a media capabilities for Augmented Reality (MeCAR) device) may receive, from a splitrendering server (SRS), a real-time control protocol (RTCP) message comprising at least a quality of experience (QoE) timing information extended report (XR) block. The QoE timing information XR block may include an indication of a block type indicating that the QoE timing information XR block comprises QoE timing information. The QoE timing information may be recorded at the SRS. The QoE timing information may include at least an estimated at time, a start to render at time, a split-rendering server output time, and a scene update time. The device may calculate at least one QoE metric based on at least one or more of the estimated at time, the start to render at time, the split-rendering server output time, or the scene update time.

[0004] The device may receive a session description protocol (SDP) a field indicating use of RCTP XR blocks for signaling the QoE timing information, wherein the RTCP message is associated with the use of the RCTP XR blocks. The device may extract the QoE timing information based on the block type indicating that the XR block comprises the timing information.

[0005] The RTCP message may include a field indicating that one or more types of the QoE timing information are present in the RTCP message and an order associated with the one or more types of theQoE timing information. The start to render at time may indicate a time a renderer associated with the SRS started to render a media frame. The estimated at time may indicate a time when pose estimation was made. The split-rendering server output time may indicate a time associated with an output of the SRS. The scene update time may indicate a time when processing of one or more actions was started by the SRS. The at least one QoE metric may include one or more of a round-trip interaction delay, a server processing delay, a user interaction delay, or an age of content. The server processing delay may be determined based on a sum of the age of content and the user interaction delay

[0006] The user interaction delay may include a duration between a first time when a user action is initiated and a second time when the user action is taken into account by a content creation engine. The user interaction delay may depend on uplink latency. The age of content may include a time duration between a first time when a content is created and a second time when the content is presented. The age of content or the user interaction delay may be determined based on at least the scene update time. The age of content may depends on downlink latency. The round-trip interaction delay may be determined as a sum of the user interaction delay and the age of content.

[0007] Systems, methods, and instrumentalities may be configured for transmission of time information data for measuring quality of experience (QoE) metrics. A device (e.g., a wireless / transmit receive unit (WTRU) and / or a media capabilities for Augmented Reality (MeCAR) device) may receive a real-time control protocol (RTCP) extended report (XR) packet with an extended report block, the extended report block comprising a field representing timing information, a block length indicator, an identifier for a source of real-time protocol (RTP) data packets, a timestamp for synchronizing media data and QoE timing information recorded at a server. The device may extract the timing information from the extended report block. The device may calculate quality of experience (QoE) metrics based on the extracted timing information.

[0008] The device may select and transmit the timing information on a condition that specific bits are configured in the field representing timing information. The timing information may be expressed in a same unit as the real-time protocol (RTP) of RTP data packets, and synchronization may be performed using the timing information. The extended report block may include a timestamp indicating a time when the timing information was recorded at the server. The device may adjust use of real-time control protocol (RTCP) bandwidth by including timing information for calculating the quality of experience (QoE) metrics in the extended report block.

[0009] The extended report block may include a type field configured to indicate an application scenario associated with the QoE timing information. When multiple application scenarios are associated with the QoE timing information, respective RTCP XR blocks related to the application scenarios may be included inthe RTCP XR packet. The timing information from the extended report block may be expressed using NTP timestamps.

[0010] Systems, methods, and instrumentalities may be configured for transmission of time information data for measuring quality of experience (QoE) metrics. A server (e.g., a split render server and / or a realtime protocol (RTP) server) may determine a real-time control protocol (RTCP) extended report (XR) packet with an extended report block, the extended report block comprising a field representing timing information, a block length indicator, an identifier for a destination of real-time protocol (RTP) data packets, a timestamp for synchronizing media data, and QoE timing information to be sent to a device. The device may embed the timing information into the extended report block. The device may transmit quality of experience (QoE) metrics based on the embedded timing information.

[0011] The device may select the timing information for embedding and transmitting on a condition that specific bits are configured in the field representing timing information. The timing information may be expressed in a same unit as the real-time protocol (RTP) of RTP data packets, and synchronization may be performed using the timing information. The extended report block may include a timestamp indicating a time when the timing information is to be sent to the server. The device may adjust use of real-time control protocol (RTCP) bandwidth by including timing information for transmitting the quality of experience (QoE) metrics in the extended report block.

[0012] The extended report block may include a type field configured to indicate an application scenario associated with the QoE timing information. When multiple application scenarios are associated with the QoE timing information, respective RTCP XR blocks related to the application scenarios may be included in the RTCP XR packet. The timing information from the extended report block may be expressed using NTP timestamps.

[0013] Each feature disclosed anywhere herein is described, and may be implemented, separately / individually and in any combination with any other feature disclosed herein and / or with any feature(s) disclosed elsewhere that may be impliedly or expressly referenced herein or may otherwise fall within the scope of the subject matter disclosed herein.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG. 1 A is a system diagram illustrating an example communications system in which one or more disclosed embodiments may be implemented.

[0015] FIG. 1 B is a system diagram illustrating an example wireless transmit / receive unit (WTRU) that may be used within the communications system illustrated in FIG. 1A according to an embodiment.

[0016] FIG. 1 C is a system diagram illustrating an example radio access network (RAN) and an example core network (CN) that may be used within the communications system illustrated in FIG. 1 A according to an embodiment.

[0017] FIG. 1 D is a system diagram illustrating a further example RAN and a further example CN that may be used within the communications system illustrated in FIG. 1A according to an embodiment.

[0018] FIG. 2 is a diagram showing an example video encoder.

[0019] FIG. 3 is a diagram showing an example of a video decoder.

[0020] FIG. 4 is a diagram showing an example of a system in which various aspects and examples may be implemented.

[0021] FIG. 5 illustrates an example pose information call flow.

[0022] FIG. 6 illustrates an example user action call flow.

[0023] FIG. 7 illustrates an example real-time control protocol (RTCP) extended report (XR) packet format.

[0024] FIG. 8 illustrates an example RTCP XR block format.

[0025] FIG. 9 illustrates an example packet format for RTCP feedback messages.

[0026] FIG. 10 illustrates an example packet format for application-defined RTCP packets.

[0027] FIG. 11 illustrates an example RTCP XR block format for quality of experience (QoE) timing information data.

[0028] FIG. 12 illustrates an example XR block format for QoE timing information data with NTP timestamps.

[0029] FIG. 13 illustrates an RTCP XR block format for QoE timing information data.

[0030] FIG. 14 illustrates RTCP XR block format for QoE timing information data with NTP timestamps.

[0031] FIG. 15 illustrates an example RTCP feedback message format for QoE timing information data.

[0032] FIG. 16 illustrates an example RTCP feedback message format for QoE timing information data with network time protocol (NTP) timestamps.

[0033] FIG. 17 illustrates an example RTCP feedback message format for QoE timing information data.

[0034] FIG. 18 illustrates RTCP feedback message format for QoE timing information data with NTP timestamps.

[0035] FIG. 19 illustrates an example RTP header extension using one-byte header format.

[0036] FIG. 20 illustrates an example RTP header extension using two-byte header format.

[0037] FIG. 21 illustrates an example RTP header extension using two-byte header format.

[0038] FIG. 22 illustrates RTP header extension using a two-byte header format.

[0039] FIG. 23 illustrates RTP header extension using a two-byte header format.DETAILED DESCRIPTION

[0040] A detailed description of illustrative embodiments will now be described with reference to the various Figures. Although this description provides a detailed example of possible implementations, it should be noted that the details are intended to be exemplary and in no way limit the scope of the application.

[0041] FIG. 1A is a diagram illustrating an example communications system 100 in which one or more disclosed embodiments may be implemented. The communications system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communications system 100 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communications systems 100 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block-filtered OFDM, filter bank multicarrier (FBMC), and the like.

[0042] As shown in FIG. 1A, the communications system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN 104 / 113, a ON 106 / 115, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and / or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a “station” and / or a “STA”, may be configured to transmit and / or receive wireless signals and may include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Fi device, an Internet of Things (loT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot and / or other wireless devices operating in an industrial and / or an automated processing chain contexts), a consumer electronics device, a device operating on commercial and / or industrial wireless networks, and the like. Any of the WTRUs 102a, 102b, 102c and 102d may be interchangeably referred to as a WTRU.

[0043] The communications systems 100 may also include a base station 114a and / or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106 / 115, the I nternet 110, and / or the other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a gNB, a NR NodeB, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and / or network elements.

[0044] The base station 114a may be part of the RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and / or the base station 114b may be configured to transmit and / or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage for a wireless service to a specific geographical area that may be relatively fixed or that may change over time. The cell may further be divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e. , one for each sector of the cell. In an embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and / or receive signals in desired spatial directions.

[0045] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0046] More specifically, as noted above, the communications system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a in the RAN 104 / 113 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 115 / 116 / 117 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and / or Evolved HSPA (HSPA+). HSPA may include High-Speed Downlink (DL) Packet Access (HSDPA) and / or High-Speed UL Packet Access (HSUPA).

[0047] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0048] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR Radio Access , which may establish the air interface 116 using New Radio (NR).

[0049] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may implement LTE radio access and NR radio access together, for instance using dual connectivity (DC) principles. Thus, the air interface utilized by WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and / or transmissions sent to / from multiple types of base stations (e.g., an eNB and a gNB).

[0050] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

[0051] The base station 114b in FIG. 1 A may be a wireless router, Home Node B, Home eNode B, or access point, for example, and may utilize any suitable RAT for facilitating wireless connectivity in a localized area, such as a place of business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a roadway, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR etc.) to establish a picocell or femtocell. As shown in FIG. 1 A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not be required to access the Internet 110 via the CN 106 / 115.

[0052] The RAN 104 / 113 may be in communication with the CN 106 / 115, which may be any type of network configured to provide voice, data, applications, and / or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have varying quality of service (QoS)requirements, such as differing throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like. The CN 106 / 115 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and / or perform high-level security functions, such as user authentication. Although not shown in FIG. 1A, it will be appreciated that the RAN 104 / 113 and / or the CN 106 / 115 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 104 / 113 or a different RAT. For example, in addition to being connected to the RAN 104 / 113, which may be utilizing a NR radio technology, the CN 106 / 115 may also be in communication with another RAN (not shown) employing a GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology.

[0053] The CN 106 / 115 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and / or the other networks 112. The PSTN 108 may include circuit- switched telephone networks that provide plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP) and / or the internet protocol (IP) in the TCP / IP internet protocol suite. The networks 112 may include wired and / or wireless communications networks owned and / or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, which may employ the same RAT as the RAN 104 / 113 or a different RAT.

[0054] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multi-mode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links). For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0055] FIG. 1 B is a system diagram illustrating an example WTRU 102. As shown in FIG. 1 B, the WTRU 102 may include a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138, among others. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0056] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application SpecificIntegrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit / receive element 122. While FIG. 1 B depicts the processor 118 and the transceiver 120 as separate components, it will be appreciated that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0057] The transmit / receive element 122 may be configured to transmit signals to, or receive signals from, a base station (e.g., the base station 114a) over the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In an embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF and light signals. It will be appreciated that the transmit / receive element 122 may be configured to transmit and / or receive any combination of wireless signals.

[0058] Although the transmit / receive element 122 is depicted in FIG. 1 B as a single element, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0059] The transceiver 120 may be configured to modulate the signals that are to be transmitted by the transmit / receive element 122 and to demodulate the signals that are received by the transmit / receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11 , for example.

[0060] The processor 118 of the WTRU 102 may be coupled to, and may receive user input data from, the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and / or the removable memory 132. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card,a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0061] The processor 118 may receive power from the power source 134, and may be configured to distribute and / or control the power to the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0062] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 116 from a base station (e.g., base stations 114a, 114b) and / or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by way of any suitable locationdetermination method while remaining consistent with an embodiment.

[0063] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functionality and / or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs and / or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a Virtual Reality and / or Augmented Reality (VR / AR) device, an activity tracker, and the like. The peripherals 138 may include one or more sensors, the sensors may be one or more of a gyroscope, an accelerometer, a hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and / or a humidity sensor.

[0064] The WTRU 102 may include a full duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for both the UL (e.g., for transmission) and downlink (e.g., for reception) may be concurrent and / or simultaneous. The full duplex radio may include an interference management unit to reduce and or substantially eliminate self-interference via either hardware (e.g., a choke) or signal processing via a processor (e.g., a separate processor (not shown) or via processor 118). In an embodiment, the WRTU 102 may include a half-duplex radio for which transmissionand reception of some or all of the signals (e.g., associated with particular subframes for either the UL (e.g., for transmission) or the downlink (e.g., for reception)).

[0065] FIG. 1 C is a system diagram illustrating the RAN 104 and the CN 106 according to an embodiment. As noted above, the RAN 104 may employ an E-UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also be in communication with the CN 106.

[0066] The RAN 104 may include eNode-Bs 160a, 160b, 160c, though it will be appreciated that the RAN 104 may include any number of eNode-Bs while remaining consistent with an embodiment. The eNode-Bs 160a, 160b, 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the eNode-Bs 160a, 160b, 160c may implement MIMO technology. Thus, the eNode-B 160a, for example, may use multiple antennas to transmit wireless signals to, and / or receive wireless signals from, the WTRU 102a.

[0067] Each of the eNode-Bs 160a, 160b, 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and / or DL, and the like. As shown in FIG. 1 C, the eNode-Bs 160a, 160b, 160c may communicate with one another over an X2 interface.

[0068] The CN 106 shown in FIG. 1 C may include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (or PGW) 166. While each of the foregoing elements is depicted as part of the CN 106, it will be appreciated that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0069] The MME 162 may be connected to each of the eNode-Bs 162a, 162b, 162c in the RAN 104 via an S1 interface and may serve as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, bearer activation / deactivation, selecting a particular serving gateway during an initial attach of the WTRUs 102a, 102b, 102c, and the like. The MME 162 may provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM and / or WCDMA.

[0070] The SGW 164 may be connected to each of the eNode Bs 160a, 160b, 160c in the RAN 104 via the S1 interface. The SGW 164 may generally route and forward user data packets to / from the WTRUs 102a, 102b, 102c. The SGW 164 may perform other functions, such as anchoring user planes during inter- eNode B handovers, triggering paging when DL data is available for the WTRUs 102a, 102b, 102c, managing and storing contexts of the WTRUs 102a, 102b, 102c, and the like.

[0071] The SGW 164 may be connected to the PGW 166, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0072] The CN 106 may facilitate communications with other networks. For example, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. For example, the CN 106 may include, or may communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that serves as an interface between the CN 106 and the PSTN 108. In addition, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to the other networks 112, which may include other wired and / or wireless networks that are owned and / or operated by other service providers.

[0073] Although the WTRU is described in FIGS. 1 A-1 D as a wireless terminal, it is contemplated that in certain representative embodiments that such a terminal may use (e.g., temporarily or permanently) wired communication interfaces with the communication network.

[0074] In representative embodiments, the other network 112 may be a WLAN.

[0075] A WLAN in Infrastructure Basic Service Set (BSS) mode may have an Access Point (AP) for the BSS and one or more stations (STAs) associated with the AP. The AP may have an access or an interface to a Distribution System (DS) or another type of wired / wireless network that carries traffic in to and / or out of the BSS. Traffic to STAs that originates from outside the BSS may arrive through the AP and may be delivered to the STAs. Traffic originating from STAs to destinations outside the BSS may be sent to the AP to be delivered to respective destinations. Traffic between STAs within the BSS may be sent through the AP, for example, where the source STA may send traffic to the AP and the AP may deliver the traffic to the destination STA. The traffic between STAs within a BSS may be considered and / or referred to as peer-to- peer traffic. The peer-to-peer traffic may be sent between (e.g., directly between) the source and destination STAs with a direct link setup (DLS). In certain representative embodiments, the DLS may use an 802.11e DLS or an 802.11 z tunneled DLS (TDLS). A WLAN using an Independent BSS (IBSS) mode may not have an AP, and the STAs (e.g., all of the STAs) within or using the IBSS may communicate directly with each other. The IBSS mode of communication may sometimes be referred to herein as an “ad-hoc” mode of communication.

[0076] When using the 802.11 ac infrastructure mode of operation or a similar mode of operations, the AP may transmit a beacon on a fixed channel, such as a primary channel. The primary channel may be a fixed width (e.g., 20 MHz wide bandwidth) or a dynamically set width via signaling. The primary channel may be the operating channel of the BSS and may be used by the STAs to establish a connection with theAP. In certain representative embodiments, Carrier Sense Multiple Access with Collision Avoidance (CSMA / CA) may be implemented, for example in in 802.11 systems. For CSMA / CA, the STAs (e.g., every STA), including the AP, may sense the primary channel. If the primary channel is sensed / detected and / or determined to be busy by a particular STA, the particular STA may back off. One STA (e.g., only one station) may transmit at any given time in a given BSS.

[0077] High Throughput (HT) STAs may use a 40 MHz wide channel for communication, for example, via a combination of the primary 20 MHz channel with an adjacent or nonadjacent 20 MHz channel to form a 40 MHz wide channel.

[0078] Very High Throughput (VHT) STAs may support 20MHz, 40 MHz, 80 MHz, and / or 160 MHz wide channels. The 40 MHz, and / or 80 MHz, channels may be formed by combining contiguous 20 MHz channels. A 160 MHz channel may be formed by combining 8 contiguous 20 MHz channels, or by combining two non-contiguous 80 MHz channels, which may be referred to as an 80+80 configuration. For the 80+80 configuration, the data, after channel encoding, may be passed through a segment parser that may divide the data into two streams. Inverse Fast Fourier Transform (IFFT) processing, and time domain processing, may be done on each stream separately. The streams may be mapped on to the two 80 MHz channels, and the data may be transmitted by a transmitting STA. At the receiver of the receiving STA, the above described operation for the 80+80 configuration may be reversed, and the combined data may be sent to the Medium Access Control (MAC).

[0079] Sub 1 GHz modes of operation are supported by 802.11 af and 802.11 ah. The channel operating bandwidths, and carriers, are reduced in 802.11 af and 802.11 ah relative to those used in 802.11 n, and 802.11ac. 802.11 af supports 5 MHz, 10 MHz and 20 MHz bandwidths in the TV White Space (TVWS) spectrum, and 802.11 ah supports 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz bandwidths using non- TVWS spectrum. According to a representative embodiment, 802.11 ah may support Meter Type Control / Machine-Type Communications, such as MTC devices in a macro coverage area. MTC devices may have certain capabilities, for example, limited capabilities including support for (e.g., only support for) certain and / or limited bandwidths. The MTC devices may include a battery with a battery life above a threshold (e.g., to maintain a very long battery life).

[0080] WLAN systems, which may support multiple channels, and channel bandwidths, such as 802.11 n, 802.11 ac, 802.11 af, and 802.11 ah, include a channel which may be designated as the primary channel. The primary channel may have a bandwidth equal to the largest common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel may be set and / or limited by a STA, from among all STAs in operating in a BSS, which supports the smallest bandwidth operating mode. In the example of 802.11 ah, the primary channel may be 1 MHz wide for STAs (e.g., MTC type devices)that support (e.g., only support) a 1 MHz mode, even if the AP, and other STAs in the BSS support 2 MHz, 4 MHz, 8 MHz, 16 MHz, and / or other channel bandwidth operating modes. Carrier sensing and / or Network Allocation Vector (NAV) settings may depend on the status of the primary channel. If the primary channel is busy, for example, due to a STA (which supports only a 1 MHz operating mode), transmitting to the AP, the entire available frequency bands may be considered busy even though a majority of the frequency bands remains idle and may be available.

[0081] In the United States, the available frequency bands, which may be used by 802.11 ah, are from 902 MHz to 928 MHz. In Korea, the available frequency bands are from 917.5 MHz to 923.5 MHz. In Japan, the available frequency bands are from 916.5 MHz to 927.5 MHz. The total bandwidth available for 802.11 ah is 6 MHz to 26 MHz depending on the country code.

[0082] FIG. 1 D is a system diagram illustrating the RAN 113 and the CN 115 according to an embodiment. As noted above, the RAN 113 may employ an NR radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 113 may also be in communication with the CN 115.

[0083] The RAN 1 13 may include gNBs 180a, 180b, 180c, though it will be appreciated that the RAN 113 may include any number of gNBs while remaining consistent with an embodiment. The gNBs 180a, 180b, 180c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the gNBs 180a, 180b, 180c may implement MIMO technology. For example, gNBs 180a, 108b may utilize beamforming to transmit signals to and / or receive signals from the gNBs 180a, 180b, 180c. Thus, the gNB 180a, for example, may use multiple antennas to transmit wireless signals to, and / or receive wireless signals from, the WTRU 102a. In an embodiment, the gNBs 180a, 180b, 180c may implement carrier aggregation technology. For example, the gNB 180a may transmit multiple component carriers to the WTRU 102a (not shown). A subset of these component carriers may be on unlicensed spectrum while the remaining component carriers may be on licensed spectrum. In an embodiment, the gNBs 180a, 180b, 180c may implement Coordinated Multi-Point (CoMP) technology. For example, WTRU 102a may receive coordinated transmissions from gNB 180a and gNB 180b (and / or gNB 180c).

[0084] The WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c using transmissions associated with a scalable numerology. For example, the OFDM symbol spacing and / or OFDM subcarrier spacing may vary for different transmissions, different cells, and / or different portions of the wireless transmission spectrum. The WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c using subframe or transmission time intervals (TTIs) of various or scalable lengths (e.g., containing varying number of OFDM symbols and / or lasting varying lengths of absolute time).

[0085] The gNBs 180a, 180b, 180c may be configured to communicate with the WTRUs 102a, 102b, 102c in a standalone configuration and / or a non-standalone configuration. In the standalone configuration, WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c without also accessing other RANs (e.g., such as eNode-Bs 160a, 160b, 160c). In the standalone configuration, WTRUs 102a, 102b, 102c may utilize one or more of gNBs 180a, 180b, 180c as a mobility anchor point. In the standalone configuration, WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c using signals in an unlicensed band. In a non-standalone configuration WTRUs 102a, 102b, 102c may communicate with / connect to gNBs 180a, 180b, 180c while also communicating with / connecting to another RAN such as eNode-Bs 160a, 160b, 160c. For example, WTRUs 102a, 102b, 102c may implement DC principles to communicate with one or more gNBs 180a, 180b, 180c and one or more eNode-Bs 160a, 160b, 160c substantially simultaneously. In the non-standalone configuration, eNode-Bs 160a, 160b, 160c may serve as a mobility anchor for WTRUs 102a, 102b, 102c and gNBs 180a, 180b, 180c may provide additional coverage and / or throughput for servicing WTRUs 102a, 102b, 102c.

[0086] Each of the gNBs 180a, 180b, 180c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and / or DL, support of network slicing, dual connectivity, interworking between NR and E- UTRA, routing of user plane data towards User Plane Function (UPF) 184a, 184b, routing of control plane information towards Access and Mobility Management Function (AMF) 182a, 182b and the like. As shown in FIG. 1 D, the gNBs 180a, 180b, 180c may communicate with one another over an Xn interface.

[0087] The CN 115 shown in FIG. 1 D may include at least one AMF 182a, 182b, at least one UPF 184a, 184b, at least one Session Management Function (SMF) 183a, 183b, and possibly a Data Network (DN) 185a, 185b. While each of the foregoing elements are depicted as part of the CN 115, it will be appreciated that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0088] The AMF 182a, 182b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via an N2 interface and may serve as a control node. For example, the AMF 182a, 182b may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, support for network slicing (e.g., handling of different PDU sessions with different requirements), selecting a particular SMF 183a, 183b, management of the registration area, termination of NAS signaling, mobility management, and the like. Network slicing may be used by the AMF 182a, 182b in order to customize CN support for WTRUs 102a, 102b, 102c based on the types of services being utilized WTRUs 102a, 102b, 102c. For example, different network slices may be established for different use cases such as services relying on ultra-reliable low latency (URLLC) access, services relying on enhanced massive mobile broadband (eMBB) access,services for machine type communication (MTC) access, and / or the like. The AMF 162 may provide a control plane function for switching between the RAN 113 and other RANs (not shown) that employ other radio technologies, such as LTE, LTE-A, LTE-A Pro, and / or non-3GPP access technologies such as WiFi.

[0089] The SMF 183a, 183b may be connected to an AMF 182a, 182b in the CN 115 via an N11 interface. The SMF 183a, 183b may also be connected to a UPF 184a, 184b in the CN 115 via an N4 interface. The SMF 183a, 183b may select and control the UPF 184a, 184b and configure the routing of traffic through the UPF 184a, 184b. The SMF 183a, 183b may perform other functions, such as managing and allocating WTRU IP address, managing PDU sessions, controlling policy enforcement and QoS, providing downlink data notifications, and the like. A PDU session type may be IP-based, non-IP based, Ethernet-based, and the like.

[0090] The UPF 184a, 184b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via an N3 interface, which may provide the WTRUs 102a, 102b, 102c with access to packet- switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices. The UPF 184, 184b may perform other functions, such as routing and forwarding packets, enforcing user plane policies, supporting multi-homed PDU sessions, handling user plane QoS, buffering downlink packets, providing mobility anchoring, and the like.

[0091] The CN 115 may facilitate communications with other networks. For example, the CN 115 may include, or may communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that serves as an interface between the CN 115 and the PSTN 108. In addition, the CN 115 may provide the WTRUs 102a, 102b, 102c with access to the other networks 112, which may include other wired and / or wireless networks that are owned and / or operated by other service providers. In one embodiment, the WTRUs 102a, 102b, 102c may be connected to a local Data Network (DN) 185a, 185b through the UPF 184a, 184b via the N3 interface to the UPF 184a, 184b, and an N6 interface between the UPF 184a, 184b and the DN 185a, 185b.

[0092] In view of Figures 1 A-1 D, and the corresponding description of Figures 1 A-1 D, one or more, or all, of the functions described herein with regard to one or more of: WTRU 102a-d, Base Station 114a-b, eNode-B 160a-c, MME 162, SGW 164, PGW 166, gNB 180a-c, AMF 182a-b, UPF 184a-b, SMF 183a-b, DN 185a-b, and / or any other device(s) described herein, may be performed by one or more emulation devices (not shown). The emulation devices may be one or more devices configured to emulate one or more, or all, of the functions described herein. For example, the emulation devices may be used to test other devices and / or to simulate network and / or WTRU functions.

[0093] The emulation devices may be designed to implement one or more tests of other devices in a lab environment and / or in an operator network environment. For example, the one or more emulation devicesmay perform the one or more, or all, functions while being fully or partially implemented and / or deployed as part of a wired and / or wireless communication network in order to test other devices within the communication network. The one or more emulation devices may perform the one or more, or all, functions while being temporarily implemented / deployed as part of a wired and / or wireless communication network. The emulation device may be directly coupled to another device for purposes of testing and / or may perform testing using over-the-air wireless communications.

[0094] The one or more emulation devices may perform the one or more, including all, functions while not being implemented / deployed as part of a wired and / or wireless communication network. For example, the emulation devices may be utilized in a testing scenario in a testing laboratory and / or a non-deployed (e.g., testing) wired and / or wireless communication network in order to implement testing of one or more components. The one or more emulation devices may be test equipment. Direct RF coupling and / or wireless communications via RF circuitry (e.g., which may include one or more antennas) may be used by the emulation devices to transmit and / or receive data.

[0095] This application describes a variety of aspects, including tools, features, examples or embodiments, models, approaches, etc. Many of these aspects are described with specificity and, at least to show the individual characteristics, are often described in a manner that may sound limiting. However, this is for purposes of clarity in description, and does not limit the application or scope of those aspects. Indeed, all of the different aspects may be combined and interchanged to provide further aspects. Moreover, the aspects may be combined and interchanged with aspects described in earlier filings as well.

[0096] The aspects described and contemplated in this application may be implemented in many different forms. FIGS. 5-23 described herein may provide some embodiments, but other embodiments are contemplated. The discussion of FIGS. 5-23 does not limit the breadth of the implementations. At least one of the aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a bitstream generated or encoded. These and other aspects may be implemented as a method, an apparatus, a computer readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the methods described, and / or a computer readable storage medium having stored thereon a bitstream generated according to any of the methods described.

[0097] In the present application, the terms “reconstructed” and “decoded” may be used interchangeably, the terms “pixel” and “sample” may be used interchangeably, the terms “image,” “picture” and “frame” may be used interchangeably.

[0098] Various methods are described herein, and each of the methods comprises one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required forproper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. Additionally, terms such as “first”, “second”, etc. may be used in various embodiments to modify an element, component, step, operation, etc., such as, for example, a “first decoding” and a “second decoding”. Use of such terms does not imply an ordering to the modified operations unless specifically required. So, in this example, the first decoding need not be performed before the second decoding, and may occur, for example, before, during, or in an overlapping time period with the second decoding.

[0099] Various methods and other aspects described in this application may (for example, be used to) modify modules, for example, pre-encoding processing 201 , intra prediction 260, entropy coding 245 and / or entropy decoding modules 330, intra prediction 360, post-decoding processing 385 of a video encoder 200 and a video decoder 300 as shown in FIG. 2 and FIG. 3 respectively. Moreover, the subject matter disclosed herein presents aspects that are not limited to VVC or HEVC, and may be applied, for example, to any type, format or version of video coding, whether described in a standard or a recommendation, whether pre-existing or future-developed, and extensions of any such standards and recommendations (e.g., including VVC and HEVC). Unless indicated otherwise, or technically precluded, the aspects described in this application may be used individually or in combination.

[0100] Various numeric values are used in examples described the present application, such as minimum and maximum value ranges (for example, 0 to 1 , 0 to N or 0 to 255), bit values for indications or determinations, default values, ID numbers (for example, for adaptation IDs), etc. These and other specific values are for purposes of describing examples and the aspects described are not limited to these specific values.

[0101] FIG. 2 is a diagram showing an example video encoder. Variations of example encoder 200 are contemplated, but the encoder 200 is described below for purposes of clarity without describing all expected variations.

[0102] Before being encoded, the video sequence may go through pre-encoding processing (201), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to get a signal distribution more resilient to compression (for instance using a histogram equalization of one of the color components). Metadata may be associated with the pre-processing, and attached to the bitstream.

[0103] In the encoder 200, a picture is encoded by the encoder elements as described below. The picture to be encoded is partitioned (202) and processed in units of, for example, coding units (CUs). Each unit is encoded using, for example, either an intra or inter mode. When a unit is encoded in an intra mode, it performs intra prediction (260). In an inter mode, motion estimation (275) and compensation (270) are performed. The encoder decides (205) which one of the intra mode or inter mode to use for encoding theunit, and indicates the intra / inter decision by, for example, a prediction mode flag. Prediction residuals are calculated, for example, by subtracting (210) the predicted block from the original image block.

[0104] The prediction residuals are then transformed (225) and quantized (230). The quantized transform coefficients, as well as motion vectors and other syntax elements, are entropy coded (245) to output a bitstream. The encoder can skip the transform and apply quantization directly to the nontransformed residual signal. The encoder can bypass both transform and quantization, i.e., the residual is coded directly without the application of the transform or quantization processes.

[0105] The encoder decodes an encoded block to provide a reference for further predictions. The quantized transform coefficients are de-quantized (240) and inverse transformed (250) to decode prediction residuals. Combining (255) the decoded prediction residuals and the predicted block, an image block is reconstructed. In-loop filters (265) are applied to the reconstructed picture to perform, for example, deblocking / SAO (Sample Adaptive Offset) filtering to reduce encoding artifacts. The filtered image is stored at a reference picture buffer (280).

[0106] FIG. 3 is a diagram showing an example of a video decoder. In example decoder 300, a bitstream is decoded by the decoder elements as described below. Video decoder 300 generally performs a decoding pass reciprocal to the encoding pass as described in FIG. 2. The encoder 200 may also generally perform video decoding as part of encoding video data. For example, the encoder 200 may perform one or more of the video decoding steps presented herein. The encoder reconstructs the decoded images, for example, to maintain synchronization with the decoder with respect to one or more of the following: reference pictures, entropy coding contexts, and other decoder-relevant state variables.

[0107] In particular, the input of the decoder includes a video bitstream, which may be generated by video encoder 200. The bitstream is first entropy decoded (330) to obtain transform coefficients, motion vectors, and other coded information. The picture partition information indicates how the picture is partitioned. The decoder may therefore divide (335) the picture according to the decoded picture partitioning information. The transform coefficients are de-quantized (340) and inverse transformed (350) to decode the prediction residuals. Combining (355) the decoded prediction residuals and the predicted block, an image block is reconstructed. The predicted block may be obtained (370) from intra prediction (360) or motion-compensated prediction (i.e., inter prediction) (375). In-loop filters (365) are applied to the reconstructed image. The filtered image is stored at a reference picture buffer (380).

[0108] The decoded picture can further go through post-decoding processing (385), for example, an inverse color transform (e.g. conversion from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping performing the inverse of the remapping process performed in the pre-encoding processing (201). Thepost-decoding processing can use metadata derived in the pre-encoding processing and signaled in the bitstream.

[0109] FIG. 4 is a diagram showing an example of a system in which various aspects and embodiments described herein may be implemented. System 400 may be embodied as a device including the various components described below and is configured to perform one or more of the aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 400, singly or in combination, may be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one example, the processing and encoder / decoder elements of system 400 are distributed across multiple ICs and / or discrete components. In various embodiments, the system 400 is communicatively coupled to one or more other systems, or other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various embodiments, the system 400 is configured to implement one or more of the aspects described in this document.

[0110] The system 400 includes at least one processor 410 configured to execute instructions loaded therein for implementing, for example, the various aspects described in this document. Processor 410 can include embedded memory, input output interface, and various other circuitries as known in the art. The system 400 includes at least one memory 420 (e.g., a volatile memory device, and / or a non-volatile memory device). System 400 includes a storage device 440, which can include non-volatile memory and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drive, and / or optical disk drive. The storage device 440 can include an internal storage device, an attached storage device (including detachable and non-detachable storage devices), and / or a network accessible storage device, as non-limiting examples.

[0111] System 400 includes an encoder / decoder module 430 configured, for example, to process data to provide an encoded video or decoded video, and the encoder / decoder module 430 can include its own processor and memory. The encoder / decoder module 430 represents module(s) that may be included in a device to perform the encoding and / or decoding functions. As is known, a device can include one or both of the encoding and decoding modules. Additionally, encoder / decoder module 430 may be implemented as a separate element of system 400 or may be incorporated within processor 410 as a combination of hardware and software as known to those skilled in the art.

[0112] Program code to be loaded onto processor 410 or encoder / decoder 430 to perform the various aspects described in this document may be stored in storage device 440 and subsequently loaded onto memory 420 for execution by processor 410. In accordance with various embodiments, one or more of processor 410, memory 420, storage device 440, and encoder / decoder module 430 can store one or more of various items during the performance of the processes described in this document. Such stored items can include, but are not limited to, the input video, the decoded video or portions of the decoded video, the bitstream, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0113] In some embodiments, memory inside of the processor 410 and / or the encoder / decoder module 430 is used to store instructions and to provide working memory for processing that is needed during encoding or decoding. In other embodiments, however, a memory external to the processing device (for example, the processing device may be either the processor 410 or the encoder / decoder module 430) is used for one or more of these functions. The external memory may be the memory 420 and / or the storage device 440, for example, a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, an external non-volatile flash memory is used to store the operating system of, for example, a television. In at least one embodiment, a fast external dynamic volatile memory such as a RAM is used as working memory for video coding and decoding operations, such as, for example, MPEG-2 (MPEG refers to the Moving Picture Experts Group, MPEG-2 is also referred to as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or WC (Versatile Video Coding, a new standard being developed by JVET, the Joint Video Experts Team).

[0114] The input to the elements of system 400 may be provided through various input devices as indicated in block 445. Such input devices include, but are not limited to, (i) a radio frequency (RF) portion that receives an RF signal transmitted, for example, over the air by a broadcaster, (ii) a Component (COMP) input terminal (or a set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Other examples, not shown in FIG. 4, include composite video.

[0115] In various embodiments, the input devices of block 445 have associated respective input processing elements as known in the art. For example, the RF portion may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a band of frequencies), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select (for example) a signal frequency band which may be referred to as a channel in certain embodiments, (iv) demodulating the downconverted and band-limited signal, (v) performing errorcorrection, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner that performs various of these functions, including, for example, downconverting the received signal to a lower frequency (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing element receives an RF signal transmitted over a wired (for example, cable) medium, and performs frequency selection by filtering, downconverting, and filtering again to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements performing similar or different functions. Adding elements can include inserting elements in between existing elements, such as, for example, inserting amplifiers and an analog-to-digital converter. In various embodiments, the RF portion includes an antenna.

[0116] Additionally, the USB and / or HDMI terminals can include respective interface processors for connecting system 400 to other electronic devices across USB and / or HDMI connections. It is to be understood that various aspects of input processing, for example, Reed-Solomon error correction, may be implemented, for example, within a separate input processing IC or within processor 410 as necessary. Similarly, aspects of USB or HDMI interface processing may be implemented within separate interface ICs or within processor 410 as necessary. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, processor 410, and encoder / decoder 430 operating in combination with the memory and storage elements to process the data stream as necessary for presentation on an output device.

[0117] Various elements of system 400 may be provided within an integrated housing. Within the integrated housing, the various elements may be interconnected and transmit data therebetween using suitable connection arrangement 425, for example, an internal bus as known in the art, including the Inter- IC (I2C) bus, wiring, and printed circuit boards.

[0118] The system 400 includes communication interface 450 that enables communication with other devices via communication channel 460. The communication interface 450 can include, but is not limited to, a transceiver configured to transmit and to receive data over communication channel 460. The communication interface 450 can include, but is not limited to, a modem or network card and the communication channel 460 may be implemented, for example, within a wired and / or a wireless medium.

[0119] Data is streamed, or otherwise provided, to the system 400, in various embodiments, using a wireless network such as a Wi-Fi network, for example IEEE 802.11 (IEEE refers to the Institute ofElectrical and Electronics Engineers). The Wi-Fi signal of these examples is received over the communications channel 460 and the communications interface 450 which are adapted for Wi-Fi communications. The communications channel 460 of these embodiments is typically connected to an access point or router that provides access to external networks including the Internet for allowing streaming applications and other over-the-top communications. Other embodiments provide streamed data to the system 400 using a set-top box that delivers the data over the HDMI connection of the input block 445. Other embodiments provide streamed data to the system 400 using the RF connection of the input block 445. As indicated above, various embodiments provide data in a non-streaming manner.Additionally, various embodiments use wireless networks other than Wi-Fi, for example a cellular network or a Bluetooth network.

[0120] The system 400 can provide an output signal to various output devices, including a display 475, speakers 485, and other peripheral devices 495. The display 475 of various embodiments includes one or more of, for example, a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 475 may be for a television, a tablet, a laptop, a cell phone (mobile phone), or other device. The display 475 can also be integrated with other components (for example, as in a smart phone), or separate (for example, an external monitor for a laptop). The other peripheral devices 495 include, in various examples of embodiments, one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 495 that provide a function based on the output of the system 400. For example, a disk player performs the function of playing the output of the system 400.

[0121] In various embodiments, control signals are communicated between the system 400 and the display 475, speakers 485, or other peripheral devices 495 using signaling such as AV. Link, Consumer Electronics Control (CEC), or other communications protocols that enable device-to-device control with or without user intervention. The output devices may be communicatively coupled to system 400 via dedicated connections through respective interfaces 470, 480, and 490. Alternatively, the output devices may be connected to system 400 using the communications channel 460 via the communications interface 450. The display 475 and speakers 485 may be integrated in a single unit with the other components of system 400 in an electronic device such as, for example, a television. In various embodiments, the display interface 470 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0122] The display 475 and speakers 485 can alternatively be separate from one or more of the other components, for example, if the RF portion of input 445 is part of a separate set-top box. In various embodiments in which the display 475 and speakers 485 are external components, the output signal maybe provided via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0123] The embodiments may be carried out by computer software implemented by the processor 410 or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 420 may be of any type appropriate to the technical environment and may be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 410 may be of any type appropriate to the technical environment, and can encompass one or more of microprocessors, general purpose computers, special purpose computers, and processors based on a multi-core architecture, as non-limiting examples.

[0124] Various implementations involve decoding. “Decoding”, as used in this application, can encompass all or part of the processes performed, for example, on a received encoded sequence in order to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, for example, entropy decoding, inverse quantization, inverse transformation, and differential decoding. In various embodiments, such processes also, or alternatively, include processes performed by a decoder of various implementations described in this application, for example, receiving a real-time control protocol (RTCP) extended report (XR) packet with an extended report block, the extended report block comprising a field representing timing information, a block length indicator, an identifier for a source of real-time protocol (RTP) data packets, a timestamp for synchronizing media data and QoE timing information recorded at a server, extracting the timing information from the extended report block, and calculating quality of experience (QoE) metrics based on the extracted timing information.

[0125] As further embodiments, in one example “decoding” refers only to entropy decoding, in another embodiment “decoding” refers only to differential decoding, and in another embodiment “decoding” refers to a combination of entropy decoding and differential decoding. Whether the phrase “decoding process” is intended to refer specifically to a subset of operations or generally to the broader decoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.

[0126] Various implementations involve encoding. In an analogous way to the above discussion about “decoding”, “encoding” as used in this application can encompass all or part of the processes performed, for example, on an input video sequence in order to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, forexample, partitioning, differential encoding, transformation, quantization, and entropy encoding. In various embodiments, such processes also, or alternatively, include processes performed by an encoder of various implementations described in this application, for example, determining a real-time control protocol (RTCP) extended report (XR) packet with an extended report block, the extended report block comprising a field representing timing information, a block length indicator, an identifier for a destination of real-time protocol (RTP) data packets, a timestamp for synchronizing media data, and QoE timing information to a device; embedding the timing information into the extended report block; and transmitting quality of experience (QoE) metrics based on the embedded timing information.

[0127] As further examples, in one embodiment “encoding” refers only to entropy encoding, in another embodiment “encoding” refers only to differential encoding, and in another embodiment “encoding” refers to a combination of differential encoding and entropy encoding. Whether the phrase “encoding process” is intended to refer specifically to a subset of operations or generally to the broader encoding process will be clear based on the context of the specific descriptions and is believed to be well understood by those skilled in the art.

[0128] Note that syntax elements as used herein, such as syntax elements that may be indicated in discussion or figures presented herein, are descriptive terms. As such, they do not preclude the use of other syntax element names.

[0129] When a figure is presented as a flow diagram, it should be understood that it also provides a block diagram of a corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow diagram of a corresponding method / process.

[0130] During the encoding process, the balance or trade-off between the rate and distortion is usually considered, often given the constraints of computational complexity. The rate distortion optimization is usually formulated as minimizing a rate distortion function, which is a weighted sum of the rate and of the distortion. There are different approaches to solve the rate distortion optimization problem. For example, the approaches may be based on an extensive testing of all encoding options, including all considered modes or coding parameters values, with a complete evaluation of their coding cost and related distortion of the reconstructed signal after coding and decoding. Faster approaches may also be used, to save encoding complexity, in particular with computation of an approximated distortion based on the prediction or the prediction residual signal, not the reconstructed one. Mix of these two approaches can also be used, such as by using an approximated distortion for only some of the possible encoding options, and a complete distortion for other encoding options. Other approaches only evaluate a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform theoptimization, but the optimization is not necessarily a complete evaluation of both the coding cost and related distortion.

[0131] The implementations and aspects described herein may be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation of features discussed can also be implemented in other forms (for example, an apparatus or program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The methods may be implemented in, for example, a processor, which refers to processing devices in general, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cell phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end-users.

[0132] Reference to “one embodiment,” “an embodiment,” “an example,” “one implementation” or “an implementation,” as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrase “in one embodiment,” “in an embodiment,” “in an example,” “in one implementation,” or “in an implementation”, as well any other variations, appearing in various places throughout this application are not necessarily all referring to the same embodiment or example.

[0133] Additionally, this application may refer to “determining” various pieces of information.Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory. Obtaining may include receiving, retrieving, constructing, generating, and / or determining.

[0134] Further, this application may refer to “accessing” various pieces of information. Accessing the information can include one or more of, for example, receiving the information, retrieving the information (for example, from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0135] Additionally, this application may refer to “receiving” various pieces of information. Receiving is, as with “accessing”, intended to be a broad term. Receiving the information can include one or more of, for example, accessing the information, or retrieving the information (for example, from memory). Further, “receiving” is typically involved, in one way or another, during operations such as, for example, storing the information, processing the information, transmitting the information, moving the information, copying theinformation, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0136] It is to be appreciated that the use of any of the following ”, “and / or”, and “at least one of”, for example, in the cases of “A / B”, “A and / or B” and “at least one of A and B”, is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C”, such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as is clear to one of ordinary skill in this and related arts, for as many items as are listed.

[0137] Also, as used herein, the word “signal” refers to, among other things, indicating something to a corresponding decoder. For example, in some embodiments the encoder signals (e.g., to a decoder) an MPD, adaptation set, a representation, a preselection, G-PCC components, a G-PCCComponent descriptor, a G-PCC descriptor or an essential property descriptor, a supplemental property descriptor, a G-PCC tile inventory descriptor, G-PCC static spatial regions descriptor, GPCCTileld descriptor GPCC3DRegionlD descriptor, among other descriptors, elements and attributes, metadata, schemas, etc. (for example, as disclosed herein), etc. In this way, in an embodiment the same parameter is used at both the encoder side and the decoder side. Thus, for example, an encoder can transmit (explicit signaling) a particular parameter to the decoder so that the decoder can use the same particular parameter.Conversely, if the decoder already has the particular parameter as well as others, signaling may be used without transmitting (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding transmission of any actual functions, a bit savings is realized in various embodiments. It is to be appreciated that signaling may be accomplished in a variety of ways. For example, one or more syntax elements, flags, and so forth are used to signal information to a corresponding decoder in various embodiments. While the preceding relates to the verb form of the word “signal”, the word “signal” can also be used herein as a noun.

[0138] As will be evident to one of ordinary skill in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal may be formatted to carry the bitstream of a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (for example,using a radio frequency portion of spectrum) or as a baseband signal. The formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor- readable medium.

[0139] The following acronyms and abbreviations may be used herein.AVP Audio-Visual ProfileAVPF Audio-Visual Profile with FeedbackBT Block TypeFB FeedbackFCI Feedback Control InformationFMT Format TypeHE Header ExtensionLSW Least Significant WordMAF Media Access FunctionMeCAR Media Capabilities for ARMSW Most Significant WordNTP Network Time ProtocolPDU Packet Data UnitPSFB Payload Specific FeedbackPD Permanent DocumentPT Payload TypeQoE Quality of ExperienceRTP Real-Time ProtocolRTCP Real-Time Control ProtocolRTSP Real Time Streaming ProtocolSAP Session Announcement ProtocolSDP Session Description ProtocolSRS Split Rendering ServerSRTP Secure Real-Time ProtocolSSRC Synchronization SourceTR Technical ReportTS Technical SpecificationUE User EquipmentURN Uniform Resource NameVoIP Voice over IPXR Extended Report

[0140] Timestamps capturing events may occur in WTRU or devices, and split rendering function may be provided, as described herein. Time information recorded at the output of a split rendering server may be used, for example to measure the server processing delay and the application delay (e.g., overall application delay), excluding the server processing delay.

[0141] QoE timing information may be provided, as described herein. A MeCAR device may send a group of pose information to a split render function (e.g., a split render function on a server) to generate rendered media frames based on the transmitted pose. A pose may be associated with time metadata, such as the time when the pose estimation was made (T1), the estimated target display time of the content (T2.estimated), and the time the group of poses was sent (TT).

[0142] The gap between the actual-target-display-time (T2.actual) and the pose estimate time (T1) may be the pose-to-render-to-photon delay, which may allow the MeCAR device to know the amount of processing time and the connection delay for a loop of split rendering. The next round of pose estimation may refer to the pose-to-render-to-photon delay for the estimation of a new T2.estimated.

[0143] The split render function in the server may refer to TT. TT may be the time when the group of poses is sent from device if multiple pairs of pose and metadata for the same target display time are received from the device. The TT information may be used to manage poses by the server, such as allowing the MeCAR device to update former estimations by resubmitting a new pose with the same estimated-target-display-time.

[0144] The split render function in the server may send rendered media frames and associated metadata. The metadata may include the pose used for the rendered frame, as well as corresponding time information, such as T1, T2.actual, and may include the time when the rendering started (T3), and may be (e.g., all) sent to the MeCAR device in order to measure the render-to-photon delay QoE metric.

[0145] The (T5) may be the time information at the output of the Split Rendering Server (SRS). The (T5) timestamp may be used to: measure the server processing delay, (T3 - T5); and / or measure the overall application delay excluding the server processing delay.

[0146] The downlink delay may be measured with T5 and the time at which the data was received by the Media Access Function (MAF) in the WTRU, and the uplink delay may be measured as (TT - T3).

[0147] If poses are stacked in the server’s pose buffer, for example, with a granularity finer than the device's supported frame rate, the split render function may select the pose closest to the display timeaccording to the previous-render-to-photon delay. The previous-render-to-photon delay from the most recent frame information may help the server to make this selection.

[0148] FIG. 5 illustrates an example pose information call flow.

[0149] Features described herein may be associated with pose information delays and QoE. The timing metadata may be used to measure the following delays: pose-to-render-to-photon delay = T2.actual - T1 ; render-to-photon delay = T2.actual - T3; and pose-to-render delay = T3 - T1

[0150] By using (e.g., all) the history of delay measurements, the application may estimate the delays for the next poses and rendered frames.

[0151] Pose and timestamp information from the device may include the following: predicted pose, which may include location and direction information; (T1): the time when the pose estimation was made; (T2.estimated): the estimated target display time for the media frame which may be rendered, using to this pose; (T1 ’): the actual time when a pose or a group of poses is sent from the device to the Split Rendering Server; and previous-render-to-photon: the render-to-photon delay for the most recent frame (T2.actual - T3).

[0152] Pose and timestamp information associated with rendered media frame from the Split Rendering Server may include the following: pose used for rendering; (T1): the time when the pose estimation was made; (T2.estimated): the estimated target display time for the media frame which is rendered; (T3): the actual time when the renderer in the Split Rendering Server starts to render the associated media frame; T5 : the time when the rendered media frame is output from the Split Rendering Server.

[0153] User interaction and QoE timing information may be used herein. In shared interactive immersive services use cases, the user interaction may be sent from a WTRU to a server. The server may handle the user’s request to the immersive media scene (e.g., changing the context such as translation, rotation, scaling, or adding a new object in the scene). With the edge assisted WTRU type, the WTRU may offload the scene rendering to the Split Rendering Server, the server may rasterize the extended reality viewport and pre-render to generate an extended reality media which may be encoded and delivered to the WTRU.

[0154] In interactive immersive services, a parameter to estimate the user quality of experience may be the roundtrip interaction delay. The roundtrip interaction delay ma be the sum of the Age of Content and the User Interaction Delay.

[0155] The user interaction QoE metrics may be described herein. The User interaction delay may be the time duration between the moment at which a user action is initiated and the time such an action is taken into account by the content creation engine. The User interaction delay may be impacted by the uplink latency of the wireless network.

[0156] The Age of content may be the time duration between the moment the content is created and the time the moment is presented to the user. The Age of content may be impacted by the downlink latency of the wireless network.

[0157] Action and timestamp information from the device may include the following: Action information: the user action information which are grouped into action sets. An action may have a unique identifier of the action; lastChangeTime the time when the user action is made. It may correspond to the lastChangeTime field in the action information defined as the timestamp of the last change to the state of the action.

[0158] Action and timestamp information associated with rendered media frame from the Split Rendering Server may include the following: Action identifiers: The identifiers of the actions which are handled by the scene manager and rendered in the associated media frame; sceneUpdateTime (T6): the time when the Scene manager processes the interaction task according to the actions in the action message from the WTRU and updates the scene. The procedures of interactivity pipeline are shown in FIG. 5.

[0159] FIG. 6 illustrates an example user action call flow. Using (e.g., all) the timestamps from the SRS and the WTRU, the application may calculate the interaction delays as below: User-interaction-delay = sceneUpdateTime (T6) - lastChangeTime; Age-of-content = T2.actual - sceneUpdateTime (T6); and / or Roundtrip-interaction-delay = T2.actual - lastChangeTime.

[0160] Delivery reports may be transmitted to a WTRU using RTCP messages. There may be a number ways to carry a variety of control information using RTCP packets. This may include: profile-specific extensions to the sender (PT=200) and receiver report (PT=201 ); application-defined RTCP packet with payload type equal to 204 (PT=204); extended reports (XR) with payload type equal to 207 (PT=207); generic RTP Feedback reports with payload type equal to 205 (RTPFB; PT=205); and / or payload-specific RTCP feedback messages with payload type equal to 206 (PSFB; PT= 206).

[0161] FIG. 7 illustrates an example real-time control protocol (RTCP) extended report (XR) packet format. An XR packet may include a header of two 32-bit words, followed by a number (e.g., zero) of extended report blocks. The type of packet may lay out in a manner consistent with RTCP packets, as concerns the version, packet type, and length information. The following parameters may pertain to the type of packet: version (V) [2 bits]: Identifies the version of RTP. This specification applies to RTP version two; padding (P) [1 bit]: If the padding bit is set, the XR packet may include (e.g., additional) padding octets at the end. The semantics of the field may be identical to the semantics of the padding field in the SR packet, as described by the RTP specification RFC 3550; reserved: 5 bits. The field may be reserved for future definition. In the absence of such definition, the bits in this field may (e.g., must) be set to zero and may (e.g., must) be ignored by the receiver; packet type (PT) [8 bits]: Contains the constant 207 to identifythis as an RTCP XR packet. The value may be registered with the Internet Assigned Numbers Authority (IANA); length: 16 bits. The length of this XR packet may be in 32-bit words minus one, including the header and any padding; SSRC [32 bits]: The synchronization source identifier for the originator of this XR packet; report blocks [variable length]: Zero or more extended report blocks. In keeping with the extended report block framework defined below, a block may (e.g., must) consist of one or more 32-bit words.

[0162] Features described herein may be associated with an extended report block framework. Extended report blocks may be stacked, one after the other, at the end of an XR packet. The value of block type (e.g., BT) field may identify the block format, and its name space may be managed by IANA.

[0163] An individual block's length may be a multiple of 4 octets. The XR header's length field may describe the total length of the packet, including the extended report blocks. A block may have block type and length fields that facilitate parsing. A receiving application may demultiplex the blocks based upon their type and may use the length information to locate each successive block, for example, in the presence of block types it does not recognize.

[0164] Seven extended report blocks may be described: block types for reporting upon received packet losses and duplicates, packet reception times, receiver reference time information, receiver inter-report delays, detailed reception statistics, and voice over IP (VoIP) metrics.

[0165] An extended report block may have a format as depicted in FIG. 8. FIG. 8 illustrates an example RTCP XR block format. The following parameters may include: A block type (BT) [8 bits]: may Identify the block format. Seven block types may be described. The field's name space may be managed by the Internet Assigned Numbers Authority (IANA); type-specific [8 bits]: The use of these may be determined by the block type definition; Block length [16 bits]: The length of this report block, including the header, may be 32-bit words minus one. If the block type definition permits, zero may be an acceptable value, signifying a block that consists of the BT, type-specific, and block length fields, with a null type-specific block contents field; and / or type-specific block contents [variable length]: The use of this field may be defined by the particular block type, subject that it may (e.g., must) be a multiple of 32 bits long. If the block type definition permits, it may be zero bits long

[0166] Features described herein may be associated with NTP time stamps. Wallclock time (e.g., absolute date and time) may be represented using the timestamp format of the Network Time Protocol (NTP), which may be in seconds relative to Oh UTC on 1 January 1900. The full resolution NTP timestamp may be a 64-bit unsigned fixed-point number with the integer part in the first 32 bits referred as most significant word (MSW) and the fractional part in the last 32 bits referred as least significant word (LSW). In fields where a compact representation is appropriate, the middle 32 bits may be used. The low 16 bits ofthe integer part and the high 16 bits of the fractional part may be used. The high 16 bits of the integer part may be determined independently.

[0167] An implementation may not be required to run the Network Time Protocol in order to use RTP. Other time sources, or none at all, may be used. Running NTP may be useful for synchronizing streams transmitted from separate hosts.

[0168] Features described herein may be associated with SDP Signaling. When initiating multimedia teleconferences, voice-over-IP calls, streaming video, or other sessions, there may be a parameter to convey media details, transport addresses, and (e.g., other) session description metadata to the participants. SDP may provide a standard representation for such information, irrespective of how that information may be transported. SDP may be a format for session description - it may not incorporate a transport protocol, and it may be intended to use different transport protocols as appropriate, including the Session Announcement Protocol, Session Initiation Protocol, Real Time Streaming Protocol, and the Hypertext Transport Protocol.

[0169] An SDP session description may include the following: session name and purpose; time(s) the session is active; the media comprising the session; and / or information needed to receive those media (addresses, ports, formats, etc.)

[0170] As resources to participate in a session may be limited, information may be desirable: information about the bandwidth to be used by the session; and / or contact information for the person responsible for the session.

[0171] SDP may convey sufficient information to enable applications to join a session (e.g., with the possible exception of encryption keys) and to announce the resources to be used to non-participants (e.g., that may need to know).

[0172] Session Description Protocol (SDP) signaling for XR blocks may be employed by applications that utilize SDP. The signaling may be defined to be used by applications that implement the SDP Offer / Answer model or by applications that use SDP to describe media and transport configurations in connection with the protocols as the Session Announcement Protocol (SAP) or the Real Time Streaming Protocol (RTSP). There may exist potential signaling methods that may not be described herein.

[0173] The XR blocks may be used without prior signaling. This may be consistent with rules governing RTCP packet types. An example in which signaling may not be used may be an application that uses (e.g., requires) one or more XR blocks. For applications that are configured at session initiation, the use of some type of signaling may be recommended. Although the use of SDP signaling for XR blocks may be optional, if used, it may (e.g., must) be used as described herein. If SDP signaling is used in an environment whereXR blocks are implemented by a fraction of the participants, the ones not implementing the XR blocks may ignore the SDP attribute.

[0174] Features described herein may be associated with RTCP feedback messages. Real-time media streams that use RTP may be resilient against packet losses. Receivers may use the base mechanisms of the Real-time Transport Control Protocol (RTCP) to report packet reception statistics and allow a sender to adapt its transmission behavior in the mid-term. This may be the means for feedback and feedback-based error repair (e.g., besides codec-specific mechanisms). An extension to the Audio-visual Profile (AVP) may enable receivers to provide, e.g., statistically, immediate feedback to the senders and allows for short-term adaptation and efficient feedback-based repair mechanisms to be implemented.

[0175] A payload format-specific SDP attribute may be described to indicate the capability of using RTCP feedback: "a=rtcp-fb". The "rtcp-fb" attribute may be used as an SDP media attribute and may not be provided at the session level. The "rtcp-fb" attribute may be used in media sessions for which the "AVPF" is specified. The "rtcp-fb" attribute may be used to indicate which RTCP FB messages may be used in a media session for the indicated payload type. A wildcard payload type (“*”) may be used to indicate that the RTCP feedback attribute applies to (e.g., all) payload types. If types of feedback are supported and / or the same feedback are specified for a subset of the payload types, “a=rtcp-fb” lines may be used.

[0176] Payload-specific FB messages transport information that is specific to a certain payload type and may be generated and acted upon at the codec layer. The packet format of an RTCP Feedback Message may be as follows in FIG. 9. FIG. 9 illustrates an example packet format for RTCP feedback messages.

[0177] The PT (payload type) field may identify the packet as being an RTCP FB message. Two values may be defined by the IANA: PT=205 for RTPFB (transport layer FB message) and PT=206 for PSFB (payload-specific FB message). The FMT (feedback message type) field may identify the type of FB message and may be interpreted relative to the payload type (e.g., RTPFB or PSFB). The FMT values for both the RTPFB payload type and the PSBF payload type may be managed by IANA.

[0178] Features described herein may be associated with an application-defined RTCP packet. The APP packet may be intended for experimental use as applications and features are developed, without requiring packet type value registration. APP packets with unrecognized names may be ignored. After testing and if wider use is justified, an APP packet may be redefined without the subtype and name fields and may be registered with IANA using an RTCP packet type.

[0179] FIG. 10 illustrates an example packet format for application-defined RTCP packets.

[0180] The semantics of the APP packets may be as follows: version (V): [2 bits] Identifies the version ofRTP, which may be the same in RTCP packets as in RTP data packets. The version described may be two (2); padding (P): [1 bit] If the padding bit is set, this individual RTCP packet may include padding octets atthe end which may not be part of the control information and may include in the length field; length: [16 bits] The length of this RTCP packet in 32-bit words minus one, including the header and any padding (e.g., the offset of one may make zero a valid length and may avoid a possible infinite loop in scanning a compound RTCP packet, while counting 32-bit words may avoid a validity check for a multiple of 4); subtype: [5 bits] May be used as a subtype to allow a set of APP packets to be described under one unique name, or for an application-dependent data; packet type (PT): [8 bits] Contains the constant 204 to identify this as an RTCP APP packet; name: 4 octets. A name chosen by the person defining the set of APP packets to be unique with respect to other APP packets this application might receive. The application creator may choose to use the application name and coordinate the allocation of subtype values to others who want to describe new packet types for the application. A name may be chosen (e.g., by others) based on the entity they represent, the use of the name may be coordinated within that entity. The name may be interpreted as a sequence of four ASCII characters, with uppercase and lowercase characters treated as distinct; and / or application-dependent data: [variable length] Application-dependent data may or may not appear in an APP packet. It may be interpreted by the application and not RTP itself. It may (e.g., must) be a multiple of 32 bits long.

[0181] Timing information data may be used to measure the QoE metrics for pose estimation and User interaction delays. The interactive Quality of Experience (QoE) may be dependent on the roundtrip interaction delay. The roundtrip interaction delay may be the sum of the Age of Content and the User Interaction Delay.

[0182] The User interaction delay, Age of content, and round-trip interaction delay measurements may be Quality of experience metrics for extended reality content. The delay measurement metrics may be calculated at the WTRU for providing a satisfactory (e.g., better) QoE to the end user. The server processing delay measurements may help the WTRU in adaptation process with the split rendering server for achieving better QoE.

[0183] Techniques may be used to transmit the time information recorded at the start of rendering in Split Rendering Server (SRS) at the output of the SRS to the WTRU, and the WTRU may measure the server processing delay. Techniques may be used to transmit SceneUpdateTime recorded in the SRS to the WTRU. SceneUpdateTime may be used to measure the user interaction delay, age of content and the round-trip interaction delay at the WTRU or MeCAR device.

[0184] The timing information recorded at the split rendering server or any RTP sending entity may be transmitted to the WTRU or MeCAR device using the below listed techniques: Transmission using RTCP extended Report (XR) packets; Transmission using the RTCP feedback messages; and / or Transmission using the RTP Header Extension mechanism.

[0185] In examples, the timing information for measuring the QoE metrics may be transmitted to the WTRU or MeCAR device using RTCP messages by extending the RTCP Extended Report (XR) framework mechanism.

[0186] In examples, the timing information data may be transmitted by extending the RTCP feedback messages with payload type equal to “payload specific feedback (PSFB)” (206).

[0187] The frequency of RTCP messages (XR packet or RTCP feedback) for transmitting QoE timing information may be negotiated during the configuration phase or during the RTP session establishment time.

[0188] In examples, the timing information data for measuring the QoE metrics may be transmitted to the WTRU or MeCAR device using the RTP header extension method. An RTP header extension method may provide the option to associate the frame rendered at a split rendering server and the recorded timing information of that frame for measurement of QoE metrics as it is carried as part of the RTP packets that carry the rendered images of a frame. The frequency of RTP header extension for transmitting timing information may be once in a frame / PDU set. Header extensions may be declared in the SDP using the “a=extmap” attribute. The header extension may be identified through an association between the URI of the header extension and an ID value that is contained as part of the extension. The QoE timing information header extension may use the following URN: “urn : 3 gpp : xr- qo e - t ime - in fo”.

[0189] RTCP messages may be used for QoE timing information transmission. T ransmission may be RTCP XR packet based. The timing information data recorded at the SRS or the RTP sender may be transmitted to the WTRU or MeCAR device by modifying (e.g., extending) the RTCP XR packets. The RTCP XR report may be identified by payload type (PT) equal to 207, which may refer to an extended report block message. For transmission of timing information data using RTCP XR messages, the block type (BT) may be extended with a value 40.

[0190] An extended report block may be for QoE timing information. The extended report block type may permit detailed reporting of timing information recorded at the split-rendering-server (SRS). The reports may be used, for example, for calculating the QoE metrics such as server processing delay, user interaction delay, age of content and the round-trip interaction delay at the WTRU or MeCAR devices.

[0191] In examples, the QoE timing information may be expressed in the same units as in the RTP timestamps of RTP data packets. For a packet, the relation may be established between the media data flowing and the corresponding QoE timing information recorded at the SRS for a specific media frame.

[0192] For use of the RTCP bandwidth, the RTCP XR block payload may include the whole or part of the timing information used to calculate the QoE metrics. In an example, when a bit is set to ONE in timejnfo field the respective timing information may be present in the payload. In an example when a bit isset to ZERO in timejnfo field, the respective time information may not be present in the payload. In examples, when the sender transmits T 1 and T3 information, the timejnfo field may be set to bO011 , and T1 and T3 information may be present in the message payload.

[0193] The identifiers of (e.g., all) actions that were processed for the rendering of a frame at a specific time may be reported in the “Rendered Pose” RTP header extension. This header extension may be identified using the “a=extmap” attribute URN: “urn:3gpp:xr-rendered-pose”. The synchronization between the various timing information present in the below XR report and the action identifiers present in the “Rendered Pose” RTP HE messages may be performed using the RTP timestamp information present in the RTP header of the packet containing the “Rendered Pose” RTP HE and the RTP timestamp field present in the below XR report block.

[0194] The QoE timing information Report Block with BT value equal to 40 may have the following format, as illustrated in FIG. 11.

[0195] FIG. 11 illustrates an example RTCP XR block format for quality of experience (QoE) timing information data or QoE timing information.

[0196] The semantics of the fields in QoE time information Extended Report (XR) block may be as follows: block type (BT) [8 bits]: A QoE time information Report Block may be identified by the constant 40; timejnfo [4 bits]: This field bits may represent the time stamps that are present in the XR report block. When T 1 is present in the XR report, the first bit (e.g., least significant bit) may be set to 1 . When the LSB is set to 0, T1 information may not be present. When T3, T5 and T6 may be present in the RTCP XR block data, bits 2, 3 and 4 may be set to 1 respectively. When T1 , T3, T5 and T6 are present in an RTCP XR block data, the timejnfo field value may be b1111 . The timing information when present may follow the order T1 , T3, T5 followed by T6. In examples, when the timejnfo field value is b0101 , the XR block may carry the T 1 information followed by T5. T3 and T6 timing information may not be present in that XR block content. The transmission frequency of T1 , T3, T5 and T6 time information in the XR report block may be negotiated during the configuration phase; resv [4 bits]: This field may be reserved for future definition. In the absence of such definition, the bits in this field may (e.g., must) be set to zero and may (e.g., must) be ignored by the receiver; block length [16 bits]: The length of this report block, including the header, in 32-bit words minus one; SSRC of source: 32 bits. The SSRC of the RTP data packet source being reported upon by this report block; RTP timestamp [32 bits]: This field may represent the RTP time stamp of the media frame at which the corresponding QoE timing information date was recorded at the SRS. This correspondence may be used for synchronization between the media data and the QoE timing information measurements recorded at the SRS for a specific media frame; estimated-at-time (T1) [32 bits]: This field represents the time when the pose estimation was made. This time information may be expressed in thesame units and with the same random offset as the RTP timestamps in data packets; Start-to-render-at- time (T3) [32 bits]: This field represents the time when the renderer in the split rendering server started to render the associated media frame. This time information may be expressed in the same units and with the same random offset as the RTP timestamps in data packets; Split-rendering-server-output-time (T5) [32 bits]: This field represents the recorded time at the output of the split rendering server. This time information may be expressed in the same units and with the same random offset as the RTP timestamps in data packets; and / or Scene-update-time(T6) [32 bits]: This field represents the time when the Scene manager processes the interaction task according to the actions in the action message from the WTRU and updates the scene. This time information may be expressed in the same units and with the same random offset as the RTP timestamps in data packets.

[0197] In examples, the QoE timing information may be expressed as NTP timestamps. The NTP timestamps report block described herein may indicate the wallclock time when the corresponding timing information was recorded at the SRS or WTRU. The association between the media data time and the corresponding QoE timing information recorded at the SRS for a specific media frame may be performed using the RTP time timestamp field present in the extended report block illustrated in FIG. 12.

[0198] The QoE timing information report block may have a format as depicted in FIG. 12. FIG. 12 illustrates an example XR block format for QoE timing information data with NTP timestamps.

[0199] The semantics of the fields in QoE time information Extended Report (XR) block may be as follows: block type (BT) [8 bits]: A QoE time information Report Block may be identified by the constant 40; timejnfo [4 bits]: This field bits represent the time stamps that are present in the XR report block. When a bit is set in this field, the respective timing information may be present in the payload. When a bit is set to zero (0) in this field, the respective time information may not be present in the payload. In examples, when T 1 is present in the XR report, the first bit (e.g., least significant bit) may be set to 1 . When the LSB is set to 0, T1 information may not be present. When T3, T5 and T6 are present in the RTCP XR block data, bits 2, 3 and 4 may be set to 1 respectively. When T1 , T3, T5 and T6 are present in an RTCP XR block data, the timejnfo field value may be b1111 . The timing information when present may follow the order T1 , T3, T5 followed by T6. In examples, when the timejnfo field value is b0101 , the XR block may carry the T1 information followed by T5. T3 and T6 timing information may not be present in the XR block content. The transmission frequency of T1 , T3, T5 and T6 time information in the XR report block may be negotiated during the configuration phase; resv: 4 bits. This field may be reserved for future definition. In the absence of such definition, the bits in this field may (e.g., must) be set to zero and may (e.g., must) be ignored by the receiver; block length [16 bits]: The length of this report block, including the header, in 32-bit words minus one; SSRC of source [32 bits]: The SSRC of the RTP data packet source being reported upon bythis report block; RTP timestamp [32 bits]: This field represents the RTP time stamp of the media frame at which the corresponding QoE timing information were recorded at the SRS. The correspondence may be used for synchronization between the media data and the QoE timing information measurements recorded at the SRS for a specific media frame; estimated-at-time (T 1 ) [64 bits]: This field represents the time when the pose estimation was made. This NTP timestamp field may be expressed in wall clock time units; Start- to-render-at-time (T3) [64 bits]: This field may represent the time when the renderer in the split rendering server started to render the associated media frame. This NTP timestamp field may be expressed in wall clock time units; Split-rendering-server-output-time (T5) [64 bits]: This field may represent the recorded time at the output of the split rendering server. This NTP timestamp field may be expressed in wall clock time units; and / or Scene-update-time (T6) [64 bits]: This field represents the time when the Scene manager processes the interaction task according to the actions in the action message from the WTRU and updates the scene. The NTP timestamp field may be expressed in wall clock time units.

[0200] In examples, the QoE timing information may be transmitted for (e.g., different) types of application scenarios, (e.g.,) split rendering server timing information, timing information recorded during viewport adaptive streaming, etc. To enable transmission of the timing information for types of application scenarios, the RTCP XR block may be extended to support the indication of the type of the application. When timing information related to more than one type of application are sent in an RTCP message, the RTCP XR blocks related to the applications may be stacked one after the other in the payload of the RTCP message (e.g., as defined in RFC 3611).

[0201] The QoE timing information report block with a BT value equal to 40 (e.g., as shown in FIG. 12) may be extended with an application type field as shown in FIG. 13.

[0202] FIG. 13 illustrates an RTCP XR block format for QoE timing information data. The semantics of the RTCP XR block format may be as follows: resv: 4 bits. This field may be reserved for future definition. In the absence of such definition, the bits in this field may (e.g., must) be set to zero and may (e.g., must) be ignored by the receiver, type [4 bits]: This field of bits may represent the type of the application scenario for which the QoE timing information is transmitted. Value 1 may indicate that the QoE timing information transmitted as part of this RTCP XR block is for a split rendering application. Other values may be reserved for future modifications (e.g., enhancements) to support other types of applications.

[0203] In examples, the QoE timing information may be expressed as NTP timestamps. The QoE timing information report block with a type of application scenario and the NTP timestamps may have a format, as depicted in FIG. 14. FIG. 14 illustrates an RTCP XR block format for QoE timing information data with NTP timestamps.

[0204] The semantics of the RTCP XR block format type field may be as follows: resv: 4 bits. This field may be reserved for future definition. In the absence of such definition, the bits in this field may (e.g., must) be set to zero and may (e.g., must) be ignored by the receiver, type [4 bits]: This field of bits may represent the type of the application scenario for which the QoE timing information is transmitted. Value 1 may indicate that the QoE timing information transmitted as part of this RTCP XR block is for a split rendering application. Other types of applications may be for future modification (e.g., enhancements).

[0205] Features described herein may be associated with SDP signaling and attributes. The SDP attribute "rtcp-xr" may be used to signal participants in a media session that they may use the specified XR blocks. The attribute may be extendable with parameters to cover a type of XR report block. The extended RTCP XR blocks with QoE time information SDP attribute may be described herein in Augmented Backus- Naur Form (ABNF).Rtcp-xr-attrib = "a=" "rtcp-xr" " :" [xr-format * (SP xr-format) ] CRLF xr-format = pkt-loss- rleI pkt-dup- rleI pkt- rcpt-timesI rcvr- rttI stat- summaryI voip-metricsI qoe-timing-infoI format-ext pkt-loss- rle = "pkt-loss-"le" ["=" max- size] pkt-dup- rle = "pkt-dup- rle" ["=" max- size] pkt- rcpt-times = "pkt - rcpt -times" ["=" max- size] rcvr- rtt = "rcvr- rtt" “=” rcvr- rtt-mode max- size] rcvr- rtt-mode = "all"I "sender" stat- summary = "stat- summary" ["=" stat-flag * (" stat-flag) ] stat-flag = "loss"" " " " / "dup"I "jitt" I "TTL" / "HL" voip-metrics = "voip-metrics" ["=" max- size] qoe-timing-info= "qoe-timing-info" ["=" max- size] max- size = 1*DIGIT ; maximum block size in octetsDIGIT = %x30-39 format-ext = non-ws- stringNon-ws- string = I* (%x21-FF)CRLF = %d!3.10

[0206] The “rtcp-xr” attribute may include zero, one, or more XR block related parameters. A parameter may signal functionality for an XR block, or a group of XR blocks. The attribute may be extensible such that parameters may be defined for any future XR block. In examples, the parameters may be extended to support delivery of QoE timing information data over RTCP packets with XR type.

[0207] The parameter names and their corresponding XR formats may be as follows:Parameter name XR block ( block type and name) pkt -loss - rle 1 Loss RLE Report Block pkt -dup- rle 2 Duplicate RLE Report Block pkt - rcpt -times 3 Packet Receipt Times Report Block stat - summary 6 Statistics Summary Report Block voip-metrics 7 VoIP Metrics Report Block qoe-timing-info 40 Timing information for QoE metrics calculation

[0208] The “qoe-timing-info”, parameter may specify an integer value. The value may indicate the largest size the whole report block may have in octets.

[0209] Features described herein may be associated with RTCP feedback message based transmission.

[0210] The timing information recorded at the SRS or the RTP sender may be transmitted to the WTRU or MeCAR device using RTCP feedback messages. The RTCP feedback message may be identified by payload type (PT) equal to 206, which may refer to a payload-specific feedback (PSFB) message. In the case of timing information for QoE measurement feedback messages, the feedback message type (FMT) may be set to the value 19. The RTCP feedback method and RTCP XR block method may involve transmitting timing information used for measuring the QoE metrics in both immediate feedback and early RTCP modes.

[0211] In examples, the QoE timing information may be expressed in the same units as in the RTP timestamps of RTP data packets. For a packet, a relation may be established between the media data flowing and the corresponding QoE timing information recorded at the SRS for a specific media frame.

[0212] For (e.g., optimum) use of the RTCP bandwidth, the RTCP message payload may contain the whole or part of the timing information required to calculate the QoE metrics. When a bit is set to ONE in timejnfo field, the respective timing information may be present in the payload. When a bit is set to ZERO in timejnfo field, the respective time information may not be present in the payload. In examples, when the sender transmits T 1 and T3 information, the timejnfo field may be set to bO011 , and T 1 and T3 information may be present in the message payload.

[0213] The feedback control information (FCI) for an RTCP feedback message with QoE timing information data may be in a format depicted in FIG. 15. FIG. 15 illustrates an example RTCP feedback message format for QoE timing information data.

[0214] The semantics of the fields in QoE time information RTCP feedback message may be as follows: timejnfo [4 bits]: This bit field indicates the timestamps that are present in the RTCP feedback message. When a bit is set in this field, the respective timing information may be present in the payload. When a bit isset to ZERO in this field, the respective time information may not be present in the payload. In examples, when T 1 is present in the RTCP feedback message, the first bit (e.g., least significant bit) may be set to 1 . When the LSB is set to 0, T1 information may not be present. When T3, T5 and T6 are present in the RTCP feedback message, bits 2, 3 and 4 may be set to 1 respectively. When T1 , T3, T5 and T6 are present in an RTCP feedback message, the timejnfo field value may be b1111 . The timing information when present may follow the order T1 , T3, T5 followed by T6. In examples, when the timejnfo field value is b0101 , the RTCP feedback message may carry the T1 information followed by T5. T3 and T6 timing information may not be present in the RTCP feedback message. The transmission frequency of T1 , T3, T5 and T6 time information in the RTCP feedback message may be negotiated during the configuration phase; resv [4 bits]: This field may be reserved for future definition. In the absence of the definition, the bits in this field may (e.g., must) be set to zero and may (e.g., must) be ignored by the receiver; RTP timestamp [32 bits]: This field represents the RTP time stamp of the media frame at which the corresponding QoE timing information was recorded at the SRS. The correspondence may be used for synchronization between the media data and the QoE timing information measurements recorded at the SRS for a specific media frame; estimated-at-time (T1) [32 bits]: This field may represent the time when the pose estimation was made. This time information may be expressed in the same units and with the same random offset as the RTP timestamps in data packets; start-to-render-at-time (T3) [32 bits]: This field represents the time when the renderer in the split rendering server started to render the associated media frame. This time information may be expressed in the same units and with the same random offset as the RTP timestamps in data packets; split-rendering-server-output-time (T5) [32 bits]: This field represents the recorded time at the output of the split rendering server. This time information may be expressed in the same units and with the same random offset as the RTP timestamps in data packets; and / or scene-update-time (T6) [32 bits]: This field represents the time when the Scene manager processes the interaction task according to the actions in the action message from the WTRU and updates the scene. This time information may be expressed in the same units and with the same random offset as the RTP timestamps in data packets.

[0215] The QoE timing information may be expressed as NTP timestamps. The association between the media data time and the corresponding QoE timing information metrics recorded at SRS for a specific media frame may be performed using the RTP time timestamp field present in the RTCP feedback message as shown below.

[0216] The feedback control information (FCI) for an RTCP feedback message with QoE timing information data may be as depicted in FIG. 16. FIG. 16 illustrates an example RTCP feedback message format for QoE timing information data with network time protocol (NTP) timestamps.

[0217] The semantics of the fields in QoE time information RTCP feedback message may be as follows: timejnfo [4 bits]: This bit field indicates the timestamps that are present in the RTCP feedback message. When a bit is set in this field, the respective timing information may be present in the payload. When a bit is set to zero in this field, the respective time information may not be present in the payload. In examples, when T 1 is present in the RTCP feedback message, the first bit (e.g., least significant bit) may be set to 1 . When the LSB is set to 0, T1 information may not be present. When T3, T5 and T6 are present, bits 2, 3 and 4 may be set to 1 respectively. When T1 , T3, T5 and T6 are present, the timejnfo field value may be b1111. The timing information when present may follow the order T1, T3, T5 followed by T6. In examples, when the timejnfo field value is b0101 , the message may carry the T1 information followed by T5. T3 and T6 timing information may not be present in that message. The transmission frequency of T1 , T3, T5 and T6 time information may be negotiated during the configuration phase; resv [4 bits]: This field is reserved for future definition. In the absence of such definition, the bits in this field may (e.g., must) be set to zero and may (e.g., must) be ignored by the receiver; RTP timestamp [32 bits]: This field represents the RTP time stamp of the media frame at which the corresponding QoE timing information were recorded at the SRS. This correspondence may be used for synchronization between the media data and the QoE timing information measurements recorded at the SRS for a specific media frame; estimated-at-time (T1) [64 bits]: This field represents the time when the pose estimation was made. This NTP timestamp field may be expressed in wall clock time units; start-to-render-at-time (T3) [64 bits]: This field represents the time when the renderer in the split rendering server started to render the associated media frame. This NTP timestamp field may be expressed in wall clock time units; split-rendering-server-output-time (T5) [64 bits]: This field represents the recorded time at the output of the split rendering server. This NTP timestamp field may be expressed in wall clock time units; and / or scene-update-time (T6) [64 bits]: This field represents the time when the Scene manager processes the interaction task according to the actions in the action message from the WTRU and updates the scene. This NTP timestamp field may be expressed in wall clock time units.

[0218] In examples, the QoE timing information RTCP feedback message may include the QoE timing information and the indication of (e.g., different) types of applications (e.g., split rendering server timing information, timing information recorded during viewport adaptive streaming, etc.). When timing information related to more than one type of application are to be transmitted, these may be sent separately over multiple RTCP feedback messages.

[0219] The feedback control information (FCI) for an RTCP feedback message with the type of application and the associated QoE timing information data may have a format, as depicted in FIG. 17. Figure 17 illustrates an RTCP feedback message format for QoE timing information data. In examples, thefeedback control information (FCI) for an RTCP feedback message with the application type information and QoE timing information represented using NTP timestamps may have a format, as depicted in FIG. 18. FIG. 18 illustrates an RTCP feedback message format for QoE timing information data with NTP timestamps

[0220] An RTCP feedback message may be associated with SDP signaling. An RTP sender supporting transmission of timing information data used for measurement of QoE metrics may signal the capability in SDP for (e.g., all) media content where QoE metrics measurements at WTRU or MeCAR device are used. QoE timing information transmission support may be offered by including the a=rtcp-fb attribute under the relevant media line scope. The QoE timing information transmission support using the RTCP feedback method may be expressed with the following parameter: qoe-timing-information. A wildcard payload type (“*”) may be used to indicate that the RTCP feedback attribute for QOE timing information transmission support signaling applies to (e.g., all) payload types. If the same QoE timing information is specified for a subset of the payload types, “a=rtcp-fb” lines may be used. An example usage of this attribute to signal QOE timing information delivery may be relative to a media line based on the RTCP feedback method: a=rtcp-fb:* qoe-timing-information.

[0221] An RTP Header extension may be associated with QoE timing information transmission. The RTP header extension mechanism may be extended to define a new RTP header extension to carry QoE timing information to the WTRU or MeCAR devices.

[0222] For use of the RTP packets bandwidth, the RTP HE payload may include the whole or part of the timing information required to calculate the QoE metrics. When a bit is set to one (1) in timejnfo field, the respective timing information may be present in the payload. When a bit is set to zero (0) in timejnfo field, the respective time information may be present in the payload. In examples, when the sender transmits T1 and T3 information, the timejnfo field may be set to bO011 , and T1 and T3 information may be present in the message payload.

[0223] The Identifiers of (e.g., all) actions that were processed for the rendering of a frame at a specific time may be reported in the “Rendered Pose” RTP header extension. The header extension may be identified using the “a=extmap” attribute URN: “urn:3gpp:xr-rendered-pose”. The synchronization between the various timing information present in the “QoE timg information” RTP HE and the action identifiers present in the “Rendered Pose” RTP HE messages may be performed using the RTP timestamp information present in the RTP header of the packets including the “Rendered Pose” RTP HE and the “QoE timg information” RTP HE.

[0224] In examples, the QoE timing information may be expressed in the same units as the RTP timestamp. The RTP header extension may be defined with one-byte extension and two-byte extension formats.The syntax and semantics of the QoE timing information header extension may be as depicted in FIG. 19 and / or FIG. 20. FIG.19 illustrates an example RTP header extension using one-byte header format. FIG. 20 illustrates an example RTP header extension using two-byte header format.

[0225] The semantics of the fields in QoE timing information RTP HE may be as follows: resv [4 bits]: This field is reserved for future definition. In the absence of such definition, the bits in this field may (e.g., must) be set to zero and may (e.g., must) be ignored by the receiver; timejnfo [4 bits]: This bit field indicates the timestamps that are present in the RTP HE message. When a bit is set in this field, the respective timing information may be present in the payload. When a bit is set to zero (0) in this field, the respective time information may not be present in the payload. In examples, when T1 is present in the RTP HE, the first bit (e.g., least significant bit) may be set to 1 . When the LSB is set to 0, T1 information may not be present. When T3, T5 and T6 are present in the RTP HE data, bits 2, 3 and 4 may be set to 1 respectively. When T1 , T3, T5 and T6 are present in an RTP HE data, the timejnfo field value may be b1111 . The timing information, when present, may follow the order T1 , T3, T5 followed by T6. In examples, when the timejnfo field value is b0101 , the RTP HE may carry the T1 information followed by T5. T3 and T6 timing information may not be present in the RTP HE. The transmission frequency of T1 , T3, T5 and T6 time information in the RTP HE may be negotiated during the configuration phase; estimated-at-time (T1) [32 bits]: This field represents the time when the pose estimation was made. The time information may be expressed in the same units and with the same random offset as the RTP timestamp in RTP header; start- to-render-at-time (T3) [32 bits]: This field represents the time when the renderer in the split rendering server started to render the associated media frame. The time information may be expressed in the same units and with the same random offset as the RTP timestamp in RTP header; split-rendering-server-output-time (T5) [32 bits]: This field represents the recorded time at the output of the split rendering server. The time information may be expressed in the same units and with the same random offset as the RTP timestamp in RTP header; and / or scene-update-time (T6) [32 bits]: This field represents the time when the Scene manager processes the interaction task according to the actions in the action message from the WTRU and updates the scene. This time information may be expressed in the same units and with the same random offset as the RTP timestamp in RTP header.

[0226] In examples, the QoE timing information carried in the RTP HE may be expressed as NTP timestamps. The NTP timestamps in the below RTP HE may indicate the wallclock time when the corresponding timing information was recorded at the SRS or WTRU. The RTP header extension may bedefined with two-byte extension format. The syntax and semantics of the QoE timing information header extension may be as depicted in FIG. 21 . FIG. 21 illustrates an example RTP header extension using two- byte header format.

[0227] The semantics of the fields in QoE timing information RTP HE may be as follows: resv [4 bits]: This field is reserved for future definition. In the absence of such definition, the bits in this field may (e.g., must) be set to zero (0) and may (e.g., must) be ignored by the receiver; timejnfo [4 bits]: This bit field indicates the timestamps that are present in the RTP HE message. When a bit is set in this field, the respective timing information may be present in the payload. When a bit is set to zero (0) in this field, the respective time information may not be present in the payload. In examples, when T1 is present in the RTP HE, the first bit (e.g., least significant bit) may be set to 1 . When the LSB is set to 0, T1 information may not be present. When T3, T5 and T6 are present in the RTP HE data, bits 2, 3 and 4 may be set to 1 respectively. When T1 , T3, T5 and T6 are present in an RTP HE data, the timejnfo field value may be b1111 . The timing information, when present, may follow the order T1 , T3, T5 followed by T6. In examples, when the timejnfo field value is b0101 , the RTP HE may carry the T1 information followed by T5. T3 and T6 timing information may not be present in the RTP HE. The transmission frequency of T1 , T3, T5 and T6 time information in RTP HE may be negotiated during the configuration phase; estimated-at-time (T1) [64 bits]: This field represents the time when the pose estimation was made. This NTP timestamp field may be expressed in wall clock time units; start-to-render-at-time (T3) [64 bits]: This field represents the time when the renderer in the split rendering server started to render the associated media frame. This NTP timestamp field may be expressed in wall clock time units; split-rendering-server-output-time (T5) [64 bits]: This field represents the recorded time at the output of the split rendering server. The NTP timestamp field may be expressed in wall clock time units; and / or scene-update-time (T6) [64 bits]: This field represents the time when the Scene manager processes the interaction task according to the actions in the action message from the WTRU and updates the scene. The NTP timestamp field may be expressed in wall clock time units.

[0228] In examples, the QoE timing information may be transmitted for (e.g., different) types of applications (e.g., split rendering server timing information, timing information recorded during viewport adaptive streaming, etc.) using the RTP Header extension mechanism. When timing information related to (e.g., different) types of applications are to be transmitted, the RTP header extension may have multiple such entries one after the other with the (e.g., appropriate) length field value.

[0229] The RTP header extension may include two-byte extension formats. The syntax and semantics of the QoE timing information header extension with application type information may be as illustrated in FIG. 22. FIG. 22 illustrates an RTP header extension using a two-byte header format.

[0230] The semantics of the RTCP header extension type field may be as follows: resv: 4 bits. This field may be reserved (e.g., for future definition). In the absence of a definition, the bits in this field may (e.g., must) be set to zero and may (e.g., must) be ignored by the receiver, type [4 bits]: This field of bits may represent the type of the application for which the QoE timing information is transmitted. Value 1 may indicate that the QoE timing information transmitted as part of the RTP header extension is for a split rendering application. Other types of applications may be for future modifications (e.g., enhancements).

[0231] The QoE timing information may be expressed using NTP timestamps. The RTP header extension with an application scenario type and the timing information using NTP timestamps may include a two-byte extension format. The syntax and semantics for the QoE timing information header extension may be as depicted in FIG. 23. FIG. 23 illustrates RTP header extension using a two-byte header format.

[0232] SDP signaling may be associated with QoE timing information RTP HE. The support for transmission of timing information data recorded at a split rendering server or an RTP sender may be signaled as an SDP extension. A sender may offer information on support for transmitting the timing information data required for QoE measurements in the initial offer-answer negotiation by signaling it in the SDP message. This may be done by including the "a=extmap" attribute under the relevant media line. The mapping may be provided per media stream (in the media-level section(s) of SDP, i.e. , after an "m=" line) or globally for (e.g., all) streams (i.e., before the first “m=" line, at session level).

[0233] The URN corresponding to QoE timing Information RTP HE is urn:3gpp:params:rtp-hdrext:qoe- timing-info. An example usage of this URN to signal transmitted QoE timing information relative to a media line may be as follows (e.g., the signaling may be part of a media line): a=extmap:11 / sendonly urn:3gpp:params:rtp-hdrext:qoe-timing-info

[0234] The number 11 in the example may be replaced with a number in the range 1 -254 using the two- byte header extension mechanism and in the range 1-15 while using the one-byte header extension mechanism.

[0235] The definitions may be either (e.g., all) session level or (e.g., all) media level; it may not be permitted to mix the two styles. As described herein, the IDs used may be unique for a stream type for a given media, or for the session for session-level declarations.

[0236] Systems, methods, and instrumentalities may be configured for transmission / reception of QoE timing information for measuring quality of experience (QoE) metrics. A device (e.g., a wireless / transmit receive unit (WTRU) and / or a media capabilities for Augmented Reality (MeCAR) device) may receive a real-time control protocol (RTCP) extended report (XR) packet with an extended report block, the extended report block comprising a field representing timing information, a block length indicator, an identifier for asource of real-time protocol (RTP) data packets, a timestamp for synchronizing media data and QoE timing information recorded at a server. The device may extract the timing information from the extended report block. The device may calculate quality of experience (QoE) metrics based on the extracted timing information.

[0237] The device may select and transmit the timing information on a condition that specific bits are configured in the field representing timing information. The timing information may be expressed in a same unit as the real-time protocol (RTP) of RTP data packets, and synchronization may be performed using the timing information. The extended report block may include a timestamp indicating a time when the timing information was recorded at the server. The device may adjust use of real-time control protocol (RTCP) bandwidth by including timing information for calculating the quality of experience (QoE) metrics in the extended report block.

[0238] The extended report block may include a type field configured to indicate an application scenario associated with the QoE timing information. When multiple application scenarios are associated with the QoE timing information, respective RTCP XR blocks related to the application scenarios may be included in the RTCP XR packet. The timing information from the extended report block may be expressed using NTP timestamps.

[0239] Systems, methods, and instrumentalities may be configured for transmission of time information data for measuring quality of experience (QoE) metrics. A server (e.g., a split render server and / or a realtime protocol (RTP) server) may determine a real-time control protocol (RTCP) extended report (XR) packet with an extended report block, the extended report block comprising a field representing timing information, a block length indicator, an identifier for a destination of real-time protocol (RTP) data packets, a timestamp for synchronizing media data, and QoE timing information to be sent to a device. The device may embed the timing information into the extended report block. The device may transmit quality of experience (QoE) metrics based on the embedded timing information.

[0240] The device may select the timing information for embedding and transmitting on a condition that specific bits are configured in the field representing timing information. The timing information may be expressed in a same unit as the real-time protocol (RTP) of RTP data packets, and synchronization may be performed using the timing information. The extended report block may include a timestamp indicating a time when the timing information is to be sent to the server. The device may adjust use of real-time control protocol (RTCP) bandwidth by including timing information for transmitting the quality of experience (QoE) metrics in the extended report block.

[0241] The extended report block may include a type field configured to indicate an application scenario associated with the QoE timing information. When multiple application scenarios are associated with theQoE timing information, respective RTCP XR blocks related to the application scenarios may be included in the RTCP XR packet. The timing information from the extended report block may be expressed using NTP timestamps.

[0242] Although features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that a feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable media include electronic signals (transmitted over wired or wireless connections) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, WTRU, terminal, base station, RNC, or any host computer.

Claims

CLAIMS1 . A method implemented by a device, the method comprising: receiving, from a split-rendering server (SRS), a real-time control protocol (RTCP) message comprising at least a quality of experience (QoE) timing information extended report (XR) block, wherein the QoE timing information XR block comprises an indication of a block type indicating that the QoE timing information XR block comprises QoE timing information, wherein the QoE timing information is recorded at the SRS, and wherein the QoE timing information comprises at least an estimated at time, a start to render at time, a split-rendering server output time, and a scene update time; and calculating at least one QoE metric based on at least one or more of the estimated at time, the start to render at time, the split-rendering server output time, or the scene update time.

2. The method of claim 1 , wherein the method further comprises: receiving a session description protocol (SDP) a field indicating use of RCTP XR blocks for signaling the QoE timing information, wherein the RTCP message is associated with the use of the RCTP XR blocks.

3. The method of claim 1 , wherein the method further comprises extracting the QoE timing information based on the block type indicating that the XR block comprises the timing information.

4. The method of claim 3, wherein the RTCP message comprises a field indicating that one or more types of the QoE timing information are present in the RTCP message and an order associated with the one or more types of the QoE timing information.

5. The method of claim 1 , wherein the device comprises a media capabilities for augmented reality (MeCar) device.

6. The method of claim 1 , wherein the start to render at time indicates a time a renderer associated with the SRS started to render a media frame.

7. The method of claim 1 , wherein the estimated at time indicates a time when pose estimation was made.

8. The method of claim 1 , wherein the split-rendering server output time indicates a time associated with an output of the SRS.

9. The method of claim 1 , wherein the scene update time indicates a time when processing of one or more actions was started by the SRS.

10. The method of claim 1 , wherein the at least one QoE metric comprises one or more of a round-trip interaction delay, a server processing delay, a user interaction delay, or an age of content.11 . The method of claim 10, wherein the server processing delay is determined based on a sum of the age of content and the user interaction delay.

12. The method of claim 10, wherein the user interaction delay is a duration between a first time when a user action is initiated and a second time when the user action is taken into account by a content creation engine.

13. The method of claim 12, wherein the user interaction delay depends on uplink latency.

14. The method of claim 10, wherein the age of content is a time duration between a first time when a content is created and a second time when the content is presented.

15. The method of claim 10, wherein the age of content or the user interaction delay is determined based on at least the scene update time.

16. The method of claim 15, wherein the age of content depends on downlink latency.

17. The method of claim 10, wherein the round-trip interaction delay is determined as a sum of the user interaction delay and the age of content.

18. A device, the device comprising: a processor configured to: receive, from a split-rendering server (SRS), a real-time control protocol (RTCP) message comprising at least a quality of experience (QoE) timing information extended report (XR) block, wherein the QoE timing information XR block comprises an indication of a block type indicating thatthe QoE timing information XR block comprises QoE timing information, wherein the QoE timing information is recorded at the SRS, and wherein the QoE timing information comprises at least an estimated at time, a start to render at time, a split-rendering server output time, and a scene update time; and calculate at least one QoE metric based on at least one or more of the estimated at time, the start to render at time, the split-rendering server output time, or the scene update time.

19. The device of claim 18, wherein the processor is further configured to: receive a session description protocol (SDP) a field indicating use of RCTP XR blocks for signaling the QoE timing information, wherein the RTCP message is associated with the use of the RCTP XR blocks.

20. The device of claim 18, wherein the processor is further configured to extract the QoE timing information based on the block type indicating that the XR block comprises the timing information.21 . The device of claim 20, wherein the RTCP message comprises a field indicating that one or more types of the QoE timing information are present in the RTCP message and an order associated with the one or more types of the QoE timing information.

22. The device of claim 18, wherein the device comprises a media capabilities for augmented reality (MeCar) device.

23. The device of claim 18, wherein the start to render at time indicates a time a renderer associated with the SRS started to render a media frame.

24. The device of claim 18, wherein the estimated at time indicates a time when pose estimation was made.

25. The device of claim 18, wherein the split-rendering server output time indicates a time associated with an output of the SRS.

26. The device of claim 18, wherein the scene update time indicates a time when processing of one or more actions was started by the SRS.

27. The device of claim 18, wherein the at least one QoE metric comprises one or more of a round-trip interaction delay, a server processing delay, a user interaction delay, or an age of content.

28. The device of claim 27, wherein the server processing delay is determined based on a sum of the age of content and the user interaction delay.

29. The device of claim 27, wherein the user interaction delay is a duration between a first time when a user action is initiated and a second time when the user action is taken into account by a content creation engine.

30. The device of claim 29, wherein the user interaction delay depends on uplink latency.31 . The device of claim 27, wherein the age of content is a time duration between a first time when a content is created and a second time when the content is presented.

32. The device of claim 27, wherein the age of content or the user interaction delay is determined based on at least the scene update time.

33. The device of claim 32, wherein the age of content depends on downlink latency.

34. The device of claim 27, wherein the round-trip interaction delay is determined as a sum of the user interaction delay and the age of content.

Images