Sending time information data for measuring QoE metrics

Abstract

Systems, methods, and means may be configured to transmit / receive QoE timing information for measuring Quality of Experience (QoE) metrics. A device (e.g., a wireless / transmit / receive unit (WTRU) and / or media function for augmented reality devices (MeCAR)) may receive a Real-Time Control Protocol (RTCP) message from a split rendering server (SRS) containing a QoE timing information enhanced report (XR) block. The QoE timing information XR block may indicate a block type of QoE timing information. The QoE timing information (e.g., recorded in the SRS) may include estimated time, rendering start time, split rendering server output time, and scene update time. The device may calculate at least one QoE metric based on the QoE timing information.

Description

[Technical Field]

[0001] Cross-Reference to Related Applications

[0001] This application claims the benefit of European Provisional Patent Application No. 23306369.2 filed on August 11, 2023 and European Patent Application No. 23306648.9 filed on September 29, 2023, the contents of which are incorporated herein by reference. [Background Art]

[0002] Background

[0002] Video coding systems may be used to compress digital video signals, for example, to reduce the storage and / or transmission bandwidth required for such signals. Video coding systems may include block-based systems, such as, for example, wavelet-based systems, object-based systems, and / or block-based hybrid video coding systems. In augmented reality applications, mechanisms for providing control of degrees of freedom boundaries may not be suitable. [Summary of the Invention]

[0003] Summary

[0003] Systems, methods, and means may be configured to transmit time information data for measuring quality of experience (QoE) metrics. A device (e.g., a wireless / transmit / receive unit (WTRU) and / or media function for augmented reality devices (MeCAR)) may receive a Real-time Control Protocol (RTCP) message from a split rendering server (SRS) which includes at least a Quality of Experience (QoE) Timing Information Enhanced Report (XR) block. The QoE Timing Information XR block may include a block-type indication that the QoE Timing Information XR block contains QoE timing information. The QoE timing information may be recorded in the SRS. The QoE timing information may include at least an estimated time, dulling start time, split rendering server output time, and scene update time. The device may calculate at least one QoE metric based on at least one of the estimated time, dulling start time, split rendering server output time, or scene update time.

[0004]

[0004] The device may receive a Session Description Protocol (SDP) which is a field indicating the use of an RCTP XR block to signal QoE timing information, and the RTCP message is associated with the use of an RCTP XR block. The device may extract the QoE timing information based on the block type which indicates that the XR block contains timing information.

[0005]

[0005] An RTCP message may include a field indicating the presence of one or more types of QoE timing information in the RTCP message, and an order associated with one or more types of QoE timing information. Rendering start time may indicate the time when the renderer associated with the SRS began rendering the media frame. Estimation time may indicate the time when pose estimation was performed. Split rendering server output time may indicate the time associated with the SRS output. Scene update time may indicate the time when the SRS began processing one or more actions. At least one QoE metric may include one or more of the following: round-trip interaction delay, server processing delay, user interaction delay, or content elapsed time. Server processing delay may be determined based on the sum of content elapsed time and user interaction delay.

[0006]

[0006] User interaction delay may include the duration between a first time when a user action is initiated and a second time when the user action is considered by the content creation engine. User interaction delay may depend on uplink latency. Content elapsed time may include the duration between a first time when the content is created and a second time when the content is presented. Content elapsed time or user interaction delay may be determined based on at least scene update time. Content elapsed time may vary depending on downlink latency. Round-trip interaction delay may be determined as the sum of user interaction delay and content elapsed time.

[0007]

[0007] Systems, methods, and means may be configured to transmit time information data to measure a Quality of Experience (QoE) metric. A device (e.g., a wireless / transmit / receive unit (WTRU) and / or media function for augmented reality devices (MeCAR)) may receive a Real-Time Control Protocol (RTCP) Enhanced Report (XR) packet having an enhanced report block, the enhanced report block including fields representing timing information, a block length indicator, an identifier for the source of the Real-Time Protocol (RTP) data packet, a timestamp for synchronizing media data, and QoE timing information recorded on a server. The device may extract timing information from the enhanced report block. Based on the extracted timing information, the device may calculate a Quality of Experience (QoE) metric.

[0008]

[0008] A device may select and transmit timing information, provided that specific bits are configured in the fields representing the timing information. The timing information may be expressed in the same units as the Real-Time Protocol (RTP) of the RTP data packets, and synchronization may be performed using the timing information. The extended report block may include a timestamp indicating the time the timing information was recorded on the server. The device may adjust the use of Real-Time Control Protocol (RTCP) bandwidth by including the timing information in the extended report block for calculating the Quality of Experience (QoE) metric.

[0009]

[0009] The extended report block may include a type field configured to indicate the application scenario associated with the QoE timing information. If multiple application scenarios are associated with the QoE timing information, each RTCP XR block associated with the application scenario may be included in the RTCP XR packet. The timing information from the extended report block may be represented using NTP timestamps.

[0010]

[0010] Systems, methods, and means may be configured to transmit time information data to measure a Quality of Experience (QoE) metric. A server (e.g., a split rendering server and / or a Real-Time Protocol (RTP) server) may determine a Real-Time Control Protocol (RTCP) Extended Report (XR) packet having an Extended Report Block, the Extended Report Block including fields representing timing information, a block length indicator, an identifier for the destination of the Real-Time Protocol (RTP) data packet, a timestamp for synchronizing media data, and QoE timing information to be sent to a device. A device may embed timing information in the Extended Report Block. Based on the embedded timing information, the device may transmit a Quality of Experience (QoE) metric.

[0011]

[0011] A device may select timing information for embedding and transmission, provided that specific bits are configured in the fields representing timing information. The timing information may be expressed in the same units as the Real-Time Protocol (RTP) of the RTP data packets, and synchronization may be performed using the timing information. The extended report block may include a timestamp indicating the time the timing information is sent to the server. The device may adjust the use of Real-Time Control Protocol (RTCP) bandwidth by including timing information for sending Quality of Experience (QoE) metrics in the extended report block.

[0012]

[0012] The extended report block may include a type field configured to indicate the application scenario associated with the QoE timing information. If multiple application scenarios are associated with the QoE timing information, each RTCP XR block associated with the application scenario may be included in the RTCP XR packet. The timing information from the extended report block may be represented using NTP timestamps.

[0013]

[0013] Each feature disclosed in any part of this Specification may be described and practiced separately / individually and in any combination with any other feature disclosed in this Specification and / or any other feature that may be implicitly or explicitly referenced in this Specification or otherwise included within the scope of the subject matter disclosed in this Specification. [Brief explanation of the drawing]

[0014] Brief explanation of the drawing [Figure 1A]

[0014] This is a system diagram illustrating an exemplary communication system in which one or more disclosed embodiments may be implemented. [Figure 1B]

[0015] This is a system diagram illustrating an exemplary wireless transmit / receive unit (WTRU) that may be used in the communication system illustrated in Figure 1A, according to one embodiment. [Figure 1C]

[0016] This is a system diagram illustrating an exemplary radio access network (RAN) and an exemplary core network (CN) that may be used in a communication system illustrated in Figure 1A according to one embodiment. [Figure 1D]

[0017] This is a system diagram illustrating further exemplary RANs and further exemplary CNs that may be used in the communication system illustrated in Figure 1A according to one embodiment. [Figure 2]

[0018] This is a diagram illustrating an example video encoder. [Figure 3]

[0019] This figure shows an example of a video decoder. [Figure 4]

[0020] This figure shows an example of a system in which various embodiments and examples can be implemented. [Figure 5]

[0021] An example of a call flow for pause information is shown. [Figure 6]

[0022] An example of a call flow for a user action is shown. [Figure 7]

[0023] An example of a Real-time Control Protocol (RTCP) Extended Report (XR) packet format is shown. [Figure 8]

[0024] An example of an RTCP XR block format is shown. [Figure 9]

[0025] An example of a packet format for an RTCP feedback message is shown. [Figure 10]

[0026] An example of a packet format for an application-defined RTCP packet is shown. [Figure 11]

[0027] An example of an RTCP XR block format for Quality of Experience (QoE) timing information data is shown. [Figure 12]

[0028] An example of an XR block format for QoE timing information data with an NTP timestamp is shown. [Figure 13]

[0029] An RTCP XR block format for QoE timing information data is shown. [Figure 14]

[0030] An RTCP XR block format for QoE timing information data with an NTP timestamp is shown. [Figure 15]

[0031] An example of an RTCP feedback message format for QoE timing information data is shown. [Figure 16]

[0032] An example of an RTCP feedback message format for QoE timing information data including a Network Time Protocol (NTP) timestamp is shown. [Figure 17]

[0033] An example of an RTCP feedback message format for QoE timing information data is shown. [Figure 18]

[0034] An RTCP feedback message format for QoE timing information data with an NTP timestamp is shown. [Figure 19]

[0035] An example of RTP header extension using a 1-byte header format is shown. [Figure 20]

[0036] An example of RTP header extension using a 2-byte header format is shown. [Figure 21]

[0037] An example of RTP header extension using a 2-byte header format is shown. [Figure 22]

[0038] This demonstrates an RTP header extension using a 2-byte header format. [Figure 23]

[0039] This demonstrates an RTP header extension using a 2-byte header format. [Modes for carrying out the invention]

[0015] Detailed explanation

[0040] The following will provide a detailed description of exemplary embodiments with reference to various figures. While this description provides detailed examples of possible embodiments, please note that the details are intended to be illustrative and not to limit the scope of this application.

[0016]

[0041] Figure 1A illustrates an exemplary communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communication system 100 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word DFT-Spread OFDM (ZT UW DTS-s OFDM), unique-word OFDM (UW-OFDM), resource block filter OFDM, filter bank multicarrier (FBMC), and similar methods.

[0017]

[0042] As shown in Figure 1A, the communication system 100 may include wireless transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, RAN 104 / 113, CN 106 / 115, public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, but it should be understood that the disclosed embodiments intend any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, and 102d may be any type of device configured to operate and / or communicate in a wireless environment. For example, WTRU102a, 102b, 102c, and 102d (all of which may be called “stations” and / or “STAs”) may be configured to transmit and / or receive radio signals and may include user equipment (UEs), mobile stations, fixed or mobile subscriber units, subscriber base units, pagers, mobile phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearables, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in industrial and / or automated processing chain settings), consumer electronics devices, and devices operating on commercial and / or industrial wireless networks. Any of WTRU102a, 102b, 102c, and 102d may be interchangeably referred to as WTUEs.

[0018]

[0043] The communication system 100 may also include base stations 114a and / or base stations 114b. Each of the base stations 114a and 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks such as CN 106 / 115, the Internet 110, and / or other networks 112. For example, base stations 114a and 114b may be base transceiver station (BTS), node B, eNode B, home node B, home eNode B, gNB, NR node B, site controller, access point (AP), wireless router, etc. Although base stations 114a and 114b are depicted as single elements, it should be recognized that base stations 114a and 114b may include any number of interconnected base stations and / or network elements.

[0019]

[0044] Base station 114a may be part of RAN 104 / 113, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), and relay nodes. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be referred to as cells (not shown). These frequencies may be in the licensed spectrum, the unlicensed spectrum, or a combination of the licensed and unlicensed spectrum. A cell can provide coverage of radio services to a particular geographic area that may be relatively fixed or change over time. A cell may be further divided into cell sectors. For example, a cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver for each sector of the cell. In one embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology, and multiple transceivers can be used for each sector of the cell. For example, beamforming can be used to transmit and / or receive signals in a desired spatial direction.

[0020]

[0045] Base stations 114a and 114b can communicate with one or more WTRUs 102a, 102b, 102c, and 102d via a radio interface 116, and the radio interface 116 may be any suitable radio communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The radio interface 116 may be established using any suitable radio access technology (RAT).

[0021]

[0046] More specifically, as described above, the communication system 100 may be a multiple access system and may employ one or more channel access schemes such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, base stations 114a and WTRUs 102a, 102b, and 102c in RAN 104 / 113 may implement radio technologies such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA) that can establish radio interfaces 115 / 116 / 117 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or Advanced HSPA (HSPA+). HSPA may include High-Speed ​​Downlink (DL) Packet Access (HSDPA) and / or High-Speed ​​UL Packet Access (HSUPA).

[0022]

[0047] In one embodiment, base stations 114a and WTRUs 102a, 102b, and 102c may implement radio technologies such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which can establish a radio interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0023]

[0048] In one embodiment, the base station 114a and WTRUs 102a, 102b, and 102c may implement wireless technologies such as NR radio access, which can establish a wireless interface 116 using New Radio (NR).

[0024]

[0049] In one embodiment, base station 114a and WTRU 102a, 102b, 102c may implement multiple radio access technologies. For example, base station 114a and WTRU 102a, 102b, 102c may implement LTE radio access and NR radio access together, for example, using the dual connectivity (DC) principle. Thus, the radio interface utilized by WTRU 102a, 102b, 102c may be characterized by transmissions between multiple types of radio access technologies and / or multiple types of base stations (e.g., eNB and gNB).

[0025]

[0050] In other embodiments, base stations 114a and WTRUs 102a, 102b, and 102c may implement wireless technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Provisional Standard 2000 (IS-2000), Provisional Standard 95 (IS-95), Provisional Standard 856 (IS-856), Global System for Mobile Communications (GSM), GSM Evolutionary High-Speed ​​Data Rate (EDGE), and GSM EDGE (GERAN).

[0026]

[0051] The base station 114b in Figure 1A may be, for example, a wireless router, home node B, home eNode B, or access point, and may utilize any suitable RAT to facilitate wireless connectivity in localized areas such as offices, homes, vehicles, campuses, industrial facilities, aerial walkways (e.g., for use by drones), and roads. In one embodiment, base stations 114b and WTRUs 102c, 102d may establish a wireless local area network (WLAN) by implementing wireless technologies such as IEEE 802.11. In another embodiment, base stations 114b and WTRUs 102c, 102d may establish a wireless personal area network (WPAN) by implementing wireless technologies such as IEEE 802.15. In yet another embodiment, base stations 114b and WTRUs 102c, 102d may establish a picocell or femtocell using cellular-based RATs (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.). As shown in Figure 1A, base station 114b may have a direct connection to the internet 110. Therefore, base station 114b may not need to access the internet 110 via CN106 / 115.

[0027]

[0052] RAN104 / 113 may communicate with CN106 / 115, which may be any type of network configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRU102a, 102b, 102c, and 102d. The data may have various quality of service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and so on. CN106 / 115 may provide call control, billing services, mobile location-based services, prepaid calls, internet connectivity, video distribution, and / or perform high-level security functions such as user authentication. Although not shown in Figure 1A, it should be understood that RAN104 / 113 and / or CN106 / 115 may communicate directly or indirectly with other RANs employing the same or different RATs as RAN104 / 113. For example, in addition to connecting to RAN104 / 113 which may be using NR radio technology, CN106 / 115 may also communicate with another RAN (not shown) employing GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0028]

[0053] CN106 / 115 may also function as a gateway for WTRU102a, 102b, 102c, and 102d to access PSTN108, the Internet 110, and / or other networks 112. PSTN108 may include a circuit-switched telephone network providing plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices using common communication protocols such as the transmission control protocol (TCP), user datagram protocol (UDP), and / or Internet protocol (IP) within the TCP / IP Internet Protocol suite. Network 112 may include wired and / or wireless networks owned and / or operated by other service providers. For example, network 112 may include another CN connected to one or more RANs that may employ the same RAT as RAN104 / 113 or a different RAT.

[0029]

[0054] Some or all of the WTRUs 102a, 102b, 102c, and 102d in the communication system 100 may include multimode functionality (for example, WTRUs 102a, 102b, 102c, and 102d may include multiple transceivers for communicating with different radio networks via different radio links). For example, WTRU 102c shown in Figure 1A may be configured to communicate with base station 114a, which may employ cellular-based radio technology, and base station 114b, which may employ IEEE 802 radio technology.

[0030]

[0055] Figure 1B is a system diagram showing an exemplary WTRU 102. As shown in Figure 1B, the WTRU 102 may comprise, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It should be understood that the WTRU 102 may include any partial combination of the above elements while maintaining consistency with one embodiment.

[0031]

[0056] The processor 118 may be a general-purpose processor, a dedicated processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functions that enable the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to a transceiver 120 which may be coupled to a transmit / receive element 122. Although Figure 1B depicts the processor 118 and the transceiver 120 as separate components, it should be understood that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0032]

[0057] The transmit / receive element 122 may be configured to transmit signals to or receive signals from a base station (e.g., base station 114a) via the radio interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In one embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive, for example, IR, UV, or visible light signals. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF signals and optical signals. It should be understood that the transmit / receive element 122 may be configured to transmit and / or receive any combination of radio signals.

[0033]

[0058] In Figure 1B, the transmit / receive element 122 is depicted as a single element, but the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for transmitting and receiving radio signals via the radio interface 116.

[0034]

[0059] The transceiver 120 may be configured to modulate the signal transmitted by the transmit / receive element 122 and demodulate the signal received by the transmit / receive element 122. As described above, the WTRU 102 may have multimode capabilities. Therefore, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0035]

[0060] The processor 118 of the WTRU102 may be coupled to a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light-emitting diode (OLED) display unit) and may receive user input data from them. The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from any type of suitable memory, such as non-removable memory 130 and / or removable memory 132, and store data in such memory. Non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. Removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from memory not physically located on the WTRU 102, such as on a server or home computer (not shown), and store data in such memory.

[0036]

[0061] The processor 118 may receive power from the power supply 134 and may be configured to distribute and / or control power to other components within the WTRU 102. The power supply 134 may be any suitable device for supplying power to the WTRU 102. For example, the power supply 134 may be one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium ion (Li ion), etc.), a solar cell, a fuel cell, etc.

[0037]

[0062] The processor 118 may also be coupled to a GPS chipset 136 which can be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to or instead of information from the GPS chipset 136, the WTRU 102 may receive location information from base stations (e.g., base stations 114a, 114b) via the radio interface 116 and / or determine its location based on the timing of signals received from two or more nearby base stations. It should be understood that the WTRU 102 may acquire location information by any suitable location determination method while maintaining consistency with one embodiment.

[0038]

[0063] The processor 118 may be further coupled to other peripherals 138, which may include one or more software modules and / or hardware modules that provide additional features, functions and / or wired or wireless connectivity. Examples of peripherals 138 include accelerometers, e-compasses, satellite transceivers, digital cameras (for photography and / or video), universal serial bus (USB) ports, vibration devices, television transceivers, hands-free headsets, Bluetooth® modules, frequency modulated (FM) wireless units, digital music players, media players, video game player modules, internet browsers, virtual reality and / or augmented reality (VR / AR) devices, activity trackers, and the like. The peripheral device 138 may include one or more sensors, which may be one or more of the following: gyroscope, accelerometer, Hall effect sensor, magnetometer, compass sensor, proximity sensor, temperature sensor, time sensor, geolocation sensor, altimeter, light sensor, touch sensor, magnetometer, barometer, gesture sensor, biometric sensor, and / or humidity sensor.

[0039]

[0064] WTRU102 may include a full-duplex radio (for example, one in which the transmission and reception of some or all of the signals associated with a particular subframe for both UL (e.g., for transmission) and downlink (e.g., for reception) may be in parallel and / or simultaneous. The full-duplex radio may include an interference management unit for reducing and / or substantially eliminating self-interference either through hardware (e.g., chokes) or signal processing via a processor (e.g., a separate processor (not shown) or processor 118). In one embodiment, WTRU102 may include a half-duplex radio for the transmission and reception of any or all of the signals (for example, one in which the signals associated with a particular subframe for either UL (e.g., for transmission) or downlink (e.g., for reception)).

[0040]

[0065] Figure 1C is a system diagram illustrating RAN104 and CN106 according to one embodiment. As described above, RAN104 employs E-UTRA wireless technology and can communicate with WTRU102a, 102b, and 102c via wireless interface 116. RAN104 can also communicate with CN106.

[0041]

[0066] RAN104 may include eNode-B160a, 160b, and 160c, but it should be understood that RAN104 may include any number of eNode-B while maintaining consistency with one embodiment. Each eNode-B160a, 160b, and 160c may include one or more transceivers for communicating with WTRU102a, 102b, and 102c via the radio interface 116. In one embodiment, eNode-B160a, 160b, and 160c may implement MIMO technology. For example, eNode-B160a may use multiple antennas to transmit radio signals to and / or receive radio signals from WTRU102a.

[0042]

[0067] Each of the eNode-B160a, 160b, and 160c may be associated with a specific cell (not shown) and may be configured to handle wireless resource management decisions, handover decisions, user scheduling in UL and / or DL, etc. As shown in Figure 1C, the eNode-B160a, 160b, and 160c may communicate with each other via the X2 interface.

[0043]

[0068] The CN106 shown in Figure 1C may include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (or PGW) 166. Although each of the above elements is depicted as part of CN106, it should be understood that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0044]

[0069] The MME162 can be connected to each of the eNode-B162a, 162b, and 162c within RAN104 via the S1 interface and can function as a control node. For example, the MME162 may be responsible for user authentication of WTRU102a, 102b, and 102c, bearer activation / deactivation, and selecting a specific serving gateway during the initial attachment of WTRU102a, 102b, and 102c. The MME162 may provide control plane functionality for switching between RAN104 and other RANs (not shown) employing other radio technologies such as GSM and / or WCDMA.

[0045]

[0070] The SGW164 can be connected to each of the eNode B160a, 160b, and 160c within RAN104 via the S1 interface. The SGW164 can generally route and forward user data packets to and from WTRU102a, 102b, and 102c. The SGW164 can perform other functions such as anchoring the user plane during eNode B handovers, triggering paging when DL data is available for WTRU102a, 102b, and 102c, managing and remembering the context of WTRU102a, 102b, and 102c, and similar functions.

[0046]

[0071] SGW164 can be connected to PGW166, thereby providing WTRU102a, 102b, and 102c with access to packet-switched networks such as the Internet 110, facilitating communication between WTRU102a, 102b, and 102c and IP-enabled devices.

[0047]

[0072] CN106 can facilitate communication with other networks. For example, CN106 can provide WTRU102a, 102b, and 102c with access to a circuit-switched network such as PSTN108 to facilitate communication between WTRU102a, 102b, and 102c and conventional land communication devices. For example, CN106 may include or communicate with an IP gateway (e.g., an IP multimedia subsystem (IMS (IP multimedia subsystem)) server) that acts as an interface between CN106 and PSTN108. In addition, CN106 may provide WTRU102a, 102b, and 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers.

[0048]

[0073] In Figures 1A to 1D, the WTRU is described as a wireless terminal, but in certain representative embodiments, it is intended that such a terminal may have access to a wired communication interface with a communication network (e.g., temporarily or permanently).

[0049]

[0074] In a typical embodiment, the other network 112 may be a WLAN.

[0050]

[0075] A WLAN in Infrastructure Basic Service Set (BSS) mode may have access points (APs) for the BSS and one or more stations (STAs) associated with those APs. APs may have access to or interfaces with a Distribution System (DS) or another type of wired / wireless network that carries traffic to and from the BSS. Traffic originating outside the BSS and destined for an STA may arrive via an AP and be delivered to the STA. Traffic originating from an STA to a destination outside the BSS may be sent to an AP and delivered to its respective destination. Traffic between STAs within the BSS may be transmitted via an AP; for example, a source STA may send traffic to an AP, which then delivers the traffic to a destination STA. Traffic between STAs within the BSS may be considered and / or referred to as peer-to-peer traffic. Peer-to-peer traffic may be transmitted (e.g., directly) between a source STA and a destination STA using a direct link setup (DLS). In certain representative embodiments, the DLS may be an 802.11e DLS or an 802.11z tunneled DLS (TDLS (tunneled DLS)). A WLAN using Independent BSS (IBSS) mode may not have access points (APs), and STAs within or using IBSS (e.g., all STAs) can communicate directly with each other. The IBSS communication mode may sometimes be referred to as the "ad hoc" communication mode in this specification.

[0051]

[0076] When using the 802.11ac infrastructure operating mode or a similar operating mode, an AP can transmit beacons on a fixed channel, such as the primary channel. The primary channel may be of a fixed width (e.g., a wide bandwidth of 20 MHz) or a width that is dynamically set via signal transmission. The primary channel may also be the operating channel of the BSS, which an STA can use to establish a connection with the AP. In a particular representative embodiment, for example in an 802.11 system, Carrier Sense Multiple Access with Collision Avoidance (CSMA / CA) may be implemented. In the case of CSMA / CA, an STA, including the AP (e.g., any STA), can sense the primary channel. If the primary channel is sensed / detected by a particular STA and / or identified as busy, that particular STA may backoff. In a given BSS, one STA (e.g., only one station) may transmit at any predetermined time.

[0052]

[0077] High-throughput (HT) STAs can use a 40MHz wide channel for communication, for example, by combining a primary 20MHz channel with adjacent or non-adjacent 20MHz channels to form a 40MHz wide channel.

[0053]

[0078] Very High Throughput (VHT) STAs can support 20MHz, 40MHz, 80MHz, and / or 160MHz wide channels. 40MHz and / or 80MHz channels can be formed by combining consecutive 20MHz channels. 160MHz channels may be formed by combining eight consecutive 20MHz channels, or by combining two non-consecutive 80MHz channels, sometimes referred to as an 80+80 configuration. In the 80+80 configuration, the channel-coded data may pass through a segment parser that can split the data into two streams. Inverse Fast Fourier Transform (IFFT) and time-domain processing may be performed separately for each stream. The streams can be mapped to two 80MHz channels, and the data can be transmitted by a transmitting STA. The receiver of a receiving STA can reverse the operation described above for the 80+80 configuration and transmit the combined data to Medium Access Control (MAC).

[0054]

[0079] Operating modes below 1 GHz are supported in 802.11af and 802.11ah. Channel operating bandwidth and carrier are reduced in 802.11af and 802.11ah compared to those used in 802.11n and 802.11ac. 802.11af supports bandwidths of 5 MHz, 10 MHz, and 20 MHz in the TV White Space (TVWS) spectrum, while 802.11ah supports bandwidths of 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz using the non-TVWS spectrum. According to a typical embodiment, 802.11ah can support meter-type control / machine-type communication, such as MTC devices, within a macro-coverage area. MTC devices may have limited functionality, including support for certain and / or limited bandwidths (e.g., support only). MTC devices may include batteries with a battery life exceeding a threshold (e.g., to maintain a very long battery life).

[0055]

[0080] WLAN systems that can support multiple channels and channel bandwidths, such as 802.11n, 802.11ac, 802.11af, and 802.11ah, include a channel that can be designated as the primary channel. The primary channel may have a bandwidth equal to the maximum common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel can be set and / or limited by the STA that supports the minimum bandwidth operating mode from among all STAs operating in the BSS. In the 802.11ah example, even if the AP and other STAs in the BSS support operating modes of 2MHz, 4MHz, 8MHz, 16MHz and / or other channel bandwidths, the primary channel may be 1MHz wide for an STA (e.g., an MTC type device) that supports 1MHz mode (e.g., only 1MHz mode). Carrier sensing and / or network allocation vector (NAV) settings may depend on the status of the primary channel. For example, if an STA (which only supports 1MHz operating mode) has its primary channel busy transmitting to an AP, the entire available frequency band may be considered busy, even though a large portion of the frequency band remains idle and could be available.

[0056]

[0081] In the United States, the available frequency band for 802.11ah is 902MHz to 928MHz. In South Korea, the available frequency band is 917.5MHz to 923.5MHz. In Japan, the available frequency band is 916.5MHz to 927.5MHz. The total available bandwidth for 802.11ah is 6MHz to 26MHz, depending on the country code.

[0057]

[0082] Figure 1D is a system diagram showing RAN113 and CN115 according to one embodiment. As described above, RAN113 employs NR radio technology and can communicate with WTRU102a, 102b, and 102c via the radio interface 116. RAN113 can also communicate with CN115.

[0058]

[0083] RAN113 may include gNB180a, 180b, and 180c, but it should be correctly recognized that RAN113 may include any number of gNBs while maintaining consistency with the embodiment. gNB180a, 180b, and 180c may each include one or more transceivers for communicating with WTRU102a, 102b, and 102c via the radio interface 116. In one embodiment, gNB180a, 180b, and 180c may implement MIMO technology. For example, gNB180a and 180b can use beamforming to transmit signals to and / or receive signals from gNB180a, 180b, and 180c. Thus, for example, gNB180a can use multiple antennas to transmit radio signals to and / or receive radio signals from WTRU102a. In one embodiment, gNB180a, 180b, and 180c may implement carrier aggregation technology. For example, gNB180a can transmit multiple component carriers to WTRU102a (not shown). A subset of these component carriers may be on the unauthorized spectrum, and the remaining component carriers may be on the authorized spectrum. In one embodiment, gNB180a, 180b, and 180c may implement coordinated multi-point (CoMP) technology. For example, WTRU102a can receive coordinated transmissions from gNB180a and gNB180b (and / or gNB180c).

[0059]

[0084] WTRU102a, 102b, and 102c can communicate with gNB180a, 180b, and 180c using transmissions associated with scalable neurology. For example, OFDM symbol intervals and / or OFDM subcarrier intervals can vary for different transmissions, different cells, and / or different parts of the radio transmission spectrum. WTRU102a, 102b, and 102c can communicate with gNB180a, 180b, and 180c using subframes or transmission time intervals (TTI) of varying or scalable lengths (e.g., containing varying numbers of OFDM symbols and / or lasting for varying absolute times).

[0060]

[0085] gNB180a, 180b, and 180c may be configured to communicate with WTRU102a, 102b, and 102c in standalone and / or non-standalone configurations. In a standalone configuration, WTRU102a, 102b, and 102c can communicate with gNB180a, 180b, and 180c without accessing other RANs (e.g., eNode-B160a, 160b, and 160c). In a standalone configuration, WTRU102a, 102b, and 102c can use one or more of gNB180a, 180b, and 180c as mobility anchor points. In a standalone configuration, WTRU102a, 102b, and 102c can communicate with gNB180a, 180b, and 180c using signals in an unauthorized band. In a non-standalone configuration, WTRU102a, 102b, and 102c can communicate / connect with gNB180a, 180b, and 180c, while also communicating / connecting with other RANs such as eNode-B160a, 160b, and 160c. For example, WTRU102a, 102b, and 102c may implement DC principles for substantially simultaneous communication with one or more gNB180a, 180b, and 180c and one or more eNode-B160a, 160b, and 160c. In a non-standalone configuration, eNode-B160a, 160b, and 160c may function as mobility anchors for WTRU102a, 102b, and 102c, and gNB180a, 180b, and 180c can provide additional coverage and / or throughput to service WTRU102a, 102b, and 102c.

[0061]

[0086] Each of the gNB180a, 180b, and 180c can be associated with a specific cell (not shown) and may be configured to handle wireless resource management decisions, handover decisions, user scheduling in UL and / or DL, support for network slicing, dual connectivity, interaction between NR and E-UTRA, routing of user plane data to User Plane Functions (UPFs) 184a and 184b, routing of control plane information to Access and Mobility Management Functions (AMFs) 182a and 182b, etc. As shown in Figure 1D, the gNB180a, 180b, and 180c can communicate with each other via the Xn interface.

[0062]

[0087] The CN115 shown in Figure 1D may include at least one AMF182a, 182b, at least one UPF184a, 184b, at least one Session Management Function (SMF)183a, 183b, and optionally a Data Network (DN)185a, 185b. Although each of the above elements is depicted as part of CN115, it should be noted that any of these elements may be owned and / or operated by an entity other than the CN operator.

[0063]

[0088] AMF182a and 182b can connect to one or more gNB180a, 180b, and 180c within RAN113 via the N2 interface and function as control nodes. For example, AMF182a and 182b may be responsible for user authentication of WTRU102a, 102b, and 102c, support for network slicing (e.g., handling different PDU sessions with different requirements), selection of specific SMF183a and 183b, management of registration areas, termination of NAS signaling, mobility management, etc. Network slicing may be used by AMF 182a and 182b to customize CN support for WTRU102a, 102b, and 102c based on the type of services utilized by WTRU102a, 102b, and 102c. For example, different network slices may be established for different use cases, such as services that rely on ultra-reliable low latency (URLLC) access, services that rely on enhanced massive mobile broadband (eMBB) access, services for machine type communication (MTC) access, and / or similar. The AMF162 may provide control plane functionality for switching between RAN113 and other RANs (not shown) employing other radio technologies such as LTE, LTE-A, LTE-A Pro, and / or non-3GPP access technologies such as WiFi.

[0064]

[0089] SMF183a and 183b can be connected to AMF182a and 182b in CN115 via the N11 interface. SMF183a and 183b can also be connected to UPF184a and 184b in CN115 via the N4 interface. SMF183a and 183b can select and control UPF184a and 184b and configure the routing of traffic passing through UPF184a and 184b. SMF183a and 183b can perform other functions such as managing and assigning WTRU IP addresses, managing PDU sessions, controlling policies and QoS, and providing downlink data notifications. PDU session types may be IP-based, non-IP-based, Ethernet-based, etc.

[0065]

[0090] UPF184a and 184b can connect to one or more gNB180a, 180b, and 180c within RAN113 via the N3 interface, thereby providing WTRU102a, 102b, and 102c with access to packet-switched networks such as the Internet 110, facilitating communication between WTRU102a, 102b, and 102c and IP-enabled devices. UPF184 and 184b can perform other functions such as packet routing and forwarding, enforcement of user plane policies, support for multi-homed PDU sessions, handling user plane QoS, buffering downlink packets, providing mobility anchoring, and similar functions.

[0066]

[0091] CN115 can facilitate communication with other networks. For example, CN115 may include, or can communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that functions as an interface between CN115 and PSTN108. In addition, CN 115 may provide WTRU102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers. In one embodiment, WTRU102a, 102b, 102c may be connected to local data networks (DN) 185a, 185b via UPF184a, 184b through an N3 interface to UPF184a, 184b and an N6 interface between UPF184a, 184b and DN185a, 185b.

[0067]

[0092] Considering Figures 1A to 1D and their corresponding descriptions, one or more of the functions described herein with respect to one or more of the WTRU102a to d, base stations 114a to b, eNode-B160a to c, MME162, SGW164, PGW166, gNB180a to c, AMF182a to b, UPF184a to b, SMF183a to b, DN185a to b, and / or any other devices described herein, one or more of the functions described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more of the functions described herein. For example, an emulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0068]

[0093] Emulation devices may be designed to perform one or more tests on other devices in a laboratory and / or operator network environment. For example, one or more emulation devices may perform one or more or all functions when fully or partially implemented and / or deployed as part of a wired and / or wireless network to test other devices in a communications network. One or more emulation devices may perform one or more or all functions when temporarily implemented / deployed as part of a wired and / or wireless network. Emulation devices may be directly coupled to another device for testing purposes and / or perform tests using wireless communications.

[0069]

[0094] One or more emulation devices may perform one or more functions, including all functions, without being implemented / deployed as part of a wired and / or wireless communication network. For example, an emulation device may be used in a test scenario in a test laboratory and / or an undeployed (e.g., test) wired and / or wireless communication network to perform testing of one or more components. One or more emulation devices may be test equipment. Wireless communication via direct RF coupling and / or RF circuitry (e.g., which may include one or more antennas) may be used by an emulation device to transmit and / or receive data.

[0070]

[0095] This application describes various embodiments, including tools, features, examples or embodiments, models, and methods. Many of these embodiments are described in detail, often in a manner that may seem restrictive, at least in order to illustrate their individual characteristics. However, this is for the purpose of clarifying the description and not to limit the application or scope of those embodiments. In fact, all of the different embodiments can be combined and interchangeable to provide further embodiments. Moreover, these embodiments can also be combined and interchangeable with embodiments described in prior applications.

[0071]

[0096] The embodiments described and contemplated herein may be implemented in many different forms. Figures 5 to 23 described herein may provide several embodiments, but other embodiments are also contemplated. The consideration of Figures 5 to 23 is not limiting the scope of embodiments. At least one of the embodiments generally relates to video encoding and decoding, and at least one other embodiment generally relates to transmitting a generated or encoded bitstream. These and other embodiments may be implemented as a computer-readable storage medium storing methods, apparatus, instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium storing a bitstream generated according to any of the described methods.

[0072]

[0097] In this application, the terms “reconstructed” and “decoded” are interchangeable, the terms “pixel” and “sample” are interchangeable, and the terms “image,” “picture,” and “frame” are interchangeable.

[0073]

[0098] Various methods are described herein, each of which includes one or more steps or actions to achieve the described method. Unless a particular order of steps or actions is required for the proper operation of the method, the order and / or use of any particular steps and / or actions may be modified or combined. In addition, terms such as “first,” “second,” etc., may be used in various embodiments to modify elements, components, steps, operations, etc. (e.g., “first decryption” and “second decryption”). The use of such terms does not imply an ordering of modified operations unless specifically required. Thus, in this example, the first decryption does not need to be performed before the second decryption, but may be performed, for example, before the second decryption, during the second decryption, or during a period overlapping with the second decryption.

[0074]

[0099] The various methods and other embodiments described herein may modify (for example, be used for modification) the modules of the video encoder 200 and video decoder 300, as shown in Figures 2 and 3, respectively, for example, the pre-coding processing 201, intra-prediction 260, entropy coding 245 and / or the entropy decoding module 330, intra-prediction 360, and post-decoding processing 385. Furthermore, the subject matter disclosed herein presents embodiments that are not limited to VVC or HEVC and may apply, for example, to any type, format or version of video coding, whether existing or future, whether described in a standard or recommendation, and to extensions of any such standards and recommendations (including, for example, VVC and HEVC). Unless otherwise indicated or technically excluded, the embodiments described herein may be used individually or in combination.

[0075]

[0100] Various numerical values, such as minimum and maximum ranges (e.g., 0 to 1, 0 to N, or 0 to 255), bit values ​​for indication or determination, default values, ID numbers (e.g., for adaptive IDs), etc., are used in the examples described herein. These and other specific values ​​are provided for illustrative purposes only, and the embodiments described are not limited to these specific values.

[0076]

[0101] Figure 2 shows an example video encoder. Modifications of the example encoder 200 are intended, but encoder 200 is described below for clarity and not all expected modifications are described.

[0077]

[0102] Before encoding, the video sequence may undergo pre-encoding processing (201), such as applying a color conversion to the input color picture (e.g., from RGB4:4:4 to YCbCr4:2:0) or performing a remapping of the input picture components to obtain a signal distribution more resistant to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0078]

[0103] In encoder 200, the picture is encoded by encoder elements as described below. The picture to be encoded is divided into units of coding units (CUs) (202) and processed. Each unit is encoded using either intra-mode or inter-mode, for example. If a unit is encoded in intra-mode, it performs intra-prediction (260). In inter-mode, motion estimation (275) and compensation (270) are performed. The encoder decides whether to use intra-mode or inter-mode to encode a unit (205), and indicates the intra / inter decision by, for example, a prediction mode flag. For example, the prediction residual is calculated by subtracting the prediction block from the original image block (210).

[0079]

[0104] The predicted residual is then transformed (225) and quantized (230). The quantized transformation coefficients, as well as the motion vector and other syntax elements, are entropy coded (245) to output a bitstream. The encoder can skip the transformation and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transformation and quantization, i.e., the residual is coded directly without applying either the transformation or quantization process.

[0080]

[0105] The encoder decodes the encoded blocks to provide a reference for further prediction. The quantized transformation coefficients are inversely quantized (240) and inversely transformed (250) to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined (255) to reconstruct the image blocks. An in-loop filter (265) is applied to the reconstructed picture to perform deblocking / SAO (sample adaptive offset) filtering, for example, to reduce encoding artifacts. The filtered image is stored in a reference picture buffer (280).

[0081]

[0106] Figure 3 shows an example of a video decoder. In the example decoder 300, the bitstream is decoded by the decoder elements as described below. The video decoder 300 generally performs a decoding path in the reverse direction of the encoding path described in Figure 2. The encoder 200 can also generally perform video decoding as part of the encoding of video data. For example, the encoder 200 can perform one or more of the video decoding steps presented herein. The encoder reconstructs the decoded image to maintain synchronization with the decoder with respect to one or more of the following, for example: a reference picture, an entropy-encoded context, and other decoder-related state variables.

[0082]

[0107] In particular, the decoder input includes a video bitstream that may be generated by the video encoder 200. The bitstream is first entropy-decoded to obtain transformation coefficients, motion vectors, and other coded information (330). Picture segmentation information indicates how the picture is segmented. Thus, the decoder can segment the picture according to the decoded picture segmentation information (335). The transformation coefficients are inversely quantized (340) and inversely transformed (350) to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined (355) to reconstruct an image block. The prediction block can be obtained from intra-prediction (360) or motion-compensated prediction (i.e., inter-prediction) (375) (370). An in-loop filter (365) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (380).

[0083]

[0108] The decoded picture can then undergo further post-decoded processing (385), such as reverse color conversion (e.g., conversion from YCbCr4:2:0 to RGB4:4:4), or reverse remapping, which performs the reverse of the remapping process performed in pre-encoded processing (201). The post-decoded processing can use metadata derived in pre-encoded processing and transmitted in a bitstream.

[0084]

[0109] Figure 4 shows an example of a system in which various embodiments and models described herein may be implemented. System 400 may be embodied as a device including various components described below and configured to perform one or more of the embodiments described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected consumer electronics, and servers. The elements of System 400 may be embodied individually or in combination as a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one example, the processing elements and encoder / decoder elements of System 400 are distributed across multiple ICs and / or separate components. In various embodiments, System 400 is communicably coupled to one or more other systems or other electronic devices, for example, via a communication bus or via dedicated input and / or output ports. In various embodiments, System 400 is configured to perform one or more of the embodiments described herein.

[0085]

[0110] System 400 includes, for example, at least one processor 410 configured to execute loaded instructions in order to implement various embodiments described herein. The processor 410 may include embedded memory, input / output interfaces, and various other circuits known in the art. System 400 includes at least one memory 420 (e.g., a volatile memory device and / or a non-volatile memory device). System 400 includes a storage device 440, which may include non-volatile memory and / or volatile memory, and these memories may include, but are not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drives, and / or optical disk drives. The storage device 440 may, in non-limiting examples, include internal storage devices, mounted storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0086]

[0111] System 400 includes, for example, an encoder / decoder module 430 configured to process data and provide encoded or decoded video, the encoder / decoder module 430 of which may include its own processor and memory. The encoder / decoder module 430 represents a module that may be included in the device to perform encoding and / or decoding functions. As is well known, the device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 430 may be implemented as a separate element of System 400, or it may be incorporated into the processor 410 as a combination of hardware and software, as is known to those skilled in the art.

[0087]

[0112] Program code loaded onto the processor 410 or encoder / decoder 430 to perform the various embodiments described herein may be stored in the storage device 440 and subsequently loaded into the memory 420 for execution by the processor 410. According to various embodiments, one or more of the processor 410, memory 420, storage device 440, and encoder / decoder module 430 may store one or more of various items during the execution of the processes described herein. Such stored items include, but are not limited to, input video, decoded video or a portion of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0088]

[0113] In some embodiments, the memory inside the processor 410 and / or encoder / decoder module 430 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory of the processing device (for example, the processing device may be either the processor 410 or the encoder / decoder module 430) is used for one or more of these functions. The external memory may be memory 420 and / or storage device 440, such as dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used, for example, to store the operating system of a television. In at least one embodiment, a high-speed external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as MPEG-2 (MPEG stands for Moving Picture Experts Group, also known as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET, the Joint Video Experts Team).

[0089]

[0114] Inputs to the elements of System 400 can be provided via various input devices as shown in Block 445. Such input devices include, but are not limited to, (i) a radio frequency (RF) portion for receiving RF signals transmitted by broadcast by a broadcasting station, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Other examples, not shown in Figure 4, include composite video.

[0090]

[0115] In various embodiments, the input device of block 445 has associated input processing elements, as is well known in the Art. For example, the RF portion may be associated with elements suitable for (i) selecting a desired frequency (also called selecting a signal, or band-limiting a signal to a frequency in a certain band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band in order to select a signal frequency band, which in a particular example may be referred to as a channel, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and / or (vi) demultiplexing to select a desired data packet stream. The RF portion of various embodiments includes one or more elements for performing these functions, e.g., a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF portion may include a tuner for performing various of these functions, e.g., down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency near the baseband) or to the baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements receive an RF signal transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to a desired frequency band. Various embodiments may rearrange the order of the elements described above (and others), remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as inserting an amplifier and an analog-to-digital converter. In various embodiments, the RF section includes an antenna.

[0091]

[0116] In addition, the USB and / or HDMI terminals may include their respective interface processors for connecting the system 400 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or within the processor 410 as needed. Similarly, aspects of USB or HDMI interface processing may be implemented, for example, in a separate interface IC or within the processor 410 as needed. Demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 410 and an encoder / decoder 430, which operate in conjunction with memory and storage elements to process the data stream as needed for presentation to an output device.

[0092]

[0117] Various elements of system 400 may be housed within a single housing. Within the single housing, the various elements may be interconnected using a suitable connection mechanism 425, such as an internal bus known in the art, including an inter-IC (I2C) bus, wiring, and a printed circuit board, and data may be transmitted between them.

[0093]

[0118] System 400 includes a communication interface 450 that enables communication with other devices via a communication channel 460. The communication interface 450 may, but is not limited to, include transceivers configured to transmit and receive data via the communication channel 460. The communication interface 450 may, but is not limited to, a modem or a network card, and the communication channel 460 may be implemented, for example, in a wired and / or wireless medium.

[0094]

[0119] In various embodiments, data is streamed to system 400 or otherwise provided using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). In these examples, the Wi-Fi signal is received via a communication channel 460 and a communication interface 450 adapted for Wi-Fi communication. In these embodiments, the communication channel 460 is typically connected to an access point or router that provides access to an external network, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, the streamed data is provided to system 400 using a set-top box that distributes data via an HDMI connection in input block 445. In other embodiments, the streamed data is provided to system 400 using an RF connection in input block 445. As described above, various embodiments provide data in non-streaming manners. In addition, various embodiments use wireless networks other than Wi-Fi, e.g., cellular networks or Bluetooth networks.

[0095]

[0120] System 400 may provide output signals to various output devices, including a display 475, a speaker 485, and other peripheral devices 495. In various embodiments, the display 475 may include, for example, one or more of a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 475 may be for a television, tablet, laptop, mobile phone, or other device. The display 475 may also be integrated with other components (for example, in a smartphone) or separate (for example, an external monitor for a laptop). In various examples of embodiments, the other peripheral devices 495 may include one or more of a standalone digital video disc (or digital multipurpose disc) (both terms refer to DVD), a disc player, a stereo system, and / or a lighting system. Various embodiments utilize one or more peripheral devices 495 that provide functions based on the output of System 400. For example, a disc player performs the function of playing back the output of System 400.

[0096]

[0121] In various embodiments, control signals are transmitted between the system 400 and the display 475, speaker 485, or other peripheral devices 495 using signal transmission such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable control between devices with or without user intervention. Output devices may be communicably coupled to the system 400 via dedicated connections through their respective interfaces 470, 480, and 490. Alternatively, output devices may be connected to the system 400 via the communication interface 450 using a communication channel 460. The display 475 and speaker 485 may be integrated into a single unit with other components of the system 400 in an electronic device such as a television. In various embodiments, the display interface 470 includes a display driver, such as a timing controller (TCon) chip.

[0097]

[0122] For example, if the RF portion of input 445 is part of a separate set-top box, the display 475 and speaker 485 can, alternatively, be separated from one or more of the other components. In various embodiments where the display 475 and speaker 485 are external components, the output signal may be provided via a dedicated output connection, such as an HDMI port, a USB port, or a COMP output.

[0098]

[0123] The embodiments may be implemented by a processor 410 or other hardware, or by computer software implemented by a combination of hardware and software. In non-limiting examples, embodiments may be implemented by one or more integrated circuits. The memory 420 may be of any type appropriate for the technical environment and may be implemented using any suitable data storage technology, such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory, in non-limiting examples. The processor 410 may be of any type appropriate for the technical environment and may include, in non-limiting examples, one or more of a microprocessor, a general-purpose computer, a dedicated computer, and a processor based on a multi-core architecture.

[0099]

[0124] Various embodiments include decoding. As used in this application, “decoding” may encompass all or part of the processes performed on a received encoded sequence to produce, for example, a final output suitable for display. In various embodiments, such processes include one or more processes typically performed by a decoder (e.g., entropy decoding, inverse quantization, inverse transform, and differential decoding). In various embodiments, in addition to or instead of such processes, such processes include receiving a Real-Time Control Protocol (RTCP) Extended Report (XR) packet having an Extended Report Block, wherein the Extended Report Block includes a field representing timing information, a block length indicator, an identifier of the source of the Real-Time Protocol (RTP) data packet, media data, and a timestamp for synchronizing QoE timing information recorded on a server; extracting timing information from the Extended Report Block; and calculating a Quality of Experience (QoE) metric based on the extracted timing information.

[0100]

[0125] In further embodiments, in one example, “decoding” refers only to entropy decoding; in another embodiment, “decoding” refers only to differential decoding; and in yet another embodiment, “decoding” refers to a combination of entropy decoding and differential decoding. Whether the expression “decoding process” is intended to specifically refer to a subset of operations or to the broader decoding process as a whole will become clear from the context of the specific description and should be well understood by those skilled in the art.

[0101]

[0126] Various embodiments include encoding. Similar to what is discussed above with respect to "decoding," as used in this application, "encoding" may encompass all or part of the processes performed on an input video sequence to generate an encoded bitstream. In various embodiments, such processes typically include one or more processes performed by the encoder (e.g., splitting, differential coding, transformation, quantization, and entropy coding). In various embodiments, in addition to or instead of such processes, such processes include processes performed by the encoder of the various embodiments described in this application, for example, determining a Real-Time Control Protocol (RTCP) Extended Report (XR) packet having an Extended Report Block, wherein the Extended Report Block includes fields representing timing information, a block length indicator, an identifier for the destination of the Real-Time Protocol (RTP) data packet, a timestamp for synchronizing media data, and QoE timing information to the device; embedding the timing information into the Extended Report Block; and transmitting a Quality of Experience (QoE) metric based on the embedded timing information.

[0102]

[0127] As further examples, in one embodiment, “encoding” refers only to entropy coding; in another embodiment, “encoding” refers only to differential coding; and in yet another embodiment, “encoding” refers to a combination of differential coding and entropy coding. Whether the expression “encoding process” is intended to specifically refer to a subset of operations or to the broader encoding process as a whole will become clear from the context of the specific description and should be well understood by those skilled in the art.

[0103]

[0128] It should be noted that the syntax elements used herein, such as those indicated in the descriptions or figures presented herein, are descriptive terms. Therefore, the use of other syntax element names is not excluded.

[0104]

[0129] When a diagram is presented as a flow chart, it should be understood that the diagram also provides a block diagram of the corresponding device. Similarly, when a diagram is presented as a block diagram, it should be understood that the diagram also provides a flow chart of the corresponding method / process.

[0105]

[0130] During the coding process, a balance or trade-off between rate and distortion is usually considered, often given the constraints of computational complexity. Rate-distortion optimization is typically formulated to minimize a rate-distortion function, which is a weighted sum of rate and distortion. There are various methods for solving rate-distortion optimization problems. For example, a method may be based on extensive testing of all coding options, including all considered mode or coding parameter values, along with a complete evaluation of the coding cost and associated distortion of the reconstructed signal after coding and decoding. Faster methods may also be used, in particular, that use approximate distortion calculations based on the predicted or predicted residual signal rather than the reconstructed signal, in order to reduce coding complexity. These two methods can also be used in combination, for example, by using approximate distortion for only some of the possible coding options and full distortion for others. Other methods evaluate only a subset of the possible coding options. More generally, many methods employ one of various techniques for performing optimization, but the optimization is not necessarily a complete evaluation of both the coding cost and associated distortion.

[0106]

[0131] The embodiments and aspects described herein may be implemented, for example, as methods or processes, apparatus, software programs, data streams, or signals. Even if considered only in the context of a single embodiment (for example, discussed only as a method), the implementation of the features under consideration may also be implemented in other forms (for example, apparatus or programs). Apparatus may be implemented, for example, in the form of appropriate hardware, software, and firmware. Methods may be implemented, for example, in a processor, where processor generally refers to processing devices including, for example, computers, microprocessors, integrated circuits, or programmable logic devices. Processors also include communication devices such as, for example, computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate the communication of information between end users.

[0107]

[0132] The terms "one embodiment," "an embodiment," "an example," "one implementation," or "an implementation," as well as references to other variations thereof, mean that the specific features, structures, characteristics, etc. described in relation to the embodiments are included in at least one embodiment. Therefore, the appearance of the expressions "in one embodiment," "in an embodiment," "in an example," "in one implementation," or "in an implementation," as well as any other variations, in various places throughout this application, does not necessarily all refer to the same embodiment or example.

[0108]

[0133] In addition, this application may refer to "determining" various types of information. Identifying information may include, for example, one or more of the following: estimating information, calculating information, predicting information, or retrieving information from memory. Acquiring may include receiving, retrieving, constructing, generating, and / or identifying.

[0109]

[0134] Furthermore, this application may refer to "accessing" various types of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, computing information, identifying information, predicting information, or estimating information.

[0110]

[0135] In addition, this application may refer to "receiving" various types of information. Receiving is intended to be a broad term, similar to "accessing." Receiving information may include, for example, accessing information or retrieving information (for example, from memory) one or more of these. Furthermore, "receiving" typically involves in some way during operations such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0111]

[0136] Please be aware that the use of any of the following " / ", "and / or", and "at least one of" is intended to include, for example, "A / B", "A and / or B", and "at least one of A and B", which encompass the selection of only the first enumerated option (A), or only the second enumerated option (B), or both options (A and B). As further examples, in the case of "A, B, and / or C" and "at least one of A, B, and C", such phrases are intended to include the selection of only the first enumerated option (A), or only the second enumerated option (B), or only the third enumerated option (C), or only the first and second enumerated options (A and B), or only the first and third enumerated options (A and C), or only the second and third enumerated options (B and C), or all three options (A, B, and C). This may be extended to a number of items listed, as will be obvious to those skilled in the art in this and related fields.

[0112]

[0137] Furthermore, as used herein, the term “signal” refers, in particular, to indicating something to the corresponding decoder. For example, in some embodiments, the encoder signals (to the decoder, for example) descriptors, elements and attributes, metadata, schemas, etc., including MPDs, adaptive sets, representations, pre-selections, G-PCC components, G-PCC component descriptors, G-PCC descriptors or required property descriptors, supplemental property descriptors, G-PCC tile inventory descriptors, G-PCC static spatial region descriptors, GPCCTileId descriptors, and GPCC3DRegionID descriptors, among others (as disclosed herein). In this way, in one embodiment, the same parameters are used on both the encoder and decoder sides. Thus, for example, the encoder can transmit certain parameters to the decoder so that the decoder can use the same particular parameters (explicit signaling). Conversely, if the decoder already has certain parameters and other parameters, signaling can be used without transmission, simply to allow the decoder to know and select certain parameters (implicit signaling). Bit saving is achieved in various embodiments by avoiding the transmission of any actual functions. It should be understood that signal transmission can be achieved in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., are used to signal information to the corresponding decoder. The above concerns the verb form of the word “signal,” but the word “signal” may also be used as a noun in this specification.

[0113]

[0138] As will be apparent to those skilled in the art, multiple embodiments can produce a variety of signals formatted to carry information that can be stored or transmitted, for example. This information may include, for example, instructions for performing a method, or data generated by one of the embodiments described. For example, a signal may be formatted to carry a bitstream from one of the embodiments described above. Such a signal may be formatted, for example, as an electromagnetic wave (using, for example, the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave using the encoded data stream. The information carried by the signal may be, for example, analog or digital information. The signal may be transmitted, as is well known, through a variety of different wired or wireless links. The signal may be stored in a processor-readable medium.

[0114]

[0139] The following acronyms and abbreviations may be used in this specification. AVP Audio-Visual Profile AVPF (Audio-Visual Profile with Feedback) BT Block Type FB Feedback FCI Feedback Control Information FMT format type HE Header Extension LSW (Left-Side Word) MAF Media Access Function Media features for MeCAR AR MSW Top-level word NTP Network Time Protocol PDU Packet Data Unit PSFB payload-specific feedback PD (Persistent Document) PT Payload Type QoE (Quality of Experience) RTP (Real-Time Protocol) RTCP Real-Time Control Protocol RTSP Real-Time Streaming Protocol SAP Session Announcement Protocol SDP Session Description Protocol SRS Split Rendering Server SRTP Secure Real-Time Protocol SSRC synchronization source TR Technical Report TS Technical Specifications UE User Equipment URN Uniform Resource Name VoIP (Voice Over IP) XR Enhanced Reports

[0115]

[0140] The timestamps capturing events can occur in the WTRU or on the device, and split rendering capabilities may be provided as described herein. Using the time information recorded in the output of the split rendering server, it is possible to measure, for example, server processing delay and application delay excluding server processing delay (e.g., overall application delay).

[0116]

[0141] QoE timing information may be provided as described herein. A MeCAR device may send a group of pose information to a split rendering function (e.g., a split rendering function on a server) to generate a rendered media frame based on the transmitted poses. A pose may be associated with time metadata such as the time the pose estimation was performed (T1), the estimated target display time of the content (T2 estimation), and the time the group of poses was transmitted (T1').

[0117]

[0142] The gap between the actual target display time (T2 actual) and the pose estimation time (T1) can be the delay from pose to rendering and then to the photon, which may allow the MeCAR device to know the amount of processing time and connection delay for the split rendering loop. The next pose estimation round may refer to the delay from pose to rendering and then to the photon for the estimation of the new T2 estimate.

[0118]

[0143] The server's split rendering function may refer to T1'. T1' may be the time when a group of poses is sent from the device if multiple pairs of poses and metadata for the same target display time are received from the device. T1' information may be used by the server to manage poses, for example, to allow the MeCAR device to update previous estimates by retransmitting new poses for the same estimated target display time.

[0119]

[0144] The server's split rendering function can transmit rendered media frames and associated metadata. The metadata may include poses used in the rendered frames and corresponding time information such as T1, T2, etc., and may include the time rendering began (T3). This metadata can be transmitted to the MeCAR device (e.g., all) to measure the delay QoE metric from rendering to photon.

[0120]

[0145] (T5) may be time information in the output of the Split Rendering Server (SRS). The (T5) timestamp may be used to measure the server processing delay (T3-T5) and / or the overall application delay excluding the server processing delay.

[0121]

[0146] Downlink delay can be measured using T5 and the time the data is received by the WTRU's Media Access Function (MAF), while uplink delay can be measured as (T1'-T3).

[0122]

[0147] For example, if poses are stacked in the server's pose buffer at a finer granularity than the device's supported frame rate, the split rendering feature can select the pose closest to the display time, based on the delay from the previous rendering to the photon. The delay from the previous rendering to the photon relative to the most recent frame information can help the server make this selection.

[0123]

[0148] Figure 5 shows an example of the call flow for pause information.

[0124]

[0149] The features described herein may be associated with pose information delay and QoE. Timing metadata may be used to measure the following delays: delay from pose to rendering and to photon = T2 actual - T1; delay from rendering to photon = T2 actual - T3; and delay from pose to rendering = T3 - T1.

[0125]

[0150] By using a history of delay measurements (e.g., all of them), the application can estimate the delay of the next pose and the frame to be rendered.

[0126]

[0151] Pose and timestamp information from the device may include: a predicted pose that may include position and orientation information; (T1): the time when the pose estimation was performed; (T2 estimation): the estimated target display time of the media frame that can be rendered using this pose; (T1'): the actual time when the pose or group of poses was sent from the device to the split rendering server; and from previous rendering to photon: the delay from rendering of the latest frame to photon (T2 actual - T3).

[0127]

[0152] The pose and timestamp information associated with a rendered media frame from the split rendering server may include: the pose used for rendering; (T1): the time when pose estimation was performed; (T2 estimation): the estimated target display time for the media frame being rendered; (T3): the actual time when the renderer on the split rendering server begins rendering the relevant media frame; T5: the time when the rendered media frame is output from the split rendering server.

[0128]

[0153] User interaction and QoE timing information may be used herein. In a use case of a shared interactive immersive service, user interaction may be transmitted from the WTRU to the server. The server may process user requests to the immersive media scene (e.g., by contextual changes such as translation, rotation, or scaling, or by adding new objects to the scene). In edge-assisted WTRU types, the WTRU can offload scene rendering to a split-rendering server, which can rasterize the augmented reality viewport and pre-rendered images to generate augmented reality media that can be encoded and delivered to the WTRU.

[0129]

[0154] In interactive and immersive services, a parameter for estimating user experience quality may be round-trip interaction delay. Round-trip interaction delay can be the sum of content elapsed time and user interaction delay.

[0130]

[0155] User interaction QoE metrics can be described herein. User interaction delay is the period between the moment a user action is initiated and the time when such action is considered by the content creation engine. User interaction delay can be affected by wireless network uplink latency.

[0131]

[0156] Content elapsed time is the period between the moment the content is created and the moment it is presented to the user. Content elapsed time can be affected by downlink latency in wireless networks.

[0132]

[0157] Action and timestamp information from the device may include the following: Action information: User action information grouped into action sets. Actions may have a unique identifier. lastChangeTime: The time when the user action was performed. This may correspond to the lastChangeTime field in the action information, which is defined as the timestamp of the last change to the action's state.

[0133]

[0158] Action and timestamp information associated with rendered media frames from the split rendering server may include the following: Action identifier: An identifier for the action processed by the scene manager and rendered in the associated media frame; sceneUpdateTime(T6): The time at which the scene manager processes the interaction task according to the action described in the action message from the WTRU and updates the scene. The interaction pipeline procedure is shown in Figure 5.

[0134]

[0159] Figure 6 shows an example of a user action call flow. Using timestamps from the SRS and WTRU (e.g., all), the application can calculate the interaction delay as follows: User interaction delay = sceneUpdateTime(T6) - lastChangeTime; Content elapsed time = T2 actual - sceneUpdateTime(T6); and / or round-trip interaction delay = T2 actual - lastChangeTime.

[0135]

[0160] Delivery reports can be sent to the WTRU using RTCP messages. There are several ways in which various control information can be transmitted using RTCP packets. These may include profile-specific extensions (PT=200) and receiver reports (PT=201) to the sender; application-defined RTCP packets with payload type equal to 204 (PT=204); extension reports (XR) with payload type equal to 207 (PT=207); generic RTP feedback reports with payload type equal to 205 (RTPFB, PT=205); and / or payload-specific RTCP feedback messages with payload type equal to 206 (PSFB, PT=206).

[0136]

[0161] Figure 7 shows an example of the Real-Time Control Protocol (RTCP) Extended Report (XR) packet format. An XR packet may contain two 32-bit word headers, followed by several (e.g., zero) extended report blocks. The packet type can be laid out in a manner consistent with RTCP packets, relating to version, packet type, and length information. Parameters related to the packet type may include: Version (V) [2 bits]: Identifies the RTP version. This specification applies to RTP version 2; Padding (P) [1 bit]: If the padding bit is set, the XR packet may include a (e.g., additional) padding octet at the end. The semantics of the field may be the same as those of the padding field in an SR packet, as described in the RTP specification RFC 3550; Spare: 5 bits. This field may be reserved for future definition. If no such definition exists, the bits in this field may be set to 0 (for example, must be set to 0) and may be ignored by the receiver (for example, must be ignored by the receiver); Packet Type (PT) [8 bits]: Contains the constant 207 to identify this as an RTCP XR packet. This value can be registered with IANA (Internet Assigned Numbers Authority); Length: 16 bits. The length of this XR packet, including the header and any padding, may be a 32-bit word minus 1; SSRC [32 bits]: Synchronization source identifier of the origin of this XR packet; Report Block [Variable Length]: 0 or more extended report blocks. According to the extended report block framework defined below, a block may consist of one or more 32-bit words (for example, must consist of one or more 32-bit words).

[0137]

[0162] The features described herein may be associated with an extended report block framework. Extended report blocks can be stacked sequentially at the end of an XR packet. The block type value field (e.g., BT) can identify the block format, and its namespace may be managed by IANA.

[0138]

[0163] The length of individual blocks can be a multiple of 4 octets. The length field in the XR header can describe the total length of the packet, including the extended report block. Blocks may have block type and length fields to facilitate parsing. Receiving applications can demultiplex blocks based on their type; for example, if there are unrecognized block types, the length information can be used to find each consecutive block.

[0139]

[0164] The following seven extended report blocks may be described: block type for reporting received packet loss and duplication, packet reception time, receiver reference time information, receiver inter-report delay, detailed reception statistics, and Voice over IP (VoIP) metrics.

[0140]

[0165] An extended report block may have the format shown in Figure 8. Figure 8 shows an example of the RTCP XR block format. The following parameters may be included: Block type (BT) [8 bits]: May identify the block format. Seven block types may be described. The namespace of the fields may be managed by IANA (Internet Assigned Numbers Authority); Type specific [8 bits]: Their use depends on the definition of the block type; Block length [16 bits]: The length of this report block, including the header, may be 32 bits minus 1. If a block type definition is permitted, zero may be an acceptable value, consisting of the BT, type specific, and block length fields, indicating that the type specific block content field is null; and / or type specific block content [variable length]: The use of this field may be defined by a particular block type, provided that its length is a multiple of 32 bits (for example, it must be a multiple of 32 bits). If a block type definition is permitted, it may be zero bits long.

[0141]

[0166] The features described herein may be associated with NTP timestamps. Wall clock time (e.g., absolute date and time) may be expressed using the Network Time Protocol (NTP) timestamp format, which may be the number of seconds from 0:00 UTC on January 1, 1900. A full-resolution NTP timestamp may be a 64-bit unsigned fixed-point number, where the first 32 bits of the integer part are called the most significant word (MSW) and the last 32 bits of the fractional part are called the least significant word (LSW). In fields where compact representation is appropriate, the middle 32 bits may be used. The lower 16 bits of the integer part and the upper 16 bits of the fractional part may be used. The upper 16 bits of the integer part may be determined independently.

[0142]

[0167] In some cases, an implementation of the Network Time Protocol (RTP) may not be necessary to use RTP. Other time sources may be used, or no time source may be used at all. NTP can be useful for synchronizing streams sent from another host.

[0143]

[0168] The features described herein may be associated with SDP signaling. When initiating a multimedia video conference, VoIP call, streaming video, or other session, there may be parameters that convey media details, transport address, and (other) session description metadata to participants. SDP may provide a standard representation of such information, regardless of how that information may be transmitted. SDP may be in the form of a session description—which may not have an embedded transport protocol and may, if necessary, be intended to use different transport protocols such as a session announcement protocol, session initiation protocol, real-time streaming protocol, or hypertext transport protocol.

[0144]

[0169] The description of an SDP session may include: the session name and purpose, the duration of the session's activity, the media that constitutes the session, and / or the information required to receive that media (such as address, port, and format).

[0145]

[0170] Because resources for participating in a session may be limited, the following information may be desirable: information about the bandwidth used by the session and / or contact information of the person in charge of the session.

[0146]

[0171] SDP can communicate enough information to allow applications to join a session (perhaps excluding encryption keys, for example) and to announce resources used by non-participants (if they need to know).

[0147]

[0172] Applications utilizing SDP may employ Session Description Protocol (SDP) signaling in the XR block. Signaling may be defined for use by applications implementing the SDP offer / answer model, or by applications using SDP to describe media and transport settings in relation to protocols such as Session Announcement Protocol (SAP) or Real-Time Streaming Protocol (RTSP). There may be signaling methods not described herein.

[0148]

[0173] XR blocks can be used without prior signaling. This may be consistent with the rules governing RTCP packet types. An example where signaling may not be required is an application that uses (e.g., requires) one or more XR blocks. For applications configured at session initiation, the use of some type of signaling may be recommended. The use of SDP signaling to XR blocks may be optional, but if used, it may (e.g., must) be used as described herein. If SDP signaling is used in an environment where XR blocks are implemented by some participants, those that do not implement XR blocks may ignore the SDP attribute.

[0149]

[0174] The features described herein may be associated with RTCP feedback messages. Real-time media streams using RTP may be tolerant of packet loss. The receiver may report packet reception statistics using the basic mechanisms of the Real-Time Transport Control Protocol (RTCP), allowing the sender to adapt transmission behavior in the medium term. This may be a means for feedback and feedback-based error remediation (apart from, for example, codec-specific mechanisms). Extensions of the Audiovisual Profile (AVP) may enable the receiver to provide immediate feedback to the sender, for example, statistically, enabling the implementation of short-term adaptation and efficient feedback-based remediation mechanisms.

[0150]

[0175] Payload format-specific SDP attributes may be written to indicate the ability to use RTCP feedback. For example, "a=rtcp-fb". The "rtcp-fb" attribute can be used as an SDP media attribute and may not be provided at the session level. The "rtcp-fb" attribute can be used in media sessions where "AVPF" is specified. The "rtcp-fb" attribute can be used to indicate which RTCP FB messages are available in a media session of a specified payload type. A wildcard payload type ("*") can be used to indicate that the RTCP feedback attribute applies to (e.g., all) payload types. The "a=rtcp-fb" line may be used if the feedback type is supported and / or the same feedback is specified for a subset of payload types.

[0151]

[0176] Payload-specific FB messages can transfer information specific to a particular payload type and may be generated and operated at the codec layer. The packet format of an RTCP feedback message may be as follows in Figure 9. Figure 9 shows an example of an RTCP feedback message packet format.

[0152]

[0177] The PT (Payload Type) field can identify a packet as an RTCP FB message. Two values ​​may be defined by IANA: PT=205 for RTPFB (Transport Layer FB Message) and PT=206 for PSFB (Payload-Specific FB Message). The FMT (Feedback Message Type) field can identify the type of FB message and can be interpreted in relation to the payload type (e.g., RTPFB or PSFB). The FMT values ​​for both RTPFB and PSFB payload types may be managed by IANA.

[0153]

[0178] The features described herein may be associated with application-defined RTCP packets. APP packets may be intended for experimental use during application and functionality development without requiring registration of packet type values. APP packets whose names are not recognized may be ignored. After testing, if a broader use is justified, APP packets can be redefined without using subtype and name fields and registered with IANA using RTCP packet types.

[0154]

[0179] Figure 10 shows an example of the packet format of an application-defined RTCP packet.

[0155]

[0180] The semantics of an APP packet can be as follows: Version (V): [2 bits] Identifies the RTP version, which in RTCP packets may be the same as the RTP data packet. The described version can be 2; Padding (P): [1 bit] If the padding bit is set, this individual RTCP packet may contain a padding octet at the end, which may not be part of the control information and may be included in the length field; Length: [16 bits] The length of this RTCP packet in a 32-bit word including the header and any padding, minus 1 (for example, an offset of 1 can make the effective length zero, avoiding an infinite loop that may occur when scanning composite RTCP packets, while counting the 32-bit word can avoid validity checks that are multiples of 4); Subtype: [5 bits] Can be used as a subtype to allow a set of APP packets to be described by a single unique name, or in the case of application-dependent data; Packet Type (PT): [8 bits] Contains the constant 204 to identify this as an RTCP app packet; Name: 4 octets. A name chosen by the person defining the set of application packets to be unique to any other application packets that this application may receive. The application creator may choose to use the application name and coordinate the assignment of subtype values ​​to other users who want to describe new packet types for the application. Names may be chosen (e.g., by others) based on the entity they represent, and the use of that name may be coordinated within that entity. Names can be interpreted as a sequence of four ASCII characters, case-sensitive; and / or application-dependent data: [variable length] Application-dependent data may or may not appear in APP packets. It may be interpreted by the application, not by the RTP itself. It may be (e.g., must be) a multiple of 32 bits in length.

[0156]

[0181] Timing information data can be used to measure QoE metrics for pose estimation and user interaction delay. Interaction experience quality (QoE) may depend on round-trip interaction delay, which can be the sum of content elapsed time and user interaction delay.

[0157]

[0182] User interaction delay, content elapsed time, and round-trip interaction delay measurements can be experience quality metrics for augmented reality content. Delay measurement metrics can be calculated in WTRU to provide end users with a satisfactory (e.g., better) QoE. Server processing delay measurements can support WTRU in the fusion process with split-rendering servers to achieve a better QoE.

[0158]

[0183] Using this technique, time information recorded at the start of rendering on the Split Rendering Server (SRS) can be sent to the Workstream Rendering Unit (WTRU) in the output of the SRS, allowing the WTRU to measure the server processing delay. Using this technique, scene update times recorded on the SRS can be sent to the WTRU. Scene update times can be used to measure user interaction delays, content elapsed time, and round-trip interaction delays on the WTRU or MeCAR device.

[0159]

[0184] Timing information recorded by a split rendering server or any RTP sending entity can be transmitted to a WTRU or MeCAR device using the following techniques: transmission using RTCP Extended Report (XR) packets, transmission using RTCP feedback messages, and / or transmission using the RTP header extension mechanism.

[0160]

[0185] In this example, timing information for measuring QoE metrics can be sent to a WTRU or MeCAR device using RTCP messages by extending the RTCP Extended Reporting (XR) framework mechanism.

[0161]

[0186] In the example, timing information data can be transmitted by extending an RTCP feedback message with a payload type equivalent to "Payload-Specific Feedback (PSFB)" (206).

[0162]

[0187] The frequency of RTCP messages (XR packets or RTCP feedback) for transmitting QoE timing information may be negotiated during the configuration phase or during the RTP session establishment time.

[0163]

[0188] In the example, timing information data for measuring the QoE metric may be sent to a WTRU or MeCAR device using the RTP header extension method. The RTP header extension method can provide an option to associate a frame rendered by a split rendering server with the recorded timing information of that frame for measuring the QoE metric when it is carried as part of an RTP packet carrying the rendered image of the frame. The frequency of RTP header extensions for sending timing information may be once per frame / PDU set. Header extensions can be declared in the SDP using the "a=extmap" attribute. Header extensions can be identified through the association between the URI of the header extension and the ID value included as part of the extension. The following URN may be used for a QoE timing information header extension: "urn:3gpp:xr-qoe-time-info".

[0164]

[0189] RTCP messages can be used to transmit QoE timing information. Transmission may be RTCP XR packet-based. Timing information data recorded on the SRS or RTP sender can be sent to a WTRU or MeCAR device by modifying (e.g., extending) the RTCP XR packet. An RTCP XR report may be identified by a payload type (PT) equal to 207, which may refer to an extended report block message. When transmitting timing information data using an RTCP XR message, the block type (BT) can be extended to a value of 40.

[0165]

[0190] An extended report block may be for QoE timing information. The extended report block type may enable detailed reporting of timing information recorded in the split rendering server (SRS). The report may be used to calculate QoE metrics such as server processing delay, user interaction delay, content elapsed time, and round-trip interaction delay in WTRU or MeCAR devices.

[0166]

[0191] In the example, QoE timing information can be expressed in the same units as the RTP timestamp of the RTP data packet. For packets, a relationship can be established between the media data in transit and the corresponding QoE timing information recorded in the SRS for a particular media frame.

[0167]

[0192] When using RTCP bandwidth, the RTCP XR block payload may contain all or part of the timing information used to calculate the QoE metric. For example, if the bits in the time_info field are set to 1, the respective timing information may be present in the payload. For example, if the bits in the time_info field are set to zero, the respective time information may not be present in the payload. In the example, when the sender sends T1 and T3 information, the time_info field may be set to b0011, and the T1 and T3 information may be present in the message payload.

[0168]

[0193] Identifiers for the actions (e.g., all) processed to render a frame at a specific time may be reported in the “rendered pose” RTP header extension. This header extension can be identified using the “a=extmap” attribute URN: “urn:3gpp:xr-rendered-pose”. Synchronization between the various timing information present in the following XR report and the action identifiers present in the “rendered pose” RTP HE message may be performed using the RTP timestamp information present in the RTP header of the packet containing the “rendered pose” RTP HE and the RTP timestamp field present in the following XR report block.

[0169]

[0194] A QoE timing information report block with a BT value equal to 40 may have the following format, as shown in Figure 11.

[0170]

[0195] Figure 11 shows an example of Quality of Experience (QoE) timing information data or QoE timing information in RTCP XR block format.

[0171]

[0196] The semantics of fields in a QoE time information augmentation report (XR) block can be as follows: Block type (BT) [8 bits]: A QoE time information report block may be identified by the constant 40; time_info [4 bits]: These field bits may represent timestamps present in the XR report block. If T1 is present in the XR report, the first bit (e.g., the least significant bit) may be set to 1. If LSB is set to 0, T1 information may not exist. If T3, T5, and T6 may be present in the RTCP XR block data, bits 2, 3, and 4 may each be set to 1. If T1, T3, T5, and T6 are present in the RTCP XR block data, the value of the time_info field may be b1111. If timing information is present, the timing information may follow the order T1, T3, T5, and then T6. In the example, if the value of the time_info field is b0101, the XR block may carry T5 after the T1 information. Timing information for T3 and T6 may not be present in the XR block content. The transmission frequency of T1, T3, T5, and T6 time information for an XR report block may be negotiated during the configuration phase; resv[4 bits]: This field may be reserved for future definition. If no such definition exists, the bits in this field may be set to 0 (e.g., must be set to 0) or ignored by the recipient (e.g., must be ignored); block length[16 bits]: The length of this report block, including the header, obtained by subtracting 1 from a 32-bit word; SSRC of source: 32 bits. The SSRC of the RTP data packet source reported by this report block; RTP timestamp[32 bits]: This field may represent the RTP timestamp of the media frame on which the date of the corresponding QoE timing information was recorded in the SRS. This correspondence may be used to synchronize media data with QoE timing information measurements recorded in the SRS for a particular media frame; Estimated time (T1)[32 bits]: This field represents the time on which the pause estimation was performed.This time information may be expressed in the same units and with the same random offset as the RTP timestamp in the data packet; Rendering Start Time (T3) [32-bit]: This field represents the time when the split rendering server's renderer started rendering the relevant media frame. This time information may be expressed in the same units and with the same random offset as the RTP timestamp in the data packet; Split Rendering Server Output Time (T5) [32-bit]: This field represents the time recorded in the split rendering server output. This time information may be expressed in the same units and with the same random offset as the RTP timestamp in the data packet; and / or Scene Update Time (T6) [32-bit]: This field represents the time when the scene manager processes the interaction task and updates the scene according to the action in the action message from the WTRU. This time information may be expressed in the same units and with the same random offset as the RTP timestamp in the data packet.

[0172]

[0197] In the example, QoE timing information may be represented as an NTP timestamp. The NTP timestamp report block described herein may indicate the wall clock time when the corresponding timing information was recorded in the SRS or WTRU. The association between media data time and the corresponding QoE timing information recorded in the SRS for a particular media frame may be performed using the RTP timestamp field present in the extended report block shown in Figure 12.

[0173]

[0198] The QoE timing information report block may have a format as shown in Figure 12. Figure 12 shows an example of the XR block format of QoE timing information data with NTP timestamps.

[0174]

[0199] The semantics of the fields in a QoE time information augmentation report (XR) block can be as follows: Block type (BT) [8 bits]: A QoE time information report block may be identified by the constant 40; time_info [4 bits]: This field bit may represent a timestamp present in the XR report block. If a bit is set in this field, the respective timing information may be present in the payload. If a bit in this field is set to zero (0), the respective time information may not be present in the payload. In the example, if T1 is present in the XR report, the first bit (e.g., the least significant bit) may be set to 1. If LSB is set to 0, T1 information may not be present. If T3, T5, and T6 are present in the RTCP XR block data, bits 2, 3, and 4 may each be set to 1. If T1, T3, T5, and T6 are present in the RTCP XR block data, the value of the time_info field may be b1111. If timing information is present, the timing information may follow the order T1, T3, T5, and then T6. In the example, if the value of the time_info field is b0101, the XR block may carry T5 after T1 information. T3 and T6 timing information may not be present in the XR block content. The transmission frequency of T1, T3, T5, and T6 time information in the XR report block may be negotiated during the configuration phase; resv: 4 bits. This field may be reserved for future definition. If no such definition exists, the bits in this field may be set to 0 (e.g., must be set to 0) or may be ignored by the recipient (e.g., must be ignored); block length [16 bits]: the length of this report block including the header, obtained by subtracting 1 from a 32-bit word; source SSRC [32 bits]: the SSRC of the RTP data packet source reported by this report block; RTP timestamp [32 bits]: this field represents the RTP timestamp of the media frame on which the corresponding QoE timing information was recorded in the SRS.Correspondence can be used to synchronize media data with QoE timing information measurements recorded by SRS for a specific media frame; Estimated Time (T1) [64-bit]: This field represents the time when pose estimation was performed. This NTP timestamp field can be expressed in wall clock time units; Rendering Start Time (T3) [64-bit]: This field can represent the time when the split rendering server's renderer started rendering the relevant media frame. This NTP timestamp field can be expressed in wall clock time units; Split Rendering Server Output Time (T5) [64-bit]: This field can represent the time recorded in the split rendering server output. This NTP timestamp field can be expressed in wall clock time units; and / or Scene Update Time (T6) [64-bit]: This field represents the time when the scene manager processes an interaction task and updates the scene according to the action in the action message from the WTRU. This NTP timestamp field can be expressed in wall clock time units.

[0175]

[0200] For example, QoE timing information may be transmitted in different types of application scenarios, such as split-rendering server timing information or timing information recorded during viewport adaptive streaming. To enable the transmission of timing information of application scenario types, RTCP XR blocks may be extended to support application type indications. If timing information related to multiple types of applications is transmitted in an RTCP message, the application-related RTCP XR blocks may be stacked sequentially within the payload of the RTCP message (for example, as specified in RFC3611).

[0176]

[0201] A QoE timing information report block with a BT value equal to 40 (for example, as shown in Figure 12) can be extended with an application type field as shown in Figure 13.

[0177]

[0202] Figure 13 shows the RTCP XR block format for QoE timing information data. The semantics of the RTCP XR block format may be as follows: resv: 4 bits. This field may be reserved for future definition. If no such definition exists, the bits in this field may be set to 0 (e.g., must be set to 0) or ignored by the recipient (e.g., must be ignored). Type "4 bits": This bit field may represent the type of application scenario to which the QoE timing information is sent. A value of 1 may indicate that the QoE timing information sent as part of this RTCP XR block is for a split-rendering application. Other values ​​may be reserved for future modifications (e.g., enhancements) to support other types of applications.

[0178]

[0203] In the example, QoE timing information may be represented as an NTP timestamp. A QoE timing information report block with an application scenario type and an NTP timestamp may have a format as shown in Figure 14. Figure 14 shows the RTCP XR block format of QoE timing information data with an NTP timestamp.

[0179]

[0204] The semantics for the RTCP XR block format type field may be as follows: resv: 4 bits. This field may be reserved for future definition. If no such definition exists, the bits in this field may be set to 0 (e.g., must be set to 0) or may be ignored by the receiver (e.g., must be ignored). Type "4 bits": This bit field may represent the type of application scenario in which QoE timing information is sent. A value of 1 may indicate that the QoE timing information sent as part of this RTCP XR block is for a split-rendering application. Other types of applications may be for future modifications (e.g., enhancements).

[0180]

[0205] The features described herein may be associated with SDP signaling and attributes. The SDP attribute "rtcp-xr" can be used to signal to participants in a media session that a specified XR block is available. This attribute may be extensible with parameters to cover types of XR report blocks. Extended RTCP XR blocks with QoE time information SDP attributes may be described herein in Extended Backusnaur Notation (ABNF).

number

[0181]

[0206] The "rtcp-xr" attribute may contain zero, one, or two or more XR block-related parameters. These parameters can signal the functionality of an XR block or group of XR blocks. This attribute may be extensible to define parameters for future XR blocks. In the example, the parameters may be extensible to support the delivery of QoE timing information data over XR-type RTCP packets.

[0182]

[0207] The parameter names and their corresponding XR formats may be as follows:

[0183] [Table 1]

[0184]

[0208] The "qoE-timing-info" parameter can be an integer value. This value can indicate the maximum size, in octets, that the entire report block can have.

[0185]

[0209] The features described herein may be associated with RTCP feedback message-based transmission.

[0186]

[0210] Timing information recorded by an SRS or RTP sender may be transmitted to a WTRU or MeCAR device using RTCP feedback messages. RTCP feedback messages may also be identified by a payload type (PT) equal to 206, which may refer to a payload-specific feedback (PSFB) message. For timing information in QoE measurement feedback messages, the feedback message type (FMT) may be set to a value of 19. The RTCP feedback method and the RTCP XR blocking method may include transmitting timing information used to measure the QoE metric in both immediate feedback mode and early RTCP mode.

[0187]

[0211] In the example, QoE timing information can be expressed in the same units as the RTP timestamp of the RTP data packet. For packets, a relationship can be established between the media data in transit and the corresponding QoE timing information recorded in the SRS for a particular media frame.

[0188]

[0212] For (e.g., optimal) use of RTCP bandwidth, the RTCP message payload may contain all or part of the timing information necessary for calculating the QoE metric. If a bit in the time_info field is set to 1, the respective timing information may be present in the payload. If a bit in the time_info field is set to 0, the respective timing information may not be present in the payload. In the example, when the sender sends T1 and T3 information, the time_info field may be set to b0011, and the T1 and T3 information may be present in the message payload.

[0189]

[0213] The feedback control information (FCI) of an RTCP feedback message containing QoE timing information data may be in the format shown in Figure 15. Figure 15 shows an example of the RTCP feedback message format for QoE timing information data.

[0190]

[0214] The semantics of the fields in a QoE time information RTCP feedback message can be as follows: time_info[4 bits]: This bit field indicates the timestamp present in the RTCP feedback message. If a bit is set in this field, the respective timing information may be present in the payload. If a bit in this field is set to 0, the respective time information may not be present in the payload. In the example, if T1 is present in the RTCP feedback message, the first bit (e.g., the least significant bit) may be set to 1. If the LSB is set to 0, T1 information may not be present. If T3, T5, and T6 are present in the RTCP feedback message, bits 2, 3, and 4 may each be set to 1. If T1, T3, T5, and T6 are present in the RTCP feedback message, the value of the time_info field may be b1111. If timing information is present, the timing information may follow the order T1, T3, T5, and then T6. In the example, if the value of the time_info field is b0101, the RTCP feedback message may carry T5 after T1 information. T3 and T6 timing information may not be present in the RTCP feedback message. The frequency of transmission of T1, T3, T5, and T6 time information in the RTCP feedback message may be negotiated during the configuration phase; resv[4 bits]: This field may be reserved for future definition. If not defined, the bits in this field may (must) be set to 0 and may be ignored by the recipient (e.g., must be ignored); RTP timestamp[32 bits]: This field represents the RTP timestamp of the media frame on which the corresponding QoE timing information was recorded in the SRS. Correspondence may be used to synchronize media data with QoE timing information measurements recorded in the SRS for a particular media frame; Estimated time (T1)[32 bits]: This field may represent the time on which the pause estimation was performed.This time information may be expressed in the same units and with the same random offset as the RTP timestamp in the data packet; Rendering Start Time (T3) [32-bit]: This field represents the time when the split rendering server's renderer started rendering the relevant media frame. This time information may be expressed in the same units and with the same random offset as the RTP timestamp in the data packet; Split Rendering Server Output Time (T5) [32-bit]: This field represents the time recorded in the split rendering server output. This time information may be expressed in the same units and with the same random offset as the RTP timestamp in the data packet; and / or Scene Update Time (T6) [32-bit]: This field represents the time when the scene manager processes the interaction task and updates the scene according to the action in the action message from the WTRU. This time information may be expressed in the same units and with the same random offset as the RTP timestamp in the data packet.

[0191]

[0215] QoE timing information can be represented as an NTP timestamp. The association between media data time and the corresponding QoE timing information metric recorded by the SRS for a particular media frame can be performed using the RTP timestamp field present in the RTCP feedback message, as shown below.

[0192]

[0216] The feedback control information (FCI) of an RTCP feedback message containing QoE timing information data may be as shown in Figure 16. Figure 16 shows an example of the RTCP feedback message format for QoE timing information data including a Network Time Protocol (NTP) timestamp.

[0193]

[0217] The semantics of the fields in a QoE time information RTCP feedback message can be as follows: time_info[4 bits]: This bit field indicates the timestamp present in the RTCP feedback message. If a bit is set in this field, the respective timing information may be present in the payload. If a bit in this field is set to 0, the respective time information may not be present in the payload. In the example, if T1 is present in the RTCP feedback message, the first bit (e.g., the least significant bit) may be set to 1. If the LSB is set to 0, T1 information may not be present. If T3, T5, and T6 are present, bits 2, 3, and 4 may each be set to 1. If T1, T3, T5, and T6 are present, the value of the time_info field may be b1111. If timing information is present, the timing information may follow the order T1, T3, T5, and then T6. In the example, if the value of the time_info field is b0101, the message may carry T5 after the T1 information. T3 and T6 timing information may not be present in that message. The transmission frequencies of T1, T3, T5, and T6 time information may be negotiated during the configuration phase; resv[4 bits]: This field may be reserved for future definition. If not defined, the bits in this field may (must) be set to 0 and may be ignored by the recipient (e.g., must be ignored); RTP timestamp[32 bits]: This field represents the RTP timestamp of the media frame on which the corresponding QoE timing information was recorded in the SRS. Correspondence may be used to synchronize media data with QoE timing information measurements recorded in the SRS for a particular media frame; Estimated time (T1)[64 bits]: This field represents the time when the pause estimation was performed. This NTP timestamp field may be expressed in wall clock time units; Rendering start time (T3)[64 bits]: This field represents the time when the renderer of the split rendering server started rendering the relevant media frame.This NTP timestamp field can be expressed in wall clock time units; Split Rendering Server Output Time (T5) [64 bits]: This field represents the time recorded in the output of the split rendering server. This NTP timestamp field can be expressed in wall clock time units; and / or Scene Update Time (T6) [64 bits]: This field represents the time it takes for the scene manager to process interaction tasks and update the scene in accordance with the action of the action message from the WTRU. This NTP timestamp field can be expressed in wall clock time units.

[0194]

[0218] In the example, a QoE timing information RTCP feedback message may include QoE timing information and indications of different types of applications (e.g., split rendering server timing information, timing information recorded during viewport adaptive streaming, etc.). If timing information related to multiple types of applications is being sent, these may be sent separately in multiple RTCP feedback messages.

[0195]

[0219] The feedback control information (FCI) and associated QoE timing information data of an RTCP feedback message with an application type may have the format shown in Figure 17. Figure 17 shows the RTCP feedback message format for QoE timing information data. In an example, the feedback control information (FCI) of an RTCP feedback message with application type information and QoE timing information represented using an NTP timestamp may have the format shown in Figure 18. Figure 18 shows the RTCP feedback message format for QoE timing information data including an NTP timestamp.

[0196]

[0220] RTCP feedback messages can be associated with SDP signaling. An RTP sender that supports the transmission of timing information data used for QoE metric measurement can signal SDP functionality for media content (e.g., all) where QoE metric measurement is used on a WTRU or MeCAR device. QoE timing information transmission support can be provided by including the a=rtcp-fb attribute in the relevant media line scope. QoE timing information transmission support using the RTCP feedback method can be expressed with the following parameters: QoE timing information. The wildcard payload type ("*") may be used to indicate that the RTCP feedback attribute for signaling QoE timing information transmission support applies to (e.g., all) payload types. The "a=rtcp-fb" line may be used when the same QoE timing information is specified for a subset of payload types. An example of using this attribute to signal QoE timing information distribution may be associated with a media line based on the RTCP feedback method: a=rtcp-fb:*QoE timing information.

[0197]

[0221] RTP header extensions can be associated with the transmission of QoE timing information. The RTP header extension mechanism can be extended to define new RTP header extensions and transmit QoE timing information to WTRU or MeCAR devices.

[0198]

[0222] To utilize the bandwidth of an RTP packet, the RTP HE payload may contain all or some of the timing information necessary to calculate the QoE metric. If the bit in the time_info field is set to 1, the respective timing information may be present in the payload. If the bit in the time_info field is set to zero (0), the respective time information may be present in the payload. In the example, when the sender sends T1 and T3 information, the time_info field may be set to b0011, and the T1 and T3 information may be present in the message payload.

[0199]

[0223] Identifiers of actions (e.g., all) processed to render a frame at a specific time may be reported in the “rendered pose” RTP header extension. The header extension may be identified using the “a=extmap” attribute URN: “urn:3gpp:xr-rendered-pose”. Synchronization between the various timing information present in the “QoE timg information” RTP HE and the action identifiers present in the “rendered pose” RTP HE message may be performed using RTP timestamp information present in the packet’s RTP header, including the “rendered pose” RTP HE and the “QoE timg information” RTP HE.

[0200]

[0224] In the example, QoE timing information may be expressed in the same units as the RTP timestamp. RTP header extensions can be defined in 1-byte and 2-byte extension formats. The syntax and semantics of a QoE timing information header extension may be as shown in Figure 19 and / or Figure 20. Figure 19 shows an example of an RTP header extension using the 1-byte header format. Figure 20 shows an example of an RTP header extension using the 2-byte header format.

[0201]

[0225] The semantics of the fields in QoE timing information RTP can be as follows: resv[4 bits]: This field may be reserved for future definition. If no such definition exists, the bits in this field may be set to 0 (e.g., must be set to 0) or may be ignored by the receiver (e.g., must be ignored); time_info[4 bits]: This bit field indicates the timestamp present in the RTP HE message. If a bit is set in this field, the respective timing information may be present in the payload. If a bit in this field is set to zero (0), the respective time information may not be present in the payload. In the example, if T1 is present in the RTP HE, the first bit (e.g., the least significant bit) may be set to 1. If the LSB is set to 0, the T1 information may not be present. If T3, T5, and T6 are present in the RTP HE data, bits 2, 3, and 4 may each be set to 1. If T1, T3, T5, and T6 are present in the RTP HE data, the value of the time_info field may be b1111. If timing information exists, it may follow the order T1, T3, T5, and then T6. In the example, if the value of the time_info field is b0101, the RTP may carry T5 after the T1 information. T3 and T6 timing information may not be present in the RTP HE. The frequency of transmission of T1, T3, T5, and T6 time information in the RTP may be negotiated during the configuration phase; Estimated time (T1) [32 bits]: This field represents the time when pose estimation was performed. The time information may be expressed in the same units and with the same random offset as the RTP timestamp in the RTP header; Rendering start time (T3) [32 bits]: This field represents the time when the split rendering server's renderer started rendering the relevant media frame. The time information may be expressed in the same units and with the same random offset as the RTP timestamp in the RTP header; Split rendering server output time (T5) [32 bits]: This field represents the time recorded in the split rendering server output.Time information may be expressed in the same units and with the same random offset as the RTP timestamp in the RTP header; and / or Scene Update Time (T6) [32 bits]: This field represents the time it takes for the Scene Manager to process an interaction task and update the scene in accordance with the action in the action message from the WTRU. This time information may be expressed in the same units and with the same random offset as the RTP timestamp in the RTP header.

[0202]

[0226] In the example, the QoE timing information transmitted in the RTP HE may be represented as an NTP timestamp. The following NTP timestamps in the RTP HE may indicate the wall clock time when the corresponding timing information was recorded in the SRS or WTRU. The RTP header extension can be defined in a 2-byte extension format. The syntax and semantics of the QoE timing information header extension may be as shown in Figure 21. Figure 21 shows an example of an RTP header extension using the 2-byte header format.

[0203]

[0227] The semantics of the fields in QoE timing information RTP HE can be as follows: resv[4 bits]: This field may be reserved for future definition. If no such definition exists, the bits in this field may be set to zero (0) (e.g., must be set to zero) and may be ignored by the receiver (e.g., must be ignored). time_info[4 bits]: This bit field indicates the timestamp present in the RTP HE message. If a bit is set in this field, the respective timing information may be present in the payload. If a bit in this field is set to zero (0), the respective time information may not be present in the payload. In the example, if T1 is present in RTP HE, the first bit (e.g., the least significant bit) may be set to 1. If LSB is set to 0, T1 information may not be present. If T3, T5, and T6 are present in the RTP HE data, bits 2, 3, and 4 may each be set to 1. If T1, T3, T5, and T6 are present in the RTP HE data, the value of the time_info field may be b1111. If timing information exists, it may follow the order T1, T3, T5, and then T6. In the example, if the value of the time_info field is b0101, the RTP may carry T5 after the T1 information. T3 and T6 timing information may not be present in the RTP HE. The frequency of transmission of T1, T3, T5, and T6 time information in the RTP is negotiated during the configuration phase. Estimated time (T1) [64 bits]: This field represents the time when the pause estimation was performed. This NTP timestamp field can be expressed in wall clock time units. Rendering start time (T3) [64 bits]: This field represents the time when the split rendering server's renderer started rendering the relevant media frame. This NTP timestamp field can be expressed in wall clock time units. Split rendering server output time (T5) [64 bits]: This field represents the time recorded in the split rendering server output.The NTP timestamp field may be expressed in wall clock time units; and / or scene update time (T6) [64 bits]: this field represents the time it takes for the scene manager to process the interaction task and update the scene in accordance with the action message from the WTRU. The NTP timestamp field may be expressed in wall clock time units.

[0204]

[0228] For example, QoE timing information may be sent to different types of applications (e.g., split rendering server timing information, timing information recorded during viewport adaptive streaming, etc.) using the RTP header extension mechanism. When timing information for different types of applications is sent, the RTP header extension may have multiple such entries successively, each with an appropriate length field value.

[0205]

[0229] RTP header extensions can include a 2-byte extension format. The syntax and semantics of a QoE timing information header extension with application type information may be as shown in Figure 22. Figure 22 shows an RTP header extension using the 2-byte header format.

[0206]

[0230] The semantics of the RTCP header extension type field may be as follows: resv: 4 bits. This field may be reserved (for example, for future definitions). If no such definition exists, the bits in this field may (for example, must be) set to 0 and may (for example, must be) ignored by the receiver; type "4 bits": This bit field may represent the type of application to which QoE timing information is being sent. A value of 1 indicates that the QoE timing information sent as part of the RTP header extension is for a split-rendering application. Other types of applications may be for future modifications (for example, enhancements).

[0207]

[0231] QoE timing information can be represented using NTP timestamps. An RTP header extension with timing information using application scenario types and NTP timestamps may include a 2-byte extension format. The syntax and semantics of a QoE timing information header extension can be shown in Figure 23. Figure 23 shows an RTP header extension using the 2-byte header format.

[0208]

[0232] SDP signaling can be associated with QoE timing information RTP HE. Support for transmitting timing information data recorded by the split rendering server or RTP sender can be signaled as an SDP extension. By signaling in the SDP message, the sender can provide information regarding support for transmitting timing information data required for QoE measurement in the initial offer-answer negotiation. This can be done by including the "a=extmap" attribute under the relevant media line. The mapping may be provided per media stream (in the media-level section of the SDP, i.e., after the "m=" line) or globally for the stream (e.g., all) (i.e., at the session level, before the first "m=" line).

[0209]

[0233] The URN corresponding to the QoE timing information RTP HE is urn:3gpp:params:rtp-hdrext:qoe-timing-info. An example of using this URN to signal QoE timing information transmitted in relation to a media line is as follows (for example, the signaling may be part of the media line): a=extmap:11 / sendonly urn:3gpp:params:rtp-hdrext:qoe-timing-info.

[0210]

[0234] In the example, the number 11 can be replaced with numbers in the range of 1 to 254 using the 2-byte header extension mechanism, and with numbers in the range of 1 to 15 using the 1-byte header extension mechanism.

[0211]

[0235] The definition can be either session-level (e.g., all) or media-level (e.g., all), and mixing the two styles may not be permitted. As described herein, the IDs used may be unique for a given media stream type or for a session in a session-level declaration.

[0212]

[0236] Systems, methods, and means may be configured for transmitting / receiving QoE timing information for measuring a Quality of Experience (QoE) metric. A device (e.g., a wireless / transmit / receiver unit (WTRU) and / or media function for an Augmented Reality (MeCAR) device) may receive a Real-Time Control Protocol (RTCP) Enhanced Report (XR) packet having an enhanced report block, the enhanced report block including fields representing timing information, a block length indicator, an identifier for the source of the Real-Time Protocol (RTP) data packet, a timestamp for synchronizing media data, and QoE timing information recorded on a server. The device may extract timing information from the enhanced report block. Based on the extracted timing information, the device may calculate a Quality of Experience (QoE) metric.

[0213]

[0237] A device may select and transmit timing information, provided that specific bits are configured in the field representing the timing information. The timing information may be expressed in the same units as the Real-Time Protocol (RTP) of RTP data packets, and synchronization may be performed using the timing information. The extended report block may include a timestamp indicating the time the timing information was recorded on the server. The device may adjust the use of Real-Time Control Protocol (RTCP) bandwidth by including timing information in the extended report block for calculating Quality of Experience (QoE) metrics.

[0214]

[0238] The extended report block may include a type field configured to indicate the application scenario associated with the QoE timing information. If multiple application scenarios are associated with the QoE timing information, each RTCP XR block associated with the application scenario may be included in the RTCP XR packet. The timing information from the extended report block may be represented using NTP timestamps.

[0215]

[0239] The system, method, and means may be configured to transmit time information data for measuring a Quality of Experience (QoE) metric. A server (e.g., a split rendering server and / or a Real-Time Protocol (RTP) server) may determine a Real-Time Control Protocol (RTCP) Extended Report (XR) packet having an Extended Report Block, the Extended Report Block including fields representing timing information, a block length indicator, an identifier for the destination of the Real-Time Protocol (RTP) data packet, a timestamp for synchronizing media data, and QoE timing information to be sent to a device. The device may embed timing information into the Extended Report Block. Based on the embedded timing information, the device may transmit a Quality of Experience (QoE) metric.

[0216]

[0240] A device may select timing information for embedding and transmission, provided that specific bits are configured in the fields representing timing information. The timing information may be expressed in the same units as the RTP in Real-Time Protocol (RTP) data packets, and synchronization may be performed using the timing information. The extended report block may include a timestamp indicating the time the timing information is sent to the server. The device can adjust the use of Real-Time Control Protocol (RTCP) bandwidth by including timing information for transmitting Quality of Experience (QoE) metrics in the extended report block.

[0217]

[0241] The extended report block may include a type field configured to indicate the application scenario associated with the QoE timing information. If multiple application scenarios are associated with the QoE timing information, each RTCP XR block associated with the application scenario may be included in the RTCP XR packet. The timing information from the extended report block may be represented using NTP timestamps.

[0218]

[0242] While features and elements are described above in specific combinations, those skilled in the art will recognize that each feature or element can be used individually or in any combination with other features and elements. In addition, the methods described herein may be implemented in computer programs, software, or firmware embedded in a computer-readable medium for execution by a computer or processor. Examples of computer-readable mediums include electronic signals (transmitted via wired or wireless connections) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks and digital multi-purpose disks (DVDs). The processor, together with the software, can be used to implement a radio frequency transceiver for use in a WTRU, WTRU terminal, base station, RNC, or any host computer.

Claims

1. A method performed by a device, Receiving a Real-Time Control Protocol (RTCP) message from a Split Rendering Server (SRS) that includes at least a Quality of Experience (QoE) Timing Information Enhancement Report (XR) block, wherein the QoE Timing Information XR block includes a block-type indication that the QoE Timing Information XR block contains QoE timing information, the QoE timing information is recorded in the SRS, and the QoE timing information includes at least an estimated time, a rendering start time, a split rendering server output time, and a scene update time. Calculate at least one QoE metric based on at least one of the following: the estimated time, the rendering start time, the split rendering server output time, or the scene update time. Methods that include...

2. The aforementioned method, Receiving a Session Description Protocol (SDP) which is a field indicating the use of an RCTP XR block for signaling the aforementioned QoE timing information, wherein the RTCP message is associated with the aforementioned use of the RCTP XR block. The method according to claim 1, further comprising:

3. The method according to claim 1, further comprising extracting the QoE timing information based on the block type indicating that the XR block contains the timing information.

4. The method according to claim 3, wherein the RTCP message includes a field indicating that one or more types of QoE timing information are present in the RTCP message, and an order associated with the one or more types of QoE timing information.

5. The method according to claim 1, wherein the device includes media functions for an augmented reality (MeCar) device.

6. The method according to claim 1, wherein the rendering start time indicates the time when the renderer associated with the SRS started rendering the media frame.

7. The method according to claim 1, wherein the estimated time indicates the time during which pose estimation was performed.

8. The method according to claim 1, wherein the split rendering server output time indicates the time associated with the output of the SRS.

9. The method according to claim 1, wherein the scene update time indicates the time at which the processing of one or more actions is initiated by the SRS.

10. The method according to claim 1, wherein the at least one QoE metric includes one or more of the following: round-trip interaction delay, server processing delay, user interaction delay, or content elapsed time.

11. The method according to claim 10, wherein the server processing delay is determined based on the sum of the elapsed time of the content and the user interaction delay.

12. The method according to claim 10, wherein the user interaction delay is the duration between a first time when a user action is initiated and a second time when the user action is considered by the content creation engine.

13. The method according to claim 12, wherein the user interaction delay depends on the uplink latency.

14. The method according to claim 10, wherein the elapsed time of the content is the duration between a first time when the content was created and a second time when the content was presented.

15. The method according to claim 10, wherein the elapsed time of the content or the user interaction delay is determined based on at least the scene update time.

16. The method according to claim 15, wherein the elapsed time of the content depends on the downlink waiting time.

17. The method according to claim 10, wherein the round-trip interaction delay is determined as the sum of the user interaction delay and the elapsed time of the content.

18. It is a device, Includes a processor, the processor is Receiving a Real-Time Control Protocol (RTCP) message from a Split Rendering Server (SRS) that includes at least a Quality of Experience (QoE) Timing Information Enhancement Report (XR) block, wherein the QoE Timing Information XR block includes a block-type indication that the QoE Timing Information XR block contains QoE timing information, the QoE timing information is recorded in the SRS, and the QoE timing information includes at least an estimated time, a rendering start time, a split rendering server output time, and a scene update time. Calculate at least one QoE metric based on at least one of the following: the estimated time, the rendering start time, the split rendering server output time, or the scene update time. A device configured to perform the following actions.

19. The aforementioned processor, The device according to claim 18, further configured to receive a session description protocol (SDP) which is a field indicating the use of an RCTP XR block for signaling the QoE timing information, wherein the RTCP message is associated with the use of the RCTP XR block.

20. The device according to claim 18, wherein the processor is further configured to extract the QoE timing information based on the block type indicating that the XR block contains the timing information.

21. The device according to claim 20, wherein the RTCP message includes a field indicating that one or more types of QoE timing information are present in the RTCP message, and an order associated with the one or more types of QoE timing information.

22. The device according to claim 18, wherein the device includes media functions for an augmented reality (MeCar) device.

23. The device according to claim 18, wherein the rendering start time indicates the time when the renderer associated with the SRS started rendering the media frame.

24. The device according to claim 18, wherein the estimated time indicates the time during which pose estimation was performed.

25. The device according to claim 18, wherein the split rendering server output time indicates the time associated with the output of the SRS.

26. The device according to claim 18, wherein the scene update time indicates the time at which the processing of one or more actions is initiated by the SRS.

27. The device according to claim 18, wherein the at least one QoE metric includes one or more of round-trip interaction delay, server processing delay, user interaction delay, or content elapsed time.

28. The device according to claim 27, wherein the server processing delay is determined based on the sum of the elapsed time of the content and the user interaction delay.

29. The device according to claim 27, wherein the user interaction delay is the duration between a first time when a user action is initiated and a second time when the user action is considered by the content creation engine.

30. The device according to claim 29, wherein the user interaction delay depends on the uplink latency.

31. The device according to claim 27, wherein the elapsed time of the content is the duration between a first time when the content was created and a second time when the content was presented.

32. The device according to claim 27, wherein the elapsed time of the content or the user interaction delay is determined at least based on the scene update time.

33. The device according to claim 32, wherein the elapsed time of the aforementioned content depends on the downlink waiting time.

34. The device according to claim 27, wherein the round-trip interaction delay is determined as the sum of the user interaction delay and the elapsed time of the content.

Images