Method for measuring and adjusting server processing task delays

Abstract

Systems and methods for high-granularity delay measurement and processing adjustment for augmented reality, extended reality, mixed reality, and / or virtual reality environments and systems are disclosed. In some embodiments, high-granularity processing task delay measurement may be performed individually or together by a client radio transmit / receive unit (WTRU) and an application server, and mitigation actions may be taken, including adjusting media quality and / or exchanging operational functions between the client and the server, to avoid individual task delays and improve perceived quality (QoE).

Description

Technical Field

[0001] (Cross - reference to Related Applications) This application claims the benefit and priority of European Patent Application No. 23305532.6, entitled "Methods for Server Processing Task Delays Measurements and Adjustments", filed on April 7, 2023, the entire disclosure of which is incorporated herein by reference in its entirety.

Background Art

[0002] Extended realities such as augmented reality, virtual reality, mixed reality, and related realities may frequently require a great deal of processing power and bandwidth. For example, to reduce the weight, complexity, and power consumption of wearable devices such as extended reality glasses, some implementations offload video processing to a remote device (e.g., a portable computer, a laptop computer, or even a server or a virtual cloud of a server via a wireless network). However, even with powerful computing devices, limitations in processing time and network throughput can result in significant delays that negatively impact the user's quality of experience (QoE). For example, a video latency exceeding 20 - 50 milliseconds can be easily noticed, may impair immersion, and may cause nausea. Measuring this latency as an end - to - end delay may be too coarse to provide practical insights for reducing latency and improving performance.

Summary of the Invention

[0003] To address these and other issues relating to Extended Reality (XR) scenarios, the methods, devices, and / or system implementations disclosed herein provide high-granularity delay measurement and processing adjustment. For example, reducing the trip time of messages transmitted and received between an XR wireless transmit-receive unit (WTRU) and a server can improve QoE, and delay measurement can precisely indicate where delays occur and how they can be mitigated. Actions may then be taken, including shifting processing tasks between local and remote devices to take advantage of the lowest additional delays. [Brief explanation of the drawing]

[0004] A more detailed understanding can be obtained from the following explanation, given as an example in conjunction with the attached drawings, where similar reference numerals in the drawings indicate similar elements. [Figure 1A] This is a system diagram showing an exemplary communication system in which one or more disclosed techniques may be implemented. [Figure 1B] This is a system diagram showing an exemplary radio transmit / receive unit (WTRU) that may be used in the communication system shown in Figure 1A, using one or more approaches disclosed herein. [Figure 1C] This is a system diagram showing exemplary radio access networks (RANs) and exemplary core networks (CNs) that may be used in the communication system shown in Figure 1A, using one or more approaches disclosed herein. [Figure 1D] This is a system diagram showing further exemplary RANs and CNs that may be used in the communication system shown in Figure 1A, using one or more approaches disclosed herein. [Figure 2A] This figure shows examples of target use cases for WTRU glasses, based on several implementation configurations. [Figure 2B]This figure shows examples of target use cases for WTRU glasses, based on several implementation configurations. [Figure 3] This diagram illustrates several examples of standalone AR WTRU (STAR)-based 5G interactive immersive service architectures. [Figure 4] This diagram illustrates examples of edge-dependent AR WTRU (EDGAR)-based 5G interactive immersive service architectures across several implementation forms. [Figure 5] This diagram illustrates examples of partitioned management architectures based on several implementation configurations. [Figure 6] This diagram illustrates examples of high-level call flows for split rendering across several implementations. [Figure 7] This diagram illustrates examples of potential time delays in a system across several implementation configurations. [Figure 8] This diagram illustrates examples of device and server processing delay task breakdowns for edge processing, based on several implementation configurations. [Figure 9] This diagram illustrates examples of device and server processing delay task breakdowns for standalone processing, based on several implementation configurations. [Figure 10] Here are some examples of flowcharts for QoE management based on various implementations. [Figure 11] This diagram illustrates examples of high-level call flows for split rendering for WTRU-centric QoE management, across several implementation configurations. [Figure 12] This figure shows examples of high-level call flows for split rendering for application server-centric QoE management, across several implementation configurations. [Figure 13] This flowchart illustrates several implementation methods for managing Quality of Experience (QoE). [Modes for carrying out the invention]

[0005] Figure 1A shows an exemplary communication system 100 in which one or more disclosed embodiments may be implemented. The communication system 100 may be a multiple access system that provides content such as voice, data, video, message transmission, and broadcast to multiple wireless users. The communication system 100 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communication system 100 can employ one or more channel access schemes such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail unique-word discrete Fourier transform spread OFDM (ZT-UW-DFT-S-OFDM), unique word OFDM (UW-OFDM), resource block filtering OFDM, and filter bank multicarrier (FBMC).

[0006] As shown in Figure 1A, the communication system 100 may include radio transmit / receive units (WTRUs) 102a, 102b, 102c, 102d, a radio access network (RAN) 104, a core network (CN) 106, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, but it will be understood that the disclosed embodiments assume any number of WTRUs, base stations, networks, and / or network elements. Each of the WTRUs 102a, 102b, 102c, and 102d may be any type of device configured to operate and / or communicate in a wireless environment. For example, WTRU102a, 102b, 102c, 102d (any of which may be referred to as a station (STA)) may be configured to transmit and / or receive radio signals and may include user equipment (UE), mobile stations, fixed or mobile subscriber units, subscription-based units, pagers, mobile phones, personal digital assistants (PDAs), smartphones, laptops, netbooks, personal computers, wireless sensors, hotspots or Mi-Fi devices, Internet of Things (IoT) devices, watches or other wearables, head-mounted displays (HMDs), vehicles, drones, medical devices and applications (e.g., remote surgery), industrial devices and applications (e.g., robots and / or other wireless devices operating in the context of industrial and / or automated processing chains), consumer electronics devices, and devices operating on commercial and / or industrial wireless networks. WTRU102a, 102b, 102c, and 102d can all be synonymously referred to as UE.

[0007] The communication system 100 may also include base stations 114a and / or base stations 114b. Each of the base stations 114a and 114b may be any type of device configured to radio interface with at least one of the WTRUs 102a, 102b, 102c, and 102d to facilitate access to one or more communication networks, such as CN 106, the Internet 110, and / or other networks 112. For example, base stations 114a and 114b may be a base transceiver station (BTS), a node B, an e-node B (eNB), a home node B, a home e-node B, a g-node B (gNB), a new radio (NR) node B, a site controller, an access point (AP), a radio router, and the like. Although base stations 114a and 114b are shown as single elements, it will be understood that base stations 114a and 114b may include any number of interconnected base stations and / or network elements.

[0008] Base station 114a may be part of RAN 104, which may also include other base stations and / or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), and relay nodes. Base station 114a and / or base station 114b may be configured to transmit and / or receive radio signals on one or more carrier frequencies, which may be called cells (not shown). These frequencies may be licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. Cells may provide coverage of radio services to a particular geographic area, which may be relatively fixed or change over time. Cells may be further divided into cell sectors. For example, a cell associated with base station 114a may be divided into three sectors. Thus, in one embodiment, base station 114a may include three transceivers, i.e., one transceiver per sector of the cell. In one embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and utilize multiple transceivers per sector of the cell. For example, beamforming may be used to transmit and / or receive signals in a desired spatial direction.

[0009] Base stations 114a and 114b may communicate with one or more WTRUs 102a, 102b, 102c, and 102d via an air interface 116, which may be any suitable radio communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0010] More specifically, as described above, the communication system 100 may be a multiple access system, but may use one or more channel access schemes such as CDMA, TDMA, FDMA, OFDMA, and SC-FDMA. For example, base stations 114a and WTRUs 102a, 102b, and 102c of RAN 104 may implement radio technologies such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which can establish an air interface 116 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed ​​Packet Access (HSPA) and / or Advanced HSPA (HSPA+). HSPA may include High-Speed ​​Downlink Packet Access (HSDPA) and / or High-Speed ​​Uplink Packet Access (HSUPA).

[0011] In one embodiment, base stations 114a and WTRUs 102a, 102b, 102c may implement radio technologies such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish an air interface 116 using Long Term Evolution (LTE) and / or LTE-Advanced (LTE-A) and / or LTE-Advanced Pro (LTE-A Pro).

[0012] In one embodiment, base stations 114a and WTRUs 102a, 102b, and 102c can implement radio technologies such as NR radio access, which can establish an air interface 116 using NR.

[0013] In one embodiment, base station 114a and WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, base station 114a and WTRUs 102a, 102b, 102c may perform LTE radio access and NR radio access together, for example, using the dual connectivity (DC) principle. Thus, the air interfaces utilized by WTRUs 102a, 102b, 102c can be characterized by multiple types of radio access technologies and / or transmissions that are transmitted and received between multiple types of base stations (e.g., eNBs and gNBs).

[0014] In other embodiments, base station 114a and WTRUs 102a, 102b, 102c may implement wireless technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi)), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), etc.

[0015] The base station 114b in Figure 1A may be, for example, a wireless router, home node B, home e-node B, or access point, but any suitable RAT may be used to facilitate wireless connectivity in local areas such as offices, homes, vehicles, campuses, industrial facilities, aerial corridors (for use by drones), roads, etc. In one embodiment, the base station 114b and WTRU 102c, 102d may implement wireless technologies such as IEEE 802.11 to establish a wireless local area network (WLAN). In another embodiment, the base station 114b and WTRU 102c, 102d may implement wireless technologies such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, base stations 114b and WTRUs 102c, 102d may establish picocells or femtocells using cellular-based RATs (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR, etc.). As shown in Figure 1A, base station 114b can connect directly to the internet 110. Therefore, base station 114b may not need to access the internet 110 via CN 106.

[0016] RAN104 may communicate with CN106, which may be any type of network configured to provide voice, data, applications, and / or Voice over Internet Protocol (VoIP) services to one or more of WTRU102a, 102b, 102c, and 102d. The data may have various quality of service (QoS) requirements, such as different throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, and mobility requirements. CN106 may provide call control, billing services, mobile location-based services, prepaid calls, internet connectivity, video distribution, and / or perform high-level security functions such as user authentication. Although not shown in Figure 1A, it will be understood that RAN104 and / or CN106 may communicate directly or indirectly with other RANs using the same RAT or a different RAT as RAN104. For example, in addition to being connected to RAN104 which may utilize NR radio technology, CN106 may also communicate with another RAN (not shown) by employing GSM, UMTS, CDMA2000, WiMAX, E-UTRA, or WiFi radio technology.

[0017] CN106 may also function as a gateway for WTRU102a, 102b, 102c, 102d to access the PSTN108, the Internet 110, and / or other networks 112. The PSTN108 may include a circuit-switched telephone network that provides a plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices, where these networks and devices use a common communication protocol such as the transmission control protocol (TCP), the user datagram protocol (UDP), and / or the internet protocol (IP) of the TCP / IP internet protocol suite. The network 112 may include a wired communication network and / or a wireless communication network that is owned and / or operated by another service provider. For example, the network 112 may include another CN connected to one or more RANs that may use the same or a different RAT as the RAN104.

[0018] Some or all of the WTRU102a, 102b, 102c, 102d in the communication system 100 may include a multi-mode function (e.g., the WTRU102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks via different wireless links). For example, the WTRU102c shown in FIG. 1A may be configured to communicate with a base station 114a that can use a cellular-based wireless technology and a base station 114b that can use IEEE802 wireless technology.

[0019] Figure 1B is a system diagram showing an exemplary WTRU 102. As shown in Figure 1B, the WTRU 102 may include, among other things, a processor 118, a transceiver 120, a transmit / receive element 122, a speaker / microphone 124, a keypad 126, a display / touchpad 128, non-removable memory 130, removable memory 132, a power supply 134, a global positioning system (GPS) chipset 136, and / or other peripherals 138. It will be understood that the WTRU 102 may include any partial combination of the aforementioned elements while maintaining consistency with a particular embodiment.

[0020] The processor 118 may be a general-purpose processor, a dedicated processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), any other type of integrated circuit (IC), a state machine, etc. The processor 118 may perform signal coding, data processing, power control, input / output processing, and / or any other functions that enable the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to a transceiver 120 which can be coupled to a transmit / receive element 122. Although Figure 1B shows the processor 118 and transceiver 120 as separate components, it will be understood that the processor 118 and transceiver 120 may be integrated together in an electronic package or chip.

[0021] The transmit / receive element 122 may be configured to transmit or receive signals to and from a base station (e.g., base station 114a) via the air interface 116. For example, in one embodiment, the transmit / receive element 122 may be an antenna configured to transmit and / or receive RF signals. In one embodiment, the transmit / receive element 122 may be an emitter / detector configured to transmit and / or receive, for example, IR signals, UV signals, or visible light signals. In yet another embodiment, the transmit / receive element 122 may be configured to transmit and / or receive both RF signals and optical signals. It will be understood that the transmit / receive element 122 may be configured to transmit and / or receive any combination of radio signals.

[0022] Although the transmit / receive element 122 is shown as a single element in Figure 1B, the WTRU 102 may include any number of transmit / receive elements 122. More specifically, the WTRU 102 may utilize MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit / receive elements 122 (e.g., multiple antennas) for sending and receiving radio signals via the air interface 116.

[0023] The transceiver 120 may be configured to modulate the signal transmitted by the transmit / receive element 122 and demodulate the signal received by the transmit / receive element 122. As described above, the WTRU 102 may have multimode capabilities. Therefore, the transceiver 120 may include multiple transceivers to enable the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11.

[0024] The processor 118 of the WTRU102 may be coupled to a speaker / microphone 124, a keypad 126, and / or a display / touchpad 128 (e.g., a liquid crystal display (LCD) display unit or an organic light-emitting diode (OLED) display unit) and may receive user input from these. The processor 118 may also output user data to the speaker / microphone 124, the keypad 126, and / or the display / touchpad 128. In addition, the processor 118 may access information from any type of suitable memory, such as non-removable memory 130 and / or removable memory 132, and may store data in memory. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from memory not physically located on the WTRU 102, such as on a server or home computer (not shown), and store data in memory.

[0025] The processor 118 may receive power from the power supply 134 and be configured to distribute and / or control power to other components in the WTRU 102. The power supply 134 may be any suitable device for supplying power to the WTRU 102. For example, the power supply 134 may include one or more dry cell batteries (e.g., nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), a solar cell, a fuel cell, etc.

[0026] The processor 118 may also be coupled to a GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or instead of, the information from the GPS chipset 136, the WTRU 102 may determine its location based on receiving location information from base stations (e.g., base stations 114a, 114b) via the air interface 116 and / or based on the timing of signals received from two or more nearby base stations. It will be understood that the WTRU 102 may acquire location information by any preferred location determination method while maintaining consistency with certain embodiments.

[0027] The processor 118 may be further coupled to other peripherals 138, which may include one or more software and / or hardware modules that provide additional features, functions, and / or wired or wireless connectivity. For example, peripherals 138 may include an accelerometer, an electronic compass, a satellite transceiver, a digital camera (for photos and / or videos), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands-free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an internet browser, a virtual reality and / or augmented reality (VR / AR) device, an activity tracker, and the like. Peripherals 138 may include one or more sensors. The sensor may be one or more of the following: gyroscope, accelerometer, Hall effect sensor, magnetometer, compass sensor, proximity sensor, temperature sensor, time sensor, geolocation sensor, altimeter, light sensor, touch sensor, barometer, gesture sensor, biometric sensor, humidity sensor, etc.

[0028] WTRU102 may include a full-duplex radio in which the transmission and reception of some or all of the UL (e.g., for transmission) and DL (e.g., for reception) signals (e.g., associated with a particular subframe) may occur in parallel and / or simultaneously. The full-duplex radio may include an interference management unit for reducing and / or substantially eliminating self-interference via signal processing either through hardware (e.g., chokes) or through a processor (e.g., via a separate processor (not shown) or processor 118). In embodiments, WRTU102 may include a half-duplex radio for the transmission and reception of some or all of the signals (e.g., associated with a particular subframe of either UL (e.g., for transmission) or DL ​​(e.g., for reception)).

[0029] Figure 1C is a system diagram showing RAN104 and CN106 according to one embodiment. As described above, RAN104 can communicate with WTRU102a, 102b, and 102c via the air interface 116 using E-UTRA radio technology. RAN104 can also communicate with CN106.

[0030] RAN104 may include e-nodes B160a, 160b, and 160c, but it will be understood that RAN104 may include any number of e-nodes B while maintaining consistency with a particular embodiment. Each of the e-nodes B160a, 160b, and 160c may include one or more transceivers for communicating with WTRU102a, 102b, and 102c via the air interface 116. In one embodiment, e-nodes B160a, 160b, and 160c may implement MIMO technology. Thus, e-node B160a may, for example, use multiple antennas to transmit radio signals to and / or receive radio signals from WTRU102a.

[0031] Each of the e-nodes B160a, 160b, and 160c can be associated with a specific cell (not shown) and can be configured to handle wireless resource management decisions, handover decisions, user scheduling in UL and / or DL, etc. As shown in Figure 1C, the e-nodes B160a, 160b, and 160c can communicate with each other via the X2 interface.

[0032] The CN106 shown in Figure 1C may include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (PGW) 166. Although these elements are shown as part of CN106, it should be understood that any of these elements may be owned and / or operated by entities other than the CN operator.

[0033] The MME162 can be connected to each of the e-nodes B162a, 162b, and 162c within RAN104 via the S1 interface and can function as a control node. For example, the MME162 may be responsible for user authentication of WTRU102a, 102b, and 102c, activation / deactivation of bearers, and selection of a specific serving gateway during the initial connection of WTRU102a, 102b, and 102c. The MME162 may provide control plane functionality for switching between RAN104 and other RANs (not shown) using other radio technologies such as GSM and / or WCDMA.

[0034] The SGW164 can connect to each of the e-nodes B160a, 160b, and 160c within RAN104 via the S1 interface. The SGW164 can generally route and forward user data packets to and from WTRU102a, 102b, and 102c. The SGW164 can perform other functions, such as fixing the user plane during handovers between e-nodes B, triggering paging when DL data is available to WTRU102a, 102b, and 102c, and managing and remembering the context of WTRU102a, 102b, and 102c.

[0035] SGW164 can connect to PGW166, which can provide WTRU102a, 102b, and 102c with access to packet-switched networks such as the Internet 110, facilitating communication between WTRU102a, 102b, and 102c and IP-enabled devices.

[0036] CN106 can facilitate communication with other networks. For example, CN106 can provide WTRU102a, 102b, and 102c with access to a circuit-switched network such as PSTN108 to facilitate communication between WTRU102a, 102b, and 102c and conventional terrestrial line communication devices. For example, CN106 may include, or communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that functions as an interface between CN106 and PSTN108. In addition, CN106 may provide WTRU102a, 102b, and 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers.

[0037] While the WTRU is shown as a wireless terminal in Figures 1A to 1D, in certain representative embodiments, it is assumed that such a terminal may be able to use a wired communication interface with a communication network (for example, temporarily or permanently).

[0038] In a typical embodiment, the other network 112 can be a WLAN.

[0039] A WLAN in Infrastructure Basic Service Set (BSS) mode may have an access point (AP) for the BSS and one or more stations (STAs) associated with the AP. The AP may have access to or interfaces with a Distribution System (DS) or another type of wired / wireless network that transmits traffic into and / or out of the BSS. Traffic originating from outside the BSS to an STA may arrive via the AP and be delivered to the STA. Traffic originating from an STA to a destination outside the BSS may be sent to the AP and delivered to its respective destination. Traffic between STAs within the BSS may be transmitted, for example, through the AP, where the source STA sends traffic to the AP, and the AP delivers the traffic to the destination STA. Traffic between STAs within the BSS may be considered and / or referred to as peer-to-peer traffic. Peer-to-peer traffic may be transmitted between a source STA and a destination STA (for example, directly between them) using a direct link setup (DLS). In certain representative embodiments, the DLS may use 802.11e DLS or 802.11z tunneled DLS (TDLS). A WLAN using Independent BSS (IBSS) mode may not have APs, and STAs within or using IBSS (e.g., all STAs) may communicate directly with one another. The IBSS communication mode may also be referred to herein as the “ad hoc” communication mode.

[0040] When using 802.11ac infrastructure mode or a similar operating mode, an AP may transmit beacons on a fixed channel, such as a primary channel. The primary channel can be of a fixed width (e.g., a wide bandwidth of 20 MHz) or a dynamically set width. The primary channel may be the operating channel of the BSS and may be used by the STA to establish a connection with the AP. In certain typical embodiments, Carrier Sense Multiple Access with Collision Avoidance (CSMA / CA) may be implemented, for example, in an 802.11 system. In the case of CSMA / CA, the STA, including the AP (e.g., all STAs), may sense the primary channel. If the primary channel is sensed / detected and / or determined to be busy by a particular STA, that STA may backoff. A single STA (e.g., only one station) may transmit on a given BSS at any given time.

[0041] A high-throughput (HT) STA may use a 40MHz wide channel for communication, for example, by combining a primary 20MHz channel with adjacent or non-adjacent 20MHz channels to form a 40MHz wide channel.

[0042] Very High Throughput (VHT) STAs can support channels with widths of 20 MHz, 40 MHz, 80 MHz, and / or 160 MHz. 40 MHz and / or 80 MHz channels can be formed by combining consecutive 20 MHz channels. 160 MHz channels can be formed by combining eight consecutive 20 MHz channels or two discontinuous 80 MHz channels, which may be referred to as an 80+80 configuration. In the 80+80 configuration, after channel encoding, data can be passed through a segment parser that can split the data into two streams. Inverse Fast Fourier Transform (IFFT) processing and time-domain processing can be performed independently on each stream. The streams may be mapped to two 80 MHz channels, and the data can be transmitted by the transmitting STA. At the receiver of the receiving STA, the above operation for the 80+80 configuration may be reversed, and the combined data can be transmitted to Medium Access Control (MAC).

[0043] Sub-1 GHz operating modes are supported in 802.11af and 802.11ah. Channel operating bandwidth and carrier are reduced in 802.11af and 802.11ah compared to those used in 802.11n and 802.11ac. 802.11af supports bandwidths of 5 MHz, 10 MHz, and 20 MHz in the TV White Space (TVWS) spectrum, while 802.11ah supports bandwidths of 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz using the non-TVWS spectrum. According to a typical embodiment, 802.11ah can support meter-type control / machine communications (MTC), such as MTC devices in a macro coverage area. MTC devices may have limited functionality, including support for specific bandwidths and / or limited bandwidths (e.g., support only). MTC devices may include batteries with battery life exceeding a threshold (e.g., to maintain very long battery life).

[0044] A WLAN system that can support multiple channels and channel bandwidths such as 802.11n, 802.11ac, 802.11af, and 802.11ah includes a channel that can be designated as the primary channel. The primary channel may have a bandwidth equal to the maximum common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel may be set and / or limited by an STA from among all STAs operating in a BSS that support the minimum bandwidth operating mode. In the 802.11ah example, even if the AP and other STAs in the BSS support operating modes of 2MHz, 4MHz, 8MHz, 16MHz, and / or other channel bandwidths, the primary channel of an STA (e.g., an MTC type device) that supports (e.g., only supports) the 1MHz mode may be 1MHz wide. Carrier detection and / or Network Allocation Vector (NAV) settings may depend on the status of the primary channel. For example, if the primary channel is busy because an STA (which only supports 1MHz operating mode) is transmitting to the AP, the entire available frequency band may be considered busy, even if the majority of the available frequency band remains idle.

[0045] In the United States, the available frequency band for 802.11ah is 902MHz to 928MHz. In South Korea, the available frequency band is 917.5MHz to 923.5MHz. In Japan, the available frequency band is 916.5MHz to 927.5MHz. The total available bandwidth for 802.11ah is 6MHz to 26MHz, depending on the country code.

[0046] Figure 1D is a system diagram showing RAN104 and CN106 according to one embodiment. As described above, RAN104 can communicate with WTRU102a, 102b, and 102c via the air interface 116 using NR radio technology. RAN104 can also communicate with CN106.

[0047] RAN104 may include gNB180a, 180b, and 180c, but it will be understood that RAN104 may include any number of gNBs while maintaining consistency with the embodiment. Each of gNB180a, 180b, and 180c may include one or more transceivers for communicating with WTRU102a, 102b, and 102c via the air interface 116. In one embodiment, gNB180a, 180b, and 180c may implement MIMO technology. For example, gNB180a and 180b may use beamforming to transmit signals to and / or receive signals from gNB180a, 180b, and 180c. Thus, gNB180a may, for example, use multiple antennas to transmit radio signals to and / or receive radio signals from WTRU102a. In some embodiments, gNB180a, 180b, and 180c may implement carrier aggregation technology. For example, gNB180a may transmit multiple component carriers to WTRU102a (not shown). A subset of these component carriers may be on the unlicensed spectrum, and the remaining component carriers may be on the licensed spectrum. In some embodiments, gNB180a, 180b, and 180c may implement coordinated multi-point (CoMP) technology. For example, WTRU102a may receive coordinated transmissions from gNB180a and gNB180b (and / or gNB180c).

[0048] WTRU102a, 102b, and 102c may communicate with gNB180a, 180b, and 180c using transmissions associated with scalable numerology. For example, OFDM symbol intervals and / or OFDM subcarrier intervals may vary for different transmissions, different cells, and / or different parts of the radio transmission spectrum. WTRU102a, 102b, and 102c may communicate with gNB180a, 180b, and 180c using subframes or transmission time intervals (TTIs) of varying or scalable lengths (e.g., containing varying numbers of OFDM symbols and / or varying lengths of absolute time).

[0049] gNB180a, 180b, and 180c can be configured to communicate with WTRU102a, 102b, and 102c in standalone and / or non-standalone configurations. In a standalone configuration, WTRU102a, 102b, and 102c can communicate with gNB180a, 180b, and 180c without accessing other RANs (e.g., e-nodes B160a, 160b, and 160c). In a standalone configuration, WTRU102a, 102b, and 102c can utilize one or more of gNB180a, 180b, and 180c as mobility anchor points. In a standalone configuration, WTRU102a, 102b, and 102c can communicate with gNB180a, 180b, and 180c using signals in the unlicensed band. In a non-standalone configuration, WTRU102a, 102b, and 102c may communicate with and connect to gNB180a, 180b, and 180c, while also communicating with and connecting to other RANs such as enodes B160a, 160b, and 160c. For example, WTRU102a, 102b, and 102c may implement DC principles for substantially simultaneous communication with one or more gNB180a, 180b, and 180c and one or more enodes B160a, 160b, and 160c. In a non-standalone configuration, enodes B160a, 160b, and 160c may function as mobility anchors for WTRU102a, 102b, and 102c, and gNB180a, 180b, and 180c may provide additional coverage and / or throughput to service WTRU102a, 102b, and 102c.

[0050] Each of the gNB180a, 180b, and 180c may be associated with a specific cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, user scheduling in UL and / or DL, support for network slicing, interworking between DC, NR and E-UTRA, routing of user plane data to User Plane Functions (UPFs) 184a and 184b, and routing of control plane information to Access and Mobility Management Functions (AM Fs) 182a and 182b. As shown in Figure 1D, the gNB180a, 180b, and 180c can communicate with each other via the Xn interface.

[0051] The CN106 shown in Figure 1D may include at least one AMF182a, 182b, at least one UPF184a, 184b, at least one Session Management Function (SMF)183a, 183b, and optionally a Data Network (DN)185a, 185b. Although the aforementioned elements are shown as part of CN106, it will be understood that any of these elements may be owned and / or operated by entities other than the CN operator.

[0052] AMF182a, 182b may be connected to one or more gNB180a, 180b, 180c within RAN104 via the N2 interface and may function as a control node. For example, AMF182a, 182b may be responsible for user authentication of WTRU102a, 102b, 102c, support for network slicing (e.g., handling different protocol data unit (PDU) sessions with different requirements), selection of specific SMF183a, 183b, management of registration areas, termination of non-access stratum (NAS) signaling, mobility management, etc. Network slicing may be used by AMF182a, 182b to customize CN support for WTRU102a, 102b, 102c based on the type of services utilized by WTRU102a, 102b, 102c. For example, various network slices can be established for various use cases, such as services that rely on ultra-reliable low latency (URLLC) access, services that rely on enhanced massive mobile broadband (eMBB) access, and MTC access services. The AMF182a and 182b can provide control plane functionality for switching between RAN104 and other RANs (not shown) that use other radio technologies such as LTE, LTE-A, LTE-A Pro, and / or non-3GPP access technologies such as WiFi.

[0053] SMF183a and 183b can be connected to AMF182a and 182b in CN106 via the N11 interface. SMF183a and 183b can also be connected to UPF184a and 184b in CN106 via the N4 interface. SMF183a and 183b can select and control UPF184a and 184b and configure the routing of traffic through UPF184a and 184b. SMF183a and 183b can perform other functions such as managing and assigning UE IP addresses, managing PDU sessions, controlling policy enforcement and QoS, and providing DL data notifications. PDU session types can be IP-based, non-IP-based, Ethernet-based, etc.

[0054] UPF184a and 184b can be connected to one or more gNB180a, 180b, and 180c within RAN104 via the N3 interface, which can provide WTRU102a, 102b, and 102c with access to packet-switched networks such as the Internet 110, facilitating communication between WTRU102a, 102b, and 102c and IP-enabled devices. UPF184 and 184b can perform other functions such as packet routing and forwarding, enforcement of user plane policies, support for multi-homed PDU sessions, processing of user plane QoS, buffering of DL packets, and providing mobility anchors.

[0055] CN106 can facilitate communication with other networks. For example, CN106 may include, or communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that functions as an interface between CN106 and PSTN108. In addition, CN106 may provide WTRU102a, 102b, 102c with access to other networks 112, which may include other wired and / or wireless networks owned and / or operated by other service providers. In one embodiment, WTRU102a, 102b, 102c can be connected to local DN185a, 185b via UPF184a, 184b through an N3 interface to UPF184a, 184b and an N6 interface between UPF184a, 184b and DN185a, 185b.

[0056] In view of Figures 1A to 1D and their corresponding descriptions, one or more of the functions described herein with respect to one or more of the WTRU102a to d, base stations 114a to b, e-node-B160a to c, MME162, SGW164, PGW166, gNB180a to c, AMF182a to b, UPF184a to b, SMF183a to b, DN185a to b, and / or any other devices described herein may be performed by one or more emulation devices (not shown). An emulation device may be one or more devices configured to emulate one or more of the functions described herein. For example, an emulation device may be used to test other devices and / or simulate network and / or WTRU functions.

[0057] Emulation devices may be designed to perform one or more tests on other devices in a laboratory and / or operator network environment. For example, one or more emulation devices may perform one or more or all functions while fully or partially implemented and / or deployed as part of a wired and / or wireless network to test other devices in a communications network. One or more emulation devices may perform one or more or all functions while temporarily implemented / deployed as part of a wired and / or wireless network. Emulation devices may be directly coupled to other devices for the purpose of performing tests and / or tests using over-the-air wireless communication.

[0058] One or more emulation devices may perform one or more functions, including all of the above, while not implemented / deployed as part of a wired and / or wireless communication network. For example, an emulation device may be used in a test laboratory test scenario, and / or in a wired and / or wireless communication network that is not deployed (e.g., for testing), to perform testing of one or more components. One or more emulation devices may be test equipment. Direct RF coupling and / or wireless communication via RF circuitry (e.g., which may include one or more antennas) may be used by the emulation device to transmit and / or receive data.

[0059] Depending on the context, there may be one or more Extended Reality (XR) services and / or applications, such as interactive immersive services, augmented reality (AR), virtual reality (VR), and / or mixed reality (MR). The systems and methods described herein may be equally applicable to interactive immersive services, XR, AR, VR, and MR, and many implementations may be interchangeable. Therefore, descriptions of specific embodiments or implementations related to XR may be equally applicable and not limited to, for example, VR.

[0060] A WTRU may correspond to any XR device / node that may take one or more form factors as described herein. In addition to the definitions already provided herein, a WTRU (for example, a WTRU may be interchangeable with an XR WTRU) may include, but are not limited to, head-mounted displays (HMDs), optical see-through glasses (sometimes called optical XR or optical AR, or optical see-through), camera see-through head-mounted displays (HMDs) (sometimes called video XR or video AR, or video see-through), mobile devices with position tracking and cameras, wearables, etc. In addition, there may be different types of XR WTRUs that are based on XR device capabilities such as one or more of the features / functions of a display, camera, sensors, sensor processing, wireless connectivity, XR / media processing, and / or power supply. These additional features / functions may be internal and / or external, and if the additional features / functions are external, they may be provided and / or associated with one or more devices, wearables, actuators, sensors, controllers, and / or accessories. One or more XR WTRUs may be grouped into a collaborative XR group to support any number of XR applications, experiences, and / or services.

[0061] In some cases, such as XR with interactive experiences, a WTRU (e.g., an XR WTRU) may interact with an XR server running on an edge computing node to offload computationally intensive tasks such as video rendering or scene state calculations (e.g., including physical simulations). For example, the XR WTRU may transmit tracking, sensor information, gesture information, attitude information, and / or interactive information to the XR server via the uplink (UL). The XR server may then generate an XR scene based on the information transmitted from the XR WTRU. The XR server may rasterize the XR viewport, perform XR pre-rendering, and generate XR media that is encoded and delivered to the XR WTRU via the downlink (DL). The XR WTRU device may decode the XR media, perform up-to-date attitude correction to address any changes in attitude, and render the XR viewport. As described herein, the application flow may involve three or more devices (e.g., WTRU <-> edge <-> server). Everything behind an edge server may be abstracted as "processing" by the edge server. Furthermore, the terms edge and cloud may be interchangeable (for example, an edge server may be equivalent to a cloud server). In some cases, an edge server and an application server (AS) may be interchangeable unless otherwise specified.

[0062] As mentioned above, XR implementations can frequently require significant processing power and bandwidth. For example, to reduce the weight, complexity, and power consumption of wearable devices such as extended reality glasses, some implementations offload video processing to remote devices (e.g., portable computers, laptop computers, or even servers or virtual clouds of servers via wireless networks). However, even with powerful computing devices, the limitations of currently available processing power and memory capacity, processing time, and network throughput can result in significant delays that negatively impact the user's perceived quality of experience (QoE). For example, XR applications may have tight round-trip interaction delays to provide an acceptable QoE for end users, such as 20ms motion-to-photon and / or 50ms for ultra-low latency applications. Latencies exceeding this are easily noticeable and can impair immersion and cause nausea. Measuring this latency as end-to-end delay may be too coarse to provide actionable insights for mitigating delays and improving performance.

[0063] In some implementations, to address these and other issues relating to Extended Reality (XR) scenarios, the methods, devices, and / or system implementations disclosed herein provide high-granularity delay measurement and processing adjustment. For example, reducing the trip time of messages transmitted between an XR radio transmit / receive unit (WTRU) and a server can improve QoE, and delay measurement can accurately indicate where delays are occurring and how they can be mitigated. Actions may then be taken, including shifting processing tasks between local and remote devices, for example, between a WTRU and an application server, to take advantage of the lowest additional delays.

[0064] For example, rendering XR video or scenes may require offloading processing to a more powerful computing unit when the WTRU does not have sufficient computing power. The following examples, shown in Figures 2A and 2B, illustrate two use cases where it may be beneficial to perform processing on an edge server to meet XR QoE requirements.

[0065] Figure 2A shows an example of a target use case 200 for one or more approaches described herein. The optical see-through device 202 (e.g., XR glasses, which may be a type of WTRU) may have limited computing and battery resources. When a user runs an AR application, the AR glasses 202 may communicate either in a standalone manner or via a smartphone (e.g., a second WTRU, not shown) that captures the user's pose (e.g., head position and / or rotation, gaze direction or angle, hand, finger, and / or controller position) and sends it to an edge application server 204. The server application then processes the video stream for rendering based on the received pose and sends it back to the AR glasses 202, possibly via the smartphone, where the video stream may be rendered on the glasses. In another use case 200' shown in Figure 2B, a video see-through device 202' (which may be a VR headset, for example, a type of WTRU) may send a first video stream to an edge server 204, which then calculates graphical information to be overlaid on the received video and then sends the rendered video stream back to the VR headset 202' (again, possibly via a second WTRU).

[0066] In some system architectures for interactive immersive services (e.g., XR), different scenarios may exist regarding how workflows are handled between XR WTRUs and cloud / edge servers. For example, in some systems (e.g., 5G), there may be standalone AR (STAR) WTRU scenarios and / or edge (server)-dependent AR (EDGAR) WTRU scenarios.

[0067] Figure 3 shows an example of a STAR-based interactive immersive service architecture 300. Similarly, Figure 4 shows an example of an EDGAR-based interactive immersive service architecture 400. In this case, most of the rendering can be achieved on server 404.

[0068] In some use cases, such as shared interactive immersive services, user interactions may be sent from the WTRU to the server. The server handles this request by modifying the immersive media scene based on the interaction (e.g., translation, rotation, scaling, and / or contextual changes such as adding / interacting with / modifying objects in the scene).

[0069] Referring first to Figure 3, in STAR Scenario 300, the UE-XR client or WTRU 302 may run an XR runtime process 312, which may communicate with an XR application 310 (e.g., a media viewer, video game, productivity application, etc.) to provide functionality. The XR runtime process 312 may function as an interface to the hardware components of the UE-XR client or WTRU 302, including sensors (e.g., IMU, accelerometer, button, dial, etc.) and cameras or microphones, as well as output components such as actuators (e.g., linear or rotary actuators for "rumble" or haptic effects, or other types of actuators), displays, speakers, or headphones. The XR source control 314 may include services, servers, subroutines, or other executable logic for interacting with the XR runtime 312 and the XR application 310 to fulfill requests for media data, and may control whether requests are provided by the local scene manager 320, presentation engine 322, and renderer 324, and / or by the remote scene manager 330 on the XR server 304.

[0070] For example, in some implementations, the UE-XR client 302 may detect attitude information or user interactions and transmit this information via the network 306 (e.g., a wired network, a wireless network, or a combination of wired and wireless networks) to the server 304 (which may include a virtual server or cloud application server provided by one or more physical computing devices) via the network interface card 318, through a media access function or application data protocol 316 (e.g., application, session, and / or transport layer protocol). In some examples, a user interaction may be a single event that is completely asynchronous with other data flows. Furthermore, the frequency of interaction events may depend on the type of interaction and the use case.

[0071] The XR application 328, executed by the server 304, may process pose information and user interactions (e.g., via the scene manager 330 and graph handler 332) to generate a processed scene which may include additional media assets 336 and / or commands for actuators or other functions instructed by XR functions 334, and send the processed scene back to the WTRU (via the media distribution function 326 protocol and NIC 318) in the form of a description update or a completely new scene description. The scene manager 320 may parse the update and / or scene description to generate a view or image for the presentation engine 322, which may use the renderer 324 to render the scene to a display.

[0072] Referring to Figure 4, in the EDGAR scenario 400, which may also be known as split rendering, the WTRU 402 may offload some or all of the scene rendering to the server 404. In such an implementation, the server's XR application 428 may include a renderer 440 (similar to renderer 324) that can rasterize the XR viewport and render images of the XR media encoded for display presentation and delivered to the WTRU. For example, a high-resolution, complex ray tracing environment may require considerable processing power to accurately calculate reflections and occlusions, while decoding and displaying the resulting video bitstream may be relatively easy. By offloading this processing to a high-performance edge server 404, the client 402 may require less power and / or processor resources, resulting in extended battery life, reduced weight, etc. In addition, processing on the edge server 404 can be highly scalable using dynamic parallel processing of the scene (e.g., distributing groups of pictures, frames, or macroblocks to different virtual servers for rendering). The media asset storage 436 may be offloaded to a dedicated storage server or application provider 440, similarly supporting the scalability of server-side processing and rendering.

[0073] Figure 5 shows an example of a split management architecture 500 having a UE-XR client or WTRU 502 and a data network (DN) 504, where DN 504 may include one or more servers (e.g., an application provider, a split rendering server, and, in some implementations, network nodes providing real-time communication application functions (RTC AFs), collectively referred to as application services or ASs). Optionally, a split rendering media service enabler may define the formats and / or protocols required to make split rendering accessible to media services and application providers. The end-to-end architecture may be depicted as shown, with WTRU 502 including the functions shown on the left and data network 504 including the functions shown on the right.Between WTRU502 and DN504, one or more of the following functions may collectively exist: a Split-Rendering Client (SRC) 506, which is responsible for acquiring WTRU media capabilities, negotiating with the Real-Time Communication (RTC) AF 510 via a media session handler 516, and agreeing on the split rendering process in the RTC AF; a Split-Rendering Server (SRS) 508, which is responsible for negotiating a split rendering (SR) session with SRC506, monitoring the server's edge resource usage, and managing / executing the split rendering process; an Application Function (AF) 510, which may also be responsible for provisioning, QoS allocation, and edge resource discovery; an Application Service Provider (512), which is an application provider that provides services; an XR application (514), which is an application running on the WTRU; and / or a Media Session Handler (Media Session Handler), which is an entity on the WTRU responsible for control plane communication with the AF 510. Handler, MSH)516.

[0074] Figure 6 shows an example of a high-level call flow 600 for split rendering. For example, there may be one or more possible scenarios in which the WTRU is the client and the AS is the server, or the AS is the client and the WTRU is the server. In step 1, the presentation engine may discover the split rendering server and set up a connection to the split rendering server. The presentation engine may provide information about its rendering capabilities and XR runtime configuration (e.g., an OpenXR configuration which may be used for this purpose). In step 2, in response, the split rendering server may create a description of the split rendering output and the input it expects to receive from the WTRU. In step 3, the presentation engine may request a buffer stream from the MAF (Media Access Function), which then establishes a connection to the split rendering server, streams the pose information, and reads the split rendering buffer. In step 4, as part of the rendering loop, the source manager may read the pose and user input from the XR runtime. In step 5, the source manager may share the pose prediction and user input actions with the split rendering server. In step 6, the split rendering server may use the information to render the frame. In step 7, the rendered frame may be encoded and streamed to the MAF. In step 8, the MAF may decode and process the buffered frame. In step 9, the MAF may pass the raw buffered frame to the presentation engine and / or the XR runtime. In step 10, the XR runtime may configure the received frame and render it to the display.

[0075] Figure 7 shows an example of potential time delays in system 700, which include a communication delay 702 for sending data to the server, a server-side delay 704 for buffering, processing, and sending responses or other data to the client, a communication delay 706 for sending data back to the client, and buffering, processing, and rendering data on the client side 708. The combination of the first three delays 702-706 can be measured as a round-trip time 710 (for example, from sending request or attitude information to receiving processed frame or scene information). Real-time communications and protocols are used primarily to meet real-time requirements, and when these requirements are not met, the resulting packet loss may lead to insufficient QoE for the end user. QoE requirements for the end user, including high-quality content in time and strict RTT delays such as motion-to-photon, cannot be solved solely by post-processing operations such as attitude correction or asynchronous time warping to mitigate such network bandwidth fluctuations.

[0076] An application may have means to calculate the overall WTRU and server processing delay 710 according to Figure 7, such as by adjusting processing in the UE or modifying communication protocols, but it may not have an idea of ​​the detailed QoE impact of individual application processing steps, such as rendering and / or encoding, which form part of the server-side delay 704. Therefore, the application does not have a way to adjust parameters that act on processing steps in real time and thus affect QoE.

[0077] The implementations of the systems and methods discussed herein address these and other issues by providing techniques and protocols for measuring and adjusting operations at either endpoint (e.g., UE / WTRU or server) or both endpoints in order to reduce time delays that may degrade the QoE of XR services / experiences. In one or more implementations of these systems, devices, and / or methods, the WTRU and / or application server (AS) (WTRU / AS) may apply one or more QoEs and transmit one or more task delay measurements and / or task delay adjustment requests by inserting one or more requests into application data units from one device (WTRU / AS) to the other. The WTRU / AS may then transmit those processing task delay measurements and / or adjustment responses. As described herein, references to the term “processing task delay” may be interchangeable with similar terms in which one or more of these words are plural, and a specific enumeration of this term in any of the possible forms is not limiting, and different permutations are interchangeable, and may be as follows: A processing task delay is a delay for processing individual tasks; a processing task delay is a delay for processing several tasks, and the delay may represent a combination of consecutive or discontinuous tasks; a processing task delay is a multiple delay for processing individual tasks; a processing task delay is a multiple delay for processing several tasks. For all of the above examples, there may be one or more measurements and / or adjustments (e.g., a request for one or more measurements at a time, and different configurations of tasks, e.g., different configurations of tasks, delays, and measurements may exist). In addition, any reference to a single delay or a single task may be interchangeable with multiple delays or multiple tasks, respectively.

[0078] In one example, the WTRU may calculate the WTRU processing task delay and may request the application server (AS) to measure and adjust one or more AS processing task delays.

[0079] The WTRU may calculate the QoE from its measured processing task delays and / or AS processing task delay measurements received from the AS. The WTRU may calculate AS and WTRU task delay adjustments to satisfy the expected application QoE, then adjust the WTRU processing task delays and / or request processing task delay adjustments from the AS.

[0080] For example, in the first flow, the WTRU may send an "AS Processing Task Delay" request in the application data unit to the AS. The request may include one or more actions to be performed by the AS, such as measuring the AS Processing Task Delay to be performed by the AS, adjusting the AS Processing Task Delay to be performed by the AS, sending the AS Processing Task Delay measurement from the AS to the WTRU, and / or sending the AS Processing Task Delay adjustment from the AS to the WTRU.

[0081] In one example, the AS may, in the second flow, send an "AS Processing Task Delay" response in the Application Data Unit to the WTRU. The response may include one or more of the measured values ​​of the AS Processing Task Delay and / or adjustments made to the AS Processing Task Delay in the AS.

[0082] In one example, the application server (AS) may calculate the AS processing task delay and may request the WTRU to measure and / or adjust the WTRU processing task delay.

[0083] The AS may calculate the QoE from its measured processing task delay and / or WTRU processing task delay measurements received from the WTRU. The AS may calculate the WTRU and AS task delay adjustments to satisfy the expected application QoE, and then adjust the AS processing task delay and / or request processing task delay adjustments from the WTRU.

[0084] For example, in the first flow, the AS may send a “WTRU Processing Task Delay” request in the application data unit to the WTRU. The request may include the actions to be performed by the WTRU, the measurement of the WTRU Processing Task Delay to be performed by the WTRU, the adjustment of the WTRU Processing Task Delay to be performed by the WTRU, the transmission of the WTRU Processing Task Delay measurement from the WTRU to the AS, and / or the transmission of the WTRU Processing Task Delay adjustment from the WTRU to the AS.

[0085] For example, in the second flow, the WTRU may send a “WTRU Processing Task Delay” response in the application data unit to the AS. The response may include one or more of the WTRU Processing Task Delay measurement and any adjustments made to the WTRU Processing Task Delay in the WTRU.

[0086] In some implementations, the delay metric may be calculated as the difference between the measured end time of a task and the measured start time of a task. While this metric may not be very accurate from an application perspective, it can provide the average deviation of delays. Other metrics may be used, such as the time from the start time of one task to the start time of the next, or from end to end.

[0087] In some implementations, a delay adjustment request may indicate reducing, increasing, or "mitigating" the processing task delay. Reducing the delay may trigger a server and / or WTRU to satisfy the required latency by degrading visual quality (e.g., by modifying bit depth, resolution, frame rate, depth of field, foveation angle or central angle where the scene is rendered at higher quality for areas beyond the central angle (to account for reduced visual acuity), shader or texture quality, or any other type and form of adjustment). Conversely, mitigating the delay or allowing for an increase in the delay may suggest increasing quality when the expected latency or QoE is met. Thus, in some implementations, the QoE and / or delay may be calculated and compared to one or more thresholds. If QoE falls below a first threshold (or delay exceeds a first threshold), delay may be reduced or visual quality may be reduced; if QoE exceeds a second threshold (or delay falls below a second threshold), delay may be mitigated or visual quality may be increased. In some implementations, if QoE or delay is between these thresholds, quality may be maintained. This provides some degree of hysteresis and can prevent abrupt visual or processing changes that could add their own delay due to changes or overhead (e.g., discarding frames already processed at one resolution and reprocessing them at a new resolution).

[0088] In some implementations, the delayed adjustment response may report any effective adjustments or adjustment feedback that the AS or WTRU will apply with respect to the requested adjustment. The response may also include additional adjustment information, such as instructions on why the adjustment was not performed or why it was performed only partially.

[0089] In some implementations, QoE may be associated with round-trip application delay requirements, such as time limits from action to image presentation. Fine-grained task delay measurements and associated deviations measured after each round trip may highlight metrics that contribute to the end-user's visual quality QoE.

[0090] In one example, the WTRU processing task delay and / or AS processing task delay may vary according to a processing delay range between a minimum delay value and a maximum delay value. This depends on different processing task parameters, as described herein. WTRU and AS may set, establish, and / or conform to parameter value ranges applicable to different processing task adjustments. WTRU and AS may set how the delay varies by updating parameter values.

[0091] In some implementations, the WTRU and AS may set, establish, and / or conform to a processing delay budget, such as an expected standard / median delay budget, a delay budget adjustment range (minimum, median, maximum), and / or a delay budget threshold (critical lower limit, critical upper limit) for triggering message transmission, such as the QoE or delay thresholds described above.

[0092] The processing delay budget can be characterized in one or more ways in various implementations. For example, in some implementations, the delay budget can be characterized by the AS or WTRU processing delays 704, 708 as a whole, as illustrated in Figure 7. In the case of an AS, delay 704 may include delays for content access, scene updates, rendering, encoding, and / or content delivery. In the case of a WTRU, delay 708 may include uplink and / or downlink processing tasks (e.g., upload or transmit, orientation and user interaction, content delivery, download or receive, content access, decoding, post-processing, display, etc.). Note that some of these delays may occur after detection of user interaction modifying the scene but before transmission of orientation or interaction information, and some may occur after reception of the rendered scene or rendering command or bitstream and before display.

[0093] In finer-grained implementations, the delay budget may be characterized by the delays of individual AS and / or WTRU processing tasks, such as the rendering task delay in the AS. For example, the delay budget may be characterized by the association of different tasks running at the same location, although the tasks may not necessarily be consecutive. Examples of such tasks include image / video processing tasks including scene updates, rendering, and encoding, WTRU content access and delivery, AS content access and delivery, and / or any other types and forms of tasks.

[0094] In some implementations, the delay budget may be characterized by the association of different tasks processed on / in different locations. Association may mean correlation between different tasks. Association may be used for QoE measurement on AS or WTRU and for QoE adjustment of associated tasks. For example, changing the encoder configuration may affect both encoding and decoding delays. Examples of tasks include: content delivery from WTRU to AS including WTRU content delivery and AS content access; content delivery from AS to WTRU including AS content delivery and WTRU content access; encoding / decoding chains including AS encoding and WTRU decoding; round-trip delivery including content delivery and content access on both WTRU and AS; AR-specific tasks including attitude and / or interaction and / or scene updates; or any other such tasks.

[0095] Figure 8 shows an example of device and server processing delay times in a system using edge processing 800, categorized into high-granularity details. As shown in the figure, the overall server processing task delay 804 may be the result of several individual task delays, which may be referred to in various ways as task delay, processing delay, parameter, or delay parameter, or by other similar terms.

[0096] For example, in some implementations, the parameter may be server content access, which may include data deserialization. In one case, it may be aperiodic, and there may be input event processing priorities and / or output message processing priorities. In another case, it may be periodic, and there may be input buffer scheduling priorities, input scheduling periodicity such as frequency, and / or output message processing priorities.

[0097] For example, in some implementations, the parameter may be a scene update, and the scene update may be an update of the scene graph according to the received data (e.g., by considering the physics of the environment, the interaction of users and / or other entities with each other or with objects, etc.). For example, the level of precision of the scene state calculation may include a parameter that affects the delay. The scene update delay may be adjusted by increasing / decreasing the precision of the scene state (e.g., by adjusting the precision of the physical simulation by using a simplified mesh geometry for collision detection).

[0098] For example, in some implementations, the parameters may be rendering. This may include selecting image resolution, frame rate, bitrate, foviation angle, and / or level of detail for rendering high-visual-quality objects, and / or having a spatial division of a large virtual scene, or any other such rendering adjustments or parameters.

[0099] For example, in some implementations, the parameter may be an encoding. This may include adjustments to the codec profile or type, compression ratio or type, quality parameter or partitioning parameter, or any other such parameter that can adjust the coding latency or delay.

[0100] For example, in some implementations, the parameters may include data serialization or other transmission-related parameters for server content delivery. In some implementations, the adjustments may include buffer management, message priority, transmit burst size or set size, retransmission rate, error correction parameters, or other such features.

[0101] The WTRU processing task delay shown in Figure 8 may similarly depend on various specific parameters.

[0102] For example, in some implementations, adjustable delay-related parameters in the UL (Pre-Transmit) direction may include attitude and user interaction, such as refresh rate for attitude prediction, and / or data serialization, in WTRU content delivery. In the case of WTRU content delivery, non-periodic modes or situations may exist, which may include input buffer management, input event processing priority, and / or output message processing priority, including message priority. In the case of WTRU content delivery, periodic modes or situations may exist, which may include input buffer scheduling priority, input scheduling periodicity such as frequency, and output message processing priority.

[0103] In some implementations, parameters in the DL (post-transmission) direction may include WTRU content access, which may involve data deserialization. For example, WTRU content access may include inputs to buffer management, which include message priority, and / or outputs to output buffer management, which also include message priority.

[0104] In some implementations, another parameter in the DL direction may be content decoding, which may include encoding parameter selection and implementation.

[0105] In some implementations, another parameter in the DL direction may be post-processing that includes attitude correction, such as time warping to adjust for the latest user attitude. For example, post-processing may include error prediction calculations that may result in unnecessary time warping. For example, post-processing may include coarse / fine grain error prediction.

[0106] In some implementations, parameters in the DL direction may be display-related, and the display may provide an API for controlling display parameters, for example, to manage the power consumption of AR glasses. Some parameters can be adjusted. For example, increasing the refresh time reduces display task delay or latency. For example, processing delay depends on the amount of computing power set for the application.

[0107] The WTRU processing task delay may depend on the AS processing task delay, and vice versa. For example, the decoding delay may depend on an encoding delay task on the AS having different quality and / or partitioning parameters; the post-processing delay may depend on the scene update processing delay, which depends on the precision required for calculating the scene state (e.g., simplified mesh geometry for collision detection, precision of physical simulation); and / or the display delay may depend on an encoding delay task on the AS, having different rendering delays at different image resolutions, frame rates, and / or bit rates.

[0108] Figure 9 shows an example of device and server processing latency in system 900 with rendering in UE / WTRU. The distribution of processing tasks is slightly different compared to Figure 8. For example, since the rendering task is not offloaded to the edge server and is part of the WTRU processing tasks, the encoding and decoding tasks are absent. As a result, the server processing latency 904 and round-trip application time latency 910 are reduced. In some cases, processing in UE / WTRU may take more time (for example, rendering may take more time than decoding a compressed or encoded image), but the overall latency can be reduced by reducing server latency and potentially reducing the required network throughput (reducing the reverse network time latency 906).

[0109] Depending on the circumstances, the WTRU and / or AS may be configured, established, or adapted to disclose their capabilities regarding either delay measurement or adjustment for different processing tasks (as disclosed herein). For example, an AS may indicate that it may consider delay measurement and / or adjustment for image / video processing tasks (e.g., scene update, encoding, rendering) but not for delivery tasks (e.g., content access and delivery).

[0110] In some implementations, when WTRU and / or AS can accommodate task delay measurement requirements, they may define some or all of the following: maximum request frequency, duration of measurement, type of measurement (e.g., periodic, event-triggered), measurement statistics mode defining how the delay measurement is performed, and corresponding parameters (e.g., standard deviation, mean, median, maximum), as well as the number of times the delay is measured and reporting periodicity. For example, a WTRU may observe when the delay reaches a lower or upper limit during a particular duration, or it may measure several times when the delay reaches a given limit. The purpose may be to avoid measurement errors or peak fluctuations that the application may have to deal with over a particular duration.

[0111] In some implementations, the WTRU and / or AS may define a delay budget characteristic such as a probability distribution (e.g., Gaussian distribution, normal distribution) that includes a set of standard deviation / variation, mean deviation / variation, median deviation / variation, minimum deviation and maximum deviation / variation, and / or probability values.

[0112] In some implementations, the WTRU and / or AS may conform to task adjustment requests and may include a maximum delay adjustment request frequency, delay adjustment reduction, delay adjustment increase, delay prediction including confidence estimation, and / or delay range or reach threshold (e.g., lower and upper limits, useless lower limit, critical upper limit).

[0113] Figure 10 shows an example of a flowchart 1000 for QoE management. As shown, this flowchart handles at least two cases: WTRU-centric QoE management, where the client is a WTRU and the server is an application server, and server-centric QoE management, where the client is an application server and the server is a WTRU. In 1002, the client may calculate / determine the application QoE by monitoring and managing processing task delays for both the client and the server. The client may need to enable measurement and adjustment of server processing task delays. In 1004, the client may send a request for measurement and / or adjustment of "server processing task delay" in the client's application data unit to a server flow in an application data unit carrying, for example, user posture and / or user interaction input data. In 1006, the server may extract the request from the application data unit received from the client, process the request, and perform the measurement and / or adjustment of the requested delay parameter or task. In some implementations, the server may perform only some of the requested adjustments and / or measurements. In 1008, the server may send a response or reply containing the measurement and / or an acknowledgment or confirmation of adjustments being made within the server's application data unit to the client flow, such as within the application data unit of a buffer frame. In 1010, the client may extract the response from the application data unit received from the server and process the response. At this point, the process may return to 1002. Process restart may be dynamic or periodic. Process restart may also be based on predetermined / pre-configured thresholds of delay, task, and / or QoE value.

[0114] Figure 11 shows an example of a high-level call flow 1100 for split rendering for WTRU-centric QoE management. As described above (for example, with respect to Figure 6), there may be an architecture for a split rendering application in which a split rendering client (SRC) communicates with a split rendering server (SRS) to enable split rendering services for XR. Figure 11 illustrates an exemplary implementation of WTRU-centric QoE management in which the WTRU of the SRC manages the expected application QoE, following a flowchart in which the XR source management function of the WTRU sends a request to the SRS for "server processing task delay" measurement and / or adjustment within the application data unit carrying user pose and / or user input data. The SRS may then send a reply or response containing the measurement and / or adjustment made within the application data unit of the buffer frame.

[0115] Accordingly, the procedure and flow 600 shown in Figure 6 and related explanations may be extended with one or more techniques, as shown in Figure 11. In 1, in some implementations, the presentation engine may discover or identify a split rendering server (SRS) and set up a connection to the SRS. The presentation engine may provide information about its rendering capabilities and XR configuration (e.g., OpenXR or similar protocols or standards). In 2, in some implementations, in response, the SRS may create a description of the split rendering output and the input it expects to receive from the WTRU. The WTRU and SRS may be configured, established, or adapted to expose processing task delay measurement and adjustment as described herein for high-granularity management of QoE. In 3, in some implementations, the presentation engine may request a buffer stream from the MAF, which may establish a connection to the SRS to stream attitude or interaction information and read the split rendering buffer.

[0116] In the rendering loop, in 4, in some implementations, the XR source manager may read pose and user input from the XR runtime. In 5a, in some implementations, the source manager may share pose prediction and user input actions with the SRS. The source manager may insert server processing task delay measurement and / or adjustment requests within the pose prediction and user input action packets. In some implementations, the source manager may insert measurement requests in some packets or requests (and monitor the responses) to determine what adjustments may be made, and then insert delay or task adjustment requests. In 6, in some implementations, the SRS may use that information to render the frame. The SRS process server may extract task delay measurement and / or adjustment requests from the application data unit and process the requests. In 7a, in some implementations, the rendered frame may be encoded and streamed to the MAF. The SRS process server may insert identification information of task delay measurements and / or adjustments made within the rendered frame (e.g., within the optional fields of the header, within the payload using a packet data unit (PDU) header extension, or elsewhere within the data stream). In 8, in some implementations, the MAF may decode and process the buffered frame. The MAF may extract acknowledgments or information of task delay measurements and / or adjustments from the application data unit and process the response. In 9, in some implementations, the MAF may pass the buffered frame to the scene manager and / or the XR runtime. In 10, the XR runtime and / or the SRC renderer may construct and render the received frame.

[0117] Figure 12 shows an example of a high-level call flow 1200 for split rendering for application server-centric QoE management. This figure illustrates an example of application server-centric QoE management where the SRS manages the expected application QoE. Similar to flow 1100 in Figure 11, but in flow 1200, in 7b, in some implementations, the SRS sends a request to the WTRU for delay measurement and / or adjustment of a WTRU processing task within the application data unit of the buffer frame. Then (in a subsequent loop iteration), in 5b, in some implementations, the WTRU's XR source management function sends a response containing the requested measurement or adjustment being made within the application data unit carrying user posture and / or user input information.

[0118] Figure 13 shows an exemplary flowchart of method 1300 for QoE management in several implementation forms. In 1302, devices such as a WTRU, UE, application server, rendering server, split rendering server, split rendering client, XR headset, wearable computing device, smartphone, portable computer, or any other such device may generate application data units. In some implementation forms, the application data unit may include pose (e.g., head tracking, eye tracking, hand tracking, controller tracking, body tracking) and / or interaction information (e.g., controller buttons or dials, gestures, eye tracking interactions), which may be transmitted to the server by the WTRU or other client device. In some implementation forms, the application data unit may include scene and / or image information, media data, physical information, encoded images, fields, frames, or portions of images, fields, or frames, or other image or media data, which may be transmitted to the WTRU or other client device by the server. In some implementation forms, the application data unit may include task delay measurements, resource consumption measurements, processing or memory consumption measurements, timestamps, or other such data. In some implementations, the application data unit may include one or more requested delay parameter adjustments or processing adjustments, acknowledgments (ACKs) or negations (NACKs), error codes, or other such information.

[0119] In some implementations, in 1304, the device may measure local processing delay. Measuring local processing delay may include measuring the start and end times of processing tasks, such as generating an application data unit, loading data into a buffer for transmission, decoding information in the buffer, rendering a frame or portion of a frame, calculating physical interactions, or any other such processing task. In some implementations, the device may monitor other resources, including processing resources, memory resources, and network resources (e.g., throughput, bandwidth, block error rate, congestion indicators, retransmission count, round-trip time, etc.). In some implementations, the measurements may be inserted into the application data unit for transmission in 1302 (e.g., at the next iteration of method 1300).

[0120] In some implementations, at 1306, the device may remotely request one or more remote delay measurements. A remote delay measurement request may include a request to another device (e.g., a client or server) to measure the start and end times of processing tasks, such as generating an application data unit, loading data into a buffer for transmission, decoding information in the buffer, rendering a frame or part of a frame, calculating physical interactions, or any other such processing task. In some implementations, the measurement request may also include requests for measurements of other resources, including processing resources, memory resources, and network resources (e.g., throughput, bandwidth, block error rate, congestion indicator, retransmission count, round-trip time, etc.). A remote delay measurement request may be transmitted at 1306 or inserted into the application data unit at 1302 for transmission.

[0121] In some implementations, in 1308, the device may receive an application data unit containing the requested remote measurements. In some implementations, the application data unit may also contain other data such as attitude and / or interaction information, or scene and / or image information, affirmative or negative responses to one or more requested delay parameter adjustments or processing adjustments, error codes, or other such information. In some implementations, the device may extract the measurements and / or processing adjustments. In some implementations, the application data unit may also contain requested adjustments to be made by the device (e.g., local adjustments rather than notifications or confirmations of remote adjustments).

[0122] In 1310, in some implementations, the device (e.g., WTRU / server) may determine whether adjustments have been requested by another device (e.g., server / WTRU) in the received application data unit, etc. In the first iteration of method 1300, such adjustment requests may not exist, but may exist in further iterations of method 1300. In some implementations, the device may decide to ignore the requested adjustments or refuse to perform them. In other implementations, in 1312 or 1314, the device may apply the requested adjustments, such as increasing or decreasing resolution, bit depth, foveation angle, shader quality, buffer size, packet priority, etc., as described above. For example, in 1312, the device may increase local processing. In some implementations, this may reduce remote processing by other devices. For example, the WTRU may enable or activate the local rendering engine so that the server can disable rendering and only provide scene or environment data. This may reduce server latency, and in some implementations, it may reduce network bandwidth requirements and associated latency. In other implementations, the device may increase local processing, such as increasing the foveation window or increasing the resolution or bit depth, due to the very high QoE. In another example, in 1314, the device may decrease local processing. In some implementations, this may increase remote processing by other devices. For example, the device may disable the local rendering engine and instruct other devices to perform rendering. In other implementations, the device may decrease local processing by reducing quality, decreasing depth of field, decreasing resolution, or narrowing the foveation window.

[0123] If local adjustment is not indicated in the received application data unit, in 1316 the device may determine whether the local and / or remote measurements indicate that the QoE is less than a first threshold t1 (or the delay is greater than the first threshold t1). If so, in 1312 the device may increase local processing via one of the methods described above. In some implementations, the device may further determine in 1320 that a corresponding remote processing adjustment is required, and in 1322 generate a request for remote processing adjustment, which may be transmitted in a subsequent application data unit in 1302 (e.g., enabling local rendering in 1312 and disabling remote rendering in 1322).

[0124] If the QoE is greater than or equal to a first threshold t1, in 1318 the device may determine whether local and / or remote measurements indicate that the QoE is greater than a second threshold t2 (or the delay is less than the second threshold t2). If so, in 1314 the device may reduce local processing via one of the methods described above. In some implementations, the device may further determine in 1320 that corresponding remote processing adjustments are required, and in 1322 generate a request for remote processing adjustments, which may be transmitted in a subsequent application data unit in 1302 (for example, disabling local rendering in 1314 and enabling remote rendering in 1322).

[0125] If the QoE is between these thresholds, in some implementations, no adjustment may be made to the local device, or adjustment may not be requested from the remote device, and another 1300 iterations may be performed.

[0126] Tables 1-4 provide examples of JSON-formatted syntax and semantics for QoE requests and responses. Other formats, semantics, and / or syntax may be used in various implementations (e.g., XML data, parameter-value pairs, concatenated strings, data arrays, etc.).

[0127] Task delay measurement requests and responses may consist of several fields. Messages may be in JSON format in some implementations and may follow the syntax and semantics shown in Table 1.

[0128] [Table 1] Table 1: Task delay measurement requests and responses name type Cardinality Description Version number 1..1 Task delay measurement QoE metadata version. taskDelayMeasurementsSets object 1..n Array of task delay measurement objects posePrediction object 0..1 An object that carries the posture prediction delay measurement value. The contents of the measurement object are defined in Table 2. posedelivery object 0..1 An object that carries attitude delivery delay measurements. The contents of the measurement object are defined in Table 2. turn object 0..1 An object that carries interaction delivery delay measurements. The contents of the measurement object are defined in Table 2. comAccessDL object 0..1 An object that carries downlink content access delay measurements. The contents of the measurement object are defined in Table 2. contentDecoding object 0..1 An object that carries content decoding delay measurements. The contents of the measurement object are defined in Table 2. postprocessing object 0..1 An object that carries post-processing delay measurements. The contents of the measurement object are defined in Table 2. contentAccessUL object 0..1 An object that carries uplink content access delay measurements. The contents of the measurement object are defined in Table 2. sceneUpdate object 0..1 An object that carries the delay measurement value for scene updates. The contents of the measurement object are defined in Table 2. night object 0..1 An object that carries the rendering delay measurement of the scene. The contents of the measurement object are defined in Table 2. contentEncoding object 0..1 An object that carries delay measurements of content encoding before distribution (e.g., encoding of rendered media frames). The contents of the measurement object are defined in Table 2. contentDeliveryDL object 0..1 An object that carries content delivery delay measurements. The contents of the measurement object are defined in Table 2.

[0129] In some implementations, the contents of the measurement object may use the following format shown in Table 2.

[0130] [Table 2] Table 2: Contents of the measurement object name type Cardinality Description query enumerate 0..1 If REPORTING=0: The WTRU or AS reports the measurement of processing task delays to the AS or WTRU, respectively. If DO_MEASUREMENT=1: The WTRU or AS requires that the measurement of processing task delays be performed by the AS or WTRU, respectively. If MEASUREMENT_REPORT=2: The WTRU or AS requests that the AS or WTRU, respectively, perform and report on the measurement of processing task delays. stdDeviation number 0..1 Standard deviation of the measured parameters min number 0..1 Minimum value of the measured parameter max number 0..1 Maximum value of the measured parameter median number 0..1 Median of the measured parameters

[0131] Task delay adjustment requests and responses may consist of several fields.

[0132] Messages may be in JSON format in some implementations and may follow the syntax and semantics shown in Table 3.

[0133] [Table 3] Table 3: Requests and responses regarding task delay adjustments name type Cardinality Description Version number 1..1 Task delay adjustment request and response versions. taskDelayAdjustmentsSets object 1..n Array of task delay adjustment objects posePredictionDelivery object 0..1 An object that carries posture prediction and delivery delay adjustment. The contents of the adjustment object are defined in Table 4. turn object 0..1 An object that carries interaction delivery delay adjustments. The contents of the adjustment object are defined in Table 4. comAccessDL object 0..1 An object that carries downlink content access delay adjustments. The contents of the adjustment object are defined in Table 4. contentDecoding object 0..1 An object that carries content decoding delay adjustments. The contents of the adjustment object are defined in Table 4. postprocessing object 0..1 An object that carries post-processing delay adjustments. The contents of the adjustment object are defined in Table 4. accessToDisplay object 0..1 An object that carries aggregated delay adjustments (including decoding and post-processing delays) for content access and display. The contents of the adjustment object are defined in Table 4. contentAccessUL object 0..1 An object that carries the uplink content access delay adjustment. The contents of the adjustment object are defined in Table 4. sceneUpdate object 0..1 An object that carries the delay adjustment for scene updates. The contents of the adjustment object are defined in Table 4. night object 0..1 An object that carries the delay adjustment for scene rendering. The contents of the adjustment object are defined in Table 4. contentEncoding object 0..1 An object that carries delay adjustments for the encoding of content before distribution (e.g., the encoding of rendered media frames). The contents of the adjustment object are defined in Table 4. contentDeliveryDL object 0..1 An object that carries content delivery delay adjustments. The contents of the adjustment object are defined in Table 4. renderToDelivery object 0..1 An object that carries the aggregated delay adjustments for scene rendering, encoding of rendered frames, and their distribution. The contents of the adjustment object are defined in Table 4.

[0134] In some implementations, the contents of the adjustment object may use the following format shown in Table 4.

[0135] [Table 4] Table 4: Contents of adjustment objects name type Cardinality Description Request object 0..1 Objects that carry adjustment requests say Boule 1..1 If FALSE, WTRU or AS will perform adjustments to process task delays. If TRUE, the WTRU or AS will adjust the processing task delay and report it to the AS or WTRU, respectively. delayAdjustment number 0..1 If positive, increment the additional delay budget that the task can spend. If negative, it reduces the delay budget that the task can spend. Answer object 0..1 Object that carries the adjustment response AdjustedDelayDone number 1..1 Regarding the requested delay adjustments, we report some of the delay adjustments that were applied.

[0136] In a first aspect, the disclosure relates to a method for measuring and managing task delays for XR processing. The method may be implemented by a client / server depending on the flow. A client (e.g., a WTRU or AS) may interact with a server (e.g., a WTRU or AS), or a server may interact with a client, in either case the underlying purpose of the interaction may be in the context of running an XR application. The client may send a measurement request relating to the server processing delay of one or more tasks. The client may receive a measurement of the server processing delay (of one or more tasks) in response to the measurement request (e.g., if the measurement is taken by the server). Based on the measurement of the server processing delay of one or more tasks, the client may calculate the perceived quality (QoE) of an Extended Reality (XR) application. Based on the XR application QoE, the client may send a request to adjust the server processing delay of one or more tasks. In some cases, the measurement request is sent within an application data unit, which is a user posture related to the XR application or a user interaction input related to the XR application. The client may receive the processing delays performed on one or more tasks in response to the measurement request. In some cases, the adjustment request may be sent within an application data unit, which is a user posture related to the XR application or a user interaction input related to the XR application. In some cases, in response to the adjustment request, the client receives an adjustment confirmation that the server processing delay adjustment for one or more tasks on the server is complete, and a second measurement request is sent to determine whether the adjustment reduced the server processing delay for one or more tasks (for example, by comparing a first calculated QoE with a second calculated QoE, or by comparing the QoE with a threshold).

[0137] In a second embodiment, the disclosure relates to a method. The method includes a first device sending a request to a second device for a measurement of processing delays related to one or more tasks. The method also includes the first device receiving a response from the second device containing the measurement of processing delays related to one or more tasks. The method also includes the first device calculating the quality of experience (QoE) of an extended reality (XR) application based on the received measurement of processing delays related to one or more tasks. The method also includes the first device sending a request to the second device to adjust the processing parameters of at least one of the tasks based on the XR application QoE.

[0138] In some implementations, at least one task includes a portion of server-client operation performed by a second device, and the method includes the first device adjusting the processing parameters of the corresponding portion of the server-client operation performed by the first device. In further implementations, at least one task includes media encoding, and the corresponding portion includes media decoding.

[0139] In some implementations, the request to adjust processing parameters includes a request to disable the rendering of XR media by the second device. In further implementations, the method includes the first device enabling the rendering of XR media by the first device.

[0140] In some implementations, requests to adjust processing parameters include requests to reduce the quality of the XR media in response to the calculated XR application QoE falling below a first threshold. In other implementations, requests to adjust processing parameters include requests to improve the quality of the XR media in response to the calculated XR application QoE exceeding a second threshold.

[0141] In some implementations, a request for measuring processing delay is transmitted within an application data unit, which includes data on user posture or user interaction input related to the XR application. In some implementations, a request for adjusting the processing parameters of at least one task is transmitted within an application data unit, which includes data on user posture or user interaction input related to the XR application. In some implementations, the method includes, in response to a request for adjusting the processing parameters of at least one task, receiving a response from a second device confirming that the adjustment of the processing parameters of at least one task has been completed, and the first device transmitting a second request to the second device for measuring processing delay related to one or more tasks.

[0142] In another aspect, the disclosure covers a radio transmit / receive unit (WTRU) configured to perform at least a portion of any one of the methods disclosed above. In another aspect, the disclosure covers at least one processor operably connected to a transceiver, wherein the processor and transceiver are configured to perform at least a portion of any one of the methods disclosed above. In another aspect, the disclosure covers a network element configured to perform at least a portion of any one of the methods disclosed above. In another aspect, the disclosure covers a server configured to perform at least a portion of any one of the methods disclosed above. In another aspect, the disclosure covers a network node in a 5G network configured to perform at least a portion of any one of the methods disclosed above. In another aspect, the disclosure covers an application server configured to perform at least a portion of any one of the methods disclosed above.

[0143] In variations of one or more examples disclosed herein (e.g., Figure 1B), there may be an architecture for an XR processing engine that may be configured to implement the methods described herein. The WTRU may be linked to other devices via their buses and / or via I / O interfaces. The WTRU may include one or more elements linked to each other by data and address buses, such as a microprocessor (or CPU), e.g., a DSP (i.e., a digital signal processor), ROM (i.e., read-only memory), RAM (i.e., random access memory), a storage device interface, an I / O interface for receiving data to send from an application, an integrated graphics processing unity (GPU), a dedicated GPU, and a power supply (e.g., a battery). In the primary example, the power supply is external to the WTRU. The WTRU may use various internal and / or external components to generate, modify, alter, enhance, display, etc., Extended Reality, Augmented Reality, and / or Virtual Reality. For example, a WTRU may use its CPU and GPU to generate a scene that may include interactive elements, objects, other users, etc. The WTRU may present this scene on the WTRU's display or through an external display via a wired or wireless connection. The external display may include a headset / glasses / helmet with a display for each eye or a single display for both eyes. As described herein, the headset, glasses, helmet, and wearable display may be interchangeable. In some cases, the WTRU may be glasses itself and / or part of glasses, and may have a hardware configuration (e.g., with respect to displays as disclosed herein) for displaying AR / XR / VR experiences. Regardless of the configuration (e.g., external or internal display), any configuration of hardware and displays (built-in or external) may be referred to as WTRU glasses.

[0144] In each of the above memories, the word “register” as used herein may correspond to a small area (a few bits) or a very large area (e.g., an entire program or a large amount of received or decoded data). ROM contains at least the program and parameters. ROM may store algorithms and instructions for performing the techniques according to this principle. When switched on, the CPU uploads the program to RAM and executes the corresponding instructions. RAM contains registers containing the program executed by the CPU and uploaded after the device is switched on, input data, intermediate data for different states of the method, and other variables used to perform the method. WTRU may be linked, for example, via a bus to a set of sensors and a set of rendering devices. Sensors may be, for example, cameras, microphones, temperature sensors, inertial measuring devices, GPS, humidity sensors, IR or UV light sensors, or wind sensors. Rendering devices may be, for example, displays, speakers, vibrators, heat, fans, etc.

[0145] Referring again to a variation of WTRU (for example, as shown in Figure 1B), a sequence of scenes (e.g., 3D scenes and / or XR scenes) may be provided to the encoder. The encoder may receive one scene or a series of scenes as input and provide a bitstream representing the input. The bitstream may be stored in memory and / or on an electronic data medium and / or transmitted over a network. The bitstream representing the series of scenes may be read from memory and / or received from the network by the decoder. The decoder receives the bitstream as input and provides the series of scenes, for example, in point cloud format.

[0146] The encoder may comprise several circuits that implement several steps. In the first step, the encoder projects each scene onto at least one picture (e.g., a 2D picture). Scene projection is any method of mapping points in at least three dimensions to a lower dimension, such as a two-dimensional plane. For example, in methods for displaying graphical data based on a two-dimensional medium of a plane (e.g., pixel information from multiple bit planes), this type of projection can be used in computer graphics, engineering, and drafting. The projection circuit provides at least one two-dimensional frame for a set of scenes. The frame includes color information and depth information representing the scene projected onto the frame. In a variant, the color information, light information, and / or depth information may be encoded in separate frames.

[0147] Metadata may be used and updated by the projection circuit. The metadata may include information about projection behavior (e.g., projection parameters) and / or information about how other information is organized within the frame.

[0148] The video encoding circuit encodes a series of frames and / or a series of pictures of a scene as video. The picture of a scene and / or a series of pictures of a scene may be encoded into a stream by the video encoder. The video data and metadata may then be encapsulated into a data stream by the data encapsulation circuit.

[0149] The encoder is compatible with encoders such as JPEG, specification ISO / CEI 10918-1 UIT-T recommendation T.81, AVC, also known as MPEG-4 AVC or h264. It is specified for both UIT-T H.264 and ISO / CEI MPEG-4 Part 10 (ISO / CEI14496-10), HEVC (T recommendation, H series, h265), 3D-HEVC (extension of HVEC, T recommendation, H series, h265), VP9, ​​and / or AV1 (AOMedia Video 1).

[0150] The data stream may be stored by the decoder in memory accessible, for example, over a network. The decoder may comprise various circuits that implement various stages of decoding. The decoder may also receive a data stream generated by the encoder as input, providing a series of scenes to be rendered and displayed by a volumetric video display device such as a WTRU and / or a head-mounted device (HMD). The decoder may obtain the stream from a source. For example, the source belongs to a set that includes local memory such as video memory or RAM (i.e., random access memory), flash memory, ROM (i.e., read-only memory), and hard disks; storage device interfaces such as interfaces to mass storage devices, RAM, flash memory, ROM, optical discs, or magnetic support; communication interfaces such as wired interfaces (e.g., bus interfaces, wide area network interfaces, local area network interfaces) or wireless interfaces (such as IEEE 802.11 interfaces or Bluetooth® interfaces); and user interfaces such as graphical user interfaces that enable user data input.

[0151] The decoder may include a circuit for extracting data encoded within a data stream. The circuit may take the data stream as input and provide metadata and a 2D video corresponding to the metadata encoded within the stream. The video may be decoded by a video decoder, which provides a series of frames. The decoded frames contain information (e.g., color and depth information, and / or any information disclosed herein). In a variant, the video decoder provides two sequences of frames, one containing color information and the other containing depth information. The circuit uses the metadata to backproject information from the decoded frames to provide a series of scenes. The series of scenes may lose precision in relation to encoding as 2D video and video compression.

[0152] The principle is described more fully below with reference to the accompanying drawings illustrating examples of the principle. However, the principle may be embodied in many alternative forms and should not be construed as being limited to the examples expressed herein. Thus, the principle is open to various modifications and alternative forms, specific examples of which are shown as examples in the drawings and described in detail herein. However, there is no intention to limit the principle to the specific forms disclosed, but rather this disclosure should be understood to encompass all modifications, equivalents, and alternatives that fall within the spirit and scope of the principle as defined in the claims.

[0153] The terminology used herein is intended solely to illustrate specific examples and is not intended to limit the principles. Where used herein, the singular forms “a,” “an,” and “the” are intended to include the plural form unless otherwise specified in the context. Where used herein, the terms “equip,” “equip,” “contain,” and / or “contain” identify the presence of the described feature, integer, step, action, element, and / or component, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof. Furthermore, where an element is referred to as “responding to” or “connecting to” another element, it may directly respond to or be connected to the other element, or there may be an intervening element. In contrast, where an element is referred to as “directly responding to” or “directly connected to” another element, there is no intervening element. Where used herein, the term “and / or” includes any combination of one or more of the enumerated items relating to it and may be abbreviated as “ / .”

[0154] In this specification, terms such as "first" and "second" may be used to describe various elements, but it should be understood that these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, without deviating from the teachings of this principle, the first element may be called the second element, and similarly, the second element may be called the first element.

[0155] Some diagrams include arrows on the communication path to indicate the primary direction of communication, but please understand that communication may occur in the opposite direction to the depicted arrow.

[0156] Some examples are illustrated with block diagrams and operation flowcharts, where each block represents a circuit element, module, or portion of code containing one or more executable instructions to implement a specified logical function. Note that in other implementations, the functions described in a block may occur in a different order than those listed. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or blocks may sometimes be executed in reverse order depending on the functions they involve.

[0157] Any reference in this specification to “by example” or “in one example” means that a particular feature, structure, or characteristic described in relation to the example may be included in at least one implementation of the principle. The occurrence of the phrase “by example” or “in one example” in various places in this specification does not necessarily refer to the same example, nor do separate or alternative examples necessarily exclude each other.

[0158] Reference numerals appearing in the claims are illustrative only and shall not limit the scope of the claims. Although not expressly described, these examples and modifications may be used in any combination or partial combination.

[0159] As described herein, the upper layers may refer to one or more layers in a protocol stack, or a specific sublayer within a protocol stack. A protocol stack may consist of one or more layers within a WTRU or network node (e.g., eNB, gNB, or other functional entity), and each layer may have one or more sublayers. Each layer / sublayer may be responsible for one or more functions. Each layer / sublayer may communicate directly or indirectly with one or more other layers / sublayers. These layers may be numbered as Layer 1, Layer 2, Layer 3, etc. For example, Layer 3 may include one or more of the Non-Access Layer (NAS), Internet Protocol (IP), and / or Radio Resource Control (RRC). For example, Layer 2 may include one or more of the Packet Data Convergence Control (PDCP), Radio Link Control (RLC), and / or Media Access Control (MAC). For example, Layer 3 may have physical (PHY) layer type operations. The higher the number of layers, the higher the level of the other layers (for example, Layer 3 is higher than Layer 1). In some cases, the examples described herein, regardless of the number of layers, may be called layers / sublayers themselves, or they may be called higher layers as described herein. For example, from the highest to the lowest, a higher layer may refer to one or more of the layers / sublayers such as the NAS layer, RRC layer, PDCP layer, RLC layer, MAC layer, and / or PHY layer. Any reference herein to a higher layer in conjunction with a process, device, or system refers to a layer higher than the layer of the process, device, or system. In some cases, a reference herein to a higher layer may refer to a function or operation performed by one or more layers described herein. In some cases, a reference herein to a higher layer may refer to information transmitted or received by one or more layers described herein.In some cases, references to higher layers in this specification may refer to configurations transmitted and / or received by one or more layers described herein.

[0160] While features and elements are described above in specific combinations (e.g., embodiments, methods, examples, etc.), those skilled in the art will understand that each feature or element may be used alone or in any combination with other features and elements. For example, there may be methods described in relation to the drawings for illustrative purposes, as disclosed herein, and those skilled in the art will understand that one or more features or elements from these methods may be used alone or in combination with one or more features from other methods described elsewhere. The symbol " / " (e.g., forward slash) may be used herein to mean "and / or," for example, "A / B" may mean "A and / or B." Where used herein, "a" and "an" and similar phrases should be interpreted as "one or more" and "at least one." Similarly, terms ending in the suffix "(s)" should be interpreted as "one or more" and "at least one." The term "may" should be interpreted as "may, for example," or to indicate that something "happens" or "may happen." In addition, the methods described herein may be implemented in computer programs, software, or firmware embedded in a computer-readable medium for execution by a computer or processor. Examples of computer-readable mediums include electronic signals (transmitted via wired or wireless connections) and computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, optical media such as CD-ROM discs, and digital multi-purpose discs (DVDs). A processor associated with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

Claims

1. It is a method, The first device sends a request to the second device for the measurement of processing delays related to one or more tasks, The first device receives a response from the second device that includes a measurement of processing delay related to one or more tasks, The first device calculates the Extended Reality (XR) Application Quality of Experience (QoE) based on the received measurements of processing delays associated with one or more tasks, A method comprising the first device sending a request to the second device to adjust the processing parameters of at least one of the one or more tasks based on the XR application QoE.

2. The method according to claim 1, wherein the at least one task includes a portion of a server-client operation performed by the second device, and the method further includes the first device adjusting processing parameters for the corresponding portion of the server-client operation performed by the first device.

3. The method according to claim 2, wherein the at least one task includes media encoding, and the corresponding portion includes media decoding.

4. The method according to claim 1, wherein the request for adjusting processing parameters includes a request for disabling the rendering of XR media by the second device.

5. The method according to claim 4, further comprising enabling the first device to render XR media by the first device.

6. The method according to any one of claims 1 to 5, wherein the request for adjusting processing parameters includes a request for reducing the quality of the XR media in response to the calculated XR application QoE falling below a first threshold.

7. The method according to any one of claims 1 to 5, wherein the request for adjusting processing parameters includes a request for increasing the quality of the XR media in response to the calculated XR application QoE exceeding a second threshold.

8. The method according to any one of claims 1 to 7, wherein the request for measurement of processing delay is transmitted within an application data unit, the application data unit includes data on user posture or user interaction input related to the XR application.

9. The method according to any one of claims 1 to 8, wherein the request for adjusting the processing parameters of at least one task is transmitted within an application data unit, the application data unit includes data on user posture or user interaction input related to the XR application.

10. In response to the request to adjust the processing parameters of at least one task, the second device receives a response confirming that the adjustment of the processing parameters of at least one task has been completed, The method according to any one of claims 1 to 9, further comprising the first device transmitting a second request to the second device for measurement of processing delays related to one or more tasks.

11. The method according to any one of claims 1 to 10, wherein the measured value of the processing delay associated with one or more tasks includes the average of multiple measured values ​​of the processing delay associated with the task, the minimum value of multiple measured values ​​of the processing delay associated with the task, the maximum value of multiple measured values ​​of the processing delay associated with the task, the average variation between multiple measured values ​​of the processing delay associated with the task, the minimum variation between multiple measured values ​​of the processing delay associated with the task, the maximum variation between multiple measured values ​​of the processing delay associated with the task, a periodic measured value of the processing delay associated with the task, a measured value of the processing delay associated with the task over a period of time, or a measured value of the processing delay associated with the task transmitted in response to a trigger.

12. A wireless transmit / receive unit (WTRU) configured to perform at least part of the method described in any one of claims 1 to 11.

13. At least one processor operably connected to a transceiver, wherein the processor and the transceiver are configured to perform at least a portion of the method according to any one of claims 1 to 11.

14. A network element configured to perform at least part of the method described in any one of claims 1 to 11.

15. A network node in a 5G network configured to perform at least a part of the method described in any one of claims 1 to 11.

Images