XR (augmented reality) device adaptation for enhanced latency performance

Abstract

The invention discloses XR (augmented reality) device adaptation for enhanced latency performance. In some embodiments, a device may include a network interface wirelessly connected to a cellular network; one or more memories; and one or more processors. The one or more memories may store a network stack that includes an application layer and a lower layer that is lower than the application layer. The one or more processors may determine that at least one of a cell load or a network congestion exists based at least on first data obtained through the network interface or from a lower layer. The one or more processors may adjust one or more operations of the application layer to generate one or more frames based at least on the first data. The one or more processors may wirelessly transmit the one or more frames over a network interface.

Description

Cross-references to related applications on adaptive XR (Extended Reality) devices for enhanced latency performance

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 713,440, filed October 29, 2024, and U.S. Non-Provisional Patent Application No. 19 / 345,686, filed September 30, 2025, which are incorporated herein by reference in their entirety. Technical Field

[0002] This disclosure generally relates to wireless communications, including but not limited to adjusting the operation of devices based on network conditions. Background Technology

[0003] Cellular communication technologies (e.g., 3G, 4G, 5G) allow for high data rate communication. In such an environment, user equipment (UE) can generate data and transmit this data to a base station. The base station can provide or forward data from the UE to its destination. A base station can also provide or forward data from another base station to another UE. Summary of the Invention

[0004] The various embodiments disclosed herein relate to a device. The device may include: a network interface wirelessly connected to a cellular network; one or more memories; and one or more processors. The one or more memories may store a network stack, including an application layer and lower layers below the application layer. The one or more processors may be configured to: determine, at least based on first data obtained through the network interface or from the lower layers, that at least one of cell load or network congestion exists. The one or more processors may be configured to: adjust one or more operations of the application layer to generate one or more frames, at least based on the first data. The one or more processors may be configured to: wirelessly transmit the one or more frames via the network interface.

[0005] The various embodiments disclosed herein relate to a method. The method may include: storing a network stack in one or more memories, the network stack including an application layer and lower layers below the application layer. The method may include: determining, by one or more processors, the presence of at least one of cell load or network congestion, based at least on first data obtained from a network interface wirelessly connected to a cellular network or from the lower layers. The method may include: adjusting one or more operations of the application layer by one or more processors, at least based on the first data, to generate one or more frames. The method may include: wirelessly transmitting the one or more frames via the network interface. Attached Figure Description

[0006] The accompanying drawings are not intended to be drawn to scale. Similar reference numerals and names in the various drawings indicate similar elements. For clarity, not every part may be labeled in each drawing.

[0007] Figure 1 is a schematic diagram of an example wireless communication system according to an exemplary embodiment of the present disclosure.

[0008] Figure 2 is a schematic diagram illustrating example components of a base station and a user equipment according to an exemplary embodiment of the present disclosure.

[0009] Figure 3 is an example of a device including a controller for adjusting device operation based on network conditions, according to an exemplary embodiment of the present disclosure.

[0010] Figure 4 illustrates an example of an adaptive system for adjusting the bit rate of an encoder-decoder (codec) based on network conditions, according to an exemplary embodiment of this disclosure.

[0011] Figure 5 illustrates an example of an adaptive system for adjusting the resolution of data (e.g., video / audio / image data) based on network conditions, according to an exemplary embodiment of this disclosure.

[0012] Figure 6 illustrates an example of an adaptive system for adjusting the frame rate of data (e.g., video / audio data) based on network conditions, according to an exemplary embodiment of this disclosure.

[0013] Figure 7 illustrates an example adaptive scheme for network-condition-based service pipelined and shaped according to an exemplary implementation of this disclosure.

[0014] Figure 8 is a flowchart illustrating a process for adjusting device operation based on network conditions according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0015] Before turning to the accompanying drawings, which illustrate certain embodiments in detail, it should be understood that this disclosure is not limited to the details or methods set forth in the specification or shown in the drawings. It should also be understood that the terminology used herein is for descriptive purposes only and should not be considered limiting.

[0016] Figure 1 illustrates an example wireless communication system 100. The wireless communication system 100 may include base stations 110A, 110B (also referred to as "wireless communication node 110" or "station 110") and user equipment (UEs) 120AA…120AN, 120BA…120BN (also referred to as "wireless communication device 120" or "terminal device 120"). The wireless communication link may be a cellular communication link conforming to 3G, 4G, 5G, 6G, or other cellular communication protocols. In one example, the wireless communication link supports, employs, or is based on orthogonal frequency division multiple access (OFDMA). In one aspect, UEs 120AA…120AN are located within a geographical boundary relative to base station 110A and can communicate with or through base station 110A. In another aspect, UEs 120BA…120BN are located within a geographical boundary relative to base station 110B and can communicate with or through base station 110B. The network between UE 120 and base station 110 may be referred to as a radio access network (RAN). In some embodiments, the wireless communication system 100 may include more, fewer, or different base stations 110 compared to the base station 110 shown in FIG. 1.

[0017] In some embodiments, UE 120 may be a user equipment, such as a mobile phone, smartphone, personal digital assistant (PDA), tablet computer, laptop computer, wearable computing device (e.g., head-mounted display, smartwatch), etc. Each UE 120 may communicate with base station 110 via a corresponding wireless communication link. For example, UE 120 may transmit data to base station 110 via a wireless communication link (e.g., 3G, 4G, 5G, 6G, or another cellular communication link), and / or receive data from base station 110 via a wireless communication link (e.g., 3G, 4G, 5G, 6G, or another cellular communication link). Example data may include audio data, image data, text, etc. Communication or data transmission from UE 120 to base station 110 may be referred to as uplink communication. Communication or data reception from base station 110 by UE 120 may be referred to as downlink communication.

[0018] In some embodiments, base station 110 may be an evolved node B (eNB), a next-generation base station (gNodeB), a femto station, or a pico station. Base station 110 may be communicatively coupled to another base station 110 or other communication devices via wireless communication links and / or wired communication links. Base station 110 may receive data (or RF signals) from UE 120 in uplink communication. Additionally or alternatively, base station 110 may provide data to another UE 120, another base station, or another communication device. Therefore, base station 110 allows communication between each UE 120 associated with base station 110, or other UEs associated with different base stations.

[0019] In some embodiments, the wireless communication system 100 includes a core network 170. The core network 170 may be a component or a collection of components that ensures reliable and secure connection of the UE 120 to the network. The core network 170 can be communicatively coupled to one or more base stations 110A, 110B via communication links. The communication link between the core network 170 and the base station 110 may be a wireless communication link (e.g., 3G, 4G, 5G, 6G, or another cellular communication link) or a wired communication link (e.g., Ethernet, optical communication link, etc.). In some embodiments, the core network 170 includes user plane functions (UPF), access and mobility management functions (AMF), policy control functions (PCF), etc. The UPF can perform packet routing and forwarding, packet inspection, quality of service (QoS) processing, and provide external protocol data unit (PDU) sessions for interconnecting data networks (DNs). The AMF can perform registration management, reachability management, connection management, etc. PCF can help operators (or operating equipment) easily create and seamlessly deploy policies in a wireless network. Core network 170 may include additional components for managing or controlling the operation of the wireless network. In one aspect, core network 170 can receive messages for performing network congestion control and perform the requested network congestion control. For example, core network 170 can receive explicit congestion notifications (ECNs) from base station 110 and / or UE 120 and perform network congestion control based on the ECN. For example, core network 170 can adjust or control the amount of data generated in response to the ECN. Additionally or alternatively, core network 170 can adjust or control the amount of data sent and / or received in response to the ECN.

[0020] In some embodiments, the wireless communication system 100 includes an application server 160. The application server 160 may be a component or device that generates, manages, or provides content data. The application server 160 may be communicatively coupled to one or more base stations 110A, 110B via a communication link. The communication link between the application server 160 and the base station 110 may be a wireless communication link (e.g., 3G, 4G, 5G, 6G, or another cellular communication link) or a wired communication link (e.g., Ethernet, optical communication link, etc.). In one aspect, the application server 160 may receive a data request from a UE 120 via base station 110 and provide the requested data to the UE 120 via base station 110. In one aspect, the application server 160 may receive a message to perform network congestion control and perform the requested network congestion control. For example, the application server 160 may receive an explicit congestion notification (ECN) from base station 110, UE 120, or core network 170 and perform network congestion control according to the ECN. For example, the application server 160 may adjust or control the amount of data generated in response to the ECN. Additionally or alternatively, application server 160 may adjust or control the amount of data sent and / or received in response to ECN. Additionally or alternatively, application server 160 may adaptively adjust or control the error correction rate. The error correction rate may be the ratio of the number of error-correcting packets (also known as "protection packets") in each group of packets used for transmission. Error-correcting packets can be provided to aid in content recovery. Application server 160 may adaptively adjust the error correction rate based on the signal quality of the signals received by UE 120 or the location of UE 120 relative to one or more base stations 110.

[0021] In some embodiments, communication between base station 110, UE 120, application server 160, and core network 170 is based on one or more layers of the Open Systems Interconnection (OSI) model. The OSI model may include multiple layers, including: the physical layer, the Medium Access Control (MAC) layer, the Radio Link Control (RLC) layer, the Packet Data Convergence Protocol (PDCP) layer, the Radio Resource Control (RRC) layer, the Non-Access Stratum (NAS) layer or the Internet Protocol (IP) layer, and another layer.

[0022] Figure 2 is a schematic diagram illustrating example components of a base station 110 and a user equipment 120 according to an exemplary embodiment of the present disclosure. In some embodiments, the UE 120 includes a wireless interface 222, a processor 224, a storage device (also referred to as memory) 226, and one or more antennas 228. These components may be embodied as hardware, software, firmware, or a combination thereof. In some embodiments, the UE 120 includes more, fewer, or different components compared to those shown in Figure 2. For example, the UE 120 may include an electronic display and / or an input device. For example, the UE 120 may include additional antennas 228 and wireless interfaces 222 in addition to those shown in Figure 2.

[0023] Antenna 228 can be a component that receives and / or transmits radio frequency (RF) signals via a wireless medium. The RF signals can be at frequencies between 200 MHz and 100 GHz. The RF signals can have packets, symbols, or frames corresponding to data used for communication. Antenna 228 can be a dipole antenna, patch antenna, loop antenna, or any suitable antenna for wireless communication. In one aspect, a single antenna 228 is used to both transmit and receive RF signals. In another aspect, different antennas 228 are used to transmit and receive RF signals. In yet another aspect, multiple antennas 228 are used to support multiple-in, multiple-out (MIMO) communication.

[0024] Wireless interface 222 includes or is embodied as a transceiver for transmitting and receiving RF signals via one or more antennas 228. Wireless interface 222 can communicate with wireless interface 212 of base station 110 via a wireless communication link. In one configuration, wireless interface 222 is coupled to one or more antennas 228. In one aspect, wireless interface 222 can receive RF signals at RF frequencies received via antenna 228 and down-convert the RF signals to baseband frequencies (e.g., 0 GHz to 1 GHz). Wireless interface 222 can provide the down-converted signals to processor 224. In one aspect, wireless interface 222 can receive baseband signals from processor 224 for transmission at baseband frequencies and up-convert the baseband signals to generate RF signals. Wireless interface 222 can transmit the RF signals via antenna 228.

[0025] Processor 224 is a data processing component. Processor 224 can be embodied as a field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), logic circuit, etc. Processor 224 can receive instructions from storage device 226 and execute these instructions. In one aspect, processor 224 can receive down-converted data at baseband frequency from wireless interface 222 and decode or process the down-converted data. For example, processor 224 can generate audio data or image data based on the down-converted data and present audio indicated by the audio data and / or images indicated by the image data to the user of UE 120. In another aspect, processor 224 can generate or acquire data for transmission at baseband frequency and encode or process the data. For example, processor 224 can encode or process image data or audio data at baseband frequency and provide the encoded or processed data to wireless interface 222 for transmission.

[0026] Storage device 226 is a component for storing data. Storage device 226 may be embodied as random access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, removable disk, compact optical disc read-only memory (CD-ROM), or any device capable of storing data. Storage device 226 may be embodied as a non-transitory computer-readable medium storing a plurality of instructions that can be executed by processor 224 to perform various functions of UE 120 disclosed herein. In some embodiments, storage device 226 and processor 224 are integrated into a single component.

[0027] In some embodiments, base station 110 includes a wireless interface 212, a processor 214, a storage device (also referred to as memory) 216, and one or more antennas 218. These components may be embodied as hardware, software, firmware, or a combination thereof. In some embodiments, base station 110 includes more, fewer, or different components than those shown in FIG. 2. For example, base station 110 may include an electronic display and / or input device. For example, base station 110 may include additional antennas 218 and wireless interfaces 212 in addition to those shown in FIG. 2.

[0028] Antenna 218 can be a component that receives and / or transmits radio frequency (RF) signals over a wireless medium. Antenna 218 can be a dipole antenna, patch antenna, loop antenna, or any suitable antenna for wireless communication. In one aspect, a single antenna 218 is used to both transmit and receive RF signals. In another aspect, different antennas 218 are used to transmit and receive RF signals. In one aspect, multiple antennas 128 are used to support multiple-input multiple-output (MIMO) communication.

[0029] Wireless interface 212 includes or is embodied as a transceiver for transmitting and receiving RF signals via one or more antennas 218. Wireless interface 212 can communicate with wireless interface 222 of UE 120 via a wireless communication link. In one configuration, wireless interface 212 is coupled to one or more antennas 218. In one aspect, wireless interface 212 can receive RF signals at RF frequencies received via antenna 218 and down-convert the RF signals to baseband frequencies (e.g., 0 GHz to 1 GHz). Wireless interface 212 can provide the down-converted signals to processor 214. In one aspect, wireless interface 212 can receive baseband signals from processor 214 for transmission at baseband frequencies and up-convert the baseband signals to generate RF signals. Wireless interface 212 can transmit the RF signals via antenna 218.

[0030] Processor 214 is a data processing component. Processor 214 can be embodied in an FPGA, ASIC, logic circuit, etc. Processor 214 can receive instructions from storage device 216 and execute these instructions. In one aspect, processor 214 can receive down-converted data at baseband frequency from wireless interface 212 and decode or process the down-converted data. For example, processor 214 can generate audio or image data based on the down-converted data. In another aspect, processor 214 can generate or acquire data for transmission at baseband frequency and encode or process the data. For example, processor 214 can encode or process image or audio data at baseband frequency and provide the encoded or processed data to wireless interface 212 for transmission. In one aspect, processor 214 can set, allocate, schedule, or allocate communication resources for different UEs 120. For example, processor 214 can set different modulation schemes, time slots, channels, frequency bands, etc., for UEs 120 to avoid interference. Processor 214 can generate data indicating communication resource configuration (or uplink configuration authorization (UL CG)) and provide the data (or UL CG) to radio interface 212 for transmission to UE 120.

[0031] Storage device 216 is a component for storing data. Storage device 216 may be embodied as RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, removable disk, CD-ROM, or any device capable of storing data. Storage device 216 may be embodied as a non-transitory computer-readable medium storing a plurality of instructions that can be executed by processor 214 to perform various functions of the base station 110 disclosed herein. In some embodiments, storage device 216 and processor 214 are integrated into a single component.

[0032] Extended reality (XR) technologies are increasingly being used in applications requiring high data rates, low latency, high reliability, and / or robust coverage and mobility to deliver seamless user experiences. With the development of XR platforms, emerging use cases such as augmented reality (AR) video calling (e.g., codec avatar communication) and multimodal artificial intelligence (MMAI) are placing increasingly stringent latency requirements on achieving real-time interaction and enhanced user engagement. Conventional methods may be insufficient to meet these demands, especially as the complexity and volume of XR data increase. Therefore, adaptive techniques capable of improving the overall quality of the user experience in XR systems are needed.

[0033] In one respect, conventional systems typically perform adaptation independently at the application layer and lower layers within the device's modem, resulting in a lack of coordinated optimization across devices. This independent adaptation may be insufficient for XR devices, which require adaptive strategies tailored to specific objectives, including use case requirements (e.g., AR video calling and / or MMAI requirements), service characteristics, latency performance targets, and key wireless channel and cellular network conditions. For example, XR applications such as AR video calling may require high bidirectional data rates, while MMAI workloads may generate bursts of downlink and / or uplink traffic. Furthermore, factors such as cell load and network congestion further complicate the adaptation process. Therefore, XR device adaptation techniques are needed to meet these diverse and dynamic needs in a unified and responsive manner. The term "cell loading" (also known as "cell load," "cellular load," or simply "load") refers to a measure of how much demand is placed on cell resources, including the number of connected devices, the amount of data transmitted, and / or the utilization of wireless resources (e.g., bandwidth and time slots). The term "network congestion" refers to a situation in a communication network where the demand for data transmission exceeds the network's available capacity, resulting in a decline in the performance of users and devices.

[0034] There is a need to address these issues. This disclosure includes systems, apparatuses, and methods for implementing device-side adaptive techniques, which can reduce latency by optimizing data processing and transmission within the device, thereby improving the overall user experience quality of XR systems.

[0035] In some embodiments, the device may include a wireless interface, one or more memories, and one or more processors configured to execute extended reality (XR) applications with low latency under variable wireless conditions. The one or more processors may be configured (e.g., via the wireless network interface) to acquire first data indicative of at least one of congestion, cell load, channel conditions, buffer status, or latency from a lower layer of the cellular network and a modem stack on the device. For example, the first data may be data received via the network interface (e.g., a low-latency, low-loss, scalable throughput (L4S) counter value received using a real-time transport control protocol (RTCP). The first data may be data indicative of the state of a buffer in a lower layer (e.g., how long data is buffered in the buffer before being transmitted to the wireless network). In some embodiments, a lower layer may refer to a layer in the protocol stack lower than the application layer. In some embodiments, a lower layer may refer to a layer in the protocol stack lower than the transport layer.

[0036] In some embodiments, the device may include a device-side cross-layer controller that connects a lower layer (e.g., modem PHY / MAC) to a higher layer (transport layer and / or application layer) to receive first data (e.g., an indication of high cell load or network congestion) and generate unified adaptive decisions for outgoing XR services (e.g., uplink XR services). The first data may include indications from the wireless network and / or proprietary or standardized indications from lower layers of the device (e.g., buffer occupancy, delay expiration, or modem API signals).

[0037] In some embodiments, the controller may implement and / or execute device-side adaptive mechanisms, which may include, but are not limited to: (i) application-layer codec bitrate adaptation; (ii) resolution adaptation at the application front end via downsampling; (iii) frame rate adaptation at the application front end via pacing; and / or (iv) XR service pipelined and shaped for bursty uplink payloads (e.g., image or video frames). The controller may select one or more mechanisms based on first data and target latency requirements.

[0038] In some embodiments, the device may include an application-layer media codec and a transport-layer congestion control. For use cases such as augmented reality (AR) video calls, low latency and high throughput can maintain a satisfactory user experience. To achieve these goals, the device may employ a combination of application-layer codec bitrate adaptation and transport-layer congestion control to achieve low-latency and high-throughput transmission.

[0039] In one aspect, in a wireless network environment, devices may encounter channel fluctuations and / or dynamic cell loads, which can lead to unpredictable throughput variations in the cellular uplink. Such variations can adversely affect the accuracy of transport layer bit rate estimation and the effectiveness of congestion control mechanisms. In some embodiments, the device may utilize cellular knowledge acquired from lower layers within the network or device and / or modems to inform the codec bit rate adaptation process. The term "modem" (modulator-demodulator) refers to hardware, firmware, software, or a combination thereof within a device (e.g., a smartphone, tablet, XR headset, or Internet of Things (IoT) device) that performs the conversion of digital data into radio signals for wireless transmission and vice versa. By incorporating inputs such as cellular load and congestion status, the device can improve the responsiveness and effectiveness of its adaptive strategies, thereby optimizing media transmission for XR applications.

[0040] In some embodiments, for codec bitrate adaptation, in addition to transport layer feedback (e.g., transport layer congestion control), one or more processors may also use network and device inputs (e.g., indications of network congestion or cell load) to determine the target media bitrate. The controller may employ a backoff ratio (determined by R) in response to congestion indications (e.g., flags associated with low latency, low loss, scalable throughput (L4S)). backoff (represented), and the new target bit rate is set to be equal to the proportionally reduced bit rate using the following equation, while limiting the back-off to its maximum value: R backoff = min (R) backoff_max , N L4S × R look-up (Equation 1), Target bit rate = (1-R) backoff ) × Current_Bitrate (Equation 2), where R backoff_max This is a predetermined value (e.g., 0.2 or 20%), N L4S (Referred to as "L4S count" or "L4S counter value") is an indicator of the number of groups marked with L4S, R look-up It is a value obtained from a lookup table (LUT) using the current bit rate (Current_Bitrate). In some embodiments, the L4S count refers to the number of explicit congestion notification (ECN) tags or other congestion signals that are counted and reported to help manage congestion in real time.

[0041] In some embodiments, the device may interoperate with transport layer congestion control (e.g., RTCP-based feedback and / or L4S signaling) while using cellular knowledge (e.g., uplink congestion signaling as discussed in 3GPP Release 19) to improve the timeliness and accuracy of bit rate decisions relative to transmission-only estimations.

[0042] In some embodiments, the device may implement a resolution and / or frame rate adaptive mechanism applicable to both video-based and image- or video-based XR applications, such as those utilizing MMAI. Resolution and / or frame rate adaptation enables applications to dynamically respond to changing network conditions, thus maintaining a consistent user experience even under low bandwidth or high latency conditions.

[0043] In some embodiments, resolution adaptation can be performed at the application front end using techniques such as downsampling. Downsampling can involve reducing the spatial resolution of video or image frames.

[0044] In some embodiments, frame rate adaptation can be performed at the application front end using techniques such as pacing. Pacing can involve adjusting the time rate at which frames are generated or transmitted.

[0045] Compared to codec bitrate adaptation, resolution and frame rate adaptation offers several advantages, including simpler implementation, reduced data transmission, and lower computational load on the device. These techniques are particularly suitable for video frames and images encountered in applications such as MMAI and other XR applications.

[0046] In some embodiments, to achieve resolution and / or frame rate adaptation, one or more processors can react immediately to detected high cell load or network congestion by reducing the resolution and / or frame rate at the application front end, thereby reducing the amount of data entering the encoder and reducing latency. This approach can be simpler and faster than encoder-level bit rate changes and can be performed on encoded frames through pacing in some pipeline.

[0047] In some embodiments, the controller may maintain triggering logic, such as a sliding window count for congestion indication (e.g., L4S flags), within defined intervals; when a threshold is exceeded, the device may invoke more aggressive adaptations (e.g., reduce bit rate, resolution, or frame rate, and / or enable traffic shaping), and may relax these adaptations when network conditions improve.

[0048] In some embodiments, the device can implement XR service pipeline and shaping mechanisms to address situations where stringent latency requirements coincide with bursty service patterns. For example, in multimodal artificial intelligence (MMAI) applications, uplink services may include image or video frames transmitted in bursts. In such cases, conventional methods of transmitting the entire payload at once may be insufficient to meet latency constraints.

[0049] To overcome these challenges, devices can be configured to divide each payload into multiple smaller blocks and send them as soon as the first usable block is generated. By segmenting the payload in this way, the network can begin scheduling resources for the initial blocks, thereby reducing overall transmission latency. This pipelining and shaping approach is particularly advantageous at cell edges and under high cellular load conditions, where timely delivery of bursty traffic is most challenging.

[0050] In some embodiments, for XR service pipelining and shaping, one or more processors can divide the payload of a frame into multiple blocks and transmit the initial blocks as soon as they become available, thereby allowing earlier scheduling opportunities and alignment with less congested sub-windows within a given latency budget. The controller can defer or advance block transmission to avoid identified high-congestion intervals while satisfying application timing constraints. In some embodiments, the controller can apply pipelining and shaping mechanisms at cell edges and / or when the cell is loaded, thereby improving not only latency but also device energy efficiency by avoiding extending maximum-power uplink transmissions during congested periods.

[0051] In some embodiments, one or more processors may allocate available bitrates among multiple substreams (e.g., video and audio, or multiple camera views) based on application requirements and user experience, and coordinate with adaptive decisions by the controller. In some embodiments, the device may be suitable for cellular and WiFi access, similarly using transport layer feedback and lower-layer indications where available; cellular examples may utilize standardized congestion signaling (e.g., air interface uplink congestion control), while WiFi embodiments may use corresponding indicators in this stack.

[0052] In some embodiments, device-side adaptation can complement RAN-side mechanisms / adaptations (e.g., latency-aware scheduling or service prioritization) to provide end-to-end latency improvements for XR applications, including AR video calls and / or MMAI, which can exhibit bidirectional high-rate streaming and bursty uplink / downlink traffic.

[0053] The embodiments in this disclosure offer at least the following advantages and benefits. Device-side adaptation can provide a useful technique for reducing latency in augmented reality (AR) wearable products by optimizing data processing and transmission within the device. This optimization is useful for delivering a seamless and enjoyable user experience. By tailoring adaptability to specific objectives, including use case requirements, business characteristics, latency performance targets, and / or wireless channel and cellular network conditions, these embodiments enable AR wearable devices to meet diverse user needs and consistently deliver a high-quality user experience. In some embodiments, device-side adaptation techniques can reduce latency by optimizing data processing and transmission within the device, thereby improving the overall user experience quality of XR systems.

[0054] In light of the foregoing, the following figures and description illustrate various examples of systems and / or methods for adjusting operations based on network conditions. It should be noted that the following figures and description are non-limiting examples and can be implemented in any of a variety of other configurations while still remaining within the scope of this disclosure.

[0055] Figure 3 is an example of a device 300 according to an exemplary embodiment of the present disclosure, the device 300 including a controller 350 for adjusting the operation of the device based on network conditions. In some embodiments, the device 300 may include one or more network interfaces 310, one or more modems 360, a controller 350, and / or a protocol stack. In some embodiments, the device 300 may have a configuration similar to that of UE 120 (see Figure 2). For example, the device 300 may include one or more processors 224 and one or more memories 226. In some embodiments, one or more network interfaces 310 may have a configuration similar to that of wireless interface 222. In some embodiments, each of the following may be implemented in hardware (e.g., a processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), one or more logic circuits, or any combination thereof), firmware, software, or a combination thereof: one or more network interfaces 310; one or more modems 360; a controller 350; and a protocol stack (e.g., including a lower layer 330 of a cellular PHY / MAC layer 320, an application layer 340, and a transport layer).

[0056] In some embodiments, the protocol stack may include a lower layer 330, a transport layer (not shown), and / or an application layer 340, wherein the lower layer 330 includes a cellular physical / medium access control (PHY / MAC) layer 320. The application layer 340 may include a video / audio codec 342 and / or one or more applications (e.g., XR applications). In some embodiments, the lower layer 330 may include one or more layers below the application layer 340, such as the PHY / MAC layer 320, the radio link control (RLC) layer, the packet data convergence protocol (PDCP) layer, the radio resource control (RRC) layer, the non-access stratum (NAS) layer, the network layer (e.g., the internet protocol (IP) layer), and / or the transport layer. In some embodiments, the lower layer 330 may include one or more layers below the transport layer, such as the PHY / MAC layer 320, the RLC layer, the PDCP layer, the RRC layer, the NAS layer, and / or the network layer. In some embodiments, the protocol stack may be stored in one or more memories of the device 300.

[0057] Referring to Figure 3, controller 350 can coordinate the adaptation of each layer in the protocol stack. Controller 350 can connect upwards to XR application 344 and video codec 342, and downwards to lower layer 330 including cellular PHY / MAC layer 320, such that controller 350 is configured to: receive first data 301 indicating wireless operating conditions (e.g., wireless network conditions), and drive or adjust application-level media operations in response to this data. First data 301 may include received indications 302 of cell load and congestion from the network, and indications 303 of channel and network conditions obtained from lower layer 330 of the device.

[0058] Controller 350 can be configured to perform one or more adaptive operations for outgoing XR services based on first data 301. These operations may include, for example, setting a target media encoding bitrate at the video codec and / or causing the XR application front-end (e.g., the part of the application that directly interacts with the user) to adjust the spatial and / or temporal characteristics of the outgoing frames (e.g., the resolution and / or frame rate of the video / audio / image data). Controller 350 can thus coordinate codec-level and application front-end operations to reduce end-to-end latency while maintaining perceived quality for XR use cases, including bidirectional high-rate AR video and bursty image / video transmissions associated with MMAI.

[0059] Controller 350 can obtain explicit indications 302 of cell load and / or network congestion from the network (e.g., via network interface 310), and indications, measurements, or status parameters 303 characterizing local buffers (e.g., the status of buffers in lower layer 330) from lower layer 330 of the device. Controller 350 can process these inputs (e.g., indications 302, 303) to determine whether high cell load or congestion exists. In response to determining the presence of high cell load or network congestion, controller 350 can command or control video codec 342 to reduce the bit rate and / or command or control XR application 344 to downsample the resolution and / or adjust the frame rate. In this way, controller 350 can act as a cross-layer coordination point between application subsystems (e.g., application layer 340) and the cellular stack (e.g., protocol stack and / or modem 360), thereby providing more timely and accurate adaptation than application-only or transport-only mechanisms.

[0060] The lower layer 330 (including the cellular PHY / MAC 320) can provide the controller with lower layer indications, such as channel quality-related information, buffer status, and / or delay-related status. The network (e.g., a cellular network) can provide congestion or load signaling suitable for device-side use through network interface 310. The controller 350 can utilize these indications to shape outgoing media characteristics before over-the-air transmission in the cellular stack, thereby mitigating latency increases caused by uplink throughput fluctuations and system load dynamics. The architecture shown in Figure 3 can be configured to complement Radio Access Network (RAN) technologies (e.g., delay-aware scheduling and service prioritization) to achieve end-to-end latency optimization.

[0061] In some embodiments, controller 350 may be implemented as software or firmware executing on one or more processors of the device. The controller's output may include control messages (e.g., bitrate targets) for video codec 342 and configuration parameters (e.g., resolution or frame rate settings) for the application front-end (XR application 344). The controller's inputs (e.g., first data 301, indications 302, 303) may be transmitted from the cellular stack via an interface, and where applicable, via standardized or vendor-specific APIs presented by modem 360. In operation, as shown in FIG3, this arrangement enables device 300 to adjust the characteristics of outgoing XR frames in near real-time based on contemporaneous knowledge of cell load, congestion, and / or channel conditions.

[0062] Figure 4 illustrates an example of an adaptive system 400 for adjusting the bit rate of an encoder-decoder (codec) 440 based on network conditions, according to an exemplary embodiment of the present disclosure. In some embodiments, the adaptive system 400 may be implemented in a device (e.g., UE 120, device 300). The adaptive system 400 may include an RTCP feedback receiver 410, a conventional congestion controller 420, a bandwidth allocator 430, a video / audio codec 440, and a controller 450. The congestion controller 420 may include loss-based congestion control 422 and / or delay-based congestion control 424. In some embodiments, the controller 450 may have a configuration similar to that of the controller 350. In some embodiments, each of the RTCP feedback receiver 410, the conventional congestion controller 420, the bandwidth allocator 430, the video / audio codec 440, and the controller 450 may be implemented in hardware (e.g., a processor, an ASIC, an FPGA, one or more logic circuits, or any combination thereof), firmware, software, or a combination thereof.

[0063] Referring to Figure 4, the system may include a transport layer feedback path (e.g., RTCP feedback receiver 410, congestion controller 420) and an application layer media pipeline (e.g., controller 450, bandwidth allocator 430, video / audio codec 440), configured to cooperatively adjust the bit rate of the video / audio codec 440 in response to congestion information / indications (e.g., L4S count NL4S from RTCP feedback receiver 410). RTCP feedback receiver 410 may be configured to acquire transport feedback of the real-time communication stream. The target bit rate of the video / audio codec 440 may be adjusted based on congestion information (e.g., L4S count NL4S). Controller 450 may perform L4S-triggered congestion control using a rate adaptation function (e.g., Equation 1) that derives a codec bit rate backoff amount based on low-latency explicit congestion information (e.g., L4S flags) transmitted in feedback packets via RTCP feedback receiver 410.

[0064] RTCP feedback receiver 410 can parse feedback (e.g., feedback packets) associated with an ongoing XR session. In some embodiments, RTCP feedback receiver 410 can determine the count NL4S of congestion indications (e.g., L4S flags) present in the current feedback interval.

[0065] In step 452, controller 450 can detect or determine that the count is non-zero. In step 454, controller 450 can use Equation 1 and lookup table (LUT) 458 to calculate the codec bit rate backoff rate R. backoff In some embodiments, the LUT 458 includes (1) a plurality of ranges of the current bit rate and (2) R corresponding to each range. look-up Value. For example, LUT458 can specify a 4% backoff value for current bit rates above 500 kbps, a 3% backoff value for current bit rates between 300 kbps and 500 kbps, a 2% backoff value for current bit rates between 150 kbps and 300 kbps, and a 1% backoff value for current bit rates below 150 kbps. The total backoff value is determined by R. backoff_max A limit (which can be a predetermined value, such as 0.2% or 20%). For example, in step 454, the controller 450 can use the current bit rate of the codec to identify R in the LUT 458. look-up and use the value N L4S and R look-up Use Equation 1 to calculate R backoff In step 456, the controller 450 can use Equation 2 based on R backoff Calculate or determine the adjusted current bit rate (also known as the "new target current bit rate").

[0066] In some embodiments, the controller 450 for performing L4S-triggered codec control can coexist with conventional loss-based control 422 and / or conventional delay-based congestion control 424, allowing the codec's target bit rate to reflect conventional transport dynamics (using conventional congestion controller 420) and to provide more immediate, finer-grained congestion signals (using controller 450) from L4S. The output of the combined control can be presented to a bandwidth allocator 430, which allocates the available bit rate among one or more sub-streams (e.g., video and audio, or multiple camera views) before configuring the video / audio codec 440. In operation, this arrangement allows the device (e.g., device 300) to react rapidly to congestion by progressively and predictably reducing the codec rate, while (e.g., using R...) backoff_max Limit the magnitude of any single adjustment to avoid disrupting the stability of the user experience.

[0067] In some embodiments, the parameters used for backoff calculation (e.g., the bitrate range in LUT 458, the backoff value for each range in LUT 458, and the maximum backoff value R) can be selected for a given XR use case (e.g., AR video calls or MMAI image / video bursts). backoff_max To balance latency reduction with perceived quality, the device can (using controller 450) combine application-layer rate adaptation of L4S notifications with other device-side adaptations (e.g., resolution control and / or frame rate control as shown in Figures 5 and 6) and lower-layer cellular knowledge (e.g., indication 303) to improve end-to-end latency performance under fluctuating uplink throughput and cell load conditions.

[0068] Figure 5 illustrates an example of an adaptive system 500 for adjusting the resolution of data (e.g., video / audio / image data) based on network conditions, according to an exemplary embodiment of the present disclosure. The adaptive system 500 may include a video / audio codec 540 and a controller 550. In some embodiments, the video / audio codec 540 may have a configuration similar to that of video / audio codecs 342 or 440. In some embodiments, the controller 550 may have a configuration similar to that of controller 350 in device 300. In some embodiments, the controller 550 may be implemented in hardware (e.g., a processor, ASIC, FPGA, one or more logic circuits, or any combination thereof), firmware, software, or a combination thereof.

[0069] Referring to Figure 5, controller 550 can implement a resolution adaptive function that responds to cell load and / or network congestion (e.g., first data 301 or indications 302, 303) by reducing the spatial resolution of the outgoing XR frame before transmission. For example, a device (e.g., device 300) may include a radio interface (e.g., network interface 310), a cellular protocol stack (including lower layer 330), an application subsystem for generating XR frames (e.g., application layer 340), a video codec, and controller 550, configured to obtain first data (e.g., first data 301) indicating at least one of cell load, congestion, channel conditions, buffer status, or delay. In step 552, controller 550 can determine that a high load or congestion condition exists. After determining that a high load or congestion condition exists, in step 554, controller 550 can cause the XR application front-end (e.g., XR application 344) and / or encoder preprocessing stage to adjust according to a predefined factor (e.g., S). backoff The frame resolution is downsampled.

[0070] In some embodiments, the controller 550 may use the following Equation 3 to calculate the adjusted resolution downsampling rate (also known as the "target resolution downsampling rate"): Target downsampling rate = (1 - S backoff ) × Current_Down-Sampling_Rate (Equation 3), where S backoff This is a predefined backoff factor selected for the current operating state. Current_Down-Sampling_Rate is the current downsampling rate used by the video / audio codec 540. Before encoding in step 560 (by the video / audio codec 540), the controller 550 can apply an adjusted downsampling rate to reduce one or more spatial dimensions of the frame (e.g., width and / or height), thereby reducing the amount of data presented to the video codec and reducing end-to-end latency under adverse wireless conditions. The downsampling factor S can be selected from a strategy or table tailored for XR use cases. backoff In order to balance latency improvement and perceived quality.

[0071] In some embodiments, controller 550 may obtain first data (e.g., first data 301) from indications provided by the network (e.g., air interface uplink congestion control signaling) and from lower-layer device measurements exposed by the modem (e.g., buffer occupancy or latency-related status provided by modem 360). Controller 550 may maintain triggering logic, such as a sliding window count of the congestion indication, and upon meeting a threshold (e.g., the sliding window count is greater than a threshold), may immediately activate resolution downsampling, then pass the downsampled frames to the video codec (back-end) for compression and subsequent wireless transmission. This fast action path can achieve rapid congestion mitigation by reducing spatial payload without waiting for the codec retuning cycle (e.g., compared to the adaptive approach shown in Figure 4).

[0072] In some embodiments, the resolution adaptation of Figure 5 can operate independently or in coordination with other device-side adaptations, including codec bitrate adaptation (see Figure 4), frame rate adaptation (see Figure 6), and pipelined and shaped operations (Figure 7). When used in coordination, the controller 550 can prioritize resolution downsampling to immediately alleviate congestion, and then coordinate with bitrate and / or frame rate control as conditions evolve to stabilize quality. Therefore, the arrangement shown in Figure 5 provides an application front-end mechanism that is simple to invoke, computationally efficient, and effective for latency mitigation in XR scenarios, including AR video and bursty MMAI traffic.

[0073] Figure 6 illustrates an example of an adaptive system 600 for adjusting the frame rate of data (e.g., video / audio data) based on network conditions, according to an exemplary embodiment of the present disclosure. The adaptive system 600 may include a video / audio codec 640 and a controller 650. In some embodiments, the video / audio codec 640 may have a configuration similar to that of video / audio codecs 342 or 440. In some embodiments, the controller 650 may have a configuration similar to that of controller 350 in device 300. In some embodiments, the controller 650 may be implemented in hardware (e.g., a processor, ASIC, FPGA, one or more logic circuits, or any combination thereof), firmware, software, or a combination thereof.

[0074] In some embodiments, controller 650 may implement a frame rate adaptive function that responds to high wireless load or congestion conditions by reducing the time density of outgoing XR frames before transmission. For example, a device (e.g., device 300) may include a wireless interface (e.g., network interface 310), a cellular protocol stack (including lower layer 330), an application subsystem for generating XR frames (e.g., application layer 340), a video codec (back-end), and controller 650 configured to obtain first data (e.g., first data 301) indicating at least one of cell load, congestion, channel conditions, buffer status, or delay. In step 652, controller 650 may determine that a high load or congestion condition exists. In step 654, when a high load or congestion condition is determined, controller 650 may use the following Equation 4 to set an adjusted frame rate (also referred to as the “target frame rate”): Target frame rate = (1 - F backoff ) × Current_Frame_Rate (Equation 4), where, F backoff This is a predefined backoff factor selected for the current operating state. Current_Frame_Rate is the current frame rate used by the video / audio codec 640. Using the target frame rate, the controller 650 can cause the application front-end (e.g., XR application 344) to synchronize frame emission accordingly. Subsequently, in step 660, the controller 650 can cause the application front-end to provide the synchronized frames to the video codec 640 (back-end) for compression and transmission.

[0075] In some embodiments, controller 650 may obtain first data (e.g., first data 301) from indications provided by the network (e.g., air interface uplink congestion control signaling) and from lower-layer device measurements exposed by the modem (e.g., buffer occupancy or latency-related status provided by modem 360). Controller 550 may maintain triggering logic (e.g., a sliding window count of the congestion indicator) and, upon meeting an initial threshold (e.g., the sliding window count is greater than the threshold), may immediately activate frame rate downselection, maintaining the downselection rate until the congestion indicator drops below a release threshold (which may be the same as the initial threshold). This arrangement allows the device to react quickly to adverse conditions by reducing transient loads without waiting for the encoder to recalibrate its cycle (e.g., compared to the adaptive approach shown in Figure 4).

[0076] In some embodiments, frame rate adaptation can be performed at the application front end (e.g., XR application 344) via a pacing mechanism that suppresses or postpones the transmission of selected frames based on a target frame rate, while preserving application timing constraints and the synchronization of associated media streams (e.g., audio). Pacing can be applied to already encoded frames in a pipeline that supports post-coding scheduling, or to pre-coded frames upstream of the codec, thereby reducing the number of frames entering the encoder and reducing data volume and computational load. Example backoff factors (e.g., F...) backoff This could include a 50% reduction under severe congestion (e.g., from 30 frames per second to 15 frames per second), with intermediate steps chosen based on a strategy.

[0077] In some embodiments, controller 650 can coordinate frame rate adaptation with other device-side adaptations, including codec bitrate adaptation (see Figure 4) and resolution adaptation (see Figure 5), as well as pipelined and sculpting (Figure 7), to achieve end-to-end latency targets while managing perceived quality. For example, frame rate pacing can be invoked to mitigate the impact immediately, and then the bitrate and / or resolution can be adjusted as conditions evolve to stabilize quality. This coordinated control is particularly useful for XR use cases, including two-way AR video and MMAI, where bursty traffic and tight latency budgets are common.

[0078] In some embodiments, parameters controlling frame rate adaptation (e.g., F) backoff The trigger threshold and / or minimum and maximum frame rate values ​​can be defined by a strategy table tailored to the target XR scenario and device capabilities, thereby balancing responsiveness with temporal smoothness and user experience considerations. Therefore, the architecture shown in Figure 6 provides a simple and fast latency mitigation mechanism that runs at the application front end while interacting with the video codec back end and the cellular stack.

[0079] Figure 7 illustrates an example adaptive scheme 700 for network condition-based service pipelined and shaping according to an exemplary embodiment of the present disclosure. As a comparative example, Figure 7 also shows a scheme 750 without adaptation, in which the complete payload 752 is transmitted in a single burst.

[0080] In some embodiments, the device (e.g., device 300) may implement or execute service pipeline and shaping mechanisms to meet the stringent latency requirements of bursty uplink payloads (e.g., image or video frames for MMAI) generated by XR applications. The device may include a radio interface (e.g., network interface 310), a cellular stack (including a lower layer 330), an application subsystem (e.g., application layer 340), and a controller (e.g., controller 350) configured to acquire first data (e.g., first data 301) indicating at least one of cell load, congestion, channel conditions, buffer status, or latency. Based on the first data, the controller may determine that transmitting the complete payload in a single burst would increase queuing latency under current network conditions. For example, referring to FIG7, assuming high cell load or network congestion exists during some intervals (e.g., sub-interval T21), the complete payload 752 may experience high cell load or network congestion during interval T2, resulting in increased queuing latency. When it is determined that transmitting the complete payload in a single burst would increase queuing delays under current network conditions, the controller may instead divide the payload into multiple blocks (e.g., block #1 (701), block #2 (702), and block #3 (703)) and transmit them in a pipelined manner.

[0081] Referring to Figure 7, the controller can divide or segment the payload into blocks #1, #2, and #3, and schedule the blocks across time sub-intervals (e.g., T1, T21, T22) within the application latency budget. The controller can enable the initial blocks (e.g., blocks #1, #2) to be sent immediately when they become available, thus advancing uplink scheduling opportunities and allowing for earlier partial delivery, while deferring subsequent blocks (e.g., block #3) to less congested sub-windows (e.g., sub-interval T22). By aligning block transmissions with sub-intervals exhibiting reduced congestion (e.g., sub-interval T22), the device can reduce line head congestion and mitigate latency increases caused by cellular load dynamics.

[0082] In some embodiments, the controller can identify one or more high-congestion intervals (e.g., sub-interval T21) and low-congestion intervals (e.g., sub-intervals T1, T22) from first data (e.g., first data 301) and schedule block transmissions accordingly (e.g., schedule blocks for sub-intervals T1, T22). For example, when the first data indicates that the first portion of a delay window (e.g., sub-interval T21 of delay window T2) is congested, the controller can postpone the transmission of at least one block (e.g., block #3) until a subsequent portion of the window expected to present low congestion (e.g., sub-interval T22), while ensuring that all blocks are transmitted within the allowable delay budget for the XR use case. This opportunistic scheduling can be particularly beneficial at cell edges and when the serving cell is loaded, where otherwise sending a single large payload would require extended maximum power transmission and increased broadcast time.

[0083] In some embodiments, the pipeline and shaping mechanisms shown in FIG7 can work in conjunction with other device-side adaptations (including codec bitrate adaptation (see FIG4) and / or resolution / frame rate adaptation (FIGs 5 and 6)) to achieve end-to-end latency targets. The controller can invoke blocks to provide immediate mitigation under bursty traffic, while concurrently or subsequently adjusting media bitrate and visual parameters (e.g., using the adaptation schemes shown in FIG4, 5, and 6) to stabilize quality as conditions evolve. In operation, the arrangement depicted in FIG7 allows the device to schedule smaller work units into less crowded sub-windows (e.g., T1, T22) while taking into account application latency constraints (e.g., allowed latency budget), thereby improving responsiveness and user experience in XR scenarios.

[0084] Figure 8 is a flowchart illustrating a process 800 for adjusting device operation based on network conditions according to an exemplary embodiment of the present disclosure. In some embodiments, process 800 is performed by a device (e.g., device 300, UE 120, or any other device configured to adapt operation based on network conditions) including one or more processors (e.g., processor 224), one or more memories (e.g., memory 226), and a transmitter or network interface (e.g., wireless interface 222) wirelessly connected to a cellular network. In some embodiments, process 800 may be performed by a UE configured to transmit data to a base station. In some embodiments, process 800 is performed by other entities. In some embodiments, process 800 includes more, fewer, or different steps compared to those shown in Figure 8.

[0085] In some embodiments, in step 802, one or more processors of the device (e.g., device 300, UE 120) may store a network stack including an application layer (e.g., application layer 340) and a lower layer below the application layer (e.g., lower layer 330).

[0086] In some embodiments, in step 804, one or more processors of the device may determine the presence of at least one of cell load or network congestion, based at least on first data obtained through the network interface (e.g., first data 301) (e.g., indication 302 based on packets with L4S tags received by RTCP feedback receiver 410) or first data obtained from a lower layer (e.g., indication 303). In some embodiments, the first data may include the number of packets indicating congestion notification (e.g., L4S count N). L4S The state of a lower-level buffer (e.g., the time data remains buffered in the buffer before being transmitted to the wireless network).

[0087] In some embodiments, in step 806, one or more processors of the device may adjust one or more operations of the application layer (e.g., adjusting the bit rate of the codec, adjusting the resolution and / or frame rate used by the codec, pipelining and shaping the uplink traffic) based at least on the first data to generate one or more frames.

[0088] In some embodiments, when adjusting one or more operations, the one or more processors may be configured (e.g., using Equations 1 and 2) to adjust the current bit rate of the encoder-decoder (codec) operation based at least on first data. A lookup table (LUT) (e.g., LUT 458) may be used to adjust the current bit rate of the codec operation based on the first data and the current bit rate of the codec.

[0089] In some embodiments, the one or more frames may include at least one of video data, audio data, or image data. During the adjustment of one or more operations, the one or more processors may be configured (e.g., using Equation 3) to adjust the resolution of the one or more frames based at least on the first data. The one or more processors may be configured to perform downsampling on the one or more frames using the adjusted resolution (e.g., step 554).

[0090] In some embodiments, when adjusting one or more operations, the one or more processors may be configured (e.g., using Equation 4) to adjust the frame rate of one or more frames based at least on first data. The one or more processors may be configured to perform pacing on one or more frames using the adjusted frame rate (e.g., step 654).

[0091] In some embodiments, the one or more processors may be configured to determine the presence of bursty uplink traffic. In some embodiments, the one or more processors may determine the presence of bursty uplink traffic using the type of application running on the system or the type of data used in the application (e.g., multimodal data including video, audio, or images). In some embodiments, the one or more processors may determine the presence of bursty uplink traffic by monitoring the magnitude of the uplink traffic (e.g., monitoring whether a large amount of data is transmitted in a short, intense burst, followed by periods of little or no activity). The one or more processors may be configured to receive the payload of a frame (e.g., payload 752). The one or more processors may be configured to divide the payload into multiple blocks (e.g., blocks 701, 702, 703). The one or more processors may be configured to wirelessly transmit each of the multiple blocks via a network interface.

[0092] In some embodiments, the one or more processors may be configured to determine a first duration (e.g., T1) during which there is no cell load or network congestion. The one or more processors may be configured to schedule a first block (e.g., block #1) of a plurality of blocks to be transmitted during the first duration (e.g., T1).

[0093] In some embodiments, the one or more processors may be configured to determine a second duration (e.g., T21) of at least one of cell load or network congestion based at least on first data. The one or more processors may be configured to postpone scheduling of a second block (e.g., block #3) among a plurality of blocks until the second duration (e.g., T21) has elapsed.

[0094] In some embodiments, in step 808, the one or more processors of the device can wirelessly transmit the one or more frames via a network interface (e.g., wireless interface 222, network interface 310).

[0095] Some illustrative embodiments have now been described, and it is clear that the foregoing, presented by way of example, is illustrative and not restrictive. In particular, although many of the examples presented herein relate to specific combinations of method actions or system elements, these actions and elements can be combined in other ways to achieve the same purpose. The actions, elements, and features discussed in connection with one embodiment are not intended to exclude similar roles in other embodiments.

[0096] Hardware and multiple data processing components used to implement the various processes, operations, illustrative logic, logic blocks, modules, and circuits described in conjunction with the various embodiments disclosed herein may be implemented or performed using components designed to perform the functions described herein, such as: a general-purpose single-chip processor or a general-purpose multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof. The general-purpose processor may be a microprocessor, or it may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of multiple computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors incorporating a DSP core, or any other such configuration. In some embodiments, specific processes and methods may be performed by circuitry for a given function. Memory (e.g., memory, storage cell, storage device, etc.) may include one or more devices (e.g., random access memory (RAM), read-only memory (ROM), flash memory, hard disk storage, etc.) for storing data and / or computer code used to perform or facilitate the various processes, layers, and modules described herein. The memory may be or may include volatile or non-volatile memory, and may include database components, object code components, script components, or any other type of information structure for supporting the various activities and information structures described herein. According to one exemplary embodiment, the memory is communicatively connected to a processor via processing circuitry and includes computer code for (e.g., by the processing circuitry and / or the processor) performing one or more of the processes described herein.

[0097] This disclosure contemplates various methods, systems, and program products on any machine-readable medium for implementing various operations. Embodiments of this disclosure can be implemented using existing computer processors, or by special-purpose computer processors combined for this and other purposes and intended for a suitable system, or by hardwired systems. Embodiments within the scope of this disclosure include program products comprising machine-readable media for carrying or having machine-executable instructions or data structures stored thereon. Such machine-readable media can be any available medium accessible to a general-purpose computer or a special-purpose computer, or other machine having a processor. As examples, such machine-readable media may include RAM, ROM, erasable programmable read-only memory (EPROM), electronically erasable read-only memory (EEPROM), or other optical disc storage, disk storage, or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of machine-executable instructions or data structures and is accessible to a general-purpose computer or a special-purpose computer, or other machine having a processor. Combinations of the above are also included within the scope of machine-readable media. Machine-executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing machine to perform a function or a set of functions.

[0098] The wording and terminology used herein are for descriptive purposes and should not be considered limiting. The use of "comprising," "including," "having," "containing," "involving," "characterized in," "characterized as," and variations thereof herein means to cover the items listed thereafter, their equivalents and additions, and to include alternative embodiments consisting exclusively of the items listed thereafter. In one embodiment, the system and method described herein may consist of one of the described elements, actions, or components, various combinations of more than one of the described elements, actions, or components, or all of the described elements, actions, or components.

[0099] Any reference to an embodiment, element, or action of a system or method mentioned in the singular herein may also cover an embodiment that includes multiple such elements, and any reference to any embodiment, element, or action mentioned herein may also cover an embodiment that includes only a single element. References in the singular or plural form are not intended to limit the systems or methods of this disclosure, their components, actions, or elements to a singular or plural configuration. A reference to any action or element based on any information, action, or element may include that action or element is at least in part based on an embodiment of that information, action, or element.

[0100] Any implementation disclosed herein may be combined with any other implementation or embodiment, and references to "an implementation," "some implementations," "one implementation," etc., are not necessarily mutually exclusive, but are intended to indicate that a particular feature, structure, or characteristic described in connection with that implementation may be included in at least one implementation or embodiment. These terms used herein do not necessarily refer to the same implementation. Any implementation may be combined with any other implementation in any manner consistent with the aspects and embodiments disclosed herein, either inclusively or exclusively.

[0101] Where reference numerals follow technical features in the accompanying drawings, detailed embodiments, or any claims, these reference numerals are already included to enhance the comprehensibility of the drawings, detailed embodiments, and claims. Therefore, the presence or absence of these reference numerals does not limit the scope of any claim element.

[0102] The systems and methods described herein may be implemented in other specific forms without departing from the nature of this document. Unless otherwise expressly stated, references to “about,” “approximately,” “substantially,” or other terms of degree include variations of plus or minus 10% (+ / - 10%) from a given measurement, unit, or range. Coupled elements may be electrically, mechanically, or physically coupled to each other, directly or through intermediate elements. Therefore, the scope of the systems and methods described herein is defined by the appended claims rather than the foregoing description, and includes variations falling within the meaning and scope of equivalents of the claims. The term “coupled” and its variations include two components directly connected or indirectly linked to each other. Such a connection may be immutable (e.g., permanent or fixed) or movable (e.g., removable or releasable). Such a connection may be achieved by two components directly coupled to each other or coupled to each other, wherein the two components are coupled to each other using a separate intermediate element and any other intermediate element coupled to each other, or the two components are coupled to each other using an intermediate element integrally formed with one of the two components as a single integral body. If "coupled" or its variants are modified by an additional term (e.g., direct coupling), then the general definition of "coupled" provided above is modified by the simple linguistic meaning of the additional term (e.g., "direct coupling" means that two components are connected without any separate intermediate component), resulting in a narrower definition than the general definition of "coupled" provided above. This coupling can be mechanical, electrical, or fluid.

[0103] A reference to “or” can be interpreted as inclusive, such that any term described using “or” can refer to a single term, more than one term, or all of the terms described. A reference to “at least one of 'A' and 'B'” can include only 'A', only 'B', or both 'A' and 'B'. Such references used in conjunction with “include” or other open-ended terms can include additional terms.

[0104] Modifications may be made to the described elements and actions without departing essentially from the teachings and advantages of the subject matter disclosed herein, such as changing the size, dimensions, structure, shape and proportion, parameter values, installation arrangement, material use, color, and orientation of various elements. For example, an element shown as a single unit may be composed of multiple parts or elements, the positions of the elements may be reversed or otherwise modified, and the nature, number, or position of the discrete elements may be changed or modified. Other substitutions, modifications, alterations, and omissions may also be made to the design, operating conditions, and arrangement of the disclosed elements and operations without departing from the scope of this disclosure.

[0105] References to the position of elements (e.g., "top", "bottom", "above", "below") are used only to describe the orientation of various elements in the accompanying drawings. The orientation of various elements may vary depending on other exemplary embodiments, and these variations are intended to be covered by this disclosure.

Claims

1. An apparatus comprising: A network interface that is wirelessly connected to a cellular network; One or more memories storing a network stack including an application layer and a lower layer below the application layer; and one or more processors configured to: determine, based at least on first data obtained through the network interface or from the lower layer, that at least one of cell load or network congestion exists; and adjust one or more operations of the application layer to generate one or more frames, based at least on the first data. And wirelessly transmit the one or more frames through the network interface.

2. The device according to claim 1, wherein, The first data includes at least one of the following: the number of packets indicating congestion notification; or the status of the lower-level buffer.

3. The device according to claim 1, wherein, In adjusting the one or more operations, the one or more processors are configured to: adjust the current bit rate of the codec operation based at least on the first data.

4. The device according to claim 3, wherein, The current bit rate of the codec operation is adjusted using a lookup table (LUT) based on the first data and the current bit rate of the codec.

5. The device according to claim 1, wherein, The one or more frames include at least one of video data, audio data, or image data.

6. The device according to claim 5, wherein, In adjusting the one or more operations, the one or more processors are configured to: adjust the resolution of the one or more frames based at least on the first data; and perform downsampling on the one or more frames using the adjusted resolution.

7. The device according to claim 5, wherein, In adjusting the one or more operations, the one or more processors are configured to: adjust the frame rate of the one or more frames based at least on the first data; and use the adjusted frame rate to pace the one or more frames.

8. The device according to claim 1, wherein, The one or more processors are configured to: determine the presence of a burst of uplink traffic; receive the payload of a frame; and divide the payload into multiple blocks; And wirelessly transmit each of the plurality of blocks through the network interface.

9. The device according to claim 8, wherein, The one or more processors are configured to: determine a first duration during which there is no cell load or network congestion; And schedule the first of the plurality of blocks to be transmitted within the first duration.

10. The device according to claim 8, wherein, The one or more processors are configured to: determine a second duration of at least one of cell load or network congestion, based at least on the first data; And postpone the scheduling of the second of the plurality of blocks until the second duration has elapsed.

11. A method comprising: A network stack is stored in one or more memories, the network stack comprising an application layer and lower layers below the application layer; The one or more processors determine, based at least on first data obtained from a network interface wirelessly connected to a cellular network or from the lower layer, that at least one of cell load or network congestion exists; the one or more processors adjust one or more operations of the application layer to generate one or more frames based at least on the first data; And wirelessly transmit the one or more frames through the network interface.

12. The method according to claim 11, wherein, The first data includes at least one of the following: the number of packets indicating congestion notification; or the status of the lower-level buffer.

13. The method according to claim 11, wherein, Adjusting one or more operations includes: adjusting the current bit rate of the codec operation based at least on the first data.

14. The method according to claim 13, wherein, The current bit rate of the codec operation is adjusted using a lookup table (LUT) based on the first data and the current bit rate of the codec.

15. The method according to claim 11, wherein, The one or more frames include at least one of video data, audio data, or image data.

16. The method of claim 15, wherein, Adjusting the one or more operations includes: adjusting the resolution of the one or more frames based at least on the first data; and performing downsampling on the one or more frames using the adjusted resolution.

17. The method of claim 15, wherein, Adjusting the one or more operations includes: adjusting the frame rate of the one or more frames based at least on the first data; and pacing the one or more frames using the adjusted frame rate.

18. The method of claim 11, further comprising: Confirm the presence of a sudden uplink traffic; receive the payload of the frame; The payload is divided into multiple blocks; And wirelessly transmit each of the plurality of blocks through the network interface.

19. The method of claim 18, further comprising: Determine the first duration during which there is no cell load or network congestion; And schedule the first of the plurality of blocks to be transmitted within the first duration.

20. The method of claim 18, further comprising: Based at least on the first data, determine a second duration of at least one of cell load or network congestion; And postpone the scheduling of the second of the plurality of blocks until the second duration has elapsed.

Images