Dynamic performance and power adjustment for split XR applications

Abstract

Aspects presented herein relate to methods and apparatus for frame processing, including devices, such as clients or servers. The apparatus may estimate a set of frame processing times for at least one first frame in the set of frames. The apparatus may also detect an actual set of frame processing times for at least one first frame of the set of frames. The apparatus may also output an indication to adjust a second frame processing time set of at least one second frame of the set of frames based on the estimated frame processing time set and the actual frame processing time set, wherein the at least one second frame is after the at least one first frame of the set of frames.

Description

Cross-reference to related applications

[0001] This application claims the benefit of U.S. nonprovisional patent application No. 18 / 473,138, filed on September 22, 2023, entitled “DYNAMIC PERFORMANCE AND POWERADJUSTMENT FOR SPLIT XR APPLICATIONS,” the entire contents of which are expressly incorporated herein by reference. Technical Field

[0002] This disclosure generally relates to processing systems, and more specifically, to one or more technologies for split-screen extended reality (XR) applications. Background Technology

[0003] Computing devices typically perform graphics and / or display processing (e.g., utilizing a graphics processing unit (GPU), a central processing unit (CPU), a display processor, etc.) to render and display visual content. Such computing devices can include, for example, computer workstations, mobile phones (such as smartphones), embedded systems, personal computers, tablet computers, and video game consoles. A GPU is configured to execute a graphics processing pipeline that includes one or more processing stages that operate together to execute graphics processing commands and output frames. A CPU controls the operation of a GPU by issuing one or more graphics processing commands to it. Modern CPUs are typically capable of executing multiple applications concurrently, each of which may require the GPU during execution. A display processor is configured to convert digital information received from the CPU into analog values ​​and can issue commands to a display panel to display visual content. Devices that provide content for visual presentation on a display may utilize a GPU and / or a display processor.

[0004] The device's GPU can be configured to execute processes within the graphics processing pipeline. Additionally, a display processor or display processing unit (DPU) can be configured to perform display processing. However, with the advent of wireless communication and smaller handheld devices, the demand for improved graphics or display processing continues to increase. Summary of the Invention

[0005] The following is a simplified summary of one or more aspects to provide a basic understanding of these aspects. This summary is not a broad overview of all anticipated aspects, nor is it intended to identify key or essential elements of all aspects, nor to describe the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that follows.

[0006] In one aspect of this disclosure, a method, computer-readable medium, and apparatus are provided. The apparatus may be a client device, a server, a display processing unit (DPU), a graphics processing unit (GPU), or any apparatus capable of performing display processing. The apparatus may obtain an indication of a set of frames before estimating the set of frame processing times. The apparatus may also train a machine learning (ML) model on the set of frames before estimating the set of frame processing times. Additionally, the apparatus may estimate the set of frame processing times for at least one first frame in the set of frames. The apparatus may also detect the actual set of frame processing times for at least one first frame in the set of frames. The apparatus may also output an indication for adjusting the set of second frame processing times for at least one second frame in the set of frames based on the estimated set of frame processing times and the actual set of frame processing times, wherein the at least one second frame follows the at least one first frame in the set of frames. Furthermore, the apparatus may adjust the set of second frame processing times for at least one second frame based on the estimated set of frame processing times and the actual set of frame processing times.

[0007] Details of one or more examples of this disclosure are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of this disclosure will become apparent from the description, the drawings, and the claims. Attached Figure Description

[0008] Figure 1 This is a block diagram illustrating the example content generation system.

[0009] Figure 2 An example graphics processing unit (GPU) is shown.

[0010] Figure 3 An example display frame including a display processor and a monitor is shown.

[0011] Figure 4 This is a diagram illustrating example communication of content / data based on the split rendering process.

[0012] Figure 5 This is a diagram illustrating an example timeline of the split rendering process.

[0013] Figure 6 Includes a diagram illustrating an example timeline of the split rendering process.

[0014] Figure 7 This is a diagram illustrating an example timeline of the split rendering process.

[0015] Figure 8 This is a diagram illustrating an example of the split rendering process.

[0016] Figure 9 This is a diagram illustrating an example timeline of the split rendering process.

[0017] Figure 10 This is a communication flowchart illustrating an example of communication between a client device, a server, and a memory.

[0018] Figure 11 This is a flowchart of an example method for frame processing.

[0019] Figure 12 This is a flowchart of an example method for frame processing. Detailed Implementation

[0020] In some aspects of split rendering, once the head pose is sent from the client device (e.g., glasses) to the server (e.g., a phone), the new scene can be rendered, encoded, decoded, and then displayed (e.g., at the client). However, the frame ultimately displayed on the client device (e.g., glasses) may be delayed by a certain amount, such as by a legacy vertical synchronization (VSync) (e.g., the frame was rendered / encoded one VSync time unit before). Similarly, certain types of data (e.g., camera data) can be delayed from the client device (e.g., glasses) to the server (e.g., a phone). However, based on several different factors (e.g., scene complexity, system workload, etc.), some frames (e.g., multiple frames) may be ready to display earlier than the time specified by the VSync timing. In some aspects, these frames, which are normally ready, may be delayed due to the occurrence of different frame problems (e.g., frame duplication or frame loss). In these types of scenes, motion-to-render-to-photon (M2R2P) latency and user experience may be less than ideal. That is, suboptimal display performance may exist due to fixed performance settings or configurations. For example, suboptimal end-to-end performance may exist in a split-rendering pipeline due to fixed performance settings of multiple components in the pipeline from a server (e.g., a phone) to a client device (e.g., glasses). Aspects of this disclosure can improve or mitigate power and / or performance issues in a split-rendering pipeline. For example, the aspects presented herein can tune performance and content quality to mitigate multiple display / frame issues (e.g., frame loss / duplication and / or M2R2P latency). Additionally, aspects of this disclosure can improve the overall user experience in a split-rendering pipeline. To this end, aspects of this disclosure can dynamically tune performance and / or content quality in a split-rendering pipeline. For example, the aspects presented herein can perform dynamic performance tuning and / or dynamic content quality tuning. That is, the aspects presented herein can improve the overall user experience in a split-rendering pipeline (e.g., improve M2R2P latency and / or frame loss / duplication issues) and reduce power and / or thermal issues by utilizing dynamic performance tuning and / or dynamic content quality tuning.

[0021] The aspects disclosed herein may include a number of benefits or advantages. For example, the aspects presented herein can improve the overall user experience of split rendering. That is, the aspects presented herein can utilize dynamic performance tuning that helps reduce frame duplication / loss and improve overall latency. Furthermore, the aspects presented herein can utilize dynamic performance tuning that helps improve content quality and / or user experience. The aspects presented herein can also adjust or reduce the amount of power used for split rendering. For example, the aspects presented herein can utilize dynamic performance tuning that can help reduce the performance of multiple subsystems in any possible way. Additionally, the aspects presented herein can utilize dynamic performance tuning that can help reduce power consumption and thermal power.

[0022] Various aspects of the systems, apparatuses, computer program products, and methods will be described more fully below with reference to the accompanying drawings. However, this disclosure may be embodied in many different forms and should not be construed as limited to any particular structure or function presented throughout this disclosure. Rather, these aspects are provided to make this disclosure comprehensive and complete, and to fully convey the scope of this disclosure to those skilled in the art. Based on the teachings herein, those skilled in the art will understand that the scope of this disclosure is intended to cover any aspect of the systems, apparatuses, computer program products, and methods disclosed herein, whether implemented independently of or in combination with other aspects of this disclosure. For example, any number of aspects set forth herein may be used to implement an apparatus or practice. Furthermore, the scope of this disclosure is intended to cover such apparatuses or methods implemented using structures, functionalities, or structures and functionalities other than or different from the various aspects of the disclosure set forth herein. Any aspect disclosed herein may be embodied by one or more elements of the claims.

[0023] Although various aspects are described herein, many variations and substitutions of these aspects fall within the scope of this disclosure. While some potential benefits and advantages of the aspects of this disclosure are mentioned, the scope of this disclosure is not intended to be limited to a particular benefit, use, or objective. Rather, the aspects of this disclosure are intended to be broadly applicable to different wireless technologies, system configurations, networks, and transmission protocols, some of which are illustrated by way of example in the accompanying drawings and the description below. The detailed description and drawings are merely illustrative and not limiting of this disclosure, and the scope of this disclosure is defined by the appended claims and their equivalents.

[0024] Several aspects are presented with reference to various apparatuses and methods. These apparatuses and methods are described in detail and illustrated in the accompanying drawings by various blocks, components, circuits, processes, algorithms, etc. (collectively referred to as "elements"). These elements can be implemented using electronic hardware, computer software, or any combination thereof. Whether such elements are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system.

[0025] For example, an element, any part of an element, or any combination of elements can be implemented as a “processing system” including one or more processors (which may also be referred to as processing units). Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), general-purpose GPUs (GPGPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, system-on-a-chip (SoCs), baseband processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic units, discrete hardware circuits, and other suitable hardware configured to perform the various functionalities described in this disclosure. One or more processors in a processing system can execute software. Software can be broadly interpreted as instructions, instruction sets, code, code segments, program code, programs, subroutines, software components, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether expressed in terms of software, firmware, middleware, microcode, hardware description languages, or other terms. The term “application” can refer to software. As described herein, one or more technologies can refer to an application, i.e., software, configured to perform one or more functions. In such examples, the application may be stored on memory (e.g., on-chip memory of a processor, system memory, or any other memory). Hardware described herein, such as a processor, may be configured to execute the application. For example, an application may be described as including code that, when executed by the hardware, causes the hardware to perform one or more technologies described herein. As an example, the hardware may access and execute code accessed from memory to perform one or more technologies described herein. In some examples, components are identified in this disclosure. In such examples, a component may be hardware, software, or a combination thereof. Each component may be a separate component or a subcomponent of a single component.

[0026] Therefore, in one or more examples described herein, the described functionality can be implemented in hardware, software, or any combination thereof. If implemented in software, the functionality can be stored or encoded as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer storage media. Storage media can be any available medium accessible to a computer. By way of example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disc storage devices, magnetic disk storage devices, other magnetic storage devices, combinations of computer-readable media of the types described above, or any other medium capable of being used to store computer-executable code in the form of instructions or data structures accessible to a computer.

[0027] In summary, this disclosure describes techniques for having a graphics processing pipeline in a single device or multiple devices, thereby improving the rendering of graphics content and / or reducing the load on processing units (i.e., any processing unit, such as a GPU, configured to perform one or more of the techniques described herein). For example, this disclosure describes techniques for performing graphics processing in any device that utilizes graphics processing. Other example benefits are described throughout this disclosure.

[0028] As used herein, instances of the term "content" can refer to "graphic content," "image," or vice versa. This is true regardless of whether these terms are used as adjectives, nouns, or other parts of speech. In some examples, as used herein, the term "graphic content" can refer to content produced by one or more processes in a graphics processing pipeline. In some examples, as used herein, the term "graphic content" can refer to content produced by a processing unit configured to perform graphics processing. In some examples, as used herein, the term "graphic content" can refer to content produced by a graphics processing unit.

[0029] In some examples, as used herein, the term "display content" can refer to content generated by a processing unit configured to perform display processing. Graphical content can be processed to become display content. For example, a graphics processing unit can output graphical content (such as frames) to a buffer (which may be referred to as a frame buffer). A display processing unit can read graphical content (such as one or more frames) from the buffer and perform one or more display processing techniques on that display processing unit to generate display content. For example, a display processing unit can be configured to perform compositing on one or more rendering layers to generate frames. As another example, a display processing unit can be configured to composite, blend, or otherwise combine two or more layers into a single frame. A display processing unit can be configured to perform scaling on frames, such as zooming in or out. In some examples, a frame can refer to a layer. In other examples, a frame can refer to two or more layers that have been blended together to form the frame, i.e., the frame comprises two or more layers, and the frame comprising two or more layers can be subsequently blended.

[0030] Figure 1 This is a block diagram illustrating an example content generation system 100 configured to implement one or more technologies of this disclosure. The content generation system 100 includes a device 104. Device 104 may include one or more components or circuitry for performing the various functions described herein. In some examples, one or more components of device 104 may be components of a System-on-a-Chip (SOC). Device 104 may include one or more components configured to perform one or more technologies of this disclosure. In the illustrated example, device 104 may include a processing unit 120, a content encoder / decoder 122, and a system memory 124. In some aspects, device 104 may include several components, such as a communication interface 126, a transceiver 132, a receiver 128, a transmitter 130, a display processor 127, and one or more displays 131. A reference to display 131 may refer to one or more displays 131. For example, display 131 may include a single display or multiple displays. Display 131 may include a first display and a second display. The first display may be a left-eye display, and the second display may be a right-eye display. In some examples, the first and second displays may receive different frames for presentation on the first and second displays. In other examples, the first and second displays may receive the same frames used for rendering on both displays. In further examples, the results of graphics processing may not be displayed on the devices; for example, the first and second displays may not receive any frames used for rendering on them. Instead, the frames or graphics processing results may be transferred to another device. In some respects, this can be referred to as split rendering.

[0031] Processing unit 120 may include internal memory 121. Processing unit 120 may be configured to perform graphics processing, such as in a graphics processing pipeline 107. Content encoder / decoder 122 may include internal memory 123. In some examples, device 104 may include a display processor (such as display processor 127) to perform one or more display processing techniques on one or more frames generated by processing unit 120 prior to presentation by one or more displays 131. Display processor 127 may be configured to perform display processing. For example, display processor 127 may be configured to perform one or more display processing techniques on one or more frames generated by processing unit 120. One or more displays 131 may be configured to display or otherwise present the frames processed by display processor 127. In some examples, the one or more displays 131 may include one or more of the following: liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, projection display device, augmented reality display device, virtual reality display device, head-mounted display, or any other type of display device.

[0032] Memory (such as system memory 124) external to processing unit 120 and content encoder / decoder 122 may be accessible to processing unit 120 and content encoder / decoder 122. For example, processing unit 120 and content encoder / decoder 122 may be configured to read from and / or write to external memory (such as system memory 124). Processing unit 120 and content encoder / decoder 122 may be communicatively coupled to system memory 124 via a bus. In some examples, processing unit 120 and content encoder / decoder 122 may be communicatively coupled to each other via this bus or a different connection.

[0033] Content encoder / decoder 122 can be configured to receive graphic content from any source, such as system memory 124 and / or communication interface 126. System memory 124 can be configured to store received encoded or decoded graphic content. Content encoder / decoder 122 can be configured to receive encoded or decoded graphic content from system memory 124 and / or communication interface 126, for example, in the form of encoded pixel data. Content encoder / decoder 122 can be configured to encode or decode any graphic content.

[0034] Internal memory 121 or system memory 124 may include one or more volatile or non-volatile memories or storage devices. In some examples, internal memory 121 or system memory 124 may include RAM, SRAM, DRAM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic data media or optical storage media or any other type of memory.

[0035] According to some examples, internal memory 121 or system memory 124 may be a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or propagating signal. However, the term "non-transitory" should not be construed as meaning that internal memory 121 or system memory 124 is immovable or that its contents are static. For example, system memory 124 may be removed from device 104 and moved to another device. Alternatively, system memory 124 may not be removable from device 104.

[0036] Processing unit 120 may be a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose GPU (GPGPU), or any other processing unit configured to perform graphics processing. In some examples, processing unit 120 may be integrated into the motherboard of device 104. In some examples, processing unit 120 may reside on a graphics card mounted in a port on the motherboard of device 104, or may otherwise be incorporated into a peripheral device configured to interoperate with device 104. Processing unit 120 may include one or more processors, such as one or more microprocessors, GPUs, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), discrete logic components, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the technology is partially implemented in software, processing unit 120 may store instructions for software in a suitable non-transitory computer-readable storage medium (e.g., internal memory 121) and may use one or more processors to execute instructions in hardware to perform the technology of this disclosure. Any of the above (including hardware, software, and combinations of hardware and software, etc.) can be considered as one or more processors.

[0037] The content encoder / decoder 122 can be any processing unit configured to perform content decoding. In some examples, the content encoder / decoder 122 may be integrated into the motherboard of device 104. The content encoder / decoder 122 may include one or more processors, such as one or more microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), video processors, discrete logic components, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the technology is partially implemented in software, the content encoder / decoder 122 may store instructions for software in a suitable non-transitory computer-readable storage medium (e.g., internal memory 123) and may use one or more processors to execute instructions in hardware to perform the technology of this disclosure. Any of the foregoing (including hardware, software, combinations of hardware and software, etc.) can be considered as one or more processors.

[0038] In some aspects, the content generation system 100 may include a communication interface 126. The communication interface 126 may include a receiver 128 and a transmitter 130. The receiver 128 may be configured to perform any of the receiving functions described herein with respect to device 104. Additionally, the receiver 128 may be configured to receive information from another device, such as eye or head positioning information, rendering commands, or location information. The transmitter 130 may be configured to perform any of the transmitting functions described herein with respect to device 104. For example, the transmitter 130 may be configured to transmit information to another device, which may include a request for content. The receiver 128 and the transmitter 130 may be combined to form a transceiver 132. In such an example, the transceiver 132 may be configured to perform any of the receiving and / or transmitting functions described herein with respect to device 104.

[0039] Refer again Figure 1In some aspects, the display processor 127 may include an adjustment component 198 configured to obtain an indication of a set of frames before estimating the set of frame processing times. The adjustment component 198 may also be configured to train a machine learning (ML) model on the set of frames before estimating the set of frame processing times. The adjustment component 198 may also be configured to estimate the set of frame processing times for at least one first frame in the set of frames. The adjustment component 198 may also be configured to detect the actual set of frame processing times for at least one first frame in the set of frames. The adjustment component 198 may also be configured to output an indication of a second set of frame processing times for at least one second frame in the set of frames based on the estimated set of frame processing times and the actual set of frame processing times, wherein the at least one second frame follows at least one first frame in the set of frames. The adjustment component 198 may also be configured to adjust the second set of frame processing times for at least one second frame based on the estimated set of frame processing times and the actual set of frame processing times. Although the following description may focus on display processing, the concepts described herein are applicable to other similar processing techniques.

[0040] As described herein, a device such as device 104 can refer to any device, apparatus, or system configured to perform one or more of the technologies described herein. For example, a device can be a server, base station, user equipment, client device, station, access point, computer (e.g., personal computer, desktop computer, laptop computer, tablet computer, computer workstation, or mainframe computer), end product, apparatus, telephone, smartphone, server, video game platform or console, handheld device (e.g., portable video game device or personal digital assistant (PDA)), wearable computing device (e.g., smartwatch, augmented reality device, or virtual reality device), non-wearable device, display or display device, television, set-top box, intermediate network device, digital media player, video streaming device, content streaming device, in-vehicle computer, any mobile device, any device configured to generate graphical content, or any device configured to perform one or more of the technologies described herein. The processes described herein may be described as being performed by a specific component (e.g., GPU), but in further embodiments, other components consistent with the disclosed embodiments (e.g., CPU) may be used to perform them.

[0041] A GPU can process various types of data or data packets within its pipeline. For example, in some aspects, a GPU can process two types of data or data packets, such as context register packets and draw call data. Context register packets can be a collection of global state information, such as information about global registers, shaders, or constant data, which can regulate how the graphics context will be handled. For example, a context register packet may include information about the color format. In some aspects of a context register packet, there may be bits indicating which workload belongs to the context register. Furthermore, there may be multiple functions or programs running simultaneously and / or in parallel. For example, a function or program may describe an operation, such as a color mode or color format. Therefore, context registers can define multiple states of the GPU.

[0042] Context states can be used to determine how individual processing units (e.g., vertex extractors (VFDs), vertex shaders (VSs), shader processors, or geometry processors) operate and / or in which mode a processing unit operates. For this purpose, the GPU can use context registers and programming data. In some aspects, the GPU can generate workloads (e.g., vertex or pixel workloads) in the pipeline based on the context register definitions of modes or states. Certain processing units (e.g., VFDs) can use these states to determine certain functions, such as how to assemble vertices. Because these modes or states can change, the GPU may need to modify the corresponding context. Additionally, the workload corresponding to a mode or state may follow the changed mode or state.

[0043] Figure 2 Example GPU 200 is illustrated according to one or more technologies according to this disclosure. For example... Figure 2 As shown, GPU 200 includes a command processor (CP) 210, a draw call group 212, a VFD 220, a VS 222, a vertex cache (VPC) 224, a triangle setup engine (TSE) 226, a rasterizer (RAS) 228, a Z-process engine (ZPE) 230, a pixel interpolator (PI) 232, a fragment shader (FS) 234, a rendering backend (RB) 236, a level 2 (L2) cache (UCHE) 238, and system memory 240. Although Figure 2 The GPU 200 shown includes processing units 220 to 238, but the GPU 200 may include multiple additional processing units. Additionally, processing units 220 to 238 are merely examples, and any combination or order of processing units may be used in the GPU according to this disclosure. The GPU 200 also includes a command buffer 250, a context register group 260, and a context state 261.

[0044] like Figure 2As shown, the GPU can use a CP (e.g., CP 210) or a hardware accelerator to resolve the command buffer into context register groups (e.g., context register group 260) and / or draw call data groups (e.g., draw call group 212). Subsequently, CP 210 can transfer the context register group 260 or the draw call group 212 to a processing unit or block in the GPU via a separate path. Furthermore, the command buffer 250 can alternate between different states of the context registers and draw calls. For example, the command buffer can be structured as follows: context register of context N, draw call of context N, context register of context N+1, and draw call of context N+1.

[0045] GPUs can render images in several different ways. In some cases, GPUs can use rendering and / or tiled rendering to render images. In a tiled rendering GPU, an image can be divided or segmented into different sections or tiles. After the image is divided, each section or tile can be rendered individually. A tiled rendering GPU divides a computer graphics image into a grid format so that each part of the grid (i.e., a tile) is rendered individually. In some aspects, during a binning pass, the image can be divided into different bins or tiles. In some aspects, during a binning pass, a visibility stream can be constructed, where visible primitives or draw calls can be identified. In contrast to tiled rendering, direct rendering does not divide the frame into smaller bins or tiles. Instead, in direct rendering, the entire frame is rendered at once. Additionally, some types of GPUs allow both tiled rendering and direct rendering.

[0046] Figure 3 This is a block diagram 300 illustrating an example display frame including a processing unit 120, system memory 124, display processor 127, and display 131, as identified by incorporable device 104.

[0047] A GPU may be included in a device that provides content for visual presentation on a display. For example, processing unit 120 may include GPU 310 configured to render graphics data for display on a computing device (e.g., device 104), which may be a computer workstation, mobile phone, smartphone or other intelligent device, embedded system, personal computer, tablet computer, video game console, etc. The operation of GPU 310 may be controlled based on one or more graphics processing commands provided by CPU 315. CPU 315 may be configured to execute multiple applications concurrently. In some cases, each of the concurrently executing applications may utilize GPU 310 simultaneously. Processing techniques may be executed by outputting frames over a physical or wireless communication channel via processing unit 120.

[0048] System memory 124, executable by processing unit 120, may include user space 320 and kernel space 325. User space 320 (sometimes referred to as "application space") may include software applications and / or application frameworks. For example, software applications may include operating systems, media applications, graphics applications, workspace applications, etc. Application frameworks may include frameworks used by one or more software applications, such as libraries, services (e.g., display services, input services, etc.), application programming interfaces (APIs), etc. Kernel space 325 may further include display driver 330. Display driver 330 may be configured to control display processor 127. For example, display driver 330 may cause display processor 127 to synthesize frames and send data of the frames to the display.

[0049] Display processor 127 includes display control block 335 and display interface 340. Display processor 127 can be configured to (e.g., based on input received from display driver 330) manipulate the functions of display 131. Display control block 335 can also be configured to output image frames to display 131 via display interface 340. In some examples, display control block 335 may additionally or alternatively perform post-processing of image data provided based on the processing unit 120's execution of system memory 124.

[0050] Display interface 340 can be configured to cause display 131 to display image frames. Display interface 340 can output image data to display 131 according to an interface protocol, such as, for example, MIPI DSI (Mobile Industry Processor Interface, Display Serial Interface) . That is, display 131 can be configured according to the MIPI DSI standard. The MIPI DSI standard supports video mode and command mode. In an example where display 131 operates in video mode, display processor 127 can continuously refresh the graphic content of display 131. For example, the entire graphic content can be refreshed in each refresh cycle (e.g., line by line). In an example where display 131 operates in command mode, display processor 127 can write the graphic content of a frame to buffer 350.

[0051] In some such examples, display processor 127 may not continuously refresh the graphics content of display 131. Instead, display processor 127 may use a vertical sync (Vsync) pulse to coordinate the rendering and consumption of graphics content at buffer 350. For example, when a Vsync pulse is generated, display processor 127 may output new graphics content to buffer 350. Thus, the generation of a Vsync pulse indicates that current graphics content has been rendered at buffer 350.

[0052] A frame is displayed on the monitor 131 based on the display controller 345, the display client 355, and the buffer 350. The display controller 345 can receive image data from the display interface 340 and store the received image data in the buffer 350. In some examples, the display controller 345 can output the image data stored in the buffer 350 to the display client 355. Therefore, the buffer 350 can represent local memory of the monitor 131. In some examples, the display controller 345 can directly output the image data received from the display interface 340 to the display client 355.

[0053] Display client 355 may be associated with a touch panel that senses interaction between the user and display 131. When the user interacts with display 131, one or more sensors in the touch panel may output signals to display controller 345 indicating which of the one or more sensors is active, the duration of the sensor activity, the pressure applied to the one or more sensors, etc. Display controller 345 may use the sensor outputs to determine how the user interacts with display 131. Display 131 may further be associated with / include other devices such as cameras, microphones, and / or speakers that operate in conjunction with display client 355.

[0054] Some processing techniques of device 104 can be performed through three stages (e.g., stage 1: rendering stage; stage 2: compositing stage; and stage 3: display / transfer stage). However, other processing techniques can combine the compositing stage and the display / transfer stage into a single stage, allowing the processing technique to be performed based on a total of two stages (e.g., stage 1: rendering stage; and stage 2: compositing / display / transfer stage). During the rendering stage, GPU 310 can process the content buffer based on the execution of the application that generates content on a pixel-by-pixel basis. During the compositing and display stages, pixel elements can be assembled to form a frame, which is then transferred to the physical display panel / subsystem (e.g., display 131) that displays the frame.

[0055] Instructions executed by the CPU (e.g., software instructions) or by the display processor can cause the CPU or display processor to search for and / or generate compositing strategies for compositing frames based on dynamic priorities and runtime statistics associated with one or more compositing strategy groups. A frame to be displayed by a physical display device (such as a display panel) may include multiple layers. Furthermore, frame compositing may be based on combining multiple layers into a frame (e.g., based on a frame buffer). After combining the multiple layers into a frame, the frame can be provided to the display panel for display on that display panel. The process of combining each of the multiple layers into a frame may be referred to as compositing, frame compositing, compositing process, compositing handling, etc.

[0056] A frame compositing process or strategy can correspond to a technique used to combine different layers from multiple layers into a single frame. Multiple layers can be stored in double data rate (DDR) memory. Each of the multiple layers can further correspond to a separate buffer. A compositor or hardware compositor (HWC) associated with a block or function determines the input to each layer / buffer and performs the frame compositing process to generate an output indicating the composite frame. That is, the input can be layers, and the output can be a frame compositing process used to synthesize the frame to be displayed on a display panel.

[0057] In some respects, the display device can present frames at different frame rates on the first and second display panels. For example, the display panels can present frames at 60 frames per second (FPS) on both the first and second display panels, or at 45 FPS on both the first and second display panels, and so on. The display device can synchronize the frame rate of the content with the refresh rate of the display panels (via a vertical synchronization process, which may be referred to as vsync, Vsync, VSync, or VSYNC). For example, the content may be available at 60 FPS, and the first and second display panels may have a refresh rate of 95 Hz. Through Vsync, the refresh rate of the first and second display panels can be set to 60 Hz to match the 60 FPS content.

[0058] As indicated in this article, VSync is a graphics technique that synchronizes the frame rate of an application / game with the refresh rate of the display (e.g., a monitor on a client device). VSync can be used as a way to handle screen tearing (i.e., sections of the screen displaying multiple frames at a time). This can cause the display to appear split along lines. Tearing can occur when the display refresh rate (i.e., how many times the display updates per second) is out of sync with the frames per second (FPS). VSync signals can synchronize the display pipeline (e.g., a pipeline that includes application rendering, compositors, and a hardware compositor (HWC) that renders the image on the display). For example, VSync signals can help synchronize the time when the application wakes up to begin rendering, the time when the compositor wakes up to composite the screen, and the display refresh cycle. This synchronization can help eliminate display refresh problems and improve visual performance. In some examples, the HWC can generate VSync events / signals and send these events / signals to the compositor.

[0059] In some aspects of graphics processing, content rendering can be performed at multiple locations and / or on multiple devices, for example, to divide the rendering workload among different devices. For instance, rendering can be split between a server and client devices; this can be referred to as "split rendering." In some cases, split rendering can be a method for bringing content to a client device, where a portion of the graphics processing can be performed outside the client device (e.g., at a server). In some aspects, the server can be at least one of the following: a telephone, a smartphone, a computer, or a cloud server. Furthermore, the client device can be at least one of the following: a headset, a head-mounted display (HMD), display glasses, or smart glasses.

[0060] Split rendering can be performed for multiple different types of applications, such as virtual reality (VR), augmented reality (AR), mixed reality (MR), and / or extended reality (XR) applications. In VR applications, the content displayed on the client device may correspond to artificial or animated content. In XR, AR, or MR content, a portion of the content displayed on the client device may correspond to real-world content (e.g., real-world objects), and a portion of the content may be artificial or animated content. Moreover, artificial or animated content and real-world content can be displayed in optical or video perspective devices, allowing users to view real-world objects and artificial or animated content simultaneously. In some respects, artificial or animated content can be referred to as augmented content, and vice versa. Split XR, AR, or MR systems may also introduce latency in delivering the rendered content to the client display. In some respects, this latency may even be higher when rendering on a server compared to client-side rendering, but it can also enable more complex XR, AR, or MR applications. Furthermore, there may be a non-negligible latency between the time it takes to calculate the camera pose and the time it takes for the content to appear on the client display. For example, there may be a certain amount of latency in split XR, AR, or MR systems.

[0061] Figure 4 Diagram 400 illustrates content / data communication based on the split rendering process. For example... Figure 4 As shown, Figure 400 includes a server 410 and a client device 450 associated with the split rendering process. Figure 4 Multiple processes executed at server 410 and client device 450 are illustrated, including encoding process 420, packaging process 430, unpacking process 470, and decoding process 480. Server 410 and client device 450 also include sending component 440 and receiving component 460, respectively.

[0062] like Figure 4As shown, on server 410, data / content associated with an image / frame can be encoded during encoding process 420. After encoding process 420, the data / content can then undergo a packetization process 430, such as a Real-time Transport Protocol (RTP) packetization process. During the packetization process, the data / content can be converted into one or more frames 442. Frames 442 can then be sent from the sending component 440 of server 410 to the receiving component 460 of client device 450. In some cases, frames can be sent via User Datagram Protocol (UDP) Internet Protocol (IP) (UDP / IP) network protocol, Transmission Control Protocol (TCP) IP (TCP / IP) network protocol, or any other network protocol. On client device 450, frames 442 can be received via receiving component 460 (e.g., via UDP / IP network protocol, TCP / IP network protocol, or any other network protocol). Frames 442 can also undergo unpacking process 470 (e.g., a Real-time Transport Protocol (RTP) unpacking process or any other protocol unpacking process), which can convert packets into data / content. After unpacking, the data / content can be decoded during the decoding process 480. Finally, the decoded data / content can be transmitted to the display or HMD of the client device 450 for display.

[0063] As indicated above, various aspects of graphics processing can handle the rendering or display of different types of content, such as virtual reality (VR) applications, augmented reality (AR) applications, mixed reality (MR) applications, and / or extended reality (XR) applications. Content can be rendered or created on a server (e.g., a computer or telephone). To display this content, users can utilize different types of headsets or display glasses, which may be referred to as client devices. In some cases, when users want to use XR glasses for extended periods without the availability of charging facilities, it is desirable to conserve power at the server or client device. Furthermore, it is desirable to conserve power at the server or client device when the battery level of the client device or server becomes low (i.e., exceeds a threshold percentage determined by the user). Additionally, if users want to actively extend battery life, it is desirable to conserve power at the server or client device and provide longer battery life to either device.

[0064] In split-rendering applications, content can be rendered on a server and encoded / streamed via Wi-Fi to an XR-based HMD. As indicated above, split-rendering means distributing the XR workload between two devices (i.e., the host / server and the client / HMD). One use case, for example, could be a smartphone connected to an HMD / AR glasses. AR glasses may not have high processing power, and heat dissipation could be an issue if all processing is performed on the client / glasses. Therefore, splitting rendering between the server and client devices is beneficial.

[0065] In one aspect, poses (e.g., six-degrees-of-freedom (6DOF) poses) can be generated on the client device. The client / HMD can then transmit the 6DOF pose data to a server via an uplink connection. An application or game can then use the transmitted 6DOF poses to render content on the server / phone. Furthermore, the encoding of the rendered content can occur on the server / phone. The encoded and compressed bitstream can then be sent from the server / phone to the HMD / client via a downlink connection. Afterward, video decoding and time-warping can be performed on the HMD / client using the latest 6DOF poses. Finally, the HMD / client can display the reprojected content.

[0066] Figure 5 Figure 500 illustrates an example timeline of the split rendering process. More specifically, Figure 5 A diagram 500 illustrates the timeline of different processing steps at a server (e.g., a phone, smartphone, or computer) and a client device (e.g., a headset, HMD, or smart glasses). For example, the client device can send multiple poses 510 (e.g., head poses) to the server. The server can then render the content of the frame at a rendering process 520 and encode the frame at an encoding process 530. Furthermore, the server can send the frame to the client device via a downlink (DL) 540. After receiving the frame, the client device can decode the frame at a decoding process 550. Figure 5 The vertical synchronization (VSync) 560 associated with each transmission is also shown.

[0067] like Figure 5As shown, head pose data can be sent from the client device to the server at a high rate (e.g., 500Hz) and / or low latency (via uplink (UL)). The client device (e.g., HMD / glasses) may be unaware of the server's (e.g., phone) rendering start time. The first frame can be rendered on the server at any time using the latest pose, followed by future frames rendered at a pre-configured frame rate per second (fps or FPS). The rendering thread can render frames as quickly as the GPU allows without any waiting time, and to limit fps, a waiting time can be added at the end of each rendering. Additionally, the rendering thread can sleep until the waiting time before the next frame's rendering begins. Furthermore, during rendering, each rendered frame can be immediately queued for encoding. Once encoding is complete, the encoded frames can be packaged and sent at any time (e.g., after the rendering time plus the encoding time) (via downlink (DL)). In some examples, the Wi-Fi modem can always be "on," allowing poses and frames to be sent with minimal latency. In addition, in some examples, multiple different types of information or data can be sent on the UL side (e.g., camera streaming data, color data or red (R) green (G) blue (B) (RGB) data, hand tracking data and / or three-dimensional (3D) rendering (3DR) data).

[0068] Split rendering can leverage several different features, such as Target Wake-up Time (TWT) and Timed Synchronization Function (TSF). The TWT feature allows the modem / radio frequency (RF) to connect at a fixed rhythm and continue for a known service period. This TWT feature can be used to save power on both server and client devices. While TWT ensures reduced power on the modem side, the choice of TWT parameters can affect XR performance, such as latency and frame multiplexing (i.e., jitter). In some cases, the TWT feature can allow UL (pose) data and DL (rendering + encoding) frame data to align with the TWT service period (on-hook period), meaning that transmission (Tx) and reception (Rx) on the client device and server can occur simultaneously. Simultaneous data transmission within the same service period provides the modem with an opportunity to sleep for a certain period, thus reducing power and thermal issues. In some split XR scenarios, each client can have a timer synchronized with the Timed Synchronization Function (TSF), which is associated with the server, for example. Additionally, early termination allows the service period to be terminated early upon detection of inactivity in DL / UL data. Early termination procedures can include End of Service Period (EOSP). For example, ESOP can terminate a service interval (i.e., change a service interval from an "on" period to an "off" period). TWT allows modems to turn on and off at a defined pace. TWT can also support early termination, such as allowing a modem to turn off if there is no data to transmit within a defined time period.

[0069] As indicated herein, there may be multiple subsystems involved in an end-to-end XR pipeline. For example, CPUs, GPUs, encoders, decoders, networks, servers (e.g., smartphones), and / or client devices (e.g., headsets, HMDs, or AR glasses) may be involved in an end-to-end XR pipeline. Some types of client devices (e.g., wireless AR glasses) may require a sleek and lightweight design / form factor, which can cause a variety of different issues, such as battery consumption (e.g., approximately 800mW for a system-on-a-chip (SoC) and double data rate (DDR) memory for some devices) and / or heat dissipation. Additionally, to achieve a high-quality user experience, servers and / or client devices may expect certain display characteristics or conditions. For example, for a high-quality user experience, a minimum amount of motion-to-render-to-photon (M2R2P) latency may be desired. Furthermore, servers and / or client devices may expect a minimum amount of frame loss or duplication.

[0070] In some respects, once the head pose is sent from the client device (e.g., AR glasses) to the server (e.g., a phone), the new scene can be rendered, encoded, decoded, and then displayed (e.g., displayed at the client). However, the frame ultimately displayed on the client device (e.g., AR glasses) may be delayed by a certain amount, such as an older VSync (e.g., the frame was rendered / encoded one VSync time unit before). Similarly, certain types of data (e.g., RGB camera data) may be encoded, sent, and decoded from the client device (e.g., AR glasses) to the server (e.g., a phone) in a delayed manner. However, based on several different factors (e.g., scene complexity, system workload, etc.), some frames (e.g., DL frames) may be ready to display earlier than the time specified by the VSync timing. In some respects, these frames, which would normally be ready, may be delayed due to various frame issues (e.g., due to variable rendering rates, higher encoding times, and higher workloads on subsystems such as CPU, GPU, encoding, decoding, higher streaming latency, etc.), resulting in frame duplication or frame loss. In these types of scenarios, motion-to-render-to-photon (M2R2P) latency and user experience may be suboptimal. That is, suboptimal display performance may exist due to fixed performance settings or configurations. For example, suboptimal end-to-end performance may exist in a split-render pipeline due to fixed performance settings of multiple components in the pipeline from the server (e.g., a phone) to the client device (e.g., an HMD or AR glasses). Based on the above, improving power and / or performance issues in the split-render pipeline may be beneficial. For example, adjusting performance and content quality to mitigate the aforementioned problems (e.g., frame loss / duplication and / or M2R2P latency) may be beneficial. Additionally, improving the overall user experience in the split-render pipeline may be beneficial.

[0071] The aspects of this disclosure can improve or mitigate power and / or performance issues in split rendering pipelines (e.g., split XR / AR / MR rendering pipelines). For example, the aspects presented herein can tune performance and content quality to mitigate multiple display / frame issues (e.g., frame loss / duplication and / or M2R2P latency). Additionally, the aspects of this disclosure can improve the overall user experience in split rendering pipelines (e.g., split XR / AR / MR rendering pipelines). To this end, the aspects of this disclosure can dynamically tune performance and / or content quality in split rendering pipelines. For example, the aspects presented herein can perform dynamic performance tuning and / or dynamic content quality tuning. That is, the aspects presented herein can improve the overall user experience in split rendering pipelines (e.g., improve M2R2P latency and / or frame loss / duplication issues) and reduce power and / or thermal issues by utilizing dynamic performance tuning and / or dynamic content quality tuning.

[0072] In some cases, aspects of this disclosure may utilize Dynamic Performance Adjustment (DPA) and Dynamic Power Adjustment. In the DPA utilized in the aspects presented herein, the performance of each subsystem in the XR pipeline can be adjusted based on predicted and current performance specifications. Furthermore, the aspects presented herein may adjust content quality based on predicted (i.e., estimated) and current (i.e., actual) performance specifications. Dynamic performance adjustment can predict or estimate the amount of frame processing time utilized in a split XR architecture. For example, dynamic performance adjustment can predict: (1) rendering time, (2) encoding time (e.g., encoding time for rendering frames and camera frames), (3) frame transmission time, (4) decoding time (e.g., decoding time for rendering frames and camera frames), and (5) other software latency in the split XR pipeline. Furthermore, the dynamic performance adjustment presented herein can monitor the current rendering time, encoding time, transmission time, and decoding time with respect to frame prediction. Furthermore, if any pipeline component takes longer than predicted, the dynamic performance tuning presented in this paper can improve subsystem performance and / or adjust content quality. By doing so, the aspects presented in this paper can reduce the amount of frame loss / duplication. Additionally, when pipeline events complete earlier than expected, the dynamic performance tuning presented in this paper may reduce the performance of subsequent models, thus benefiting power. This can also improve the content quality produced in a split XR pipeline.

[0073] Furthermore, the aspects presented herein can predict / estimate the number of frame processing times for frames (e.g., at least one first frame) in a set of frames associated with split rendering (e.g., split XR rendering involving both client devices and servers). For example, the aspects presented herein can predict or estimate one or more of the following frame processing times for at least one first frame: pose time of at least one first frame, frame rendering time of at least one first frame, frame encoding time of at least one first frame, frame encryption time of at least one first frame, frame transmission time of at least one first frame, frame decryption time of at least one first frame, frame decoding time of at least one first frame, and / or frame reprojection time of at least one first frame. Pose time can be the time when the pose of at least one first frame is available at the server and / or the time when the server generates pose information for at least one first frame. This can include pose generation time on the client side, pose transmission time from the client to the server, and any other 6DOF latency. Frame rendering time can be the time when at least one first frame is rendered at the server or client device. Furthermore, frame encoding time can be the time when at least one first frame is encoded or compressed at the server. Frame encryption time can be the time when at least one first frame is encrypted at the server. Frame transmission time can be the time when at least one first frame is transmitted from the server to the client device. Furthermore, frame decryption time can be the time when at least one first frame is decrypted at the client device. Frame decoding time can be the time when at least one first frame is decoded at the client device. Frame reprojection time can be the time when at least one first frame is reprojected at the client device. Additionally, the aspects presented herein can detect / calculate the number of actual or current frame processing times for frames (e.g., at least one first frame) in a set of frames associated with split rendering (e.g., split XR rendering involving both client devices and servers). For example, the aspects presented herein can detect or calculate one or more of the following actual / current frame processing times for at least one first frame: pose time of at least one first frame, frame rendering time of at least one first frame, frame encoding time of at least one first frame, frame encryption time of at least one first frame, frame transmission time of at least one first frame, frame decryption time of at least one first frame, frame decoding time of at least one first frame, and / or frame reprojection time of at least one first frame.

[0074] The aspects presented in this paper may include a number of different details to tune performance and / or content quality within the split rendering pipeline. For example, to tune performance within the split rendering pipeline, the aspects presented in this paper may utilize task affinity (e.g., silver, gold, etc.) to migrate / move workloads between CPU cores. Furthermore, to tune performance within the split rendering pipeline, the aspects presented in this paper may utilize clock or corner tuning for certain components (e.g., CPU, GPU, encoder / decoder, etc.). Additionally, to tune performance within the split rendering pipeline, the aspects presented in this paper may utilize network performance tuning (e.g., modulation and decoding scheme (MCS) rate tuning, etc.). In some aspects, to tune content quality within the split rendering pipeline, the aspects presented in this paper may utilize encoding compression tuning (e.g., bitrate, group of pictures (GOP), etc.). Finally, to tune content quality within the split rendering pipeline, the aspects presented in this paper may adjust content resolution.

[0075] The aspects presented in this paper relate to dynamic performance and power tuning for end-to-end split-rendering XR systems and architectures. As indicated in this paper, an end-to-end split-rendering XR pipeline may involve multiple subsystems (e.g., CPU, GPU, encoder, decoder, etc.). Furthermore, client devices (e.g., AR glasses) may require lightweight construction due to potential battery consumption challenges. Additionally, for a high-quality user experience, minimal possible M2R2P latency and / or minimal (e.g., zero) frame loss or duplication may be desired. In some respects, once the head pose is sent from the client to the server and the new scene is rendered, encoded, decoded, and then displayed on the client device, the frames displayed on the client are already old. Therefore, M2R2P latency and user experience may not be optimal or suitable. Suboptimal end-to-end performance exists due to fixed performance settings of multiple components on the server and client devices (e.g., HMDs or AR glasses). The aspects presented in this paper can focus on improving the user experience of split rendering (e.g., M2R2P latency) and reducing thermal power using dynamic performance tuning and dynamic content quality tuning. The aspects presented in this paper can provide dynamic performance tuning, where each subsystem in the split rendering pipeline can be tuned (and / or content quality can be tuned) based on predicted / estimated and actual / current performance specifications or conditions. For example, dynamic performance tuning can estimate / predict performance specifications based on frame rendering time, frame encoding time, frame sending time, frame decoding time, and other split rendering pipeline latencies. Furthermore, the dynamic performance tuning in this paper can monitor the current / actual frame, current / actual frame rendering time, current / actual frame encoding time, current / actual frame sending time, and current / actual frame decoding time relative to the estimates / predictions. By doing so, if any component takes longer than predicted / estimated, the aspects presented in this paper can improve subsystem performance (and / or adjust content quality). This can also help reduce frame drops or frame duplication. Additionally, whenever a pipeline event completes earlier than expected, the aspects presented in this paper can reduce the performance of the next / subsequent model to favor power (and / or improve content quality).

[0076] Figure 6Figures 600, 602, 604, 610, 612, and 614 illustrate example timelines of a split rendering process involving client devices and servers. More specifically, Figures 600, 602, 604, 610, 612, and 614 depict timelines of different frame processing times at the server (e.g., a phone, smartphone, or computer) and client devices (e.g., headsets, HMDs, or smart glasses). That is, Figures 600 through 614 depict the difference between predicted frame processing times and actual frame processing times. For example, Figures 600, 602, and 604 include multiple predicted frame times (e.g., predicted pose time 622, predicted rendering time 624, predicted encoding time 626, predicted network streaming / transmission time 628, predicted decoding time 630, and predicted Vsync time 632). In some aspects, network (NW) streaming / transmission time may include DL transmission time, UL transmission time, and / or other software delays. Furthermore, Figures 610, 612, and 614 include multiple actual frame times (e.g., actual pose time 652, actual rendering time 654, actual encoding time 656, actual network streaming / transmission time 658, actual decoding time 660, and actual Vsync time 662).

[0077] like Figure 6 As shown, the predicted pose time 622 is the predicted / estimated pose time at the client device or server, and the actual pose time 652 is the actual / current pose time at the client device or server. The predicted rendering time 624 is the predicted / estimated rendering time at the client device or server, and the actual rendering time 654 is the actual / current rendering time at the client device or server. Furthermore, the predicted encoding time 626 is the predicted / estimated encoding time at the client device or server, and the actual encoding time 656 is the actual / current encoding time at the client device or server. The predicted network streaming / transmission time 628 is the predicted / estimated network transmission time of the frame, and the actual network streaming / transmission time 658 is the actual / current transmission time of the frame. The predicted decoding time 630 is the predicted / estimated decoding time at the client device or display device, and the actual decoding time 660 is the actual / current decoding time at the client device or display device. The predicted Vsync time 632 is the predicted / estimated Vsync time, and the actual Vsync time 662 is the actual / current Vsync time.

[0078] As shown in Figures 600 and 610, the actual rendering time 654 may take longer than the predicted rendering time 624. In other words, the actual / current rendering time may take longer than the predicted / estimated rendering time. This can lead to an increase in overall latency (e.g., M2R2P latency) and various frame issues (e.g., frame duplication and frame loss). To address this issue, the aspects presented herein can improve / enhance the performance of the current module (e.g., the GPU) and / or improve / enhance the performance of the next module (e.g., the encoding module as shown in 656, where the encoding module is enhanced).

[0079] Additionally, as shown in Figures 602 and 612, the actual rendering time 654 may take longer than the predicted rendering time 624. Furthermore, the actual encoding time 656 may take longer than the predicted encoding time 626. In other words, the actual / current rendering time and actual / current encoding take longer than the predicted / estimated rendering time and predicted / estimated encoding. This leads to an increase in overall latency (e.g., M2R2P latency) and various frame issues (e.g., frame duplication and frame loss). To address this issue, the aspects presented herein can improve / enhance the performance of the current module (e.g., GPU) and / or the next module (e.g., rendering module, encoding module, etc.), as well as improve / enhance network transmission performance.

[0080] Furthermore, as shown in Figures 604 and 614, the actual rendering time 654 may take longer than the predicted rendering time 624. Additionally, the actual encoding time 656 takes longer than the predicted encoding time 626. And the actual network streaming / transmission time 658 takes longer than the predicted network streaming / transmission time 628. In other words, the actual / current rendering time, actual / current encoding time, and actual / current DL take more time than the predicted / estimated rendering time, predicted / estimated encoding time, and predicted / estimated network streaming / transmission time. This leads to an increase in overall latency (e.g., M2R2P latency) and various frame issues (e.g., frame duplication and frame loss). To address this issue, the aspects presented herein can improve / increase the performance of the current module (e.g., GPU) and / or improve / increase the performance of the next module (e.g., encoding module), as well as improve / increase the performance of encoding time and network streaming / transmission time. The aspects presented herein can also improve / increase the performance of decoding time.

[0081] Figure 7Figures 700, 702, 704, 710, 712, and 714 illustrate example timelines of a split rendering process involving client devices and servers. More specifically, Figures 700, 702, 704, 710, 712, and 714 depict timelines of different frame processing times at the server (e.g., a phone, smartphone, or computer) and client devices (e.g., headsets, HMDs, or smart glasses). That is, Figures 700 through 714 depict the difference between predicted frame processing times and actual frame processing times. For example, Figures 700, 702, and 704 include multiple predicted frame times (e.g., predicted pose time 722, predicted rendering time 724, predicted encoding time 726, predicted network streaming / transmission time 728, predicted decoding time 730, and predicted Vsync time 732). In addition, Figures 710, 712 and 714 include multiple actual frame times (e.g., actual pose time 752, actual rendering time 754, actual encoding time 756, actual network streaming / transmission time 758, actual decoding time 760 and actual Vsync time 762).

[0082] like Figure 7 As shown, the predicted pose time 722 is the predicted / estimated pose time at the client device or server, and the actual pose time 752 is the actual / current pose time at the client device or server. The predicted rendering time 724 is the predicted / estimated rendering time at the client device or server, and the actual rendering time 754 is the actual / current rendering time at the client device or server. Furthermore, the predicted encoding time 726 is the predicted / estimated encoding time at the client device or server, and the actual encoding time 756 is the actual / current encoding time at the client device or server. The predicted network streaming / transmission time 728 is the predicted / estimated network transmission time of the frame, and the actual network streaming / transmission time 758 is the actual / current network transmission time of the frame. The predicted decoding time 730 is the predicted / estimated decoding time at the client device or display device, and the actual decoding time 760 is the actual / current decoding time at the client device or display device. The predicted Vsync time 732 is the predicted / estimated Vsync time, and the actual Vsync time 762 is the actual / current Vsync time.

[0083] As shown in Figures 700 and 710, the actual rendering time 754 may take less time than the predicted rendering time 724. In other words, the actual / current rendering time may take less time than the predicted / estimated rendering time. This allows frames to be available and displayed earlier than the rendering FPS and VSync specified. To address this issue, the aspects presented in this paper can slow down / degrade the performance of the next module, which helps reduce power consumption.

[0084] Additionally, as shown in Figures 702 and 712, the actual rendering time 754 may take less time than the predicted rendering time 724. Furthermore, the actual encoding time 756 takes less time than the predicted encoding time 726. In other words, the actual / current rendering time and actual / current encoding take less time than the predicted / estimated rendering time and predicted / estimated encoding. This allows frames to be available and displayed earlier than the rendering FPS and VSync specified times. To address this issue, the aspects presented herein can slow down / degrade the performance of the next module, which helps reduce power consumption, as well as slow down / adjust the performance of network transmissions to reduce power consumption.

[0085] Furthermore, as shown in Figures 704 and 714, the actual rendering time 754 may take less time than the predicted rendering time 724. Additionally, the actual encoding time 756 takes less time than the predicted encoding time 726. And the actual network streaming / transmission time 758 takes less time than the predicted network streaming / transmission time 728. In other words, the actual / current rendering time, actual / current encoding time, and actual / current DL time may take less time than the predicted / estimated rendering time, predicted / estimated encoding time, and predicted / estimated network streaming / transmission time. This allows frames to be available and displayed earlier than the rendering FPS and VSync specified times. To address this issue, the aspects presented herein slow down the next module (e.g., the encoding module) and slow down / adjust the performance of network transmissions to reduce power consumption. The aspects presented herein can also slow down / reduce the performance of decoding time.

[0086] As indicated herein, the aspects presented herein can predict rendering, encoding, decoding, and network streaming / transmission times. The aspects of this disclosure can be implemented in a variety of different ways for prediction. For example, the aspects presented herein can utilize dynamically calculated statistics from previous times (e.g., the nth percentile, 95th percentile, etc.). That is, the aspects presented herein can utilize 95% of the rendering / encoding / decoding / network streaming / transmission times calculated at a specific pace (e.g., FPS / 4) during runtime, and this value is used by a dynamic performance tuning module. The aspects presented herein can also utilize a machine learning (ML) regression model. This model can be constructed using variables to predict future values ​​using regression. ML regression can also utilize input features such as previous rendering time, previous encoding time, previous network streaming / transmission time, previous decoding time, and the amount of usage and statistical clocks (e.g., CPU, GPU, encoding, DDR usage statistics and clocks). ML regression can also utilize outputs such as rendering time, encoding time, network streaming / transmission time, and / or decoding time.

[0087] Figure 8 This is diagram 800, an example block diagram illustrating the split rendering process. More specifically, diagram 800 depicts a split rendering process that includes dynamic performance tuning. Figure 8 As shown in the figure, Figure 800 includes a statistical model 810, an algorithm model 830, and an adjustment component model 850. Figure 8 The input is fed to a statistical model 810, which includes CPU statistics 812, GPU statistics 814, encoder statistics 816, DDR statistics 818, and network statistics 820. The output of the statistical model 810 is fed to an algorithm model 830, which includes a pipeline threshold 832 and current pipeline information 834. Next, it is determined whether adjustments are needed. This determination is then fed to either a performance adjustment component 842 or a quality adjustment component 844. The output of the algorithm model 830 is fed to an adjustment component model 850, which includes a CPU adjuster 852 (for scaling CPU performance), a GPU adjuster 854 (for scaling GPU performance), an encoder adjuster 856 (for scaling encoder performance), a DDR adjuster 858 (for scaling DDR performance), a network adjuster 860 (for scaling network performance), and a content quality adjuster 862 (for adjusting content quality). The algorithm model 830 then outputs the result.

[0088] Figure 9 This is diagram 900, illustrating an example timeline of the split rendering process. More specifically, diagram 900 depicts the functional components and sequence diagrams of the split XR framework. Figure 9As shown, Figure 900 includes a Dynamic Performance Adjustment (DPA) component 902, an XR client 904, a Real-Time Transport Protocol (RTP) client 906, an RTP server 908, an XR server 910, a rendering component 912, an encoding component 914, a decoding component 916, an Asynchronous Time Warp (ATW) component, a display component 918, and a display 920. Figure 9 It also describes several frame processing times, such as UL time 930, rendering time 940, encoding time 950, network streaming / transmission time 960, decoding time 970, and display time 980. Figure 9 As depicted, the DPA component 902 interacts with the rendering component 912, the encoding component 914, the decoding component 916, and the networking module to calculate the predicted frame processing time and the current frame processing time. Based on these values, the DPA component 902 can adjust the performance of each of the other components / modules.

[0089] like Figure 9 As shown, at 922, DPA component 902 monitors frame information. At 932, XR client 904 transmits pose information to RTP client 906. At 934, RTP client 906 transmits pose packet information to RTP server 908. At 936, RTP server 908 transmits new pose information to XR server 910. At 938, RTP client 906 transmits statistics (e.g., frame processing time) to DPA component 902. At 942, rendering component 912 transmits head pose information to XR server 910. At 944, XR server 910 transmits pose information to rendering component 912. At 948, rendering component 912 transmits statistics (e.g., frame processing time) to DPA component 902. At 952, rendering component 912 transmits frame information to encoding component 914. At 958, encoding component 914 transmits statistics (e.g., frame processing time) to DPA component 902. At 962, encoding component 914 transmits frame information to RTP server 908. At 964, RTP server 908 transmits frame packet information to RTP client 906. At 968, RTP server 908 transmits statistics (e.g., frame processing time) to DPA component 902. At 972, RTP client 906 transmits frame information to decoding component 916. At 978, decoding component 916 transmits statistics (e.g., frame processing time) to DPA component 902. At 982, decoding component 916 transmits frame information to ATW and display component 918. At 988, ATW and display component 918 transmits statistics (e.g., frame processing time) to DPA component 902. At 990, DPA component 902 monitors frame information.

[0090] In addition, such as Figure 9As shown, the DPA component 902 can estimate the set of frame processing times for at least one first frame in the frame set. To estimate the set of frame processing times for at least one first frame, the DPA component 902 can estimate the set of frame processing times for at least one first frame based on a target vertical synchronization (Vsync) time. Furthermore, to estimate the set of frame processing times for at least one first frame, the DPA component 902 can estimate the set of frame processing times for at least one first frame based on at least one of the following: a machine learning (ML) model, an artificial intelligence (AI) model, or a statistical model. The ML or AI model can be trained offline on a computer or on edge computing, and the current frame processing time can be used to remotely improve the model via transfer learning. The DPA component 902 can also detect the actual set of frame processing times for at least one first frame in the frame set. To detect the actual set of frame processing times for at least one first frame in the frame set, the DPA component 902 can detect the actual set of frame processing times for at least one first frame based on a target Vsync time. Furthermore, to detect the actual set of frame processing times for at least one first frame in the frame set, the DPA component 902 can compute the actual set of frame processing times for at least one first frame. Furthermore, the DPA component 902 can output an indication of a second frame processing time set for at least one second frame in the adjusted frame set based on the estimated frame processing time set and the actual frame processing time set, wherein the at least one second frame follows at least one first frame in the frame set.

[0091] In some aspects, the DPA component 902 can also adjust the second frame processing time set of at least one second frame based on the estimated frame processing time set and the actual frame processing time set. To adjust the second frame processing time set of at least one second frame, the DPA component 902 can perform several different actions. For example, the DPA component 902 can increase the second frame processing time set of at least one second frame based on the fact that the actual frame processing time set of at least one first frame is less than the estimated frame processing time set of at least one first frame. The DPA component 902 can also decrease the second frame processing time set of at least one second frame based on the fact that the actual frame processing time set of at least one first frame is greater than the estimated frame processing time set of at least one first frame. Furthermore, the DPA component 902 can adjust the workload associated with the central processing unit (CPU) or graphics processing unit (GPU) for at least one second frame. The DPA component 902 can also adjust the clock or corner associated with at least one of the following for at least one second frame: CPU, GPU, encoder, decoder, or double data rate (DDR) memory. The DPA component 902 can also adjust the data rate or modulation and decoding scheme (MCS) rate of at least one second frame. In addition, the DPA component 902 can adjust at least one of the following for at least one second frame: encoding rate, compression rate, bit rate, or group of picture (GOP) rate. The DPA component 902 can also adjust the content resolution of at least one second frame.

[0092] The aspects of this disclosure may include multiple benefits or advantages. For example, the aspects presented herein can improve the overall user experience of split rendering. That is, the aspects presented herein can utilize dynamic performance tuning that helps reduce frame duplication / loss and improve overall latency. Furthermore, the aspects presented herein can utilize dynamic performance tuning that helps improve content quality and / or user experience. The aspects presented herein can also adjust or reduce the amount of power used for split rendering. For example, the aspects presented herein can utilize dynamic performance tuning that can help reduce the performance of multiple subsystems in any possible way. Furthermore, the aspects presented herein can utilize dynamic performance tuning that can help reduce power consumption and thermal power. The aspects of this disclosure can also improve or mitigate power and / or performance issues on split rendering pipelines (e.g., split XR / AR / MR rendering pipelines). For example, the aspects presented herein can tune performance and content quality to mitigate multiple display / frame issues (e.g., frame loss / duplication and / or M2R2P latency). Furthermore, the aspects of this disclosure can improve the overall user experience in split rendering pipelines (e.g., split XR / AR / MR rendering pipelines). Therefore, the aspects of this disclosure can dynamically adjust the performance and / or content quality in a split rendering pipeline. For example, the aspects presented herein can perform dynamic performance tuning and / or dynamic content quality tuning. That is, the aspects presented herein can improve the overall user experience in a split rendering pipeline (e.g., improve M2R2P latency and / or frame loss / duplication issues) and reduce power and / or thermal issues by utilizing dynamic performance tuning and / or dynamic content quality tuning.

[0093] Figure 10 This is a communication flowchart 1000 for frame processing according to one or more techniques of this disclosure. For example... Figure 10 As shown, Figure 1000 includes example communication between a client device 1002 (e.g., a client, headset, HMD, AR glasses, server, telephone, or smartphone), a server 1004 (e.g., a server, telephone, smartphone, client, headset, HMD, or AR glasses), and a memory 1006 (e.g., a memory or cache) according to one or more technologies of this disclosure.

[0094] At 1010, client device 1002 may obtain an indication of the frame set before estimating the frame processing time set (e.g., client device 1002 may receive indication 1012 from server 1004). In some aspects, estimating the frame processing time set of at least one first frame includes estimating the frame processing time set of at least one first frame based on the indication of the frame set.

[0095] At 1020, client device 1002 may train a machine learning (ML) model on the set of frames before estimating the set of frame processing times. In some aspects, estimating the set of frame processing times for at least one first frame includes: estimating the set of frame processing times for at least one first frame based on the ML model.

[0096] At 1030, client device 1002 can estimate the set of frame processing times for at least one first frame in the frame set. In some aspects, estimating the set of frame processing times for at least one first frame may include estimating the set of frame processing times for at least one first frame based on the target vertical synchronization (Vsync) time. Furthermore, estimating the set of frame processing times for at least one first frame may include estimating the set of frame processing times for at least one first frame based on at least one of the following: a machine learning (ML) model, an artificial intelligence (AI) model, or a statistical model. In some aspects, the estimated set of frame processing times for at least one first frame may include one or more of the following: the pose time of at least one first frame, the frame rendering time of at least one first frame, the frame encoding time of at least one first frame, the frame encryption time of at least one first frame, the frame transmission time of at least one first frame, the frame decryption time of at least one first frame, the frame decoding time of at least one first frame, or the frame reprojection time of at least one first frame. Additionally, pose time can be the time when the pose of at least one first frame is available at the server or the time when the server generates pose information for at least one first frame; frame rendering time is the time when at least one first frame is rendered at the server or client device; frame encoding time can be the time when at least one first frame is encoded or compressed at the server; frame encryption time can be the time when at least one first frame is encrypted at the server; frame transmission time can be the time when at least one first frame is sent from the server to the client device; frame decryption time can be the time when at least one first frame is decrypted at the client device; frame decoding time can be the time when at least one first frame is decoded at the client device; and frame reprojection time is the time when at least one first frame is reprojected at the client device.

[0097] At 1040, client device 1002 can detect the actual frame processing time set of at least one first frame in the frame set. In some aspects, detecting the actual frame processing time set of at least one first frame may include: detecting the actual frame processing time set of at least one first frame based on a target Vsync time. Furthermore, detecting the actual frame processing time set of at least one first frame may include: calculating the actual frame processing time set of at least one first frame. Additionally, detecting the actual frame processing time set of at least one first frame includes: obtaining an indication of the actual frame processing time set of at least one first frame. Furthermore, obtaining an indication of the actual frame processing time set of at least one first frame includes: receiving an indication of the actual frame processing time set of at least one first frame from a client device or a server. In some aspects, the actual frame processing time set of at least one first frame may include one or more of the following: the pose time of at least one first frame, the frame rendering time of at least one first frame, the frame encoding time of at least one first frame, the frame encryption time of at least one first frame, the frame transmission time of at least one first frame, the frame decryption time of at least one first frame, the frame decoding time of at least one first frame, or the frame reprojection time of at least one first frame. Additionally, pose time can be the time when the pose of at least one first frame is available at the server or the time when the server generates pose information for at least one first frame; frame rendering time is the time when at least one first frame is rendered at the server or client device; frame encoding time can be the time when at least one first frame is encoded or compressed at the server; frame encryption time can be the time when at least one first frame is encrypted at the server; frame transmission time can be the time when at least one first frame is sent from the server to the client device; frame decryption time can be the time when at least one first frame is decrypted at the client device; frame decoding time can be the time when at least one first frame is decoded at the client device; and frame reprojection time is the time when at least one first frame is reprojected at the client device.

[0098] At 1050, client device 1002 can output an indication of the second frame processing time set for adjusting at least one second frame in the frame set, based on the estimated frame processing time set and the actual frame processing time set, wherein the at least one second frame follows at least one first frame in the frame set. In some aspects, outputting the indication of adjusting the second frame processing time set of at least one second frame includes sending the indication to a client device or server (e.g., client device 1002 may send indication 1052 to server 1004). Furthermore, outputting the indication of adjusting the second frame processing time set of at least one second frame includes storing the indication of adjusting the second frame processing time set of at least one second frame in memory or a cache (e.g., client device 1002 may store indication 1054 in memory 1006).

[0099] At 1060, client device 1002 can adjust the second frame processing time set of at least one second frame based on the estimated frame processing time set and the actual frame processing time set. In some aspects, adjusting the second frame processing time set of at least one second frame may include increasing the second frame processing time set of at least one second frame based on the fact that the actual frame processing time set of at least one first frame is less than the estimated frame processing time set of at least one first frame. Furthermore, adjusting the second frame processing time set of at least one second frame may include decreasing the second frame processing time set of at least one second frame based on the fact that the actual frame processing time set of at least one first frame is greater than the estimated frame processing time set of at least one first frame. Furthermore, adjusting the second frame processing time set of at least one second frame may include adjusting the workload associated with a central processing unit (CPU) or a graphics processing unit (GPU) for at least one second frame. Furthermore, adjusting the second frame processing time set of at least one second frame may include adjusting the clock or corner associated with at least one of the following for at least one second frame: a central processing unit (CPU), a graphics processing unit (GPU), an encoder, a decoder, or double data rate (DDR) memory. Furthermore, adjusting the second-frame processing time set of at least one second frame may include: adjusting the data rate or modulation and decoding scheme (MCS) rate of at least one second frame. Additionally, adjusting the second-frame processing time set of at least one second frame may include: adjusting at least one of the following for at least one second frame: encoding rate, compression rate, bit rate, or group of picture blocks (GOP) rate. Furthermore, adjusting the second-frame processing time set of at least one second frame may include: adjusting the content resolution of at least one second frame.

[0100] Figure 11 This is a flowchart 1100 of an example method for image processing according to one or more techniques of this disclosure. The method may be performed by a client device, a headset, an HMD, AR glasses, a server, a telephone, a smartphone, a DPU (or other display processor), a CPU (or other central processing unit), a DPU driver, a DDIC, a GPU (or other graphics processing unit), a means for display processing, a wireless communication device, and / or an apparatus capable of performing such a combination. Figures 1 to 10 The example uses any device for frame processing to perform.

[0101] At 1106, the client device can estimate the frame processing time set of at least one first frame in the frame set, such as by combining Figures 1 to 10 As described in the examples. For example, such as Figure 10 As described in 1030, client device 1002 can estimate the frame processing time set of at least one first frame in the frame set. Furthermore, step 1106 can be performed by... Figure 1The display processor 127 in the display process performs the following: In some aspects, estimating the frame processing time set of at least one first frame includes: estimating the frame processing time set of at least one first frame based on an ML model. In some aspects, estimating the frame processing time set of at least one first frame may include: estimating the frame processing time set of at least one first frame based on a target vertical synchronization (Vsync) time. In some aspects, estimating the frame processing time set of at least one first frame includes: estimating the frame processing time set of at least one first frame based on an indication of a set of frames. Furthermore, estimating the frame processing time set of at least one first frame may include: estimating the frame processing time set of at least one first frame based on at least one of the following: a machine learning (ML) model, an artificial intelligence (AI) model, or a statistical model. In some aspects, the estimated frame processing time set of at least one first frame may include one or more of the following: the pose time of at least one first frame, the frame rendering time of at least one first frame, the frame encoding time of at least one first frame, the frame encryption time of at least one first frame, the frame transmission time of at least one first frame, the frame decryption time of at least one first frame, the frame decoding time of at least one first frame, or the frame reprojection time of at least one first frame. Additionally, pose time can be the time when the pose of at least one first frame is available at the server or the time when the server generates pose information for at least one first frame; frame rendering time is the time when at least one first frame is rendered at the server or client device; frame encoding time can be the time when at least one first frame is encoded or compressed at the server; frame encryption time can be the time when at least one first frame is encrypted at the server; frame transmission time can be the time when at least one first frame is sent from the server to the client device; frame decryption time can be the time when at least one first frame is decrypted at the client device; frame decoding time can be the time when at least one first frame is decoded at the client device; and frame reprojection time is the time when at least one first frame is reprojected at the client device.

[0102] At 1108, the client device can detect at least one set of actual frame processing times for the first frame in the frame set, such as in combination with Figures 1 to 10 As described in the examples. For example, such as Figure 10 As described in 1040, client device 1002 can detect the actual frame processing time set of at least one first frame in the frame set. Furthermore, step 1108 can be performed by... Figure 1The display processor 127 in the display process executes the following. In some aspects, detecting the actual frame processing time set of at least one first frame may include: detecting the actual frame processing time set of at least one first frame based on a target Vsync time. Furthermore, detecting the actual frame processing time set of at least one first frame may include: calculating the actual frame processing time set of at least one first frame. Additionally, detecting the actual frame processing time set of at least one first frame includes: obtaining an indication of the actual frame processing time set of at least one first frame. Furthermore, obtaining an indication of the actual frame processing time set of at least one first frame includes: receiving an indication of the actual frame processing time set of at least one first frame from a client device or a server. In some aspects, the actual frame processing time set of at least one first frame may include one or more of the following: the pose time of at least one first frame, the frame rendering time of at least one first frame, the frame encoding time of at least one first frame, the frame encryption time of at least one first frame, the frame transmission time of at least one first frame, the frame decryption time of at least one first frame, the frame decoding time of at least one first frame, or the frame reprojection time of at least one first frame. Additionally, pose time can be the time when the pose of at least one first frame is available at the server or the time when the server generates pose information for at least one first frame; frame rendering time is the time when at least one first frame is rendered at the server or client device; frame encoding time can be the time when at least one first frame is encoded or compressed at the server; frame encryption time can be the time when at least one first frame is encrypted at the server; frame transmission time can be the time when at least one first frame is sent from the server to the client device; frame decryption time can be the time when at least one first frame is decrypted at the client device; frame decoding time can be the time when at least one first frame is decoded at the client device; and frame reprojection time is the time when at least one first frame is reprojected at the client device.

[0103] At 1110, the client device can output an indication of the second frame processing time set for at least one second frame in the adjusted frame set, based on the estimated frame processing time set and the actual frame processing time set, wherein the at least one second frame follows at least one first frame in the frame set, as in combination with... Figures 1 to 10 As described in the examples. For example, such as Figure 10 As described in 1050, client device 1002 can output an indication of a second frame processing time set for at least one second frame in the adjusted frame set based on an estimated frame processing time set and an actual frame processing time set, wherein the at least one second frame follows at least one first frame in the frame set. Furthermore, step 1110 can be performed by… Figure 1The display processor 127 in the display is executed. In some aspects, outputting an indication for adjusting the second frame processing time set of at least one second frame includes: sending the indication for adjusting the second frame processing time set of at least one second frame to a client device or a server (e.g., client device 1002 may send indication 1052 to server 1004). Furthermore, outputting an indication for adjusting the second frame processing time set of at least one second frame includes: storing the indication for adjusting the second frame processing time set of at least one second frame in memory or a cache (e.g., client device 1002 may store indication 1054 in memory 1006).

[0104] Figure 12 This is a flowchart 1200 of an example method for image processing according to one or more techniques disclosed herein. The method may be performed by a client device, a headset, an HMD, AR glasses, a server, a telephone, a smartphone, a DPU (or other display processor), a CPU (or other central processing unit), a DPU driver, a DDIC, a GPU (or other graphics processing unit), a means for display processing, a wireless communication device, and / or an apparatus capable of performing such a combination. Figures 1 to 10 The example uses any device for frame processing to perform.

[0105] At 1202, the client device can obtain an indication of the frame set before estimating the frame processing time set, such as in conjunction with Figures 1 to 10 As described in the examples. For example, such as Figure 10 As described in 1010, client device 1002 can obtain an indication of the frame set before estimating the frame processing time set. Furthermore, step 1202 can be performed by... Figure 1 The display processor 127 in the middle is executed.

[0106] At 1204, the client device can train a machine learning (ML) model on the frame set before estimating the frame processing time set, such as by combining Figures 1 to 10 As described in the examples. For example, such as Figure 10 As described in 1020, the client device 1002 can train a machine learning (ML) model on the frame set before estimating the frame processing time set. Furthermore, step 1204 can be performed by... Figure 1 The display processor 127 in the middle is executed.

[0107] At 1206, the client device can estimate the frame processing time set of at least one first frame in the frame set, such as by combining Figures 1 to 10 As described in the examples. For example, such as Figure 10 As described in 1030, client device 1002 can estimate the frame processing time set of at least one first frame in the frame set. Furthermore, step 1206 can be performed by... Figure 1The display processor 127 in the display process performs the following: In some aspects, estimating the frame processing time set of at least one first frame includes: estimating the frame processing time set of at least one first frame based on an ML model. In some aspects, estimating the frame processing time set of at least one first frame may include: estimating the frame processing time set of at least one first frame based on a target vertical synchronization (Vsync) time. In some aspects, estimating the frame processing time set of at least one first frame includes: estimating the frame processing time set of at least one first frame based on an indication of a set of frames. Furthermore, estimating the frame processing time set of at least one first frame may include: estimating the frame processing time set of at least one first frame based on at least one of the following: a machine learning (ML) model, an artificial intelligence (AI) model, or a statistical model. In some aspects, the estimated frame processing time set of at least one first frame may include one or more of the following: the pose time of at least one first frame, the frame rendering time of at least one first frame, the frame encoding time of at least one first frame, the frame encryption time of at least one first frame, the frame transmission time of at least one first frame, the frame decryption time of at least one first frame, the frame decoding time of at least one first frame, or the frame reprojection time of at least one first frame. Additionally, pose time can be the time when the pose of at least one first frame is available at the server or the time when the server generates pose information for at least one first frame; frame rendering time is the time when at least one first frame is rendered at the server or client device; frame encoding time can be the time when at least one first frame is encoded or compressed at the server; frame encryption time can be the time when at least one first frame is encrypted at the server; frame transmission time can be the time when at least one first frame is sent from the server to the client device; frame decryption time can be the time when at least one first frame is decrypted at the client device; frame decoding time can be the time when at least one first frame is decoded at the client device; and frame reprojection time is the time when at least one first frame is reprojected at the client device.

[0108] At 1208, the client device can detect the actual frame processing time set of at least one first frame in the frame set, such as in combination with Figures 1 to 10 As described in the examples. For example, such as Figure 10 As described in 1040, client device 1002 can detect the actual frame processing time set of at least one first frame in the frame set. Furthermore, step 1208 can be performed by... Figure 1The display processor 127 in the display process executes the following. In some aspects, detecting the actual frame processing time set of at least one first frame may include: detecting the actual frame processing time set of at least one first frame based on a target Vsync time. Furthermore, detecting the actual frame processing time set of at least one first frame may include: calculating the actual frame processing time set of at least one first frame. Additionally, detecting the actual frame processing time set of at least one first frame includes: obtaining an indication of the actual frame processing time set of at least one first frame. Furthermore, obtaining an indication of the actual frame processing time set of at least one first frame includes: receiving an indication of the actual frame processing time set of at least one first frame from a client device or a server. In some aspects, the actual frame processing time set of at least one first frame may include one or more of the following: the pose time of at least one first frame, the frame rendering time of at least one first frame, the frame encoding time of at least one first frame, the frame encryption time of at least one first frame, the frame transmission time of at least one first frame, the frame decryption time of at least one first frame, the frame decoding time of at least one first frame, or the frame reprojection time of at least one first frame. Additionally, pose time can be the time when the pose of at least one first frame is available at the server or the time when the server generates pose information for at least one first frame; frame rendering time is the time when at least one first frame is rendered at the server or client device; frame encoding time can be the time when at least one first frame is encoded or compressed at the server; frame encryption time can be the time when at least one first frame is encrypted at the server; frame transmission time can be the time when at least one first frame is sent from the server to the client device; frame decryption time can be the time when at least one first frame is decrypted at the client device; frame decoding time can be the time when at least one first frame is decoded at the client device; and frame reprojection time is the time when at least one first frame is reprojected at the client device.

[0109] At 1210, the client device can output an indication of the second frame processing time set for at least one second frame in the adjusted frame set, based on the estimated frame processing time set and the actual frame processing time set, wherein the at least one second frame follows at least one first frame in the frame set, as in combination with... Figures 1 to 10 As described in the examples. For example, such as Figure 10 As described in 1050, client device 1002 can output an indication of a second frame processing time set for at least one second frame in the adjusted frame set based on an estimated frame processing time set and an actual frame processing time set, wherein the at least one second frame follows at least one first frame in the frame set. Furthermore, step 1210 can be performed by… Figure 1The display processor 127 in the display is executed. In some aspects, outputting an indication for adjusting the second frame processing time set of at least one second frame includes: sending the indication for adjusting the second frame processing time set of at least one second frame to a client device or a server (e.g., client device 1002 may send indication 1052 to server 1004). Furthermore, outputting an indication for adjusting the second frame processing time set of at least one second frame includes: storing the indication for adjusting the second frame processing time set of at least one second frame in memory or a cache (e.g., client device 1002 may store indication 1054 in memory 1006).

[0110] At 1212, the client device can adjust the second frame processing time set of at least one second frame based on the estimated frame processing time set and the actual frame processing time set, such as by combining... Figures 1 to 10 As described in the examples. For example, such as Figure 10 As described in 1060, client device 1002 can adjust the second frame processing time set of at least one second frame based on the estimated frame processing time set and the actual frame processing time set. Furthermore, step 1212 can be performed by... Figure 1 The display processor 127 in the display processor 127 performs the operation. In some aspects, adjusting the second frame processing time set of at least one second frame may include: increasing the second frame processing time set of at least one second frame based on the fact that the actual frame processing time set of at least one first frame is less than the estimated frame processing time set of at least one first frame. Furthermore, adjusting the second frame processing time set of at least one second frame may include: decreasing the second frame processing time set of at least one second frame based on the fact that the actual frame processing time set of at least one first frame is greater than the estimated frame processing time set of at least one first frame. Furthermore, adjusting the second frame processing time set of at least one second frame may include: adjusting the workload associated with the central processing unit (CPU) or graphics processing unit (GPU) for at least one second frame. Furthermore, adjusting the second frame processing time set of at least one second frame may include: adjusting the clock or corner associated with at least one of the following for at least one second frame: the central processing unit (CPU), the graphics processing unit (GPU), the encoder, the decoder, or double data rate (DDR) memory. Furthermore, adjusting the second frame processing time set of at least one second frame may include: adjusting the data rate or modulation and decoding scheme (MCS) rate of at least one second frame. Furthermore, adjusting the second-frame processing time set for at least one second frame may include adjusting at least one of the following for at least one second frame: encoding rate, compression rate, bit rate, or group of picture blocks (GOP) rate. Additionally, adjusting the second-frame processing time set for at least one second frame may include adjusting the content resolution of at least one second frame.

[0111] In various configurations, a method or apparatus for display processing is provided. This apparatus may be a client device, a headset, an HMD, AR glasses, a server, a telephone, a smartphone, a DPU (or other display processor), a CPU (or other central processing unit), a DPU driver, a DDIC, a GPU (or other graphics processing unit), a device for display processing, a wireless communication device, and / or some other processor capable of performing display processing. In various aspects, the apparatus may be a display processor 127 within device 104, or may be some other hardware within device 104 or another device. The apparatus (e.g., display processor 127) may include components for estimating a set of frame processing times for at least one first frame in a set of frames. The apparatus (e.g., display processor 127) may also include components for detecting the actual set of frame processing times for at least one first frame in the set of frames. The apparatus (e.g., display processor 127) may also include components for outputting an indication for adjusting the set of second frame processing times for at least one second frame in the set of frames based on the estimated set of frame processing times and the actual set of frame processing times, wherein the at least one second frame follows the at least one first frame in the set of frames. The apparatus (e.g., display processor 127) may further include components for adjusting a second frame processing time set of at least one second frame based on an estimated frame processing time set and an actual frame processing time set. The apparatus (e.g., display processor 127) may also include components for obtaining an indication of the frame set before estimating the frame processing time set. The apparatus (e.g., display processor 127) may also include components for training a machine learning (ML) model on the frame set before estimating the frame processing time set.

[0112] The subjects described herein can be implemented to achieve one or more benefits or advantages. For example, the described display processing techniques can be implemented by client devices, headsets, HMDs, AR glasses, servers, telephones, smartphones, DPUs (or other display processors), CPUs (or other central processing units), DPU drivers, DDICs, GPUs (or other graphics processing units), devices for display processing, wireless communication devices, or some other processors capable of performing display processing to implement the dynamic performance and power tuning techniques described herein. This can also be achieved at a lower cost compared to other display processing techniques. Furthermore, the display processing techniques of this invention can improve or accelerate data processing or execution. In addition, the display processing techniques of this invention can improve resource or data utilization and / or resource efficiency. Furthermore, aspects of this disclosure can utilize dynamic performance and power tuning techniques to improve memory bandwidth efficiency and / or increase processing speed at client devices, servers, GPUs, DPUs, and / or CPUs.

[0113] It should be understood that the specific order or hierarchy of the boxes in the disclosed process / flowcharts is merely an example of the exemplary method. It should be understood that the specific order or hierarchy of the boxes in the process / flowcharts may be rearranged based on design preferences. Furthermore, some boxes may be combined or omitted. The appended method claims present elements of various boxes in a sample order, but this does not imply limitation to the specific order or hierarchy presented.

[0114] The foregoing description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but should be given the full scope consistent with the language of the claims, wherein references to elements in the singular form, unless specifically stated otherwise, are not intended to mean “one and only one,” but rather “one or more.” The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

[0115] Unless otherwise specified, the term "some" refers to one or more, and unless otherwise specified in the context, the term "or" may be interpreted as "and / or". Combinations such as "at least one of A, B, or C", "one or more of A, B, or C", "at least one of A, B, and C", "one or more of A, B, and C", and "A, B, C, or any combination thereof" include any combination of A, B, and / or C, and may include multiple A, multiple B, or multiple C. Specifically, combinations such as "at least one of A, B, or C", "one or more of A, B, or C", "at least one of A, B, and C", "one or more of A, B, and C", and "A, B, C, or any combination thereof" may be only A, only B, only C, A and B, A and C, B and C, or A and B and C, wherein any such combination may include one or more members of A, B, or C. The various aspects described throughout this disclosure are all structural and functional equivalents known now or hereafter to those skilled in the art, and are expressly incorporated herein by reference and intended to be covered by the claims. Furthermore, nothing disclosed herein is intended to be offered to the public, whether or not such disclosure is expressly stated in the claims. Terms such as “module,” “mechanism,” “element,” and “device” cannot replace the word “component.” Therefore, no claim element will be construed as a functional component unless the element is expressly described using the phrase “component for…”.

[0116] In one or more examples, the functionality described herein may be implemented in hardware, software, firmware, or any combination thereof. For example, although the term "processing unit" is used throughout this disclosure, such a processing unit may be implemented in hardware, software, firmware, or any combination thereof. If any functionality, processing unit, technique, or other module described herein is implemented in software, then such functionality, processing unit, technique, or other module may be stored on or transmitted on a computer-readable medium as one or more instructions or code.

[0117] According to this disclosure, unless otherwise specified in the context, the term "or" may be understood as "and / or". Additionally, while phrases such as "one or more" or "at least one" may be used for some features disclosed herein but not others, features not using such language may be understood to have such implied meaning unless otherwise specified in the context.

[0118] In one or more examples, the functionality described herein may be implemented in hardware, software, firmware, or any combination thereof. For example, although the term “processing unit” is used throughout this disclosure, such a processing unit may be implemented in hardware, software, firmware, or any combination thereof. If any functionality, processing unit, technique, or other module described herein is implemented in software, then the functionality, processing unit, technique, or other module described herein may be stored on or transmitted on a computer-readable medium as one or more instructions or code. A computer-readable medium may include computer data storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described herein. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage devices, magnetic disk storage devices, or other magnetic storage devices. As used herein, disks and optical discs include: compact optical discs (CDs), laser optical discs, optical discs, digital multifunction optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media. Computer program products may include computer-readable media.

[0119] The code can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), arithmetic logic units (ALUs), field-programmable arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the above-described structures or any other structure suitable for implementing the techniques described herein. Furthermore, these techniques can be fully implemented in one or more circuit or logic elements.

[0120] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless mobile phones, integrated circuits (ICs), or IC sets (e.g., chipsets). Various components, modules, or units are described in this disclosure to emphasize functional aspects of a device configured to perform the disclosed techniques, but they do not necessarily need to be implemented by different hardware units. Rather, as described above, various units can be combined in any hardware unit or provided by a collection of interoperable hardware units (including one or more processors as described above) combined with suitable software and / or firmware. Therefore, the term "processor" as used herein can refer to any of the above-described structures or any other structure suitable for implementing the techniques described herein. Furthermore, these techniques can be fully implemented in one or more circuit or logic elements.

[0121] The following aspects are merely illustrative and may be combined with other aspects or teachings described herein without limitation.

[0122] Aspect 1 is an apparatus for display processing, the apparatus comprising at least one memory; and at least one processor coupled to the at least one memory, and configured, at least in part, based on information stored in the at least one memory, to: estimate a set of frame processing times for at least one first frame in a set of frames; detect an actual set of frame processing times for the at least one first frame in the set of frames; and output an indication, based on the estimated set of frame processing times and the actual set of frame processing times, to adjust a second set of frame processing times for at least one second frame in the set of frames, wherein the at least one second frame follows the at least one first frame in the set of frames.

[0123] Aspect 2 is the apparatus according to aspect 1, wherein the at least one processor is configured, individually or in any combination, to adjust the second frame processing time set of the at least one second frame based on the estimated frame processing time set and the actual frame processing time set.

[0124] Aspect 3 is the apparatus according to aspect 2, wherein, in order to adjust the second frame processing time set of the at least one second frame, the at least one processor is configured individually or in any combination to increase the second frame processing time set of the at least one second frame based on the fact that the actual frame processing time set of the at least one first frame is less than the estimated frame processing time set of the at least one first frame.

[0125] Aspect 4 is the apparatus according to aspect 2, wherein, in order to adjust the second frame processing time set of the at least one second frame, the at least one processor is configured individually or in any combination to reduce the second frame processing time set of the at least one second frame based on the fact that the actual frame processing time set of the at least one first frame is greater than the estimated frame processing time set of the at least one first frame.

[0126] Aspect 5 is an apparatus according to any one of Aspects 2 to 4, wherein, in order to adjust the set of second frame processing times for the at least one second frame, the at least one processor is configured individually or in any combination to adjust the workload associated with the central processing unit (CPU) or graphics processing unit (GPU) for the at least one second frame.

[0127] Aspect 6 is an apparatus according to any one of aspects 2 to 5, wherein, in order to adjust the set of second frame processing times for the at least one second frame, the at least one processor is configured individually or in any combination to adjust a clock or corner associated with at least one of the following for the at least one second frame: a central processing unit (CPU), a graphics processing unit (GPU), an encoder, a decoder, or a double data rate (DDR) memory.

[0128] Aspect 7 is an apparatus according to any one of aspects 2 to 6, wherein, in order to adjust the second frame processing time set of the at least one second frame, the at least one processor is configured individually or in any combination to adjust the data rate or modulation and decoding scheme (MCS) rate of the at least one second frame.

[0129] Aspect 8 is an apparatus according to any one of aspects 2 to 7, wherein, in order to adjust the second frame processing time set of the at least one second frame, the at least one processor is configured individually or in any combination to adjust at least one of the following for the at least one second frame: encoding rate, compression rate, bit rate, or group of picture (GOP) rate.

[0130] Aspect 9 is an apparatus according to any one of aspects 2 to 8, wherein, in order to adjust the second frame processing time set of the at least one second frame, the at least one processor is configured individually or in any combination to: adjust the content resolution of the at least one second frame.

[0131] Aspect 10 is an apparatus according to any one of aspects 1 to 9, wherein at least one of the estimated frame processing time set or the actual frame processing time set of the at least one first frame includes one or more of the following: the pose time of the at least one first frame, the frame rendering time of the at least one first frame, the frame encoding time of the at least one first frame, the frame encryption time of the at least one first frame, the frame transmission time of the at least one first frame, the frame decryption time of the at least one first frame, the frame decoding time of the at least one first frame, or the frame reprojection time of the at least one first frame.

[0132] Aspect 11 is the apparatus according to aspect 10, wherein the pose time is the time when the pose of the at least one first frame is available at the server or the time when the server generates the pose information of the at least one first frame, wherein the frame rendering time is the time when the at least one first frame is rendered at the server or client device, wherein the frame encoding time is the time when the at least one first frame is encoded or compressed at the server, wherein the frame encryption time is the time when the at least one first frame is encrypted at the server, wherein the frame transmission time is the time when the at least one first frame is transmitted from the server to the client device, wherein the frame decryption time is the time when the at least one first frame is decrypted at the client device, wherein the frame decoding time is the time when the at least one first frame is decoded at the client device, and wherein the frame reprojection time is the time when the at least one first frame is reprojected at the client device.

[0133] Aspect 12 is an apparatus according to any one of aspects 1 to 11, wherein, in order to estimate the set of frame processing times for the at least one first frame, the at least one processor is configured individually or in any combination to estimate the set of frame processing times for the at least one first frame based on a target vertical synchronization (Vsync) time; and wherein, in order to detect the actual set of frame processing times for the at least one first frame, the at least one processor is configured individually or in any combination to detect the actual set of frame processing times for the at least one first frame based on the target Vsync time.

[0134] Aspect 13 is an apparatus according to any one of aspects 1 to 12, wherein, in order to estimate the set of frame processing times for the at least one first frame, the at least one processor is configured individually or in any combination to estimate the set of frame processing times for the at least one first frame based on at least one of: a machine learning (ML) model, an artificial intelligence (AI) model, or a statistical model.

[0135] Aspect 14 is an apparatus according to any one of aspects 1 to 13, wherein, in order to detect the actual frame processing time set of the at least one first frame, the at least one processor is configured individually or in any combination to: calculate the actual frame processing time set of the at least one first frame.

[0136] Aspect 15 is an apparatus according to any one of aspects 1 to 14, wherein, in order to detect the actual frame processing time set of the at least one first frame, the at least one processor is configured individually or in any combination to: obtain an indication of the actual frame processing time set of the at least one first frame.

[0137] Aspect 16 is an apparatus according to aspect 15, the apparatus further comprising at least one of an antenna or a transceiver coupled to the at least one processor, wherein, in order to obtain the indication of the actual frame processing time set for the at least one first frame, the at least one processor is configured individually or in any combination to receive, via the antenna or at least one of the transceivers, the indication of the actual frame processing time set for the at least one first frame from a client device or a server.

[0138] Aspect 17 is an apparatus according to any one of aspects 1 to 16, wherein the at least one processor is further configured, individually or in any combination, to obtain an indication of the frame set before estimating the frame processing time set, wherein, in order to estimate the frame processing time set of the at least one first frame, the at least one processor is configured, individually or in any combination, to estimate the frame processing time set of the at least one first frame based on the indication of the frame set.

[0139] Aspect 18 is an apparatus according to any one of aspects 1 to 17, wherein the at least one processor is further configured, individually or in any combination, to train a machine learning (ML) model for the set of frames prior to estimating the set of frame processing times, wherein, in order to estimate the set of frame processing times for the at least one first frame, the at least one processor is configured, individually or in any combination, to estimate the set of frame processing times for the at least one first frame based on the ML model.

[0140] Aspect 19 is an apparatus according to any one of aspects 1 to 18, wherein, in order to output the indication for adjusting the set of second frame processing times of the at least one second frame, the at least one processor is configured individually or in any combination to send the indication for adjusting the set of second frame processing times of the at least one second frame to a client device or a server.

[0141] Aspect 20 is an apparatus according to any one of aspects 1 to 19, wherein, in order to output the indication for adjusting the set of second frame processing times for the at least one second frame, the at least one processor is configured individually or in any combination to store the indication for adjusting the set of second frame processing times for the at least one second frame in a memory or cache.

[0142] Aspect 21 is a method for implementing the display processing of any one of aspects 1 to 20.

[0143] Aspect 22 is an apparatus for display processing, the apparatus including components for implementing any one of aspects 1 to 20.

[0144] Aspect 23 is a computer-readable medium (e.g., a non-transitory computer-readable medium) storing computer-executable code that, when executed by at least one processor, causes the at least one processor to implement any one of aspects 1 to 20.

Claims

1. An apparatus for frame processing, the apparatus comprising: At least one memory; and At least one processor, coupled to the at least one memory, and configured individually or in any combination, based at least in part on information stored in the at least one memory, to: Estimate the set of frame processing times for at least one first frame in the frame set; Detect the actual frame processing time set of at least one first frame in the frame set; as well as An indication is output based on the estimated set of frame processing times and the actual set of frame processing times to adjust the set of second frame processing times for at least one second frame in the set of frames, wherein the at least one second frame follows the at least one first frame in the set of frames.

2. The apparatus of claim 1, wherein the at least one processor is further configured, alone or in any combination, to: The second frame processing time set of the at least one second frame is adjusted based on the estimated frame processing time set and the actual frame processing time set.

3. The apparatus of claim 2, wherein, in order to adjust the second frame processing time set of the at least one second frame, the at least one processor is configured individually or in any combination to: The second frame processing time set of the at least one second frame is increased based on the fact that the actual frame processing time set of the at least one first frame is less than the estimated frame processing time set of the at least one first frame.

4. The apparatus of claim 2, wherein, in order to adjust the second frame processing time set of the at least one second frame, the at least one processor is configured individually or in any combination to: The second frame processing time set of the at least one second frame is reduced based on the fact that the actual frame processing time set of the at least one first frame is greater than the estimated frame processing time set of the at least one first frame.

5. The apparatus of claim 2, wherein, in order to adjust the second frame processing time set of the at least one second frame, the at least one processor is configured individually or in any combination to: Adjust the workload associated with the central processing unit (CPU) or graphics processing unit (GPU) for the at least one second frame.

6. The apparatus of claim 2, wherein, in order to adjust the second frame processing time set of the at least one second frame, the at least one processor is configured individually or in any combination to: For the at least one second frame, adjust the clock or corner associated with at least one of the following: central processing unit (CPU), graphics processing unit (GPU), encoder, decoder, or double data rate (DDR) memory.

7. The apparatus of claim 2, wherein, in order to adjust the second frame processing time set of the at least one second frame, the at least one processor is configured individually or in any combination to: Adjust the data rate or modulation and decoding scheme (MCS) rate of the at least one second frame.

8. The apparatus of claim 2, wherein, in order to adjust the second frame processing time set of the at least one second frame, the at least one processor is configured individually or in any combination to: For the at least one second frame, adjust at least one of the following: encoding rate, compression rate, bit rate, or group of pictures (GOP) rate.

9. The apparatus of claim 2, wherein, in order to adjust the second frame processing time set of the at least one second frame, the at least one processor is configured individually or in any combination to: Adjust the content resolution of at least one second frame.

10. The apparatus of claim 1, wherein at least one of the estimated frame processing time set or the actual frame processing time set of the at least one first frame comprises one or more of the following: The pose time of at least one first frame. The frame rendering time of at least one first frame The frame encoding time of at least one first frame The frame encryption time of at least one first frame The frame transmission time of at least one first frame The frame decryption time of at least one first frame The frame decoding time of at least one first frame, or The frame reprojection time of at least one first frame.

11. The apparatus of claim 10, wherein the pose time is the time when the pose of the at least one first frame is available at the server or the time when the server generates the pose information of the at least one first frame. The frame rendering time is the time it takes for the at least one first frame to be rendered on the server or client device. The frame encoding time is the time when the at least one first frame is encoded or compressed at the server. The frame encryption time is the time when the at least one first frame is encrypted at the server. The frame transmission time is the time when the at least one first frame is sent from the server to the client device. The frame decryption time is the time when the at least one first frame is decrypted at the client device. The frame decoding time is the time it takes for the at least one first frame to be decoded at the client device, and The frame reprojection time is the time during which the at least one first frame is reprojected at the client device.

12. The apparatus of claim 1, wherein, in order to estimate the set of frame processing times for the at least one first frame, the at least one processor is configured individually or in any combination to: estimate the set of frame processing times for the at least one first frame based on a target vertical synchronization (Vsync) time; and In order to detect the actual frame processing time set of the at least one first frame, the at least one processor is configured individually or in any combination to detect the actual frame processing time set of the at least one first frame based on the target Vsync time.

13. The apparatus of claim 1, wherein, in order to estimate the set of frame processing times for the at least one first frame, the at least one processor is configured individually or in any combination to: The frame processing time set of the at least one first frame is estimated based on at least one of the following: a machine learning (ML) model, an artificial intelligence (AI) model, or a statistical model.

14. The apparatus of claim 1, wherein, in order to detect the actual set of frame processing times for the at least one first frame, the at least one processor is configured individually or in any combination to: Calculate the actual frame processing time set for the at least one first frame.

15. The apparatus of claim 1, wherein, in order to detect the actual set of frame processing times for the at least one first frame, the at least one processor is configured individually or in any combination to: Obtain an indication of the actual frame processing time set for the at least one first frame.

16. The apparatus of claim 15, further comprising at least one of an antenna or transceiver coupled to the at least one processor, wherein, in order to obtain the indication of the actual frame processing time set of the at least one first frame, the at least one processor is configured individually or in any combination to: The indication of the actual frame processing time set for the at least one first frame is received from the client device or server via at least one of the antenna or the transceiver.

17. The apparatus of claim 1, wherein the at least one processor, alone or in any combination, is further configured to: Before estimating the set of frame processing times, an indication of the set of frames is obtained, wherein, in order to estimate the set of frame processing times for the at least one first frame, the at least one processor is configured individually or in any combination to estimate the set of frame processing times for the at least one first frame based on the indication of the set of frames.

18. The apparatus of claim 1, wherein the at least one processor, alone or in any combination, is further configured to: A machine learning (ML) model is trained on the set of frames prior to estimating the set of frame processing times, wherein, in order to estimate the set of frame processing times for the at least one first frame, the at least one processor is configured individually or in any combination to estimate the set of frame processing times for the at least one first frame based on the ML model.

19. The apparatus of claim 1, wherein, in order to output the indication for adjusting the set of second frame processing times for the at least one second frame, the at least one processor is configured individually or in any combination to: Send the instruction to the client device or server to adjust the set of processing times for the at least one second frame.

20. The apparatus of claim 1, wherein, in order to output the indication for adjusting the set of processing times for the at least one second frame, the at least one processor is configured individually or in any combination to: The indication for adjusting the set of processing times for the at least one second frame is stored in memory or cache.

21. A method for frame processing, the method comprising: Estimate the set of frame processing times for at least one first frame in the frame set; Detect the actual frame processing time set of at least one first frame in the frame set; as well as An indication is output based on the estimated set of frame processing times and the actual set of frame processing times to adjust the set of second frame processing times for at least one second frame in the set of frames, wherein the at least one second frame follows the at least one first frame in the set of frames.

22. The method according to claim 21, further comprising: The second frame processing time set of the at least one second frame is adjusted based on the estimated frame processing time set and the actual frame processing time set.

23. The method of claim 22, wherein adjusting the second frame processing time set of the at least one second frame comprises at least one of the following: The second frame processing time set of the at least one second frame is increased based on the fact that the actual frame processing time set of the at least one first frame is less than the estimated frame processing time set of the at least one first frame. The second frame processing time set of the at least one second frame is reduced based on the fact that the actual frame processing time set of the at least one first frame is greater than the estimated frame processing time set of the at least one first frame. Adjust the workload associated with the central processing unit (CPU) or graphics processing unit (GPU) for the at least one second frame; For the at least one second frame, adjust the clock or corner associated with at least one of the following: the CPU, the GPU, the encoder, the decoder, or the double data rate (DDR) memory; Adjust the data rate or modulation and decoding scheme (MCS) rate of the at least one second frame; For the at least one second frame, adjust at least one of the following: encoding rate, compression rate, bit rate, or group of pictures (GOP) rate; or Adjust the content resolution of at least one second frame.

24. The method of claim 21, wherein at least one of the estimated frame processing time set or the actual frame processing time set of the at least one first frame includes one or more of the following: The pose time of at least one first frame. The frame rendering time of at least one first frame The frame encoding time of at least one first frame The frame encryption time of at least one first frame The frame transmission time of at least one first frame The frame decryption time of at least one first frame The frame decoding time of at least one first frame, or The frame reprojection time of at least one first frame The pose time refers to the time when the pose of the at least one first frame is available at the server or the time when the server generates the pose information of the at least one first frame. The frame rendering time is the time it takes for the at least one first frame to be rendered on the server or client device. The frame encoding time is the time when the at least one first frame is encoded or compressed at the server. The frame encryption time is the time when the at least one first frame is encrypted at the server. The frame transmission time is the time when the at least one first frame is sent from the server to the client device. The frame decryption time is the time when the at least one first frame is decrypted at the client device. The frame decoding time is the time it takes for the at least one first frame to be decoded at the client device, and The frame reprojection time is the time during which the at least one first frame is reprojected at the client device.

25. The method of claim 21, wherein estimating the set of frame processing times for the at least one first frame comprises: The set of frame processing times for the at least one first frame is estimated based on the target vertical synchronization (Vsync) time; and The detection of the actual frame processing time set of the at least one first frame includes: detecting the actual frame processing time set of the at least one first frame based on the target Vsync time.

26. The method of claim 21, wherein estimating the set of frame processing times for the at least one first frame comprises: The set of frame processing times for the at least one first frame is estimated based on at least one of the following: a machine learning (ML) model, an artificial intelligence (AI) model, or a statistical model; and Detecting the actual frame processing time set of the at least one first frame includes: calculating the actual frame processing time set of the at least one first frame, or obtaining an indication of the actual frame processing time set of the at least one first frame.

27. The method according to claim 21, further comprising: Obtaining an indication of the frame set before estimating the frame processing time set, wherein estimating the frame processing time set of the at least one first frame includes: estimating the frame processing time set of the at least one first frame based on the indication of the frame set.

28. The method according to claim 21, further comprising: Training a machine learning (ML) model for the set of frames prior to estimating the set of frame processing times, wherein estimating the set of frame processing times for the at least one first frame includes: estimating the set of frame processing times for the at least one first frame based on the ML model.

29. An apparatus for frame processing, the apparatus comprising: A component for estimating the set of frame processing times for at least one first frame in a set of frames; A component for detecting the actual frame processing time set of at least one first frame in the frame set; and A component for outputting an indication of adjusting a second frame processing time set for at least one second frame in the frame set based on an estimated set of frame processing times and the actual set of frame processing times, wherein the at least one second frame follows the at least one first frame in the frame set.

30. A computer-readable medium storing computer-executable code for display processing, said code, when executed by at least one processor, causing said at least one processor to: Estimate the set of frame processing times for at least one first frame in the frame set; Detect the actual frame processing time set of at least one first frame in the frame set; as well as An indication is output based on the estimated set of frame processing times and the actual set of frame processing times to adjust the set of second frame processing times for at least one second frame in the set of frames, wherein the at least one second frame follows the at least one first frame in the set of frames.

Images