A method for reduced display processing unit transfer time to compensate for delayed graphics processing unit rendering time

Abstract

The present disclosure provides systems, devices, apparatuses and methods for reducing DPU transfer time to compensate for delayed GPU rendering time, including computer programs encoded on storage media. After completing the rendering of a second frame following a first frame, a frame processor determines whether the first frame is currently being transferred to a display panel or has been transferred to the display panel. When it is determined that the first frame is currently being transferred, a first set of clock speeds is used with at least one clock, and when it is determined that the first frame has been transferred, a second set of clock speeds is used with at least one clock, and the second set of clock speeds is faster than the first set of clock speeds. After completing the transfer of the first frame, the second frame is transferred based on the set of clock speeds.

Description

Technical Field

[0001] The present disclosure relates generally to processing systems and, more particularly, to one or more techniques for display and / or graphics processing. Background Art

[0002] Computing devices typically perform graphics processing (e.g., using a graphics processing unit (GPU)) to render graphics data for display by the computing device. Such computing devices may include, for example, computer workstations, mobile phones such as smartphones, embedded systems, personal computers, tablet computers, and video game consoles. The GPU is configured to execute a graphics processing pipeline that includes one or more processing stages that operate together to execute graphics processing commands and output frames. A central processing unit (CPU) can control the operation of the GPU by issuing one or more graphics processing commands to the GPU. Modern CPUs are typically capable of executing multiple applications simultaneously, each of which may need to utilize a GPU during execution. Devices that provide content for visual presentation on a display may utilize a GPU.

[0003] In some cases, the GPU may render frames at a variable rate, which may cause the frame processor to drop frames. Therefore, in view of the variable frame rendering rate, there is a need for improved frame delivery techniques. Summary of the invention

[0004] A simplified summary of one or more aspects is given below to provide a basic understanding of these aspects. The content of the present invention is not an extensive overview of all intended aspects, and its purpose is neither to identify the key or critical elements of all aspects, nor to delineate the scope of any or all aspects. Its only purpose is to present some concepts of one or more aspects in a simplified form as a prelude to a more detailed description presented later.

[0005] The display content processing technique may be performed in at least two processing stages, such as: (1) GPU rendering; and (2) frame transfer / display. The GPU rendering of frame N+1 may be performed simultaneously with the transfer of the previous frame N to the physical display panel. In the case where the GPU rendering of frame N+1 exceeds the vertical synchronization (V SYNC ) time interval, a variable refresh rate technique (such as, Synchronous (Q SYNC ) function) to extend the length of the frame time to prevent frame N from being displayed repeatedly in two consecutive frame times. While extending the frame time may allow slightly more time for GPU rendering, implementing the variable refresh rate feature may result in variable frames per second (FPS) processing rates, which may result in frame drops in later frames based on the display processing unit (DPU) hardware configuration.

[0006] Therefore, when the length of the first frame is extended by the variable refresh rate feature, the second frame after the first frame may be shortened by the same length as the length of the first frame extended, so as to compensate for the extension of the first frame and to maintain a stable FPS processing rate over multiple frames. Specifically, after completing the rendering of the second frame, the DPU may determine whether the first frame is currently being transferred to the display panel or has been transferred to the display panel, wherein the fact that the first frame is currently being transferred to the display panel after completing the rendering of the second frame indicates that the GPU rendering is in the initial V SYNC The first frame is transferred to the display panel before the second frame is rendered, indicating that the GPU rendering of the second frame is delayed beyond the initial V SYNC When it is determined that the first frame is currently being transferred, at least one clock is maintained at a set of default clock speeds for transferring the second frame to the display panel. Alternatively, when it is determined that the first frame has been transferred, the at least one clock is accelerated via a second set of clocks, wherein the second set of speeds is configured to complete the transfer of the second frame in a shortened time period and compensate for the extended time period / delay of the previous frame.

[0007] In one aspect of the present disclosure, a method, computer-readable medium, and apparatus for frame processing are provided. The apparatus may be configured to determine whether a first frame is currently being transferred or has been transferred after completing rendering of a second frame, the second frame following the first frame. The apparatus may also be configured to use a first set of clock speeds with at least one clock when determining that the first frame is currently being transferred, and to use a second set of clock speeds with at least one clock when determining that the first frame has been transferred, the second set of clock speeds being faster than the first set of clock speeds. After completing the transfer of the first frame, the second frame is transferred based on the set of clock speeds used.

[0008] To achieve the above and related ends, one or more aspects include features fully described below and particularly pointed out in the claims. The following description and the accompanying drawings set forth in detail certain illustrative features of one or more aspects. However, these features are merely indicative of some of the various ways in which the principles of the various aspects may be employed, and the present description is intended to include all such aspects and their equivalents. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram illustrating an example content generation system in accordance with one or more techniques of this disclosure.

[0010] Figure 2 An example GPU in accordance with one or more techniques of this disclosure is shown.

[0011] Figure 3A-3B The impact of implementing a variable refresh rate feature to extend the length of the frame is shown.

[0012] Figure 4 A video mode timing diagram is shown where the transfer time of the DPU is reduced to compensate for the long rendering time of the GPU.

[0013] Figure 5 is a flow chart illustrating determining whether one or more clocks need to be accelerated to reduce the transmission time of a frame.

[0014] Figure 6 is a flowchart of an example method in accordance with one or more techniques of this disclosure.

[0015] Figure 7 is a conceptual data flow diagram illustrating the data flow between different units / components in an example apparatus. . DETAILED DESCRIPTION

[0016] Various aspects of the system, device, computer program product and method are described more fully below with reference to the accompanying drawings. However, the present disclosure can be implemented in many different forms and should not be interpreted as being limited to any specific structure or function presented in the present disclosure. On the contrary, these aspects are provided to make the present disclosure comprehensive and complete, and to fully convey the scope of the present disclosure to those skilled in the art. Based on the teachings of this article, a person skilled in the art should understand that the scope of the present disclosure is intended to cover any aspect of the system, device, computer program product and method disclosed herein, regardless of whether it is implemented independently of other aspects of the present disclosure or implemented in combination with other aspects of the present disclosure. For example, any number of aspects set forth herein can be used to implement a device or practice method. In addition, the scope of the present disclosure is intended to cover such a device or method, which is practiced using other structures, functions or structures and functions in addition to or different from the various aspects of the present disclosure set forth herein. Any aspect disclosed herein can be implemented by one or more elements of the claims.

[0017] Although various aspects are described herein, many variations and permutations of these aspects fall within the scope of the present disclosure. Although some potential benefits and advantages of aspects of the present disclosure are mentioned, the scope of the present disclosure is not limited to specific benefits, uses or objectives. On the contrary, various aspects of the present disclosure are intended to be widely applicable to different wireless technologies, system configurations, networks and transmission protocols, some of which are illustrated by way of example in the drawings and the following description. The detailed description and drawings are merely illustrative of the present disclosure and are not limiting, and the scope of the present disclosure is defined by the appended claims and their equivalents.

[0018] Several aspects are described with reference to various apparatus and methods. These apparatus and methods are described in the following detailed description and illustrated in the accompanying drawings by various blocks, components, circuits, processes, algorithms, etc. (collectively, "elements"). These elements can be implemented using electronic hardware, computer software, or any combination thereof. Whether these elements are implemented as hardware or software depends on the specific application and design constraints imposed on the overall system.

[0019] For example, an element, any part of an element, or any combination of elements may be implemented as a "processing system", which includes one or more processors (which may also be referred to as processing units). Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), general-purpose GPUs (GPGPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems on chip (SOCs), baseband processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functions described in this disclosure. One or more processors in a processing system may execute software. Software may be broadly interpreted as referring to instructions, instruction sets, codes, code segments, program codes, programs, subroutines, software components, applications, software applications, software packages, routines, subroutines, objects, executable files, threads in execution, processes, functions, etc., regardless of whether it refers to software, firmware, middleware, microcode, hardware description languages, or other.

[0020] The term application may refer to software. As described herein, one or more technologies may refer to an application (e.g., software) configured to perform one or more functions. In these examples, the application may be stored in a memory (e.g., an on-chip memory of a processor, a system memory, or any other memory). The hardware described herein (such as a processor) may be configured to execute an application. For example, an application may be described as including a code that causes the hardware to perform one or more technologies described herein when executed by the hardware. As an example, the hardware may access the code from the memory and execute the code accessed from the memory to perform one or more technologies described herein. In some examples, components are identified in the present disclosure. In these examples, the component may be hardware, software, or a combination thereof. The component may be a separate component or a subcomponent of a single component.

[0021] Therefore, in one or more examples described herein, the functions can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored on a computer-readable medium or encoded as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media. Storage media can be any available medium that a computer can access. As an example and not limitation, such computer-readable media may include random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), optical disk storage, magnetic disk storage, other magnetic storage devices, a combination of computer-readable media of the above types, or any other medium that can be used to store computer executable code in the form of instructions or data structures that can be accessed by a computer.

[0022] In general, this disclosure describes techniques for performing graphics processing in a single device or multiple devices that can improve rendering of graphical content and / or reduce the load on a processing unit (e.g., any processing unit configured to perform one or more of the techniques described herein, such as a GPU). For example, this disclosure describes techniques suitable for performing graphics processing in any device that utilizes a graphics processor. Other potential advantages of such techniques are described in this disclosure.

[0023] As used herein, instances of the term "content" may refer to "graphic content," "images," and the like, regardless of whether such terms are used as adjectives, nouns, or other parts of speech. In some examples, the term "graphic content" as used herein may refer to content produced by one or more processes of a graphics processing pipeline. In a further example, the term "graphic content" as used herein may refer to content produced by a processing unit configured to perform graphics processing. In yet another example, as used herein, the term "graphic content" may refer to content produced by a graphics processing unit.

[0024] In an example, the term "display content" used herein may refer to content generated by a processing unit configured to perform display processing. In a further example, the term "display content" used herein may refer to content generated by a display processing unit. Graphics content may be processed to become display content. For example, a graphics processing unit may output graphics content (such as a frame) to a buffer (which may be referred to as a frame buffer). The display processing unit may read graphics content (such as one or more frames) from the buffer and perform one or more display processing techniques thereon to generate display content. For example, the display processing unit may be configured to perform synthesis on one or more rendered layers to generate a frame. As another example, the display processing unit may be configured to synthesize, mix or otherwise merge two or more layers into one frame. The display processing unit may be configured to perform scaling (e.g., zooming in or out) on a frame. In some examples, a frame may refer to a layer. In other examples, when a frame includes two or more layers, a frame may refer to two or more layers that have been mixed together to form a frame. Alternatively, a frame including two or more layers may be subsequently mixed.

[0025] Figure 1 1 is a block diagram showing an example content generation system 100 configured to implement one or more techniques of the present disclosure. The content generation system 100 includes a device 104. The device 104 may include one or more components or circuits for performing various functions described herein. In some examples, one or more components of the device 104 may be components of a SOC. The device 104 may include one or more components configured to perform one or more techniques of the present disclosure. In the example shown, the device 104 may include a processing unit 120 and a system memory 124. In some aspects, the device 104 may include many optional components (e.g., a communication interface 126, a transceiver 132, a receiver 128, a transmitter 130, a display processor 127, and one or more displays 131). The display 131 may refer to one or more displays 131. For example, the display 131 may include a single display or multiple displays, and the multiple displays may include a first display and a second display. The first display may be a left-eye display, and the second display may be a right-eye display. In some examples, the first and second displays may receive different frames for presentation thereon. In other examples, the first and second displays may receive the same frame for presentation thereon. In a further example, the results of the graphics processing may not be displayed on the device, for example, the first and second displays may not receive any frames for rendering thereon. Instead, the frames or graphics processing results may be transferred to another device. In some aspects, this may be referred to as split rendering.

[0026] Processing unit 120 may include internal memory 121. Processing unit 120 may be configured to perform graphics processing using graphics processing pipeline 107. In some examples, device 104 may include a display processor (such as display processor 127) to perform one or more display processing techniques on one or more frames generated by processing unit 120 before being displayed by one or more displays 131. For example, display processor 127 may be configured to perform one or more display processing techniques on one or more frames generated by processing unit 120. One or more displays 131 may be configured to display or otherwise present frames processed by display processor 127. In some examples, one or more displays 131 may include one or more of a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, a projection display device, an augmented reality display device, a virtual reality display device, a head mounted display, or any other type of display device.

[0027] Memory external to processing unit 120, such as system memory 124, may be accessible by processing unit 120. For example, processing unit 120 may be configured to read and / or write from external memory, such as system memory 122. Processing unit 120 may be communicatively coupled to system memory 124 via a bus. In some examples, processing unit 120 may be communicatively coupled to internal memory 121 via a bus or via a different connection. Internal memory 121 or system memory 124 may include one or more volatile or non-volatile memory or storage devices. In some examples, internal memory 121 or system memory 124 may include RAM, static random access memory (SRAM), dynamic random access memory ("DRAM"), erasable programmable ROM (EPROM), EEPROM, flash memory, magnetic data media or optical storage media, or any other type of memory.

[0028] According to some examples, the internal memory 121 or the system memory 124 may be a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not implemented in a carrier wave or a propagating signal. However, the term "non-transitory" should not be interpreted to mean that the internal memory 121 or the system memory 124 is non-removable or that its contents are static. As an example, the system memory 124 may be removed from the device 104 and moved to another device. As another example, the system memory 124 may not be removed from the device 104.

[0029] The processing unit 120 may be a CPU, a GPU, a GPGPU, or any other processing unit that may be configured to perform graphics processing. In some examples, the processing unit 120 may be integrated into the motherboard of the device 104. In a further example, the processing unit 120 may be present on a graphics card installed in a port of the motherboard of the device 104, and the processing unit 120 may be otherwise incorporated into a peripheral device configured to interact with the device 104. The processing unit 120 may include one or more processors, such as one or more microprocessors, GPUs, ASICs, FPGAs, arithmetic logic units (ALUs), DSPs, discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuits, or any combination thereof. If the technology is partially implemented in software, the processing unit 120 may store the instructions of the software in an appropriate non-temporary computer-readable storage medium (e.g., an internal memory 121), and may use one or more processors to execute instructions in hardware to perform the technology of the present disclosure. Any of the above items (including hardware, software, a combination of hardware and software, etc.) may be considered as one or more processors.

[0030] In some aspects, the content generation system 100 may include an optional communication interface 126. The communication interface 126 may include a receiver 128 and a transmitter 130. The receiver 128 may be configured to perform any receiving functions described herein with respect to the device 104. In addition, the receiver 128 may be configured to receive information from another device, such as eye or head position information, rendering commands, or position information. The transmitter 130 may be configured to perform any sending functions described herein with respect to the device 104. For example, the transmitter 130 may be configured to send information to another device, which may include a request for content. The receiver 128 and the transmitter 130 may be combined into a transceiver 132. In such examples, the transceiver 132 may be configured to perform any receiving functions and / or sending functions described herein with respect to the device 104.

[0031] Reference again Figure 1 In a particular aspect, the graphics processing pipeline 107 may include a frame processing component 198 configured to: determine whether a first frame is currently being transferred or has been transferred after completing rendering of a second frame, the second frame following the first frame; when the first frame is determined to be currently being transferred, use a first set of clock speeds with at least one clock, and when the first frame is determined to have been transferred, use a second set of clock speeds with at least one clock, the second set of clock speeds being faster than the first set of clock speeds; and after completing the transfer of the first frame, transfer the second frame based on the set of clock speeds used.

[0032] Devices such as device 104 may refer to any device, apparatus or system configured to perform one or more technologies described herein. For example, a device may be a server, a base station, a user device, a client device, a station, an access point, a computer (such as a personal computer), a desktop computer, a laptop computer, a tablet computer, a computer workstation, or a host, a final product, an apparatus, a phone, a smart phone, a server, a video game platform or console, a handheld device such as a portable video game device or a personal digital assistant (PDA), a wearable computing device such as a smart watch, an augmented reality device or a virtual reality device, a non-wearable device, a display or display device, a television, a television set-top box, an intermediate network device, a digital media player, a video streaming device, a content streaming device, a vehicle-mounted computer, any mobile device, any device configured to generate graphics content, or any device configured to perform one or more technologies described herein. The process herein may be described as being performed by a specific component (e.g., a GPU), but in further embodiments, other components (e.g., a CPU) consistent with the disclosed embodiments may be used to perform.

[0033] Figure 2 1 is a block diagram 200 illustrating an exemplary display framework including a processing unit 120 , a system memory 124 , a display processor 127 , and a display 131 , as may be identified in connection with the exemplary device 104 .

[0034] The GPU is typically included in a device that provides content for visual presentation on a display. For example, processing unit 120 may include GPU 210, which is configured to render graphics data for display on a computing device (e.g., device 104), which may be a computer workstation, a mobile phone, a smart phone or other smart device, an embedded system, a personal computer, a tablet computer, a video game console, etc. The operation of GPU 210 can be controlled based on one or more graphics processing commands provided by CPU 215. CPU 215 can be configured to execute multiple applications simultaneously. In some cases, each of the multiple applications executed concurrently can utilize GPU 210 simultaneously. Processing techniques can be performed via processing unit 120 to output frames through a physical or wireless communication channel.

[0035] The system memory 124, which can be executed by the processing unit 120, may include a user space 220 and a kernel space 225. The user space 220 (sometimes referred to as an "application space") may include software applications and / or application frameworks. For example, software applications may include an operating system, a media application, a graphics application, a workspace application, etc. The application framework may include a framework used by one or more software applications, such as a library, a service (e.g., a display service, an input service, etc.), an application program interface (API), etc. The kernel space 225 may further include a display driver 230. The display driver 230 may be configured to control the display processor 127. For example, the display driver 230 may cause the display processor 127 to change the display rate (e.g., the number of frames per second (FPS)) of the generated frames.

[0036] The display processor 127 includes a display control block 235 and a display interface 240. The display processor 127 may be configured to manipulate the functions of the display 131 (e.g., based on input received from the display driver 230). For example, the display control block 235 may be configured to receive instructions from the display driver 230 to change the FPS display rate of the display 131. The display control block 235 may also be configured to output image frames to the display 131 via the display interface 240 based on the display refresh rate determined by the display driver 230. The display driver 230 may output refresh rate information indicating a new display refresh rate / change to the current display refresh rate. The display control block 235 may receive the refresh rate information and cause the display interface 240 to output image frames to the display 131 based on the refresh rate information. In some examples, the display control block 235 may additionally or alternatively perform post-processing of image data provided based on execution of the system memory 124 by the processing unit 120.

[0037] The display interface 240 may be configured to cause the display 131 to display image frames and / or to establish a specific display rate (e.g., a specific FPS display rate) at which the display 131 displays image frames. The display interface 240 may output image data to the display 131 according to an interface protocol (e.g., MIPIDSI (Mobile Industry Processor Interface, Display Serial Interface)). That is, the display 131 may be configured according to the MIPIDSI standard. The MIPIDSI standard supports a video mode and a command mode. In an example in which the display 131 is operating in video mode, the display processor 127 may continuously refresh the graphics content of the display 131. For example, the entire graphics content may be refreshed on a refresh cycle-by-refresh basis (e.g., line-by-line basis). In an example in which the display 131 is operating in command mode, the display processor 127 may write the graphics content of the frame into the buffer 250.

[0038] In some such examples, display processor 127 may not continuously refresh the graphical content of display 131. Instead, display processor 27 may use vertical synchronization (V SYNC ) pulses to coordinate the rendering and consumption of graphics content at buffer 250. For example, when V SYNC When the V SYNC The generation of the pulse may indicate that the current graphics content has been rendered at the buffer 250. However, it should also be understood that V SYNC The generation of pulses may indicate a time period based on, for example, the current FPS display rate of display 131 .

[0039] Based on the display controller 245, the display client 255 and the buffer 250, the frame is displayed on the display 131. The display controller 245 can receive image data from the display interface 240 and store the received image data in the buffer 250. In some examples, the display controller 245 can output the image data stored in the buffer 250 to the display client 255. Therefore, the buffer 250 can represent a local memory for the display 131. In some examples, the display controller 245 can output the image data received from the display interface 240 directly to the display client 255. It should be further understood that determining whether to change the FPS display rate of the display 131 can be performed by the display controller 245, the display control block 235 and / or the display driver 230.

[0040] The display client 255 may be associated with a touch panel that senses interactions between a user and the display 131. When a user interacts with the display 131, one or more sensors in the touch panel may output signals to the display controller 245 indicating which of the one or more sensors have sensor activity, duration of sensor activity, pressure applied to the one or more sensors, etc. The display controller 245 may use the sensor outputs to determine the manner in which the user has interacted with the display 131. The display 131 may further be associated with / include other devices that operate in conjunction with the display client 255, such as a camera, a microphone, and / or a speaker.

[0041] Some processing techniques of device 104 may be performed over three stages (e.g., stage 1: rendering stage; stage 2: composition stage; stage 3: display / transfer stage). However, other processing techniques may combine the composition stage and the display / transfer stage into a single stage, so that the processing techniques may be performed based on a total of two stages (e.g., stage 1: rendering stage; stage 2: composition / display / transfer stage). During the rendering stage, GPU 210 may process content buffers for execution of an application that generates content on a pixel-by-pixel basis. During the composition and display stages, pixel elements may be assembled to form a frame, which is transferred to a physical display panel / subsystem (e.g., display 131) that displays the frame.

[0042] In order to maintain a constant frame per second (FPS) transfer rate of frames to the physical display panel / subsystem, the various stages of the processing pipeline can be performed based on a constant time period. In some cases, the time period can be controlled by the desired FPS transmission rate of the frame. For example, a 60Hz FPS display panel with a frame time of 16.6ms may need to complete the various stages of the processing pipeline within 16.6ms to prevent frame drops, while a 120Hz FPS display screen with a frame time of 8.3ms may need to complete the various stages of the processing pipeline within 8.3ms to prevent frame drops. Specifically, in order to ensure that the display panel / subsystem displays content smoothly, the time period over which the processing pipeline is executed should be less than or equal to the frame time of the display panel / subsystem. If the total time required to complete the various stages of the processing pipeline exceeds the frame time of the display panel, frame drops may cause non-smooth / perceptible pauses in the displayed content.

[0043] Figure 3A-3B 312 (such as Q SYNC The variable refresh rate technique is configured to perform frame updates at a rate that can be dynamically varied. An associated display can synchronize the display rate with the rate at which frames are being received, thereby reducing frame stuttering and / or tearing that may otherwise be caused by a varying frame rate. In aspects of the frame processing technique, the GPU may render frame N+1 at the same time that the DPU displays / transfers frame N to the display panel. In diagram 300, the rendering of frame N+2 takes longer than period T 302, which may correspond to V SNYCTime 304 and / or the frame time of the display panel. For example, depending on the configuration of the display panel, period T 302 may be set to 16.6 ms or 8.3 ms. Since the rendering of frame N+2 extends beyond the frame time and into the subsequent frame, the DPU may repeat frame N+1 for two consecutive frames and delay the transfer of frame N+2 to the next frame, in which the rendering of frame N+3 is to begin. The frame drop caused by delaying the transfer of frame N+2 to the next frame may cause a perceptible pause in the displayed content, known as jank.

[0044] In order to reduce jank caused by the GPU rendering frames too slowly and exceeding the frame time, the diagram 310 introduces a variable refresh rate feature 312 to extend the length of the frame time of the corresponding frame. That is, when it is expected that the rendering of the frame will not be completed within the initial frame time, the variable refresh rate feature 312 provides flexibility to increase the V SYNC Time 304 replaces the previously fixed frame time with the extended frame time. Although extending the frame time can allow slightly more time to render the frame, the variable V SYNC Time 304 may also cause the FPS transfer rate to become variable and cause other frames to drop in the following frames based on the DPU hardware configuration. Therefore, a stable FPS transfer rate can further improve the way the end user visually perceives the frame.

[0045] Without implementing the variable refresh rate feature 312, the FPS transfer rate is usually a fixed value for each corresponding frame. For example, for a 60Hz display panel, the FPS transfer rate can be fixed at 16.6ms. However, in the case of implementing the variable refresh rate feature 312, the FPS transfer rate can be dynamically changed based on the frame rendering time of the GPU. For example, frame N+1 can be initially set to 16.6ms, frame N+2 can be extended to 20ms, and frame N+3 can be changed to some other value. Then, the task of the DPU is to manipulate the refresh rate of the display panel based on the length of each corresponding frame. Therefore, if the frame rendering time takes 20ms, the variable refresh rate feature 312 can extend the frame from 16.6ms to 20ms to compensate for the additional time spent, thereby preventing the jank problem. On the contrary, if the frame rendering time only takes 15ms, the variable refresh rate feature 312 does not need to manipulate the 16.6ms frame, because the GPU can perform rendering within the frame time. As previously described, the variable FPS transfer rate may cause some frames in the following frames to drop. However, the impact of frame drops on the displayed content is not as obvious as the impact caused by jank.

[0046] Figure 4A video mode timing diagram 400 is shown in which the transfer time of the DPU is reduced to compensate for the extended rendering time of the GPU. The reduced transfer time is configured to maintain a stable FPS transfer rate, even for gaming applications that place heavy demands on the graphics processing pipeline. In one aspect, when the variable refresh rate feature 404 is in effect to extend the frame, the display serial interface (DSI) clock can be dynamically changed to speed up the display / transfer time and / or composition time, so that the data stream processing can be maintained at a stable FPS processing rate. That is, when the time taken by the GPU to render the previous frame is longer than the initial frame time, one or more clocks can be accelerated to increase the display / transfer time and / or composition time.

[0047] In the video mode timing diagram 400, the initial frame time is represented by period T 410. SYNC 402, when each of the three frames N, N+1 and N+2 in the diagram 400 has a length of period T 410, the data stream is processed at a constant FPS rate. However, when the variable refresh rate feature 404 is in effect, the duration of frame N is extended to the duration of T+a by delaying "a" 414. Under the conventional method, this may cause the FPS to change due to the length of frame N being longer than each of the frames N+1 and N+2. Accordingly, one or more clocks may be accelerated to reduce the transfer time of the application data (AP data) 406 in frame N+1, thereby shortening the length of frame N+1 to period X 412. Period X 412 may be equal to Ta to maintain a stable FPS processing rate. Specifically, the processing rate in FPS over the time periods T+a, Ta and T is the same as the FPS processing rate when all three frames N, N+1 and N+2 each have a period T 410. Additionally, the execution time of the data dictionary (DDIC) 408 may also be reduced in Frame N+1 based on the cycle X 412 .

[0048] Diagram 400 may be implemented for both a video mode panel and / or a command mode panel. In one aspect, the DPU may be switched from the command mode panel to the video mode panel to provide a stable FPS processing rate when a high frame rate is detected, such as during execution of a gaming application.

[0049] For each frame submission, when GPU rendering of the frame is completed, the DPU line counter can be compared with the vertical total (V TOTAL ) line quantity to determine whether the data transfer is also completed. V TOTAL Indicates the total number of display lines (e.g., lines 1 to L) in the display panel / subsystem. The display panel / subsystem can be divided into active lines and inactive lines (e.g., vertical black (V BLACK)) line, and is counted by the line counter. V TOTAL Can be equal to the total number of active and inactive lines. For example, a smartphone may have 1988 active lines and 20 inactive / V BLACK Line, a total of 2008 V TOTAL Other platforms may have different configurations (e.g., for V2030 TOTAL The DPU line count increases as the DPU processes frames line by line and can be equal to V in some configurations. TOTAL The DPU line counter can be reset after finishing rendering a frame.

[0050] Since the DPU line counter is updated with V when a rendered frame (e.g., frame N+1) is completed, TOTAL For comparison, it is less than V TOTAL The line count of indicates that the data transfer of the previous frame (e.g., frame N) is still in progress. This means that the GPU rendering of the current frame has been completed in time (e.g., within period T 410), and the default DSI clock, mobile display processor (MDP) clock, and other default settings can be maintained for the data transfer and / or composition in subsequent frames. However, when the DPU line count is greater than V when the rendered frame is completed, the DPU line count is greater than V. TOTAL Indicates that the variable refresh rate feature 404 is in effect to extend the frame. This means that the GPU rendering of the current frame is not completed in time (e.g., within cycle T 410), and one or more clocks should be accelerated for subsequent frames to reduce the frame transfer time and / or composition time to cycle X 412, which thereby compensates for the delay a 414 included in the extended GPU rendering time.

[0051] The length of delay a 414 may be identified via a DPU line counter, one or more DSI clocks, and / or other timing parameters. In particular, period X 412 may be equal to period T 410 minus delay a 414. That is, if the GPU rendering takes an additional time (e.g., 2 ms) beyond period T 410, the DPU may reduce the transfer time by the same amount of time (e.g., 2 ms) to compensate for the increased rendering overhead. The new period for subsequent frames (e.g., period X 412, which is equal to Ta) is used together with the panel resolution and other DSI timing parameters (e.g., vertical porch and horizontal porch) to determine adjustments to one or more clocks (such as DSI bit clock, pixel clock, or byte clock). This adjustment is implemented to ensure that the DPU transfer time and / or composition time can be performed according to the new period. For example, the DSI clock may be accelerated while maintaining other current DSI timing parameters (such as vertical porch, horizontal porch, and / or active frame area). The MDP core clock and other related clocks may be accelerated to match the accelerated DSI clock.

[0052] Figure 5 500 is a flowchart illustrating determining whether one or more clocks should be accelerated to reduce the transfer time of a frame. At 502, a display submission is generated that ultimately determines a frame for rendering by the GPU. Specifically, the display submission makes one or more tentative changes to data used to generate the frame that is unchanged prior to rendering the frame (e.g., a display submission 502 that is "submitted" via the GPU for actions related to rendering the frame). At 504, a variable refresh rate feature is implemented. For example, the time required for the GPU to render a frame may exceed the current frame time, causing the variable refresh rate function to activate and extend the frame time. At 506, the processing system waits for the GPU to complete the frame rendering so that at 508, the processing system can prepare the frame for transfer to the display panel. For example, at 508, the DPU can synthesize one or more rendered layers of the frame, such as by assembling pixel elements to form a frame.

[0053] At 510 , the processing system determines whether LINE_COUNT is greater than V TOTAL In one aspect, the line counter that outputs LINE_COUNT can be hardware reset to an initial value after each frame in the data stream is completed. In another aspect, the initial value of LINE_COUNT can be based on the start of the first line of each frame in the data stream, where LINE_COUNT continues to increase with each line. If, in some embodiments, LINE_COUNT is not greater than V TOTAL , then the variable refresh rate feature can operate freely as expected without any adjustment to the DPU transfer time. That is, at 512, the default MDP clock and DSI clock are used to transfer frames.

[0054] However, if LINE_COUNT is greater than V TOTAL , then the variable refresh rate feature may extend the frame due to the slow speed at which the GPU renders the frame. Therefore, at 514, the promoted / accelerated time for the MDP clock and the DSI clock is calculated for transferring the frame. Although the default clock speed can be based on the fixed frame time of the physical display panel, the accelerated clock speed can be based on the GPU rendering time. For example, when the GPU rendering time of frame N+1 is shortened from time T to Ta because the GPU rendering time of the previous frame N is longer than time T, the clock may be accelerated to a rate that allows frame N+1 to be transferred to the display panel within the shortened time Ta. In any case, the variable refresh rate feature ends at 516, and at 518, the frame is transferred to the physical display panel for display.

[0055] Figure 6 600 is a flowchart of an example method according to one or more techniques of the present disclosure. The method 600 may be performed by combining Figure 1-5 The examples are performed using a frame synthesizer, a display processor, a DPU, a GPU, an apparatus for graphics processing, a wireless communication device, etc.

[0056] At 602, the frame processor may switch from command mode to video mode when a constant display rate is requested. Figure 4 The frame processor may switch to the video mode represented by video mode timing diagram 400 after receiving a request for a constant display rate in command mode (e.g., based on detection of a high frame rate, initiation of a high frame rate application, manual input by a user, etc.).

[0057] At 604, the frame processor may determine whether the first frame is currently being transferred or has been transferred after completing rendering of the second frame, wherein the second frame follows the first frame. Figure 4 , the frame processor can determine whether frame N of the video mode timing diagram 400 has been extended to T+a by the variable refresh rate feature 404, wherein the delay a 414 indicates that frame N has been transferred to the display panel when the rendering of frame N+1 is completed; or determine whether the length of frame N remains at period T 410 (e.g., there is no delay a 414), wherein the length of period T 410 indicates that frame N is still currently being transferred to the display panel. Therefore, in various aspects, the frame processor can determine whether the first frame is currently being transferred or has been transferred based on whether the delay is measured by the DSI clock.

[0058] At 606, determining whether the first frame is currently being transferred or has been transferred may further include determining whether the DPU line counter indicates that the first frame is being transferred to the display buffer or has been completely transferred to the display buffer. Figure 5 If the frame processor determines at 510 that LINE_COUNT is greater than V TOTAL , then the indication is that frame N has been completely transferred to the buffer, which further indicates that frame N has been transferred to the display panel. Alternatively, if the frame processor determines at 510 that LINE_COUNT is less than V TOTAL , it indicates that frame N has not been completely transferred to the buffer, which further indicates that frame N is currently being transferred to the display panel.

[0059] At 608, when it is determined that the first frame is currently being transferred, at least one clock can be used with the first set of clock speeds. The at least one clock can include at least one of a DSI clock or an MDP clock. For example, referring to Figure 5 If the frame processor determines at 510 that LINE_COUNT is less than V TOTAL , the frame processor may apply the default MDP clock and DSI clock at 512. That is, the first set of clock speeds may be a set of default clock speeds for providing transfer of frames within the first period T.

[0060] At 610, when the first frame has been transferred, V SYNC The period can be extended from the first period T to a second period T+a. Based on the second period T+a, at least one clock can be used with a second set of clock speeds that is faster than the first set of clock speeds. For example, referring to Figure 4 , the variable refresh rate feature 404 can extend the original V by delaying a 414 SYNC 402, where the original V SYNC 402 provides an initial period T 410 for frame N. Since the extended period T+a is longer than period T 410, it may be necessary to speed up the set of clock speeds for frame N+1 to maintain a stable FPS processing rate.

[0061] At 612, a second set of clock speeds may be determined based on reducing the transfer completion time of the second frame so as to provide transfer of the second frame within the third period Ta. Figure 4 , the frame processor may determine a second set of clock speeds based on reducing the transfer completion time of frame N+1 to period X 412, where period X 412 is equal to Ta.

[0062] At 614, when it is determined that the first frame has been transferred, at least one clock can be used with a second set of clock speeds, wherein the second set of clock speeds is faster than the first set of clock speeds. Figure 5 If the frame processor determines at 510 that LINE_COUNT is greater than V TOTAL , the frame processor may calculate and boost the MDP clock and the DSI clock (e.g., increase their rates) at 514. That is, the second set of clock speeds may be a set of accelerated clock speeds for providing transfer of frames within the third period Ta. The second set of clock speeds may be based on the variable refresh rate of the second frame.

[0063] At 616, the frame processor may transfer the second frame based on the set of clock speeds used after completing the transfer of the first frame. That is, block 616 may be performed based on the first set of clock speeds or the second set of clock speeds. For example, referring to Figure 5 When the variable refresh rate feature has been completed at 516 , the frame processor transfers frames at 518 based on the default MDP clock and DSI clock applied at 512 or the enhanced MDP clock and DSA clock at 514 .

[0064] Figure 7 700 is a conceptual data flow diagram illustrating the data flow between different units / components in an example apparatus 702. Apparatus 702 may be a frame compositor, a frame processor, a display processor, a DPU, a wireless communication device, or other similar apparatus. Apparatus 702 includes a receiving component 704 that receives an indication from GPU 740 that frame N+1 has finished rendering. For example, as described in connection with 506, apparatus 702 may have to wait for the GPU rendering to complete before receiving component 702 receives the indication.

[0065] The device 702 includes a switching component 706 that switches the device 702 from the command mode to the video mode. For example, as described in conjunction with 602, the switching component 706 can switch from the command mode to the video mode when a constant display rate is requested, a high frame rate is detected, a high frame rate application is started, etc. In a configuration where the device 702 is already in the video mode when the indication is received from the GPU 740, the receiving component 704 can directly provide the V mode to the determining component 708. SYNC Additionally or alternatively, the determination component 708 may be provided with an indication of the V period after the switching component 706 has switched from the command mode to the video mode. SYNC Cycle indication.

[0066] The determining component 708 further included in the apparatus 702 is configured to determine the transfer status of the previous frame N. For example, as described in conjunction with 604, the determining component 706 can determine that frame N is still currently being transferred to the display panel (e.g., display panel 750). Therefore, the clock speed can be maintained for transferring frame N+1. As described in conjunction with 606, the determining component 708 can determine that frame N is still currently being transferred based on an indication from the DPU line counter that frame N is being transferred to the display buffer. As further described in conjunction with 604, the determining component 708 can determine that frame N has been transferred to the display panel (e.g., display panel 750). Specifically, as described in conjunction with 606, the determining component 708 can determine that frame N has been transferred based on an indication from the DPU line counter that frame N has been completely transferred to the display buffer.

[0067] The apparatus 702 includes an extension component 710 that extends V when the frame N has been transferred to the display panel 750. SYNC For example, as described in conjunction with 610, when frame N has been transferred, the extension component 710 can SYNC The period is extended from the first period T to the second period T+a. Extend V SYNC The period may require an increase in clock speed for transferring frame N+1. As described in conjunction with 612, the determination component 708 can determine a second set of (increased) clock speeds for transferring frame N+1 based on reducing the transfer completion time of frame N+2 to the third period Ta.

[0068] The apparatus 702 includes a using component 712, wherein the component uses an increased clock speed or a maintained clock speed with at least one clock, depending on whether the determining component 708 determines whether the frame N is currently being transferred or has been transferred. For example, as described in conjunction with 608, when the frame N is currently being transferred, the using component 712 uses a first set of clock speeds with at least one clock, and as described in conjunction with 714, when the frame N has been transferred, the using component 712 uses a second set of clock speeds with at least one clock, the second set of clock speeds being faster than the first set of clock speeds. The sending / transferring component 714 included in the apparatus 702 is configured to send / transfer the frame N+1 to the display panel 750 based on the clock speed used by the using component 712.

[0069] The device 702 may include a device for performing the above Figure 6 Additional components for each box of the algorithm in the flowchart. Therefore, Figure 6Each block in the flowchart of can be performed by a component, and the device 702 may include one or more of these components. The component can be one or more hardware components specifically configured to implement the process / algorithm, implemented by a processor configured to execute the process / algorithm (e.g., logic and / or code executed by the processor), stored in a computer-readable medium for processor implementation, or some combination of the above.

[0070] Therefore, when the length of the first frame is extended by the variable refresh rate feature, the second frame after the first frame can be shortened by the same length as the length of the first frame extended to compensate for the extension of the first frame and maintain a stable FPS processing rate over multiple frames. Specifically, after completing the rendering of the second frame, the frame processor (e.g., DPU) can determine whether the first frame is currently being transferred to the display panel or has been transferred to the display panel, wherein the first frame is currently still being transferred to the display panel indicating that the GPU rendering is in the initial V SYNC The time interval has completed, and the first frame has been transferred to the display panel indicating that GPU rendering has been delayed beyond the initial V SYNC When it is determined that the first frame is currently being transferred, at least one clock may be maintained at a set of default clock speeds for transferring the second frame to the display panel. Alternatively, when it is determined that the first frame has been transferred, at least one clock may be accelerated / set to a second set of clock speeds configured to complete the transfer of the second frame within a shortened time period and compensate for the extended time period / delay of the previous frame.

[0071] It should be understood that the specific order or hierarchy of blocks in the disclosed process / flowchart is an illustration of an example method. Based on design preferences, it is understood that the specific order or hierarchy of blocks in the process / flowchart can be rearranged. In addition, some blocks can be combined or omitted. The accompanying method claims present elements of the various blocks in a sample order and are not meant to be limited to the specific order or hierarchy presented.

[0072] The foregoing description is provided to enable those skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not limited to the aspects shown herein, but should conform to the full scope consistent with the claim language, wherein, unless otherwise specified, references to singular elements do not mean "one and only one", but "one or more". The word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any aspect described herein as "exemplary" is not necessarily to be construed as superior or preferred over other aspects.

[0073] Unless expressly specified otherwise, the term "some" refers to one or more, and the term "or" may be interrupted as "and / or" if the context does not dictate otherwise. Combinations such as "at least one of A, B, or C," "one or more of A, B, and C," "at least one of A, B, and C," and "A, B, C, or any combination thereof" include any combination of A, B, and / or C, and may include multiples of A, multiples of B, or multiples of C. Specifically, combinations such as "at least one of A, B, or C," "one or more of A, B, or C," "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, C, or any combination thereof" may be only A, only B, only C, A and B, A and C, B and C, or A and B and C, where any such combination may include one or more members of A, B, or C. All structural and functional equivalents to the elements of the various aspects described in this disclosure that are known or later known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be included in the claims. Furthermore, nothing disclosed herein is intended to be dedicated to the public, regardless of whether such disclosure is explicitly recited in the claims. Words such as "module," "mechanism," "element," and "device" cannot replace "unit." Therefore, unless any claim element is explicitly recited using the phrase "unit for...", the claim element shall not be interpreted as a functional module.

[0074] In one or more examples, the functions described herein can be implemented in hardware, software, firmware, or any combination thereof. For example, although the term "processing unit" has been used in the present disclosure, such processing unit can be implemented in hardware, software, firmware, or any combination thereof. If any function, processing unit, technique, or other module described herein is implemented in software, the function, processing unit, technique, or other module described herein can be stored or sent as one or more instructions or codes on a computer-readable medium.

[0075] Computer-readable media may include computer data storage media or communication media, the latter including any media that facilitates the transfer of a computer program from one place to another. In this manner, a computer-readable medium may generally correspond to: (1) a tangible computer-readable storage medium that is non-transitory; or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to obtain instructions, codes, and / or data structures to implement the techniques described in the present disclosure. By way of example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, compact disk read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage, or other magnetic storage devices. Disks and optical disks used herein include compact disks (CDs), laser optical disks, optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray disks, wherein disks generally reproduce data magnetically, while optical disks generally reproduce data optically with lasers. The above combinations should also be included within the scope of computer-readable media. A computer program product may include a computer-readable medium.

[0076] The technology of the present disclosure can be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs) or a group of ICs (e.g., chipsets). Various components, modules or units are described in the present disclosure to emphasize the functional aspects of the equipment configured to perform the disclosed technology, but it is not necessarily necessary to implement them by different hardware units. On the contrary, as described above, various units can be combined in any hardware unit, or provided by a group of interoperable hardware units including one or more processors as described above in combination with appropriate software and / or firmware. Therefore, the term "processor" used in this article can refer to any structure in the above structure or any other structure suitable for implementing the technology described herein. In addition, these technologies can be fully implemented in one or more circuits or logic elements.

[0077] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A frame processing method, comprising: After completing rendering of the second frame, determining whether the first frame is currently being transferred to the display panel or has been transferred to the display panel, the second frame following the first frame; using a first set of clock speeds with at least one clock when it is determined that the first frame is currently being transferred, and using a second set of clock speeds with the at least one clock when it is determined that the first frame has been transferred, the second set of clock speeds being faster than the first set of clock speeds; as well as After the transfer of the first frame is completed, the second frame is transferred to the display panel based on a set of clock speeds used.

2. The method according to claim 1, wherein: Determining whether the first frame is currently being transferred or has been transferred includes determining whether a display processing unit (DPU) line counter indicates that the first frame is being transferred into a display buffer of the display panel or has been completely transferred into the display buffer of the display panel.

3. The method according to claim 1, wherein: The at least one clock includes at least one of a display serial interface (DSI) clock or a mobile display processor (MDP) clock.

4. The method according to claim 1, wherein: The first set of clock speeds is a set of default clock speeds that provide for the transfer of frames within a first period T.

5. The method according to claim 1, further comprising: When the first frame has been transferred, the vertical synchronization (V SYNC ) period is extended from a first period T to a second period T+a, wherein the at least one clock is used with the second set of clock speeds.

6. The method according to claim 5, further comprising: The second set of clock speeds is determined based on reducing the transfer completion time of the second frame so as to provide transfer of the second frame within a third period Ta.

7. The method according to claim 1, wherein: The second set of clock speeds is based on a variable refresh rate of the second frame.

8. The method according to claim 1, further comprising: Switches from command mode to video mode when a constant display rate is requested.

9. A device for frame processing, comprising: Memory; as well as at least one processor coupled to the memory and configured to: After completing rendering of the second frame, determining whether the first frame is currently being transferred to the display panel or has been transferred to the display panel, the second frame following the first frame; using a first set of clock speeds with at least one clock when it is determined that the first frame is currently being transferred, and using a second set of clock speeds with the at least one clock when it is determined that the first frame has been transferred, the second set of clock speeds being faster than the first set of clock speeds; as well as After the transfer of the first frame is completed, the second frame is transferred to the display panel based on a set of clock speeds used.

10. The device according to claim 9, wherein: In order to determine whether the first frame is currently being transferred or has already been transferred, the at least one processor is also configured to determine whether a display processing unit (DPU) line counter indicates that the first frame is being transferred to a display buffer of the display panel or has already been completely transferred to the display buffer of the display panel.

11. The device according to claim 9, wherein: The at least one clock includes at least one of a display serial interface (DSI) clock or a mobile display processor (MDP) clock.

12. The device according to claim 9, wherein: The first set of clock speeds is a set of default clock speeds that provide for the transfer of frames within a first period T.

13. The device according to claim 9, wherein: The at least one processor is further configured to: set vertical synchronization (V SYNC ) period is extended from a first period T to a second period T+a, wherein the at least one clock is used with the second set of clock speeds.

14. The device according to claim 13, wherein: The at least one processor is further configured to determine the second set of clock speeds based on reducing a transfer completion time of the second frame to provide transfer of the second frame within a third period Ta.

15. The device according to claim 9, wherein: The second set of clock speeds is based on a variable refresh rate of the second frame.

16. The device according to claim 9, wherein: The at least one processor is further configured to switch from the command mode to the video mode when a constant display rate is requested.

17. The device according to claim 9, wherein: The apparatus is a wireless communication device.

18. A computer readable medium storing computer executable code, which, when executed by a processor, causes the processor to perform the following operations: After completing rendering of the second frame, determining whether the first frame is currently being transferred to the display panel or has been transferred to the display panel, the second frame following the first frame; using a first set of clock speeds with at least one clock when it is determined that the first frame is currently being transferred, and using a second set of clock speeds with the at least one clock when it is determined that the first frame has been transferred, the second set of clock speeds being faster than the first set of clock speeds; as well as After the transfer of the first frame is completed, the second frame is transferred to the display panel based on a set of clock speeds used.

19. The computer-readable medium of claim 18, wherein: In order to determine whether the first frame is currently being transferred or has been transferred, the processor is also configured to: determine whether a display processing unit (DPU) line counter indicates that the first frame is being transferred to a display buffer of the display panel or has been completely transferred to the display buffer of the display panel.

20. The computer-readable medium of claim 18, wherein: The first set of clock speeds is a set of default clock speeds that provide for the transfer of frames within a first period T.

Images