Video data processing based on sampling rate

Abstract

In general, aspects disclosed herein provide techniques for processing video data. Certain aspects provide a method for processing video data including a plurality of frames for display on a display as discussed herein. The method includes determining a sampling rate (N) based on a frame rate for displaying the video data on the display. The sampling rate N is greater than 1. The method further includes processing, for every N frames of the plurality of frames, one or more statistics associated with the frames to determine one or more display settings. The method further includes outputting the corresponding N frames of the plurality of frames to the display using the one or more display settings.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to display panels, and more particularly, to one or more techniques for display or graphics processing for display panels. BACKGROUND

[0002] Computing devices often utilize a graphics processing unit (GPU) to accelerate rendering of graphics data for display. Such computing devices can include, for example, computer workstations, mobile phones such as so-called smartphones, embedded systems, personal computers, tablet computers, and video game consoles. The GPU executes a graphics processing pipeline that includes one or more processing stages that operate together to execute graphics processing commands and output frames. A central processing unit (CPU) can control the operation of the GPU by issuing one or more graphics processing commands to the GPU. Today, CPUs are often capable of concurrently executing multiple applications, each of which can require utilization of the GPU during execution.

[0003] Computing devices can present graphics content on a display at various frame rates, such as 24 Hz, 30 Hz, 60 Hz, 120 Hz, or others. The higher the frame rate, the more fluid the user’s perception of the graphics content, such as when the graphics content includes rapidly changing visual effects, such as during scrolling, playing games, video playback or recording, and other applications that require high frame rates. However, requiring high frame rates can result in the opposite, interrupted display effects if the CPU or GPU is unable to timely handle the processing tasks. For example, when the CPU or GPU is too busy to timely complete processing for a high frame rate, the display can stall (or slow down) on a frame, resulting in interrupted display effects such as user interface stutters. SUMMARY

[0004] The following presents a simplified summary of one or more aspects in order to provide a basic understanding of such aspects. This summary is not an extensive overview of all contemplated aspects, and is intended to neither identify key nor critical elements of all aspects nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more aspects in a simplified form as a prelude to the more detailed description that is presented later.

[0005] Certain aspects of the present disclosure provide a method for processing video data including a plurality of frames for display on a display, as discussed herein. The method includes determining a sampling rate (N) based on a frame rate for displaying the video data on the display. The sampling rate N is greater than 1. The method further includes, for every N frames of the plurality of frames, processing one or more statistics associated with the frames to determine one or more display settings. The method further includes outputting the corresponding N frames of the plurality of frames to the display using the one or more display settings.

[0006] Certain aspects of the present disclosure provide a computing device. The computing device further includes a memory and a processor. The memory and processor are configured to determine a sampling rate N based on a frame rate for displaying video data including a plurality of frames on a display. The sampling rate N is greater than 1. The processor is configured to process one or more statistics associated with every N frames of the plurality of frames to determine one or more display settings. The processor is further configured to output the corresponding N frames of the plurality of frames to the display using the one or more display settings.

[0007] Certain aspects of the present disclosure provide a non-transitory computer- readable medium storing instructions that, when executed by a computing device as discussed herein, cause the computing device to operate as discussed herein. For example, the non-transitory computer-readable medium stores instructions that, when executed by a computing device, cause the computing device to determine a sampling rate N based on a frame rate for displaying video data including a plurality of frames on a display. The sampling rate N is greater than 1. The non-transitory computer-readable medium stores instructions that, when executed by a computing device, cause the computing device to process one or more statistics associated with every N frames of the plurality of frames to determine one or more display settings, and output the corresponding N frames of the plurality of frames to the display using the one or more display settings.

[0008] Certain aspects of the present disclosure provide a computing device including means for determining a sampling rate N based on a frame rate for displaying video data including a plurality of frames on a display. The sampling rate N is greater than 1. The computing device further includes means for processing one or more statistics associated with every N frames of the plurality of frames to determine one or more display settings. The computing device further includes means for outputting the corresponding N frames of the plurality of frames to the display using the one or more display settings.

[0009] The details of one or more examples of the present disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF DRAWINGS

[0010] So that the manner in which the above-recited features of the present disclosure can be understood in detail, a more particular description, briefly summarized above, can be had by reference to each of the aspects, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only certain typical aspects of this disclosure and are therefore not to be considered limiting of its scope, for the description can admit to other equally effective aspects.

[0011] Figure 1 is a block diagram illustrating an example computing device configured to process video data, in accordance with one or more aspects of the present disclosure.

[0012] Figure 2 An example block diagram of video data processing is illustrated in accordance with one or more aspects of the disclosure.

[0013] Figure 3 An example flowchart of video data processing is illustrated in accordance with one or more aspects of the disclosure.

[0014] Figure 4 Example operations of a computing device operable to process video data in accordance with certain aspects of the disclosure are illustrated.

[0015] The same reference numbers can indicate the same elements. DETAILED DESCRIPTION

[0016] Generally, aspects disclosed herein provide techniques for processing video data. In certain aspects, these techniques can be applicable to scenarios related to processing video data at high frame rates (as measured in frames per second, or FPS in Hz), or other situations that include a large amount of frame-by-frame analysis or computation, such as real-time rendering. For example, certain aspects herein provide systems and methods for adapting the frequency at which statistical data is processed for video data (e.g., by adjusting a sampling rate), and for determining display settings for displaying frames of video data on a display. For example, one or more statistical data can include one or more of a histogram or a noise level of one or more frames, for determining (e.g., adjusting) display settings for one or more subsequent frames. One or more display settings can include one or more of a color setting, a tone mapping setting, a display saturation setting, a brightness control setting, or a backlight setting. Such aspects can advantageously reduce or eliminate interrupting display effects such as user interface choppiness.

[0017] In certain aspects, video data includes a plurality of frames corresponding to images that can be displayed on a display at a frame rate (such as measured by frames per second (FPS)). Frame rate indicates the number of times per time period (e.g., within a second) that an entire frame of new data is fed to a display by a source. While older devices can use lower frame rates, such as from 24Hz to 30Hz, today’s devices can operate at higher frame rates, such as 60Hz or higher (such as 144Hz, 240Hz, or higher), as required by gaming or other visually intensive applications. Moreover, many high frame rate devices are mobile devices, such as smartphones, tablets, or laptops. The mobility of high frame rate devices constrains power consumption, limiting how powerful a CPU and / or GPU can be used or operated in. For example, a battery-powered smartphone can limit CPU / GPU performance when battery life is a concern or other power consumption is increased, such as increasing display brightness in sunlight. Such power limitations can prevent a CPU or GPU from timely performing required analysis or alterations to graphical content, resulting in dropped or delayed responses. In certain aspects, to address such issues, the present disclosure provides techniques for processing video data corresponding to image frames by adapting a sampling rate based on a frame rate, such as a frame rate in a high frame rate device.

[0018] In some cases, a processor (e.g., CPU and / or GPU) can analyze content of one or more frames of video data and use the analysis to adjust one or more display settings for displaying one or more subsequent frames of video data. For example, the processor can process one or more statistics based on content of one or more frames of video data. The one or more statistics can include, for example, one or more of a noise level associated with the one or more frames of video data or a histogram associated with the one or more frames of video data. In certain aspects, a histogram is an approximate representation of a distribution of numerical or categorical data, such as hue values of a frame. In certain aspects, a histogram can be global (e.g., across an entire frame) or local (e.g., across a portion of a frame). In certain aspects, a histogram can reflect several aspects of frame content, such as brightness and color. The processor can adjust display settings of a display to display one or more subsequent frames of video data based on the one or more statistics, such as by correcting for under- or over-exposure and removing or adding color shifts to achieve a desired distribution. For example, the processor can perform one or more processes on the display / one or more subsequent frames based on the one or more statistics, such as changing brightness to conserve power, performing global or local tone mapping, increasing sharpness or softness, increasing color saturation, incorporating reference data, smoothing skin tone areas, etc. These one or more processes can thus include setting or adjusting one or more display settings, such as one or more of a color setting, a tone mapping setting, a display saturation setting, a brightness control setting, or a backlight setting.

[0019] Unlike a strict per-frame processing model in which a processor collects and computes statistics and performs adjustments to display settings for each frame of video data, the present disclosure reduces overhead by determining a sampling rate based on a frame rate and processing one or more statistics and adjusting display settings according to the sampling rate. For example, in a high frame rate system with a 120Hz or 240Hz frame rate, the system can save power and still provide smooth performance by determining a sampling rate N greater than 1 such that one set of one or more display settings is used to display each of N frames for every N frames in the total number of frames. Thus, in certain aspects, one or more statistics can be computed and processed only once for N frames to adjust one or more display settings, rather than computing and processing a separate one or more statistics for each frame. In certain aspects, N = floor(frame rate in Hz / 60Hz). Thus, in some aspects, a subset of the total number of frames, rather than each frame, is used to collect and compute one or more statistics for adjusting one or more display settings. For example, in certain aspects, when N = 2, one or more statistics are processed every other frame and used to determine that one or more display settings are to be applied to two frames, rather than just one frame, and the overall analysis and processing is halved. Thus, the processor has sufficient capacity to handle the statistical analysis and processing without producing interrupting display effects such as a user interface stutter.

[0020] In some aspects, determining a sampling rate N is adapted to a workload and / or processing capacity. Thus, the value of N can change from time to time. For example, the system can begin to apply N when the system determines that the cost of processing each frame is too high or can not result in smooth output. In other cases, the system can increase the value of N when the system determines that using the current sampling rate N can not result in smooth output.

[0021] In some aspects, processing one or more statistics can include determining a number of subtasks (also referred to as task partitioning) to complete the processing. For each subtask, a respective duration is determined for completion. For each subtask, a corresponding frame of the N frames is determined for performing the subtask. For example, a subtask can have a similar duration to the display duration of a frame based on the frame rate, such as 6.94ms when the frame rate is 144Hz. When two subtasks are performed to complete a cycle of statistical processing and display setting adjustment, the sampling rate N = 2, and the overall amount of computation is reduced by half. Thus, with adaptive sampling rate and task partitioning, the processing load is significantly reduced. Thus, user interface (UI) performance can be improved.

[0022] Accordingly, certain aspects of certain devices and systems included herein enjoy several technical advantages over conventional approaches for processing video data. First, the disclosed techniques avoid, in certain aspects, overloading of processing units and ensure smooth display performance. Second, the disclosed techniques adaptively change, in certain aspects, the sampling rate to properly utilize CPU and / or GPU resources. Third, the disclosed techniques provide, in certain aspects, reduced computational burden through task partitioning. In some aspects, the disclosed techniques are applicable to histogram, noise level, or other statistics-related display features and video features. The disclosed techniques can be used in high frame rate mobile devices using statistics-related display features with good UI performance and power balance. Finally, certain aspects are applicable to any frame content analysis based display and video features. Determining the algorithm frame content subsampling rate at runtime is an important and useful solution in high frame rate devices.

[0023] Various aspects of systems, apparatuses, computer program products, and methods will be described below with reference to the accompanying drawings. However, the disclosure can take many different forms and should not be construed as limited to any specific structure or function presented throughout this disclosure. Rather, these aspects are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. Based on the teachings herein, one skilled in the art should appreciate that the scope of the disclosure is intended to cover any aspect of the systems, apparatuses, computer program products, and methods disclosed herein, whether implemented independently of, or combined with, other aspects of the disclosure. For example, an apparatus can be implemented or a method can be practiced using any number of the aspects set forth herein. In addition, the scope of the disclosure is intended to cover such an apparatus or method which is practiced using, in addition to or in place of the aspects set forth herein, other structures, functionalities, or structures and

[0024] Although various aspects are described herein, many variations and permutations of these aspects fall within the scope of the disclosure. While the disclosure is susceptible to various modifications and alternative forms, specific aspects of the disclosure are shown by way of example in the drawings and are described in detail herein. It should be understood that the intention is not to limit the disclosure to the particular aspects described but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure including aspects licensed to the patentee by the U.S. Government. The detailed description explains the principles of the disclosure, the practical application of which is demonstrated by the description of the drawings and the appended claims.

[0025] Several aspects are presented with reference to various apparatus and methods. These apparatus and methods are described in the following detailed description and illustrated in the accompanying drawings by various blocks, components, circuits, processes, algorithms, etc. (collectively referred to as “elements”). These elements can be implemented using electronic hardware, computer software, or any combination thereof. Whether such elements are implemented as hardware or software depends on the particular application and design constraints imposed on the overall system.

[0026] By way of example, an element, or any portion of an element, or any combination of elements can be implemented with a “processing system” that includes one or more processors (which can also be referred to as processing units). Examples of processors include microprocessors, microcontrollers, graphics processing units (GPUs), general purpose GPUs (GPGPUs), central processing units (CPUs), application processors, digital signal processors (DSPs), reduced instruction set computing (RISC) processors, systems on a chip (SoC), baseband processors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), state machines, gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described throughout this disclosure.

[0027] One or more processors in the processing system can execute software. Software can be broadly interpreted to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software components, applications, software applications, software packages, routines, subroutines, objects, executables, execution threads, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. The term application can refer to software. As described herein, one or more techniques can refer to an application, i.e., software, configured to perform one or more functions. In such examples, the application can be stored on a memory, e.g., on-chip memory of a processor, system memory, or any other memory.

[0028] Hardware described herein, such as a processor, can be configured to execute an application. For example, an application can be described as including code that, when executed by the hardware, causes the hardware to perform one or more techniques described herein. As an example, the hardware can access code from a memory and execute the code accessed from the memory to perform one or more techniques described herein. In some examples, components are identified in this disclosure. In such examples, a component can be hardware, software, or a combination thereof. A component can be a separate component or a subcomponent of a single component.

[0029] Accordingly, in one or more examples described herein, the functions described can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored on or encoded as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer storage media. Storage media can be any available media that can be accessed by a computer. By way of example, and not limitation, such computer-readable media can comprise a random-access memory (RAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), compact disk ROM (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, combinations of the aforementioned types of computer-readable media, or any other medium that can be used to store computer executable code in the form of instructions or data structures that can be accessed by a computer.

[0030] Generally, the examples disclosed herein provide techniques for processing video data. For example, these techniques allow high frame rate video content to be processed using a processor that is limited in computing and power resources. Other example benefits are described throughout this disclosure.

[0031] As used herein, examples of the term "content" can refer to "graphics content," "images," and vice versa. Whether these terms are used as adjectives, nouns, or other parts of speech. In some examples, as used herein, the term "graphics content" can refer to content produced by one or more processes of a graphics processing pipeline. In some examples, as used herein, the term "graphics content" can refer to content produced by a processing unit configured to perform graphics processing. In some examples, as used herein, the term "graphics content" can refer to content produced by a graphics processing unit.

[0032] In some examples, as used herein, the term "display content" can refer to content generated by a processing unit configured to perform display processing. In some examples, as used herein, the term "display content" can refer to content generated by a display processing unit. Graphics content can be processed into display content. For example, a graphics processing unit can output graphics content, such as a frame, to a buffer (which can be referred to as a frame buffer). A display processing unit can read graphics content, such as a frame or frames, from the buffer and perform one or more display processing techniques thereon to generate display content. For example, a display processing unit can be configured to perform compositing on one or more render layers to generate a frame.

[0033] As another example, the display processing unit can be configured to combine, blend, or otherwise join two or more layers together into a single frame. The display processing unit can be configured to perform scaling, e.g., zooming in or out, on a frame. In some examples, a frame can refer to a layer. In other examples, a frame can refer to two or more layers that have been blended together to form a frame, i.e., a frame includes two or more layers, and the frame including the two or more layers can subsequently be blended.

[0034] Figure 1 FIG. 1 is a block diagram illustrating an example system 100 including an example computing device 104 configured to process video data in accordance with one or more techniques of this disclosure. The computing device 104 can include one or more components or circuits to perform the various functions described herein. In some examples, one or more components of the computing device 104 can be components of a SOC. The computing device 104 can contain one or more components configured to perform one or more techniques of this disclosure. In the illustrated example, the computing device 104 can include a processor 120 and a system memory 124.

[0035] In some examples, the computing device 104 can include a number of additional or alternative components, e.g., a communication interface 126, a transceiver 132, a receiver 128, a transmitter 130, a display processor 127, and a display client 131. Reference to the display client 131 can refer to one or more displays. For example, the display client 131 can include a single display or multiple displays. The display client 131 can include a first display and a second display, or a foldable or detachable display. In other examples, the results of the graphics processing can not be displayed on the device, e.g., the first and second displays can not receive any frames for presentation thereon. Rather, the frames or graphics processing results can be communicated to another device. In some aspects, this can be referred to as split rendering.

[0036] The processor 120 can include an internal memory 121. The processor 120 can be configured to perform graphics processing, such as in a graphics processing pipeline 107. In some examples, the computing device 104 can include a display processor or display processing unit, such as the display processor 127, to perform one or more display processing techniques on one or more frames generated by the processor 120 prior to presentation at the display client 131. The display processor 127 can be configured to perform display processing. For example, the display processor 127 can be configured to perform one or more display processing techniques on one or more frames generated by the processor 120. The display processor 127 can output image data to the display client 131 according to an interface protocol, such as, for example, MIPI DSI (Mobile Industry Processor Interface, Display Serial Interface).

[0037] The graphics processing pipeline 107 is configured to process input graphics data into signals that can be used by the display processor 127 or the display client 131. The graphics processing pipeline 107 can depend on the software and hardware used and the target display characteristics. In some implementations, some graphics application programming interfaces (APIs) such as Direct3D and OpenGL can be used to control the graphics processing pipeline 107. The steps taken by the graphics processing pipeline 107 can be implemented in hardware and optimized therein. In some cases, the various steps of the graphics processing pipeline 107 can run concurrently and be stalled until the slowest step is complete. In some cases, the graphics processing pipeline 107 can include steps of application, geometry, and rasterization. The application step can involve user input and can be performed by software on a CPU or GPU. The geometry step can include operations that perform various operations and calculations. The rasterization step can output a raster image for each frame. Details of an example of the operation of the graphics processing pipeline 107 are illustrated in Figure 2

[0038] Turning now to Figure 2 , Figure 2 an example block diagram 200 for video data processing (which can be performed in the graphics processing pipeline 107) is shown. As Figure 3 indicated, the video data processing receives various inputs, including one or more of: a system display panel frame rate 205 (e.g., the frame rate at which the display is capable of running), a video data frame rate 210 (e.g., the frame rate at which the video data itself is generated, which can be the same or different than the frame rate 205), one or more statistics 215 for one or more frames (such as sequential frame histogram similarity), and a frame drop or UI stutter status 220. The frame drop or UI stutter status 220 refers to a report or monitoring result that indicates the number of unprocessed frames (whether dropped or frozen (or in a stutter state)). At 235, the processing block of the histogram task subsample determination policy uses information and status related to the system display panel frame rate 205 and the system display primary content frame rate 210 to determine a sampling rate (also referred to as a subsample division factor) N. The sampling rate N can vary to reduce the number of frames to be processed in a given time period. For example, for N = 2 in a 120 Hz FPS example, a statistical analysis or histogram can be collected and processed every N = 2 frames. The effective processing rate is thus reduced to 60 FPS. N can vary from 1 to any larger number. In certain aspects, N is greater than 1, such as for high frame rates. For example, when N = 1, the device performs statistical analysis and corresponding adjustments for every frame, so the effective processing rate for the same example is equal to the system display primary content FPS 210, which is 120 FPS. For this example, when N = 3, the device performs statistical analysis and adjustments once every three frames, for an effective processing rate of 40 FPS.

[0039] ​In some aspects, the sampling rate N can be adjusted at 230 according to the incoming workload and processing capacity. For example, a larger sampling rate value, such as N = 3 or higher, can be used when the processing capacity is occupied by higher priority tasks and the frame drop or UI stutter condition 220 shows an increasing trend. On the other hand, a lower sampling rate value can be used when the processing capacity is sufficient and the frame drop or UI stutter condition 220 is maintained, such as zero. The adjustment 230 of the sampling rate can depend on the specific hardware and software configuration.

[0040] At 225, the processor can divide the statistical processing and computation tasks into N sequential subtasks. The processor divides and / or reschedules the statistical collection and computation for adjustment into multiple subtasks. In certain aspects, each subtask can occupy the same processor execution time, 1 / N of the overall task processor time, 1 / N corresponding to the system display panel frame rate 205 and the system display primary content frame rate 210. For example, assume the system display primary content frame rate 210 has a frame rate of 120 FPS, with 1 second divided by 120 frames of task processor time per frame, equaling 8.333 milliseconds. When N = 2, each of the two subtasks occupies the same processor execution time of 4.167 milliseconds. In some cases, the content used to process / compute one or more statistics can be from one or more frames prior to N frames, for example, individually referred to as frame 0 for ease of illustration. The processing of the one or more statistics then occurs between frame 0 and frame N, such that the display settings at frame N are adjusted based on the processing of the one or more statistics. In certain aspects, content from one or more of the N frames can also be used to process the one or more statistics. For example, the one or more statistics can be updated for a subtask that executes after such one or more N frames based on content from one or more of the N frames. In some cases, the updated display settings based on a lookup table (LUT) and the processed one or more statistics can be applied to the Nth frame. In one illustrative example, when N = 2, the statistical processing and display setting adjustment occurs once for every 2 frames of content. Figure 1 An example is provided in FIG. 3, showing a subtask timeline 300 for N = 4 and a subtask timeline 305 for N = 1.

[0041] For example, in the N=4 sub-task timeline 300, statistics are collected for frame 0. Further, based on N=4, the processing of the statistics is divided into 4(N) sub-tasks 0-3. Sub-task 0 is performed between frame 0 and frame 1, sub-task 1 is performed between frame 1 and frame 2, sub-task 2 is performed between frame 2 and frame 3, and sub-task 3 is performed between frame 3 and frame 4. Thus, display settings based on the processing of the statistics between sub-tasks 0-3 are adjusted for display at frame 4(N). In certain aspects, additional statistics can be collected and used for processing in only certain sub-tasks. For example, statistics for frame 1 can be collected and can also be used to perform one or more of sub-tasks 1-3 as they occur after frame 1. Further, in certain aspects, some display settings can be adjusted for one or more frames prior to frame N. For example, if sub-task 0 indicates to update a subset of display settings, these settings can be adjusted to display frame 1 and possibly the remaining frames in N frames.

[0042] As shown in the N=l timeline 305, statistics are collected for frame 0. Further, only one task, task 0, is performed between frame 0 and frame 1, and the display settings for frame 1 are adjusted based on task 0 using the statistics from frame 0. Further, statistics are again collected for frame 1, and one task, task 1, is performed between frame 1 and frame 2, and the display settings for frame 2 are adjusted based on task 1 using the statistics from frame 1.

[0043] At 240, a post-processing algorithm of the display is executed based on the scheduled divided sub-tasks at 225. The post-processing algorithm processes one or more statistics associated with the frame and determines one or more display settings, such as color settings, tone mapping settings, display saturation settings, brightness control settings, or backlight settings. For example, color settings can include white balance adjustments, color filtering, or other settings. Tone mapping settings can map one set of colors to another set of colors. Display saturation settings can change image color saturation to enhance or suppress color vividness. Brightness control settings can adjust image brightness, such as how bright high light areas should be. Backlight settings can adjust backlight brightness based on ambient lighting conditions.

[0044] At 245, the processor applies the adjustments in the Nth frame, completing one task cycle. At 250, the processed frame is output to an adjustment block, such as for backlight adjustment, global tone adjustment, and local tone adjustment. The adjustment block can forward the processed results to the display processor 127 and the display client 131.

[0045] The display client 131 can be configured to display or otherwise present frames processed by the display processor 127. In some examples, the display client 131 can include one or more of a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, a projection display device, an augmented reality display device, a virtual reality display device, a head-mounted display, or any other type of display device.

[0046] In Figure 1 In the illustrated example, the display client 131 includes a display controller 132, a buffer 134, and a display 136. The example display 136 includes a plurality of pixel elements for displaying image data. The display controller 132 can receive image data from the display processor 127 and store the received image data in the buffer 134. In some examples, the display controller 132 can output image data stored in the buffer 134 to the display 136. Thus, the buffer 134 can represent local memory of the display client 131. In some examples, the display controller 132 can output image data received from the display processor 127 to the display 136.

[0047] Further, as described above, the display client 131 can be configured according to the MIPI DSI standard. The MIPI DSI standard supports a video mode and a command mode. In examples where the display client 131 operates in the video mode, the display processor 127 can continuously refresh graphics content of the display client 131. For example, the entire graphics content can be refreshed at each refresh cycle (e.g., line by line).

[0048] In examples where the display client 131 operates in the command mode, the display processor 127 can write graphics content of a frame to the buffer 134. In some such examples, the display processor 127 can not continuously refresh graphics content of the display client 131. Rather, the display processor 127 can use a vertical sync (Vsync) pulse to coordinate rendering and consumption of graphics content at the buffer 134. For example, when a Vsync pulse is generated, the display processor 127 can output new graphics content to the buffer 134. Thus, production of the Vsync pulse can indicate when the current graphics content at the buffer 134 has been rendered.

[0049] Memory external to the processor 120, such as system memory 124, can be accessed by the processor 120. For example, the processor 120 can be configured to read from and / or write to the external memory, such as system memory 124. The processor 120 can be communicatively coupled to the system memory 124 by a bus. In some examples, the processor 120 and the system memory 124 can be communicatively coupled to each other by a bus or a different connection.

[0050] It should be appreciated that, in some examples, the computing device 104 can include a content encoder / decoder configured to receive graphical and / or display content from any source, such as the system memory 124 and / or the communication interface 126. The system memory 124 can be configured to store received encoded or decoded content. In some examples, the content encoder / decoder can be configured to receive encoded or decoded content in the form of encoded pixel data, for example, from the system memory 124 and / or the communication interface 126. In some examples, the content encoder / decoder can be configured to encode or decode any content.

[0051] The internal memory 121 or the system memory 124 can include one or more volatile or non-volatile memories or storage devices. In some examples, the internal memory 121 or the system memory 124 can include RAM, SRAM, DRAM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, a magnetic data medium, or an optical storage medium, or any other type of memory.

[0052] According to some examples, the internal memory 121 or the system memory 124 can be a non-transitory storage medium. The term “non-transitory” can indicate that the storage medium is not embodied in a carrier wave or a propagating signal. However, the term “non-transitory” should not be interpreted to mean that the internal memory 121 or the system memory 124 is not movable or that its contents are static. As one example, the system memory 124 can be removed from the computing device 104 and moved to another device. As another example, the system memory 124 can not be removable from the computing device 104.

[0053] The processor 120 can be a central processing unit (CPU), a graphics processing unit (GPU), a general purpose GPU (GPGPU), or any other processing unit that can be configured to perform graphics processing. In some examples, the processor 120 can be integrated into a motherboard of the computing device 104. In some examples, the processor 120 can reside on a graphics card that is installed in a port of the motherboard of the computing device 104, or can be otherwise incorporated within a peripheral device that is configured to interoperate with the computing device 104. The processor 120 can include one or more processors, such as one or more microprocessors, GPUs, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), arithmetic logic units (ALUs), digital signal processors (DSPs), discrete logic, software, hardware, firmware, other equivalent integrated or discrete logic circuitry, or any combinations thereof. If the techniques are implemented partially in software, a processor 120 can store instructions for the software in suitable, non- transitory computer-readable storage media (e.g., the internal memory 121) and can execute the instructions in hardware using one or more processors to perform the techniques of the present disclosure. Any of the above, including hardware, software, a combination of hardware and software, etc., can be considered a processor or processors. In some aspects, the processor 120 can include or be integrated with a display processor 127.

[0054] In some aspects, the content generation system 100 can include a communication interface 126. The communication interface 126 can include a receiver 128 and a transmitter 130. The receiver 128 can be configured to perform any of the receiving functions described herein with respect to the computing device 104. Additionally, the receiver 128 can be configured to receive information from another device, such as eye or head position information, rendering commands, or position information. The transmitter 130 can be configured to perform any of the transmitting functions described herein with respect to the computing device 104. For example, the transmitter 130 can be configured to transmit information to another device, which can include a request for content. The receiver 128 and the transmitter 130 can be combined into a transceiver 132. In such examples, the transceiver 132 can be configured to perform any of the receiving functions and / or transmitting functions described herein with respect to the computing device 104.

[0055] In some examples, the graphical content from the processor 120 for display via the display client 131 is not static and can be changing. Accordingly, the display processor 127 can periodically refresh the graphical content displayed via the display client 131. For example, the display processor 127 can periodically retrieve the graphical content from the system memory 124, where the graphical content can have been updated by execution of an application that outputs the graphical content to the system memory 124 (and / or the processor 120).

[0056] It should be understood that while shown as separate components in Figure 1 some examples, the display client 131 (sometimes referred to as a “display panel”) can include the display processor 127.

[0057] As shown in Figure 4 some aspects, the display processor 127 (also referred to as a display processing unit (DPU)) can be configured to operate the functions of the display client 131. For example, in some aspects, the display processor 127 is configured to output, to the display controller 132, a plurality of codewords such as corresponding to a frame. Each codeword can be represented by a binary number in a digital domain. Each codeword can correspond to a pixel (e.g., red, green, blue, white, etc.) of the display 136.

[0058] The display controller 132 can be configured to convert the codewords received from the display processor 127 into analog signals for driving the pixels of the display 136. In some aspects, for each codeword corresponding to a pixel, the display controller 132 is configured to convert the codeword into an analog signal that drives the pixel to a particular luminance level. Thus, in some aspects, the codewords and / or the analog signal(s) correspond to a luminance level of the pixel.

[0059] The following description uses the processor 120 as an example, and similar techniques, methods, or processes performed by the processor 120 can also be performed in or in conjunction with the display processor 127.

[0060] As described herein, a device such as the computing device 104 can refer to any device, apparatus, or system configured to perform one or more techniques described herein. For example, a device can be a server, a base station, a user device, a client device, a station, an access point, a computer (e.g., a personal computer, a desktop computer, a laptop computer, a tablet computer, a computer workstation, or a mainframe computer), an end product, an apparatus, a telephone, a smartphone, a server, a video game platform or console, a handheld device (e.g., a portable video game device or a personal digital assistant (PDA)), a wearable computing device (e.g., a smartwatch, an augmented reality device, or a virtual reality device), a non-wearable device, a display or display device, a television, a television set-top box, an intermediate network device, a digital media player, a video streaming device, a content streaming device, an in-vehicle computer, any mobile device, any device configured to generate graphical content, or any device configured to perform one or more techniques described herein. Processes herein can be described as being performed by a particular component (e.g., a GPU), but in further embodiments, can be performed using other components (e.g., a CPU) that are consistent with the disclosed embodiments.

[0061] Figure 1FIG. 13 illustrates example operations 1300 that can be performed by a computing device to process video data. In some examples, operations 1300 can be performed by a processor 120 or display processor 127 of the computing device 100 of FIG. 1. ​ For example, the processor 120 or display processor 127 can process video data including a plurality of frames for display on the display 136.

[0062] Operations 1300 begin, at 1305, by determining a sampling rate (N) based on a frame rate of video data for display including a plurality of frames for display on a display. In some aspects, the sampling rate is greater than one (e.g., at least two). In some aspects, the sampling rate can vary depending on the complexity of the video data and the capabilities of the processor. For example, operations 1300 can be triggered when the processor does not have sufficient resources to process each frame of the video data (e.g., when N = 1) and has suffered from frame drops. In other cases, a higher sampling rate value can be determined (e.g., N is increased from 2 to 3 as needed) if the processor does not have sufficient resources to process the video data at the current sampling rate. On the other hand, the value of the sampling rate can be decreased if too many processing resources are idle. The sampling rate N can be determined and dynamically updated using specific hardware and software configurations.

[0063] At 1310, for every N frames of the plurality of frames, one or more statistics associated with the frames are processed to determine one or more display settings. In some aspects, the one or more statistics can include one or more of a histogram or a noise level. The one or more display settings can include one or more of a color setting, a tone mapping setting, a display saturation setting, a brightness control setting, or a backlight setting. The one or more statistics can also be associated with at least one additional frame. The frame can be a frame occurring before the N frames. The at least one additional frame can include one or more frames occurring before the N frames. The at least one additional frame can include one or more of the N frames. For example, the one or more statistics can correspond (e.g., directly) to statistics of a frame occurring before the N frames, to a combination of statistics of a plurality of frames occurring before the N frames, or to statistics of one or more frames occurring before the N frames and statistics of one or more of the N frames. For example, statistics of one of the N frames can be used to update statistics for one or more sub-tasks occurring after the one of the N frames.

[0064] In some examples, processing the one or more statistics can include determining a plurality of sub-tasks for performing the processing. For example, in certain aspects, for a sample rate N, each time period of a content frame includes N sub-tasks. For each of the plurality of sub-tasks, a corresponding duration is determined to complete the corresponding sub-task. For example, in certain aspects, each sub-task can take up 1 / (N*FPS) per second. For each of the plurality of sub-tasks, a corresponding frame of the N frames during which to perform the corresponding sub-task is determined based on the corresponding duration.

[0065] At 415, the corresponding N frames of the plurality of frames are output to a display using the one or more display settings. In some cases, one or more transformations are applied to the corresponding N frames using the one or more display settings.

[0066] In some aspects, the operations 400 further include determining one or more statistics based on image data of the frame. For example, the image data can correspond to one or more of unrendered layer data of the frame, uncomposited rendered layer data of the frame, or composited rendered layer data of the frame.

[0067] In one configuration, a method or apparatus for display processing is provided. The apparatus can be a processing unit, a display processor, a display processing unit (DPU), a graphics processing unit (GPU), a video processor, or some other processor that can perform display processing. In some examples, the apparatus can be the processor 120 within the computing device 104, or can be some other hardware within the computing device 104, or another device.

[0068] According to the present disclosure, the term “or” can be construed as “and / or” unless the context otherwise dictates. Additionally, while phrases such as “one or more” or “at least one” can have been used in relation to some features disclosed herein but not others, features for which such language has not been used can be construed to imply such a meaning where the context so dictates.

[0069] In one or more examples, the functions described herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media can include computer data storage media or communication media including any medium that facilitates transferring computer program from one place to another. In this manner, computer-readable media can include a non-transitory computer-readable storage medium or a communication medium.

[0070] Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation of the techniques described in this disclosure. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices. Disk and disc, as used herein, includes compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and blu-ray discs where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media. The computer program product can include a computer-readable medium.

[0071] The code can be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), arithmetic logic units (ALUs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor," as used herein can refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, the techniques could be fully implemented in one or more circuits or logic elements.

[0072] The techniques of this disclosure can be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described herein as being stored in or on one or more types of computer-readable media, such as computer-readable storage media. Generally, the computer-readable media can include tangible storage media that is non-transitory. By way of example, and not limitation, such computer-readable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other storage medium that is used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any

[0073] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method of processing video data, the video data comprising a plurality of frames for display on a display, the method comprising: determining a sampling rate N, where N is greater than 1, based on a frame rate for displaying the video data on the display and based on a processing capability of a computing device; and for every N frames of the plurality of frames: processing one or more statistics computed based on the N frames to determine one or more display settings; and outputting the corresponding N frames of the plurality of frames to the display using the one or more display settings, where the frame rate indicates a number of times a frame is sent to the display per time period and the sampling rate N indicates that statistics are processed once for every N frames. The one or more statistics comprise one or more of a histogram or a noise level.

2. The method of claim 1, wherein, The one or more display settings comprise one or more of a color setting, a tone mapping setting, a display saturation setting, a brightness control setting, or a backlight setting.

3. The method of claim 1, wherein, The one or more statistics are further associated with at least one additional frame.

4. The method of claim 1, wherein, Processing the one or more statistics comprises:

5. The method of claim 1, wherein, determining a plurality of subtasks for performing the processing; for each of the plurality of subtasks, determining a corresponding duration for completing the corresponding subtask; and for each of the plurality of subtasks, determining, based on the corresponding duration, a corresponding frame of the N frames during which the corresponding subtask is to be performed. Outputting the corresponding N frames of the plurality of frames to the display using the one or more display settings comprises applying one or more transformations to the corresponding N frames using the one or more display settings.

6. The method of claim 1, wherein, Determining the one or more statistics based on image data of the frame, the image data corresponding to one or more of unrendered layer data of the frame, uncomposited rendered layer data of the frame, or composited rendered layer data of the frame.

7. The method of claim 1, further comprising:

8. A computing device comprising: a memory; and a processor coupled to the memory, wherein the memory and the processor are configured to perform the method of any of claims 1-7.

9. A computing device comprising: various means for performing the method of any of claims 1-7.

10. A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform the method of any of claims 1-7.

11. A computer program product comprising computer instructions, which when executed by a processor, cause the processor to perform the method of any of claims 1-7. ​ ​

Images