Power management on an electronic device

Abstract

A method comprising: obtaining information indicative of quality of an image 626 generated by a graphics-related operation (i.e. application) 608 to display the image on a display; obtaining informati

Description

Field

[001] The present application generally relates to power management of an electronic device. In particular, the present application provides a method and apparatus for concurrently controlling hardware parameters and software parameters for the electronic device while running an application which includes displaying an image. Background

[002] Hardware power management is a known technique for enabling and disabling parts of the hardware, changing the hardware operating properties (e.g. frequency / voltage) and / or swapping between different hardware modules in order to increase or decrease the performance as needed. Power management is usually implemented in software due to the need to tweak and change according to the different scenarios of hardware usage.

[003] Dynamic Voltage and Frequency Scaling (DVFS) is one of the key components of power management. DVFS can increase or decrease the voltage and frequency provided to a component depending upon usage. Typically, the voltage and frequency are decreased in order to reduce power usage and such control is particularly useful in battery-powered and resource constrained devices, such as laptops, tablets or mobile telephones, to conserve power. DVFS typically increases the voltage and frequency provided to the component in order to increase performance, for example when processing load has increased. An example of a power management system which uses DVFS is described in EP3125072 to the present applicant.

[004] There are limitations to existing DVFS methods. For example, they are typically unable to identify and optimise trade-offs. An increase in frequency will generally create an increase in power consumption and an increase of perceived performance generated by the hardware unit. While performance and frequency / voltage increases are linear, an increase in power is a squared function. This leads to a problem that conventional DVFS techniques normally do not try to solve, namely finding a balance point between perceived performance and performance per Watt. That is, when rendering images on a resource constrained device, current methods often only allow for power management by adjusting hardware parameters. Under some circumstances the required performance can be high enough to allow some degradation, in order to get closer to the point where the maximum performance per Watt power is achieved, without compromizing the user experience. However, in other circumstances, a user experience when rendering images on a resource constrained device, for example, during gaming may be suboptimal.

[005] The applicant has therefore identified the need for an improved method and apparatus for power management, particularly in resource constrained electronic devices. Summary

[006] In a first approach of the present techniques, there is provided a computer-implemented method for power management on an electronic device, the method comprising: obtaining operation information related to a graphics-related operation of an application which is running on the electronic device; obtaining load information relating to a load generated by the graphics-related operation; determining a scale factor to be applied during the graphics-related operation using the obtained operation information and load information; intercepting a command from the graphics-related operation which is sent to a graphics processing unit, GPU in the electronic device, wherein the command comprises writing image data to a framebuffer; modifying the command using the scale factor by scaling a region within the framebuffer which is to receive the image data by the scale factor; and sending the modified command to the GPU for execution whereby power is reduced when the modified command is executed by the GPU.

[007] The method described above may be wholly or partly performed on an apparatus, i.e. an electronic device. Thus according to another aspect, there is provided an electronic device, comprising a display, a graphics processing unit, GPU, for rendering at least one image on the display and an operating system control module for controlling a load on the GPU. The operating system control module comprises a processor and an interceptor. The processor is for obtaining operation information related to a graphics-related operation of an application which is running on the electronic device to generate at least one image to be displayed on the display, wherein operation information is indicative of quality of the at least one image output by the graphics-related operation. The processor is also for obtaining load information relating to a load on hardware in the electronic device generated by the graphics-related operation. The processor is also for determining a scale factor to be applied during the graphics-related operation using the obtained operation information and load information. The interceptor is for intercepting a command from the graphics-related operation which is sent to the GPU, wherein the command comprises writing image data of the at least one image to a framebuffer; modifying the command by using the scale factor to downscale a region within the framebuffer into which the image data is to be written; and sending the modified command to the GPU for execution. As will be appreciated, power is reduced when the modified command is executed by the GPU.

[008] The following features apply to both aspects.

[009] Hardware denotes a physical component included in an electronic device. For example, the hardware may include a central processing unit (CPU) or a graphic processing unit (GPU), and may be any hardware on which power management may be performed.

[010] The method is dynamic, and the scale factor is adjusted dynamically based on the operation and load information. It will be appreciated that the steps above are repeated whilst the application is running on the electronic device. The load information may comprise some or all of an indication of the maximum load on the GPU and / or a CPU, and temperature of the GPU and / or the CPU. The operation information may comprise some or all of a number of frames which are being output by the graphics-related operation, frames per second, stability and quality. More generally, operation information denotes information that is expressed to the user about an operation directly or indirectly experienced by the user. In other words, the operation information denotes information about an operation experienced by the user or provided to the user at an end of a processing pipeline of an electronic device. For example, the user may not know an accurate value of a number of frames per unit time of a screen output to the user, but the user may directly experience the number of frames based on smooth displaying of the screen or on a degree of breaks in the displaying of the screen or may indirectly experience the number of frames when the number of frames calculated by a GPU is expressed as a numerical value.

[011] The obtaining and determining steps may be carried outby a quality of service controller or processor. Intercepting, modifying and sending commands may be implemented by an interceptor or interceptor module in the electronic device. By intercepting and then modifying the command before it is sent to the GPU, there is no need to modify the application itself. In other words, the application which is running on the electronic device is the original, unmodified application. The region within the framebuffer which is scaled may be known by different terms depending on the specific graphics-related operation. For example, the region may be termed a viewport or a viewing region because it is the region within the framebuffer in which the image data can be viewed. By downscaling the region, fewer pixels will be rendered to the framebuffer when the modified command is executed by the GPU and thus the load on the GPU will be reduced.

[012] When determining the scale factor, at least one hardware parameter may also be determined, and the electronic device may be configured based on the determined hardware parameter. The determined parameter is likely to be a modification to a current value of the hardware parameter and may thus be termed a modified hardware parameter. In other words, both software (i.e. application / operation) parameters and hardware parameters are modified together to achieve a balance between the load on the device (i.e. on the hardware) and the quality of the graphics-related operation. Configuring the electronic device based on the modified hardware parameter may comprise applying dynamic voltage and frequency scaling, DVFS.

[013] Determining a scale factor (and a modified hardware parameter) to be applied may comprise predicting performance of the graphics-related operation and the hardware using the operation information and the load information, including how the performance differs for different values of the scale factor and the hardware parameter(s). Predicting the performance of the graphics-related operation may comprise predicting frames per second, stability and / or quality of an image(s) generated by the operation. Predicting performance of the hardware may comprise predicting power and / or thermal values of the hardware. The prediction may be done using any suitable technique, including a rules based method, a machine learning, ML model, and an estimation technique. The prediction may be for a time t. The scale factor (and hardware parameters) which optimise the predicted performance may then be output, e.g. they may be selected from the multiple values which were used to generate the predictions or from the ML model.

[014] Predicting performance of the graphics-related operation may comprise predicting at least one of frames per second, stability and quality. Each of these is dependent on the scale factor and the hardware parameters and the performance may be predicted for multiple values of the scale factor and hardware parameters. Predicting performance of the hardware may comprises predicting power and / or thermal performance and again this may be predicted for multiple values of the scale factor and hardware parameter. The method may further comprise determining an overall quality of service metric using a combination of some or all of the predicted frames per second, stability, quality, power and thermal performance and selecting optimal values from the multiple values used in the predicting step. The selected values are those which maximise the quality of service metric.

[015] It will be appreciated that fixed parameters may also be used when predicting performance of the graphics-related operation and the hardware. The fixed parameters may include some or all of a target size of an image to be displayed as a result of the graphics related operation, CPU and / or GPU baseline performance data, maximum threshold for CPU and / or GPU, and thermal limits for the device power. These fixed parameters may be obtained in a pre-processing or set-up stage before obtaining load and / or operation information.

[016] The graphics-related operation generates an image to be displayed on a display of the electronic device. The image may be generated using a single command and in this example, the image data may represent the generated image. More generally, the graphics-related operation uses multiple image components which are combined when generating (i.e. rendering) an image. Such components may include for example shadow generation, reflection textures generation, skybox environment generation, deferred shadingand other suitable components. Thus, the graphics-related operation comprises a plurality of commands each of which when executed by the GPU writes image data to a framebuffer. In this example, the image data which is written to each framebuffer comprises image component data relating to each component used in generating the image. Alternatively, when the image is generated in a single command or in the final stage which combines all components, the image data which is written to each framebuffer comprises image data relating to the image which is generated. When writing image component data to a framebuffer, the framebuffer may be termed an intermediate framebuffer and when writing image data to a framebuffer, the framebuffer may be termed a final framebuffer.

[017] The graphics-related operation may thus comprise a plurality of commands in which image data is written into framebuffers. Some or all of the plurality of commands may be intercepted by the interceptor and the intercepted commands may be intercepted at different times. As explained above, the method is intended to dynamically adjust the scale factor and thus an updated scale factor may be determined between intercepting different commands, e.g. an earlier command and a later command. Determining a scale factor to be applied during the graphics-related operation may comprises determining a plurality of scale factors, including a first scale factor to be applied at a first time and a second scale factor to be applied at a second, later time. The time of intercepting thus determines the scale factor which is applied during the modifying. The method may comprise intercepting the earlier command and applying a first scale factor to the earlier command and intercepting the later command and modifying the later command by an updated or second scale factor. In other words, each intercepted command may be modified by a different scale factor.

[018] As an alternative, commands may be modified in batches, for example for a whole pipeline or part of a pipeline and the batch of modified commands is sent to the GPU before executing the pipeline. Each batch of commands may be modified using the same scaling factor.

[019] When components have been scaled with a different scale factor, the scale factor attached to each component determines which pixels in each component are to be combined to form a new component or the generated image. If the scale factor is not taken into account, combining components may result in suboptimal generated images, containing, for example, empty sections, stretched areas, or missing components (shadows, textures, etc).. The method may further comprise linking each component with an attribute indicating the scale factor which has been used to modify a command associated with the component. For example, the command which is modified may be the command which writes image component data relating to the component into a framebuffer (i.e. an intermediate framebuffer).

[020] The graphics-related operation generates an image to be displayed on a display of the electronic device. The method may thus comprise displaying the at least one generated image on a display of the electronic device. This may comprise executing, by the GPU, the modified command to write image data for the generated image into the downscaled region within the framebuffer and outputting the framebuffer. Hardware on the display may then read the image data from the downscaled region within the framebuffer and display this image data on the display. The downscaled region is smaller than the target size and thus the display hardware (e.g. processor) may repeat and / or stretch pixels within the downscaled region to fill the target size. Although this display may be considered a rescaling of the downscaled image data, one advantage of the proposed technique is that there is no storing of any upscaled image.. The displaying is thus done automatically and may be done using any suitable technique.

[021] For the displaying to be automatic in the final stage, the image data should be written into the same part of the framebuffer and thus the rescaling of the region in which data is written is not arbitrarily applied. Similarly, when there are multiple commands using image component data, this should also be written into the same part of the framebuffer (e.g. the intermediate framebuffer). The method may thus comprise modifying the or each command so that regions of the framebuffer outside the downscaled region are left unchanged.

[022] In a related approach of the present techniques, there is provided a computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out any of the methods described herein.

[023] As will be appreciated by one skilled in the art, the present techniques may be embodied as a system, method or computer program product. Accordingly, present techniques may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[024] Furthermore, the present techniques may take the form of a computer program product embodied in a computer readable medium having computer readable program code embodied thereon. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable medium may be, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[025] Computer program code for carrying out operations of the present techniques may be written in any combination of one or more programming languages, including object oriented programming languages and conventional procedural programming languages. Code components may be embodied as procedures, methods or the like, and may comprise subcomponents which may take the form of instructions or sequences of instructions at any of the levels of abstraction, from the direct machine instructions of a native instruction set to high-level compiled or interpreted language constructs.

[026] Embodiments of the present techniques also provide a non-transitory data carrier carrying code which, when implemented on a processor, causes the processor to carry out any of the methods described herein.

[027] The techniques further provide processor control code to implement the abovedescribed methods, for example on a general purpose computer system or on a digital signal processor (DSP). The techniques also provide a carrier carrying processor control code to, when running, implement any of the above methods, in particular on a non-transitory data carrier. The code may be provided on a carrier such as a disk, a microprocessor, CD- or DVD- ROM, programmed memory such as non-volatile memory (e.g. Flash) or read-only memory (firmware), or on a data carrier such as an optical or electrical signal carrier. Code (and / or data) to implement embodiments of the techniques described herein may comprise source, object or executable code in a conventional programming language (interpreted or compiled) such as Python, C, or assembly code, code for setting up or controlling an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), or code for a hardware description language such as Verilog (RTM) or VHDL (Very high speed integrated circuit Hardware Description Language). As the skilled person will appreciate, such code and / or data may be distributed between a plurality of coupled components in communication with one another. The techniques may comprise a controller which includes a microprocessor, working memory and program memory coupled to one or more of the components of the system.

[028] It will also be clear to one of skill in the art that all or part of a logical method according to embodiments of the present techniques may suitably be embodied in a logic apparatus comprising logic elements to perform the steps of the above-described methods, and that such logic elements may comprise components such as logic gates in, for example a programmable logic array or application-specific integrated circuit. Such a logic arrangement may further be embodied in enabling elements for temporarily or permanently establishing logic structures in such an array or circuit using, for example, a virtual hardware descriptor language, which may be stored and transmitted using fixed or transmittable carrier media.

[029] In an embodiment, the present techniques may be realised in the form of a data carrier having functional data thereon, said functional data comprising functional computer data structures to, when loaded into a computer system or network and operated upon thereby, enable said computer system to perform all the steps of the above-described method.

[030] Predicting and selecting optimal values for the scale parameter and / or hardware parameter may be done using a machine learning or artificial intelligence model. The model may be processed by an artificial intelligence-dedicated processor designed in a hardware structure specified for artificial intelligence model processing. The artificial intelligence model may be obtained by training. Here, "obtained by training" means that a predefined operation rule or artificial intelligence model configured to perform a desired feature (or purpose) is obtained by training a basic artificial intelligence model with multiple pieces of training data by a training algorithm. The artificial intelligence model may include a plurality of neural network layers. Each of the plurality of neural network layers includes a plurality of weight values and performs neural network computation by computation between a result of computation by a previous layer and the plurality of weight values.

[031] A function associated with the ML / AI model may be performed through the non-volatile memory, the volatile memory, and the processor. The processor may include one or a plurality of processors. At this time, one or a plurality of processors may be a general purpose processor, such as a central processing unit (CPU), an application processor (AP), or the like, a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and / or an Al-dedicated processor such as a neural processing unit (NPU). The one or a plurality of processors control the processing of the input data in accordance with a predefined operating rule or artificial intelligence (Al) model stored in the non-volatile memory and the volatile memory. The predefined operating rule or artificial intelligence model is provided through training or learning. Here, being provided through learning means that, by applying a learning algorithm to a plurality of learning data, a predefined operating rule or Al model of a desired characteristic is made. The learning may be performed in a device itself in which Al according to an embodiment is performed, and / o may be implemented through a separate server / system.

[032] The Al model may consist of a plurality of neural network layers. Each layer has a plurality of weight values, and performs a layer operation through calculation of a previous layer and an operation of a plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural network (CNN), deep neural network (DNN), recurrent neural network (RNN), restricted Boltzmann Machine (RBM), deep belief network (DBN), bidirectional recurrent deep neural network (BRDNN), generative adversarial networks (GAN), and deep Q-networks.

[033] The learning algorithm is a method for training a predetermined target device (for example, a robot) using a plurality of learning data to cause, allow, or control the target device to make a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. Brief description of the drawings

[034] Implementations of the present techniques will now be described, by way of example only, with reference to the accompanying drawings, in which:

[035] Figure 1 shows a schematic representation of an electronic device used in the present techniques;

[036] Figure 2 is a flowchart showing how components of the electronic device of Figure 1 control software and hardware parameters for power management within the electronic device;

[037] Figures 3a and 3b show how a quality of service may be affected by hardware constraints;

[038] Figures 3c and 3d show how a quality of service may be improved by using the present techniques;

[039] Figure 4 is a flowchart showing the steps for using a software parameter calculated using the method of Figure 2 to manage power within the electronic device;

[040] Figure 5 schematically illustrates some of the steps of Figure 4;

[041] Figure 6 shows an example of a more complex image rendering process which may be implemented on the electronic device of Figure 1;

[042] Figure 7 is a flowchart showing the steps for using a software parameter calculated using the method of Figure 2 in the process shown in Figure 6; and

[043] Figure 8 shows a general example of a more complex image rendering process. Detailed description of the drawings

[044] Broadly speaking, embodiments of the present techniques provide a method and apparatus for power management. In particular, the present techniques relate to a method and apparatus for optimising a scale factor of an image to be computed by a hardware component of the apparatus and preferably also simultaneously optimising a hardware configuration. The hardware configuration and scale factor are determined by taking into account operation information on the application (e.g. perceived performance) and load information which may be measured using hardware parameters and thermal parameters.

[045] The perceived performance or number of frames (FPS) generated by a GPU or other processor, particularly when used to generate display output, can advantageously be used to improve the determinations made by a power management / DVFS algorithm. Conventional power management / DVFS for GPUs generally base their determination merely on the processing load. Perceived performance factors, such as FPS, are not typically part of conventional DVFS algorithms, i.e. they are not used in / by conventional DVFS algorithms (as an input / value / variable) to determine the desired hardware component configuration. However, power usage is not equal to perceived performance for the purposes of effective power management. In conventional DVFS, power usage will increase as long as it provides better performance to the demanding application, leading to very high power usage in some cases, which is neither beneficial to the user nor optimal. Thus, the present techniques relate to power management and image rendering techniques and can take into account power, performance, temperature values and quality. In particular, the present techniques can balance all of these parameters to find a better operation point.

[046] Figure 1 shows a schematic representation of an electronic device 100 including power management used in the present techniques for the example of computer gaming. The electronic device 100 comprises a central processing unit CPU 102, a graphics processing unit GPU 104 and an application engine 106 (which may be a game engine when the application is a computer game). In operation, the electronic device (e.g., a processor of the electronic device) may execute at least one application according to an application execution command. The application execution command may be generated by the application engine 106. Here, hardware for processing a related process may be required to execute the at least one application. It will be appreciated that these are the standard minimum components required for image processing to render images for the application, e.g. the computer game. There may be other standard components, for example a display, a user input interface and so on which are omitted for clarity. The GPU 104 may generate / render / compute (the terms are interchangeable) images to be displayed on the display.

[047] The electronic device 100 may be any suitable apparatus, including for example a smartphone, tablet, laptop, computer or computing device. It will be understood that this is a non-limiting and non-exhaustive list of apparatuses. The at least one central processing unit 102 may comprise one or more of: a microprocessor, a microcontroller, and an integrated circuit. The electronic device may also comprise memory (not shown) which may comprise volatile memory, such as random access memory (RAM), for use as temporary memory, and / or non-volatile memory such as Flash, read only memory (ROM), or electrically erasable programmable ROM (EEPROM), for storing data, programs, or instructions, for example. The memory 124 may also, for example, be coupled to the CPU and to other components within the electronic device.

[048] The present techniques propose a method for controlling power and thermal properties of the CPU and GPU whilst balancing any trade off in performance, resolution and stability of a running application, e.g. a game. Figure 1 shows that such a control function may be separated into three control modules: a hardware control module 110, an operating system OS control module 112 and an application control module 114. Each of the control modules may be termed a controller and the terms are used interchangeably.

[049] The hardware control module 110 which may also be termed a kernel thermal control module comprises a DVFS module 120 and an optional thermal off-switch 124 which stores a thermal limit for the device. If the thermal limit is exceeded, the device cannot function properly anymore and the off-switch can be activated to prevent this.

[050] As indicated by the arrows, the DVFS manages the values for the voltage and frequency in both the GPU and CPU. By controlling the voltage and frequency, the hardware control module 110 ensures that a temperature of the GPU and / or CPU does not exceed a thermal limit temperature. The temperature may be measured by an optional temperature sensor 122 for the GPU and / or CPU. That is, the DVFS may take into account a measured temperature and a thermal limit temperature to control a voltage and frequency of the GPU and / or CPU to remain within safe thermal limits.

[051] As indicated by arrow 130, the hardware control module 110 sends values of parameters for the operation information such as frames per second (FPS), and values of parameters for the load information such as thermal limit, temperature and / or frequency to the OS control module 112. The OS control module may comprise three separate modules. The QoS (Quality of Service) predictor module 140 and selector module 142 are shown as two separate components but it will be appreciated that their functionality could be combined in one module termed a QoS controller. The QoS controller may also be termed a processor and may comprise one or more of: a microprocessor, a microcontroller, and an integrated circuit. These modules / controller take into account software and hardware parameters to determine if and how the game parameters need to be modified before being passed to the GPU and / or CPU. The predictor module 140 may be used to predict the quality of service, for example using the parameters received from the hardware control module 110. The predicted values may be passed to the selector module 142 to select the best parameters. The OS control module 112 optimises power (of the CPU / GPU), thermal properties of the CPU and GPU, as well as performance, resolution and stability of the application running on the hardware concurrently. Thus, as indicated by the arrow 132, the frequency (or other hardware parameters) may be sent to the GPU and / or CPU, e.g. from the selector module 142.

[052] In the example of a game, the QoS controller may ensure that hardware capacity is used as best possible while obtaining the best possible gaming parameters and ensures stable, not lagging, game play. Additionally, controlling thermal properties is an important factor, especially for mobile user devices, as these are often handheld devices. Consequently, there is not just a limit at which a CPU and GPU will no longer function properly, but there may also be a thermal limit at which a user can no longer comfortably hold a mobile device. Optimising to take into account thermal properties may therefore take into account thermal properties and requirements of the hardware and a user experience when holding the mobile device.

[053] In addition to sending parameters to the GPU and / or CPU, the selector module 142 sends a parameter as indicated by arrow 134 to an interceptor module 144. The interceptor module may also be termed an interceptor and the terms are used interchangeably. As shown in Figure 1, the interceptor module 144 intercepts a command 152 from the application control module 114 and sends a modified command 154 to the GPU and / or CPU. As explained in more detail below, the modified command is modified using the parameter received from the selector module 142 and this parameter is a scaling factor or scale which is typically less than 1 to ensure downscaling. More generally, the QoS module (including the predictor module and selector module) may be considered to be controlling the interceptor module.

[054] Figure 1 also shows the application control module 114 and in this example, the application is a game. The application control module 114 tracks the application’s performance, for example using a performance tracking module 162 and a control module 164. The control module 164 sends the required application parameters relating to the operation information, e.g. resolution and quality to the application engine 106 which in turn sends commands to the GPU and CPU. As shown in Figure 1, the GPU commands which relate to displaying an image are intercepted by the interceptor module 144 and adjusted before sending the modified commands to the GPU. Other CPU commands may be sent as shown by arrow 156 direct to the CPU 102. By adjusting the image commands sent to the hardware, it can be ensured that requirements placed on the hardware remain within the limits set for the hardware. When the hardware operates, a load may be generated accordingly. The load may refer to the amount of processing jobs (e.g., executing commands, processing data, calculating numbers, etc.) that the hardware is tasked with. The load may also indicate a utilization rate of available resources. Therefore, while an image required by the application may be displayed at a reduced resolution, or otherwise reduced, the GPU and / or CPU will be able to compute a required output within a time frame that ensures that there is no lag in the application.

[055] The steps carried out by the OS control module, more specifically the QoS controller and the interceptor, are now described in more detail with reference to Figure 2 to show how the Quality of Service (QoS) controller controls and optimises software and hardware parameters concurrently and how the software parameters are used by the interceptor. First, the QoS controller obtains any fixed software and hardware parameters at step S200. In particular, the QoS controller obtains a target size of an output image (e.g. the size of a display screen on which the image will be displayed to a user) and device parameters. The target size may be a fixed parameter and thus remains unchanged by the QoS controller throughout the steps executed by the QoS controller. That is, other parameters controlled by the QoS controller may need to be changed and / or adapted to accommodate the target size.

[056] The QoS controller may also obtain other fixed device parameters. These parameters may, for example, comprise: CPU and / or GPU baseline performance data, i.e. data about the maximum performance that can be expected from the CPU and / or GPU. These parameters may also include CPU and / or GPU power. Further device parameters may be data about thermal properties of the device, such as the device’s thermal conductivity and / or the device’s thermal capacitance, as well as any thermal limits for the device and its components. The thermal limits may include a safe temperature limit for a user to hold the device as well as safe operational limits for the CPU and / or GPU. For these limits, the device’s thermal conductivity and thermal capacitance are important factors. For example, a device made of metal, such as aluminium may conduct heat much more quickly than a device made from a thermally less conductive material, such as glass. This impacts whether temperatures inside the device may be permitted to rise, and to what extent, while still providing a device with a temperature that is safe for holding by a user. A device’s thermal capacitance may determine how much heat may be stored inside the device, and thus may, for example, determine an amount of time needed for the device to cool down to a certain temperature again after a large amount of heat has been generated by the CPU and / or GPU.

[057] At step S202 other information is obtained, and this information may reflect the current values of various parameters. The QoS controller may obtain values of hardware parameters, for example from the hardware control module as shown above. Hardware parameters may represent information regarding the current load of the GPU / CPU (and / or the computer device / other components). The format of this information can vary, but will typically comprise an indication (e.g. percentage) of the maximum load of the component currently being utilised. At step S202, a measurement of the air temperature and / or temperature inside a mobile device may also be obtained, e.g. from the sensor shown in Figure 1. This measurement may be necessary to predict a temperature at the CPU / GPU of the device as well as a temperature of the outside of the device that may be held by a user. The QoS controller may also obtain values representing the operation of the application, particularly information related to a graphics-related operation of the application which is running on the electronic device. The operation information may represent the user experience of the application, e.g. number of frames. For example, a game may be rendered at 60 FPS when the frequency increases whereas a benchmark may not normally reach 60 FPS. An FPS threshold value may be set relatively higher. In addition, because a normal user interace (III) operation may not significantly increase the temperature of a computer device, power and temperature conditions may be ignored for general UI operations.

[058] The skilled person will understand that the inputs (i.e. the information which has been obtained) may be used directly (unchanged) as values / variables in the power management algorithm. Alternatively, the inputs may be processed further before being used as values / variables in the power management algorithm. For example, the data collected in the hardware control module may correspond to the time between images / frames rendered by the GPU, and may be converted to FPS, e.g. using: FPS = number_of_frames / time.

[059] At step S204, the QoS controller may take into account some or all of the parameters obtained at steps S200 and S202 to predict the performance of the hardware, e.g. power and thermal properties as well as to predict the performance of the software, e.g. resolution and stability. Stability may represent the FPS generated by the GPU, as measured by a display driver or display controller of the device. In some embodiments, the device may comprise an Android™ based mobile device, and the FPS is measured at the SurfaceFlinger level, which is a service in the Android™ operating system that takes care of refreshing the display. However, the skilled person will appreciate that this type of data may be obtained in different ways, depending on the computer system / device. The skilled person will also understand that other indicators of perceived performance could be used instead of (or in addition to) FPS, such as III response time (e.g. time between the press of a button and when a corresponding action is completed, either a sound, vibration, image, III change, or feedback to the user); application completion time (e.g. collected by the information the application provides to the UI trough loading indicators in “%”,“bars” or loading animations) or the current speed of processing / installing / downloading in the case of processing operations.

[060] Any suitable method for predicting the performance may be used. One example is a rule based method may be used which includes formulae which make assumptions on how the various parameters (fixed and measureable / variable) affect the Quality of Service. As an example, the rule based method may use a formula such as: QoS = Cpu_Freq*x+Gpu_Freq*y+Scale*z where QoS is the quality of service, CPU_Freq and Gpu_Freq are the frequencies of the CPU and GPU, Scale is a scaling factor for image results and x, y and z are weights. As an alternative to a rule based method, a machine learning, ML, model which has been trained to predict how parameters affect QoS may be used. Any architecture is possible, provided that the architecture takes input variables as well as past values and predicts the QoS and / or the optimal values of the parameters which can be adjusted by the system.

[061] In the example below, we detail an estimation based method which uses formulae with tuneable parameters that are estimated and updated with data. It will be appreciated that these exemplified formulae are simply examples and different, e.g. more detailed, formulae can be used.

[062] The total time (and therefore FPS) may depend on CPU and GPU times, i.e. the amount of time needed by the CPU and GPU to complete the requested calculations: FPS = 1 / FrameTime FrameTime = f (CPU Jime, GPU_tline) CPUjime = CPU_work / CPU_frequency / CPU_baseline_performance(work / Hz) GPU_time = GPU_per_pixel_work * (Scale * Width) * (Scale * Height) / GPU-frequency I GPU_baseline_performance(work / Hz) wherein FrameTime describes the amount of time needed to calculate / render one frame, i.e. one image. FrameTime may in turn be a function of the amount of time CPU_time and GPU_time needed by the CPU and GPU, respectively to complete the requested calculations. CPU_time depends on the work done by the CPU (CPU_work), the frequency at which the CPU is operating (CPU_frequency) and the amount of work the CPU is capable of doing (CPU _baseline performance), i.e. what type of CPU is being used. The CPU_work will vary depending on the application, i.e. game, being used and needs to be estimated or predicted. The CPU_frequency is controllable and can be adjusted to control the CPU_time. More generally, the FPS at time t can be expressed as: FPS(t) = PredFps(CpuFreq, Gpupreq, Scale)

[063] GPU time similarly depends on the amount of GPU work which is this case is measured per pixel (e.g. the number of computations required per pixel, or another measure of work done by the GPU), the frequency at which the GPU is operating (GPU_frequency) and the amount of work the GPU is capable of doing (GPU _baseline performance), i.e. what type of GPU is being used. Additionally, as the work done by the GPU depends on the work done per pixel, the time required by the GPU to render an image will also depend on the size of the image. This is represented by the Width and Height terms in the equation above. To reduce GPU computation time, the QoS controller is able to adjust a scale factor (Scale) of the image being rendered. In this way, when the amount of computation per pixel is high, the image may be downscaled to reduce the total amount of computation required per image. The GPU_frequency is also controllable and can be adjusted to control the GPU_time. In other words, software and hardware parameters are jointly controlled to control the GPU_time.

[064] Another set of formulae approximate the total power, the power required from the CPU (CPU_power) and the power required from the GPU (GPU_power) as: Power = CPU_power + GPU_power CPU_power = FrameTime / CPUjime * CPU_Power(CPU-frequency) GPU_power = FrameTime I GPU_time * GPU _Power(GPU-frequency) As in the previous set of equations, CPU_frequency and the GPU_frequency are controllable to control the power at each component. Note that CPU_Power and GPU_Power represent the maximum thresholds for the power levels at each component which should not be exceeded. These thresholds are fixed. More generally, predicting the power at time (t) this can be expressed as: Power(t) = Predpower(CpuFreq, GpuFreq, Scale)

[065] Thermal properties of the device and / or GPU / CPU, i.e. the temperature of the device, are determined by the power being used, the surrounding temperature or air temperature and some other device characteristics. This can be approximated as: Thermal(t) = Thermal^ — 1) + ((T_Air(t — 1) — Thermal(t — 1)) * Dev_TConductivity + Power(ty) / Device_TCapacitance where t is time, Thermal are the thermal properties of the device, which may be a temperature 5 and / or other thermal properties, such as a thermal score given in arbitrary units. The temperature and / or thermal score may each be associated with a temperature and / or thermal score limit and the goal of the QoS controller is to keep these quantities below their limits. T_Air is the air and / or surrounding temperature, i.e. the temperature around the device. Dev_TConductivity is the device’s thermal conductivity. Device_TCapacitance is the device’s 10 thermal capacitance and Power is the power expanded by the GPU and / or CPU at a given point in time t. In the above equation, the following values may be fixed values (i.e. unchanged for any one particular device or type of apparatus): Dev_TConductivity and Device_TCapacitance. In the above equation, the following values may be dependent on application requirements and / or surroundings of the user device: CPU_work, 15 GPU_pixel_work, Width, Height, T_Air(t-1). More generally, the thermal value at time t may be predicted using: Thermal(t) = PredThermai(Thermal(t-1), Power(t))

[066] Additional predictions can be made, e.g. a quality measure which is just a function of the 20 scale factor: Quality(t) = Q(Scale) There may also be a quality measure, e.g. stability, for the frames per second: FPSstabiiity (t) = StabFpS(FPS(t-1), FPS(t)) 25

[067] In summary, at step S204 the QoS controller may predict and / or estimate, for example, some or all of the quantities described in the equations above: power, thermal properties of the CPU and GPU and frametime (or another quality measure(s)). As mentioned above, there is a need for balance in the controllable parameters, and thus when predicting the performance, the QoS controller may determine an overall QoS metric which combines the 3D controllable parameters which include for the software: the optimum scale factor (scale) and for the hardware: an optimum GPU frequency and an optimum CPU frequency. Any suitable metric may be used and the simplest version may be defined is: QoS = CPU_frequency * x + GPU_frequency * y + Scale * z, where x,y and z may be weights that can be set so as to weight the importance of each of 35 these factors.

[068] For example, when the QoS is determined for playing a computer game, the amount of computation required from the CPU may be relatively constant throughout the game, as the CPU usually makes calculations that do not involve any rendering of images / graphics. For example, the CPU may determine the location of objects in each rendered image or make other calculations necessary for the game to function. Thus, as the QoS may significantly depend on the stability of a framerate per second, the CPU may have less bearing on a perceived QoS. However, the CPU also produces heat, so the CPU’s frequency has to be taken into account into these calculations. The GPU on the other makes graphic-intensive calculations and thus the GPU’s load may vary significantly based on the complexity of an image that is being calculated. Thus, the frequency of the GPU may be weighted differently than that of the CPU.

[069] In the estimation-based approach and using the equations described above with reference to step S204, an alternative QoS metric may be defined as: QoS(CPU-frequency, GPU-frequency, Scale) = wl * FPS stability (CPU -frequency, GPU -frequency, Scale) + w2 * PerceivedUserQualityFunction(FPS(CPU-frequency, GPU-frequency, Scale), Resolution) - w3 * Thermal(CPU-frequency, GPU-frequency, Scale) where wl, w2 and w3 are weighting functions that may be used to weight each of the quantities listed above. wl,w2 and w3 may take any suitable form, and may be constants, linear functions, polynomials or any other suitable function. The first term represents the stability of the FPS, FPS_stability, and this is a function of the three controllable software and hardware parameters: CPU-frequency, GPU-frequency and Scale. Similarly, the second term represents the perceived user quality, PerceivedUserQualityFunction, and includes a function representing FPS and based on the three controllable parameters, as well as a resolution term. The final term represents the thermal information (Thermal) and is also a function of the three controllable parameters. It will be appreciated that these functions may be constructed in any suitable way, so as to enable estimation of a QoS. An alternative version of the QoS metric above may be estimated using square root functions as: QoS(c, g,s) = wl * FPS_stability(CPU-frequency, GPU -frequency, Scale) + w2 * sqrt(FPS(CPU-frequency, GPU-frequency, Scale)) * sqrt(Resolution) - w3 * Thermal(CPU -frequency, GPU -frequency, Scale) with wl, w2 and w3 similarly being functions used to weight each component of the equation and the other terms having the same meaning as above. More generally, the QoS metric may be expressed as: QoS(t) = QoSFunction(FPS(t), FPSstab(t), Thermal(t), Quality(t))

[070] At step S206, there is a calculation of the optimal values for each of the controllable software and hardware parameters. Any suitable maximisation function may be used, for example: Best(c, g, s') TYlCLXcpu frequency ,GPU frequency .Scale ^QoS(VCP UfVCgUGTlCy^ 'VgGPUfV6gUCTiCyf VScale^^ Using this equation, all possible QoS metric values for each possible value of the controllable parameters is calculated and then the optimal or best value is selected. For example, the following computations may be made and the bold value is selected as the optimal value, i.e. the highest score: QoS(1.2GHz, 200Mhz, 0.8) = 100*0.05 + 14.4 - 0.1*42 = 15.2 QoS(1.3GHz, 200Mhz, 0.8) = 100*0.05 + 14.8 - 0.1*43 = 15.3 QoS(1.2GHz, 300Mhz, 0.8) = 100*0.05+ 14.7-0.1*44= 15.3 QoS(1.2GHz, 300Mhz, 0.75) = 100*0.05 + 14.7 - 0.1*43 = 15.4 QoS(1.2GHz, 300Mhz, 0.85) = 100*0.05+ 14.4-0.1*43= 15.1

[071] That is, the equations may be solved to maximise QoS, CPU (c) and GPU (g) frequencies and scale (s) concurrently. The scale (and any other software parameters) calculated at step S206 is sent to the interceptor at step S208. The CPU and GPU frequencies calculated at step S206 are used to configure the hardware of the device / apparatus, in other words they are sent to the GPU and CPU, e.g. via the DFVS at step S210. That is, the GPU and CPU frequencies are set to the calculated frequencies. The method of calculating the optimised parameters then loops back to step S202. The optimised parameters may be calculated at regular intervals and may not occur for all frames or may occur multiple times per frame.

[072] Steps S212 to S218 show the steps carried out by the interceptor. At step S212, the interceptor receives the software parameter (e.g. scale) from the QoS controller. Step S212 may be asynchronous with the generation of the scale is steps S200 to S210 and may repeat at regular intervals which are different from the regular intervals for calculating the scale. Step S214 shows the interception of a command from the application, i.e. from the application engine, to the GPU / CPU. As explained below, the intercepted command can be modified to take into account the above calculations. In particular, after the command has been intercepted, the resolution required of the image that is to be calculated is reduced. It will be appreciated that although this step is shown after the optimised parameters have been calculated, the steps for calculating the optimised parameters are typically operating in a loop operating at a fixed time interval. Thus, the intercepting step could take place at any time during the calculation process and will use the most recently calculated scale factor.

[073] As explained in more detail below, each intercepted command may comprise multiple embedded commands and these multiple commands may be modified as a batch or separately. Each command relates to generation of an image and an image may be a frame within a video, or a sub-frame. Thus, at step S216, the interceptor modifies the command, e.g. by applying the received scale as described in more detail below. The interceptor then outputs the modified command to the GPU and / or CPU at step S218. Steps S202 to S218 are then repeated throughout the running of the application. As shown steps S202 to S210 may also be on a repeat loop.

[074] Figures 3a and 3b show how a quality of service may be affected by hardware constraints. In Figures 3a and 3b a computer game is rendered at a constant resolution of 1440 pixels. The blue (bottom) line in Figure 3a shows a change over time in framerate per second (FPS), i.e. how many images are rendered per second, while the orange (middle) line shows a GPU load in percent and the grey (top) line shows the resolution of images being rendered. While the resolution of images remains constant, in the high load area (pink), the GPU is at maximum capacity, meaning that the GPU requires more time to render each image at constant resolution. This in turn results in a drop in FPS that is output.

[075] Figure 3b shows a corresponding Quality of Service over time, calculated as: QoS = sqrt(resolution) * sqrt^FPS) / 100 + 7 * abs(FPS(t -2)- FPS(tJ) I FPS^t) This is a simplified QoS metric which includes stability as abs(Fps(T-2) - Fps(t)) I Fps(t). The quality of service calculated in this graph is a proxy for user experience during the time interval shown in Figures 3a and 3b. Because users perceive shifts in the framerate per second that is being output during a game as less desirable / disruptive to game play, it is desirable to keep the FPS as constant as possible. This is not happening in Figure 3a, as represented by the calculated QoS in Figure 3b.

[076] Figures 3c and 3d show how a quality of service may be improved by using the present techniques. Figure 3c plots a change in framerate per second (FPS) over time (bottom line), a change over time in a GPU load in percent (middle line) and the change in resolution of images being rendered (top line). In Figures 3c and 3d, a computer game is rendered at a resolution of 1080pixels, corresponding to 75% of the full / original resolution in the High Load Area (highlighted in pink). In the high load area shown in Figure 3c, the present techniques reduce a resolution of an image being rendered by the GPU, thus reducing the amount of computing power required to render these images. The GPU is still rendering images at maximum capacity, but due to the resolution of the rendered images having been reduced, the FPS remains generally constant. That is, the computations during the high load area are still more demanding, but by reducing the resolution of the images the computational requirements are reduced such that the FPS is generally constant. Similarly as shown in Figure 3d, the QoS remains relatively constant.

[077] Figure 4 is a flowchart describing the steps carried out by the interceptor and the GPU to enable dynamic scaling of an image that is to be rendered by the GPU and hence to reduce the resolution of the displaying image in the target area. Figure 5 illustrates these of dynamic rescaling of images. By contrast, prior art techniques typically only enable static scaling of an image that is to be calculated. That is, in other methods, a scale factor is determined at the beginning and can no longer be changed when hardware and / or software requirements change. Additionally, prior art techniques typically only scale the resulting image and do not support multiple stages in which scaling is applied.

[078] In a first step S402 of Figure 4, the interceptor intercepts a command from the application to the GPU to render an image in a viewing region. As illustrated in Figure 5, the interception prevents the original command reaching the GPU and thus prevents the original image 502 from being displayed in the full sized viewing region which may also be termed the world coordinate window or the target area, e.g. the display screen of the apparatus. In this simplified example, the intercepted command is a render command which renders the image in a single pass, for example using a single pipeline. It will be appreciated that are many different methods for rendering images: Vulkan, GL and / or Metal are examples of suitable graphics APIs. The inputs to and nature of the pipeline will depend on the specific API which is being used.

[079] The viewing region may have different terms depending on the API being used. The viewing region is typically a rectangle and represents an area expressed in rendering-device specific coordinates, such as pixels, in which the image may be rendered. For example, the viewing region may be termed a viewport . The Interceptor module then applies the scale factor which has been determined as described above to the intercepted command. As shown at step S404. the Interceptor module modifies the command by modifying the viewing region (i.e. Viewport) size. The viewing region size may be modified by being scaled by the scale factor. As shown in Figure 5, the scaled viewing region 506 is smaller than the target area and thus when the GPU renders the image in the scaled viewing region 506, the image is downscaled. The resolution of the scaled image is lower than the original unsealed image.

[080] As shown in Figure 5, the scaled viewing region 506 only occupies a part of the frame buffer 504. Accordingly, as shown in Figure 4 at step S406, a size of the image buffer is maintained by leaving random and / or previously saved buffer content in the part of the buffer outside the scaled region. In other words, the rest of the buffer may be left “as is”. These unchanged areas of the image buffer are shown in black in Figure 5. At step S408, the modified command, i.e. the command to generate an image buffer with an unchanged region and a downscaled viewing region (viewport) is input into the GPU.

[081] The modified command is then executed by the GPU at step S410. Execution may include the GPU rendering the image based on the size of the modified viewing region. The GPU may also leave unchanged the regions of the frame buffer outside the viewing region as instructed in the modified command. As shown at step S412, the image buffer with the scaled image is output. The data for the scaled image is then read to generate a final image of target size at step S414 and is output as the final image at step S416. These steps are shown as being carried out by the GPU but any suitable hardware or software method may be used for the upscaling. For example, for Android™ operating systems, the operating system may automatically scale any image received from an image buffer to a target size, i.e. a display size. In this case, upscaling automatically takes place using a lightweight bilinear scaling algorithm.

[082] Figure 5 shows the output final image 508 on the display which has the same size as the original image 502 but has a lower resolution (i.e. lower pixel density) because of the scaling applied to the viewing region by the interceptor. By scaling the viewing region, scaling is effectively applied to the image, and this means that the computational cost of rendering this scaled image is reduced compared to rendering the image at full resolution.

[083] The method shown in Figures 4 and 5 is applied to a single image. The scale factor can dynamically be changed for the computation of any subsequent images, e.g. subsequent video frames, making optimum use of available hardware resources at any given point in time. The scaling factor which is applied by the interceptor may be updated at any time, for example the value of the scaling factor can change per frame or even mid-frame.

[084] As explained above, Figures 4 and 5 relate to the simplified example of processing a single image using a single command. The command which is modified may be termed the viewport command because the viewing region is scaled. In practice, there may be more than one command which needs to be intercepted and modified. Additionally, one or more of a rendering command, a shader command to use texture and uniforms, a texture creation / modification command, a framebuffer / texture attachment command and / or a copies command may be intercepted and modified.

[085] Figure 6 shows an example of a more complex image rendering process. When rendering a complex final image, components of the image are usually calculated in multiple steps. For example, a light map may need to be calculated first. A light map may map how light from a light source and / or reflections may be distributed throughout the final image. The light map may then need to be used to compute the colours of the final pixels in the image to reflect lighting conditions that are to be shown in the final image. Another example of a component may be shadows, that may be calculated in a similar way. That is, a shadow map may also be calculated. The shadow map may then map the location of one or more shadows in the final image by using data about the location of a light source.

[086] The input image may be input in further components. As a further example, a complete image that is to be rendered may comprise the same smaller components that are to be rendered in several places. Each of these smaller components may, for example, represent a different surface pattern. When one pattern is calculated, this may then be reused to calculate another part of the image. The complete image may then, for example be split up to calculate patterns, shading, light(ing) and / or texture. Each of these components may be reused several times and / or combined with other components and / or parts of the input image to generate the output image which is displayed to the user.

[087] For example, most of the final image may be calculated and components corresponding to lighting may be applied when most of the image has already been calculated. The CPU may determine the location of a light source and thus this may be used to place components corresponding to lighting within the complete image. The lighting components may then, for example, be merged with other components of the complete image to create the impression of light coming from and / or being reflected off a light source. In another example, surface textures may be used to calculate several parts of the final image. For example, if the final image contains water and a sky, the texture for “water” and “sky” may share common elements. Thus, a pattern computed for “water” may then be reused to compute “sky”.

[088] Components may thus be understood to mean precursors to the final image. Components can be maps, mapping a certain feature (e.g. lighting, shadows, texture) to the final image. Components may also be patterns that repeat several times in the image and / or need to be combined with other patterns, textures, lighting and / or shading to create a pixel of the final image. Calculating one component may rely on having calculated a previous component. That is, pixel values in a component may or may not affect pixel values in a later component. It is usually necessary to make this computation to determine if a component affects a subsequent component. Because these patterns may not be updated for every complete image that is being computed but instead may only be updated after a certain number of frames, a method is needed for ensuring the components can be combined to form the output image.

[089] A pipeline may be understood as a sequence of commands to combine different images and / or image components to obtain a resulting image which is displayed to the user. A pipeline may mean a simple sequence of commands, such as combining two image components or a complex sequence of commands, such as combining two image components that are then combined further with other image components. A pipeline may be of arbitrary size, i.e. include an arbitrary number of commands and / or steps to combine images and / or image components. A pipeline may be specific to a certain image or image component, for example, there may be a specific pipeline to obtain a texture for “wood” or “cloud” and this specific pipeline may combine specific image components to obtain the desired texture. The pipeline may then be adapted to these specific image components. A pipeline may also be generic and may be reusable to calculate a variety of different outputs. For example, combining two different components to obtain an output may be considered a pipeline. This pipeline may be applicable to a large number of different desired outputs. Several pipelines may be combined to form a larger and more complex pipeline. For example, when two components are combined to obtain an output, the output may form an input for another pipeline that in turn combines the output with another component. These pipelines may then be considered to form one pipeline for obtaining a further output.

[090] Merely as an example Figure 6 shows three different components of an input image which include vertex data 602, image texture 604 and shading data 606. The vertex data is a data structure which describes the position of a point or multiple points in 2D or 3D space. The image texture is a set of metrics which quantifies the perceived texture of an image and typically gives information about the spatial arrangement of colour or intensities in an image. The shading data may be in the form of uniform which acts as the parameters for the shader program within the pipeline. The uniform is so named because their value is uniform at each stage.

[091] Figure 6 shows an application 608, e.g. a gaming application, which requires an output image 626 to be displayed. In this example, the overall computational pipeline comprises two pipelines both of which comprise a render stage 610, 620 and the output of the first pipeline is required for the second pipeline. Figure 6 shows which commands within the overall computational pipeline commands need to be intercepted and modified by the interceptor module to account for a scale factor. In the first pipeline, two components 602 and 604 are input to a render stage 610 which when executed by the GPU renders an output texture. In a similar manner to that described in relation to Figures 4 and 5, the viewport command 612 is intercepted and modified by scaling the viewing region for the texture and by defining any actions on the regions outside the viewing regions. The viewing region for the texture may be scaled using the same scale factor as the scale factor that is used for any other components of the image. That is, the viewing region for the texture may be adapted when the scale factor is changed by the QoS controller. Alternatively, the viewing region for the texture may remain constant. For example, for some textures, best results may be achieved by not scaling the texture, i.e. not changing the viewing region of the texture. This corresponds to setting a scale factor of the texture to 1 and not changing this, even when the QoS controller changes the scale factor for other components of the image. Thus, when this viewport command 612 is executed by the GPU, the GPU generates a scaled texture 614 which occupies only part of the buffer 615. The buffer size is maintained by leaving in the unchanged data as described above.

[092] The first frame buffer 615 is input to a second render stage 620 together with shading data 606. When executed by the GPU, the output from this second render stage 620 is the complete image. As indicated in Figure 6, the texture command 616 which sends the first frame buffer containing the texture to the second render stage is intercepted and modified by the interceptor. The interceptor modifies the command by setting the scale which was applied to the viewing region in the modified viewport command 612. Thus, when executing this modified texture command, the GPU knows the scale which was applied to the texture. Furthermore, the second render command 620 also needs to be intercepted and modified to ensure that when executing the command, the GPU upscales the scaled texture which is received from the first stage. The modification will involve modifying shaders and / or setting uniforms according to sampling at “coord*scale_factor”. That is, the command needs to be modified to accommodate a changed coordinate location of the scaled texture 614 in the buffer 615. When a location value of an unsealed texture in the buffer 615 is, e.g. location value 100 in the buffer, the corresponding scaled location value can be obtained by multiplying the unsealed location value by the scale factor. I.e. the scaled location value is then coord*scale_factor = 100 * 0.5, when the scale factor is 0.5 (i.e. the size of the image is halved). A location value of the unsealed texture in the buffer 615 corresponds to a pixel of the unsealed texture saved at this location. Accordingly, a location value of the scaled texture corresponds to the corresponding scaled pixel saved at this location value. In this way, any command may be modified to account for a scaled image and / or component of an image being saved in a buffer. That is, to retrieve a pixel value for an unsealed image or image component saved at a particular location value in the buffer, the unsealed location value is multiplied by the scale factor to obtain a scaled location value and corresponding pixel saved at the scaled location value for the scaled image.

[093] The framebuffers (also termed buffer) are the input and output of functions within the GPU pipeline for rendering an image. The functions which are executed include “shaders”. Merely as a simple example for illustration, a GPU might execute: o[x,y] = i[x,y]*i[x,y-1] When changing the viewport, the range of x,y is reduced (that is fewer pixels are written and read). If the input is scaled as well, then we not only modify the viewport (x,y), but also unscale the input with the input scale s (might be different to the output scale). For example the shader function now would look like: o[x,y] = i[x*s,y*s] * i[x*s,(y-1)*s]

[094] In a similar manner to that described in relation to Figures 4 and 5, a second viewport command 620 which inputs the result of the second render command 618 into a second frame buffer 624 is intercepted and modified by the interceptor by scaling the viewing region for the image and by defining any other instructions for creating the framebuffer. Thus, when executing this modified texture command, the GPU generates a scaled image 622 which occupies only part of the buffer 624. The buffer size is maintained by retaining unchanged data as described above. The size of the first buffer 615 may be the same or different from the size of the second buffer 624. Similarly, the scaling which is applied to the viewport commands 612 and 620 may be the same or different. In particular, the scaling which is applied to command 620 may be the scaling which is generated by the QoS controller as described above. The scaling which is applied to command 612 may be the “current” texture scale and may also be generated by the QoS controller as described above. Alternatively, as described above, the scale factor for a texture may be a fixed constant value, such as 1. However, even when no scaling takes place, the scale factor is still linked to the texture to enable the texture to be combined with other components that may be scaled. The scaled image 622 within the second buffer 624 is then upscaled to generate the final output image 626 which is displayed on the display. The interceptor is thus operating between the application and the display.

[095] Figure 7 is a flowchart representing the general steps which are carried out when executing the pipeline shown in Figure 6. In a first step S702, the interceptor intercepts a viewport command for the output from the texture render stage. The viewport command is a command which sets the size of the viewing region of the texture and may also set which region of the frame buffer to ignore when rendering a texture and / or other component. The viewport command may also be termed a framebuffer write command (and the terms may be used interchangeably). At step S704, the viewport command is modified by the interceptor using the current scale factor. As explained above, this modification involves scaling the viewing region (i.e. the viewport). As shown at step S720, this modified command may then be sent to the GPU and executed.

[096] In a next step S706, the interceptor intercepts a texture command which sends the output from the texture render stage to the next render stage. At step S708, the interceptor modifies the texture command by adding or setting the scale which was used in step S704 to the command. As an alternative, the scale may be attached as an attribute to the output texture. As shown at step S722, this modified command may then be sent to the GPU and executed by the hardware.

[097] In a next step S710, the interceptor intercepts a render command which when used by the GPU will render the final image based on the input texture and any other inputs. At step S712, the interceptor modifies the render command to make sure that the scale which was set to the previous command or attributed to the texture is used when rendering the final image. For example, the render command may be modified to upscale the texture. As shown at step S724, this modified command may then be sent to the GPU and executed.

[098] In this example, the final command which is intercepted at step S716 is a viewport command for the output from the image render stage. At step S718, the viewport command is modified by the interceptor using the current scale factor. As explained above, this modification involves scaling the viewing region (i.e. the viewport) in this case for the final image which will be output on the display (after rescaling). As shown at step S726, this modified command may then be sent to the GPU and executed. The GPU can then output the result of the pipeline at step S728. As explained above, the output may be a frame buffer with the scaled final image which is rescaled to the target size using any appropriate means. Alternatively, the output from the GPU may be the scaled final image which is rescaled to the target size.

[099] As indicated in Figure 7, each command may be modified on the fly and immediately executed by the GPU. As an alternative, commands may be modified in batches, for example for a whole pipeline or part of a pipeline and the batch of modified commands is sent to the GPU before executing the pipeline. In this way, the number of computations to modify the pipeline may be reduced and may be avoided during execution of the batch of commands. Furthermore, some pipelines may be used several times, but with different inputs and outputs. It may be possible to modify a particular pipeline only once and then reuse the modified pipeline several times. It will be appreciated that the modifications may depend on a number of factors, such as the current scale factor of an input component / image and / or output component / image of a pipeline.

[100] Figure 8 shows a more complex image rendering process comprising four pipelines 702, 704, 706, 708, each of which executes at least one command. Such a process may be used when rendering a plurality of images r, for example, for a computer game, one rendered image is often used to generate another rendered image. As shown in Figure 7, when the pipeline for rendering an image involves combining several image components, the interceptor module needs to make changes when scaling each of these components and also when combining the scaled components. In particular, all pipelines need to be modified to include an extra input argument 710 containing the scale factor applied to each component 712, 714, 716, 718 so that there is no stretching, glitches or errors in the pipelines. Components may be reused over several frames and the scale factor may change between these frames. When components have been scaled with a different scale factor, the scale factor attached to each component determines which pixels in each component are to be combined to form a new component or the full image. If the scale factor is not taken into account, combining components may result in suboptimal final images, containing, for example, blurring or distortions.

[101] The scale factor being attached to each component is shown in Figure 8 as a dot 710 attached to each of the components. Thus, the Interceptor module not only needs to intercept and modify application commands but also has to attach a scale factor to each image and / or image component that is being rendered. This is similar to step S708 in which the scale is “set” to track the scaling applied to the texture. In summary, each framebuffer which is used in the pipeline to render a resolution of an image or a component of an image (e.g. texture) will be modified. Thus intermediate framebuffers which are used for preparatory steps such as rendering shadows and / or reflections, etc. are modified as well as the final image framebuffer.

[102] All commands that affect pipeline setup, for example, texture setup and framebuffer output need to be “tracked” and modified to take into account any scale factor which has been applied. For example, if an application uses a component in the form of pre-calculated texture data when computing the final image, the scale factor of this saved texture data is set to 1 (i.e. no scaling has been applied). That is, a scale factor needs to be added to all components used in computing the final image, including those components that are not computed by the GPU but are imported / pre-set by an application and then used at some point in the pipeline for calculating a final image. Similarly, the first inputs 712, namely the inputs into each of the first pipelines 702, 704 may be set with an initialised scale factor of 1, when no scaling has been applied. However, after processing in the first pipelines 702, 704, a scaling factor will have been applied and thus the outputs 714 will also be associated with a scale factor 710.

[103] The overall scale factor of the final image 720 is the scale factor set by the QoS controller at a time when the final image 720 is rendered. This means that there may be several different scale factors that need to be combined in the pipeline. For example, previously calculated outputs such as the outputs 714 from the first pipelines and the output 718 from the second pipeline 706 may have a higher or lower scale factor than the current scale factor that has been set by the QoS controller. Care needs to be taken with the different scale factors when combining these outputs as they are input into the third pipeline 708. Components with a different scale factor are made compatible with the current scale factor by attaching the scale factor of each component to the component (or the command that acts on that component) as an attribute. For example, while running one of the first pipelines 702, a component 714 is computed and written to a frame buffer, using a viewport command. The Interceptor module has modified this viewport command by scaling the viewing region to the desired scale value, i.e. the current scale value set by the QoS controller. When the component is reused in the third pipeline 708, the scale factor may have been modified and by attaching the scaling factor to the component 714, account can be taken of the scaling which was applied when generating this component when processing the component in the third pipeline. This is an efficient way of computing an image because each component is scaled to the desired scale factor at the point at which it is generated and thus the desired power management is achieved.

[104] Using the attributes, components may be rescaled before they are used to compute another component and / or the final image. Alternatively, rescaling may only happen in a final step, when the final image is computed, for example, in the third pipeline as shown in Figure 8. Advantageously, in the techniques described above, it is possible to compute at what points rescaling / accounting of different scale factors needs to occur in order for a complete pipeline to run as desired. It is noted that there is no attribute attached to the inputs 716 to the second pipeline because they are 3D inputs. Similarly, the output is a 3D output and thus is not scaled.

[105] In summary, the method and system described above aim to control SoC frequencies at the CPU and GPU to achieve as stable as possible FPS whilst also balancing FPS and resolution as well as tracking temperature and reducing SoC frequencies to control heat. In this way, we achieve a balance between the elements which affect the user experience. In other words, we achieve a balance between keeping power and heat as low as possible, keeping performance and stability as high as possible and keeping image quality (i.e. resolution) as high as possible. The pipeline design described above is able to provide a holistic control of thermal properties and power which are hardware factors as well as performance, stability and quality of an application (i.e. software) running on that hardware. As explained above, the method is executed without any prior knowledge of the application code which is being applied, including any details such as shaders or other techniques used in image generation.

[106] Those skilled in the art will appreciate that while the foregoing has described what is considered to be the best mode and where appropriate other modes of performing present techniques, the present techniques should not be limited to the specific configurations and methods disclosed in this description of the preferred embodiment. Those skilled in the art will recognise that present techniques have a broad range of applications, and that the embodiments may take a wide range of modifications without departing from any inventive 5 concept as defined in the appended claims. 03 09 24

Claims

1. A computer-implemented method for power management of an electronic device, the method comprising:obtaining operation information related to a graphics-related operation of an application which is running on the electronic device to generate at least one image to be displayed on a display of the electronic device, wherein operation information is indicative of quality of the at least one image output by the graphics-related operation;obtaining load information relating to a load on hardware in the electronic device generated by the graphics-related operation;determining a scale factor to be applied during the graphics-related operation using the obtained operation information and load information, and when determining the scale factor, determining at least one hardware parameter, wherein determining the scale factor and at least one hardware parameter comprises:predicting performance of the graphics-related operation and the hardware using the obtained operation information, the obtained load information and values for the scale factor and the at least one hardware parameter;selecting an optimal value for the scale factor and an optimal value for the at least one hardware parameter which optimises the predicted performance;configuring hardware in the electronic device based on the determined at least one hardware parameter;intercepting a command from the graphics-related operation which is sent to a graphics processing unit, GPU, in the electronic device, wherein the command comprises writing image data of the at least one image to a framebuffer;modifying the command by using the scale factor to downscale a region within the framebuffer into which the image data is to be written; andsending the modified command to the GPU for execution whereby power is reduced when the modified command is executed by the GPU.

2. The method of claim 1, wherein the determined hardware parameter includes at least one of an operating frequency of the GPU, an operating voltage of the GPU, an operating frequency of a central processing unit, CPU, of the electronic device and an operating voltage of the CPU.

3. The method of claim 1 or 2, wherein predicting performance of the graphics-related operation comprises predicting at least one of frames per second, stability and quality for03 09 24multiple values of the scale factor and hardware parameters and wherein predicting performance of the hardware comprises predicting thermal performance for multiple values of the scale factor and hardware parameters.

4. The method of claim 3, wherein determining an optimal value for the scale factor and an optimal value for the at least one hardware parameter comprises:calculating an overall quality of service metric using a combination of at least some of the predicted frames per second, stability, quality, thermal performance and multiple values of the scale factor and hardware parameters; andselecting optimal values of the scale factor and the at least one hardware parameter which maximise the quality of service metric.

5. The method of any one of the preceding claims, wherein the graphics-related operation comprises a plurality of commands each of which, when executed by the GPU, writes image data to a framebuffer and wherein the image data which is written to each framebuffer is either image component data relating to one of multiple components which are combined to generate the image or image data relating to the image which is generated.

6. The method of claim 5,wherein determining a scale factor to be applied during the graphics-related operation comprises determining a first scale factor to be applied at a first time and a second scale factor to be applied at a second, later time, wherein the first scale factor is different from the second scale factor;wherein intercepting a command from the graphics-related operation comprises intercepting, at the first time, a first command which comprises writing image component data to a first framebuffer and intercepting, at the second time, a second command which comprises writing image data to a second framebuffer; andwherein modifying the command comprises modifying the first command by using the first scale factor to downscale a region within the first framebuffer into which the image component data is to be written and modifying the second command by using the second scale factor to downscale a region within the second framebuffer into which the image data is to be written.

7. The method of claim 5 or 6, further comprising linking each component with an attribute indicating the scale factor which has been used to modify a command associated with the component.03 09 248. The method of any one of the preceding claims, further comprising displaying the at least one generated image on a display by:executing, by the GPU, the modified command to write image data for the generated image into the downscaled region within the framebuffer;outputting the framebuffer;rescaling the downscaled image data within the framebuffer to a target size for an image to be displayed on the display; anddisplaying the rescaled image data.

9. The method of any one of the preceding claims, further comprising modifying the command so that regions of the framebuffer outside the downscaled region are unchanged when the modified command is executed.

10. A computer-readable storage medium comprising instructions which, when executed by a processor, causes the processor to carry out any of the methods described herein.

11. An electronic device comprising:a display,a graphics processing unit, GPU, for rendering at least one image on the display andan operating system control module for controlling a load on the GPU, wherein the operating system control module comprises a processor and an interceptor andwherein the processor is for:obtaining operation information related to a graphics-related operation of an application which is running on the electronic device to generate at least one image to be displayed on the display, wherein operation information is indicative of quality of the at least one image output by the graphics-related operation;obtaining load information relating to a load on hardware in the electronic device generated by the graphics-related operation;determining a scale factor to be applied during the graphics-related operation using the obtained operation information and load information, and when determining the scale factor, determining at least one hardware parameter, wherein determining the scale factor and at least one hardware parameter comprises:predicting performance of the graphics-related operation and the hardware using the obtained operation information, the obtained load information and values for the scale factor and the at least one hardware parameter;03 09 24selecting an optimal value for the scale factor and an optimal value for the at least one hardware parameter which optimises the predicted performance; andsending the determined at least one hardware parameter to a hardware control module to implement the determined at least one hardware parameter; andwherein the interceptor is for:intercepting a command from the graphics-related operation which is sent to GPU, wherein the command comprises writing image data of the at least one image to a framebuffer;modifying the command by using the scale factor to downscale a region within the framebuffer into which the image data is to be written; andsending the modified command to the GPU for execution.

12. The electronic device of claim 11, further comprising a central processing unit, CPU and wherein the determined hardware parameter includes at least one of an operating frequency of the GPU, an operating voltage of the GPU, an operating frequency of the CPU and an operating voltage of the CPU.

13. The electronic device of any one of claims 11 to 12, wherein the graphics-related operation comprises a plurality of commands each of which, when executed by the GPU, writes image data to a framebuffer and wherein the image data which is written to each framebuffer is either image component data relating to one of multiple components which are combined to generate the image or image data relating to the image which is generated.

14. The electronic device of claim 13, wherein the processor is fordetermining a scale factor to be applied during the graphics-related operation by determining a first scale factor to be applied at a first time and determining a second scale factor to be applied at a second, later time, wherein the first scale factor is different from the second scale factor; andwherein the interceptor is forintercepting a command from the graphics-related operation by intercepting, at the first time, a first command which comprises writing image component data to a first framebuffer and intercepting, at the second time, a second command which comprises writing image data to a second framebuffer; andmodifying the command by modifying the first command by using the first scale factor to downscale a region within the first framebuffer into which the image component data is to be written and modifying the second command by using the second scale factor03 09 24to downscale a region within the second framebuffer into which the image data is to be written.

15. The electronic device of claim 13 or 14, wherein the interceptor is for linking each component with an attribute indicating the scale factor which has been used to modify a command associated with the component.

16. The electronic device of any one of claims 11 to 15, wherein the GPU is forexecuting the modified command to write image data for the generated image into the downscaled region within the framebuffer; andoutputting the framebuffer; andwherein the electronic device further comprising a central processing unit, CPU, for rescaling the downscaled image data within the framebuffer to a target size for an image to be displayed on the display.

17. The electronic device of any one of claims 11 to 16, wherein the interceptor is for modifying the command to leave unchanged regions of the framebuffer outside the downscaled region.

Images