Frequency adjustment method of image processor, electronic device, chip system, storage medium and program product

Abstract

The embodiment of the invention provides a frequency adjustment method of an image processor, electronic equipment, a chip system, a storage medium and a program product, and relates to the technical field of terminals. According to the invention, the frequency of the image processor in the electronic equipment can be adjusted in time, which is beneficial for avoiding the lagging phenomenon when the electronic equipment is switched to a scene with a higher image processing capability requirement. The method comprises the steps that a first task is acquired, wherein the first task is a task needing to be processed by a GPU; in response to the first task, obtaining a first duration required for completing the first task and at least one second task, the at least one second task including a task of which the priority is higher than that of the first task in the tasks to be processed by the GPU; under the condition that the first duration is larger than a target duration, the frequency of the GPU is increased, and the target duration comprises the duration needed for processing the first task and the at least one second task under the condition that the GPU meets the task processing requirement.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of terminal, and in particular, to a frequency adjustment method of an image processor, an electronic device, a chip system, a storage medium and a program product. BACKGROUND

[0002] With the development of electronic device technology, the supported functions of electronic devices are increasing, and the requirements for display capabilities of electronic devices are gradually increasing.

[0003] In some scenarios, the content displayed by the display interface of the electronic device is more detailed and complex, such as a game interface, a video interface under high resolution, and the like. Such scenarios have higher requirements for the image processing capability of the electronic device, and the electronic device may have a lag phenomenon when switching to such an interface, affecting the user experience. SUMMARY

[0004] The embodiments of the present application provide a frequency adjustment method of an image processor, an electronic device, a chip system, a storage medium and a program product, which can timely adjust the frequency of the image processor in the electronic device, and is beneficial to avoid the lag phenomenon when the electronic device switches to a scenario with high requirements for image processing capability.

[0005] In a first aspect, the embodiments of the present application provide a frequency adjustment method of an image processor, applied to an electronic device including a GPU. The method comprises: obtaining a first task, the first task being a task that needs to be processed by the GPU; in response to the first task, obtaining a first time length required for completing the first task and at least one second task, the at least one second task including a task with a higher priority than the first task in the tasks to be processed by the GPU; and in a case where the first time length is greater than a target time length, increasing the frequency of the GPU, the target time length including a time length required for the GPU to complete the first task and the at least one second task under the condition that the task processing demand is met.

[0006] Optionally, the first task and the at least one second task can be tasks related to image processing, such as an image rendering task (which can be understood as having a similar meaning to the rendering drawing task indicated by the description of the drawing instruction 1), an image synthesis task (which can be understood as having a similar meaning to the synthesis drawing task indicated by the description of the drawing instruction 2), and the like, or a task for accelerating the analysis and processing of data flow such as deep learning, artificial intelligence (AI) calculation, or other arbitrary GPU supported tasks, which are not limited by the present application.

[0007] In the embodiments of the present application, the electronic device obtains a first time length required for completing a first task and at least one second task in response to the first task, and the at least one second task includes a task with a higher priority than the first task in the tasks to be processed by the GPU, which means that the time length required for the GPU to obtain the first task and complete the first task is the first time length. If the first time length is greater than the target time length, it means that the GPU cannot complete the task within the set time, which indicates that the frequency of the GPU does not meet the task processing requirement. Therefore, in the case where the first time length is greater than the target time length, the frequency of the GPU is increased so that the first task can be completed within the target time length, thereby improving the processing efficiency of the GPU. The method of the embodiments of the present application can avoid waiting for the GPU usage rate to maintain a preset threshold (for example, 90%) for a period of time, which is beneficial to timely adjusting the GPU frequency and avoiding the phenomenon of lag when the electronic device switches to a scene with high image processing requirement.

[0008] Further, the electronic device can execute the method of the present application for each task obtained, which is beneficial to adjusting the GPU frequency for the scene that needs to be frequency-boosted, and is beneficial to avoiding the waste of energy consumption caused by blind and unified frequency-boosting.

[0009] The target time length can be understood as a specified time length set by the electronic device for a task from generation to completion. If the time length for the electronic device to process a task exceeds this specified time length, the device performance may be affected. For example, if the task related to image processing cannot be completed in time, the screen refresh rate requirement may not be met, causing screen lag; if the task related to data analysis and processing cannot be completed in time, the user's waiting time for the result may be prolonged, resulting in a poor user experience.

[0010] In combination with the first aspect, in some implementations of the first aspect, the method for obtaining the first time length includes: obtaining a first predicted time length required for the GPU to execute the first task, and a predicted time length required for each of the at least one second task based on the first task, the at least one second task, and a first mapping relationship, the first mapping relationship including a mapping relationship between different tasks and their predicted time lengths; and summing the predicted time length required for each of the at least one second task and the first predicted time length to obtain the first time length.

[0011] In the embodiments of the present application, the first mapping relationship can be constructed based on known different tasks and their predicted time lengths. When the output corresponding to a specific input needs to be queried, the mapping relationship-based lookup method almost does not require calculation time, has extremely fast response speed, and consumes low calculation resources, which is beneficial to obtaining the result more quickly.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the first mapping relationship is obtained at the lowest frequency of the GPU, and the method also includes: adjusting the frequency of the GPU to the lowest value when the first duration is less than or equal to the target duration.

[0013] In this embodiment of the application, the first mapping relationship is obtained at the lowest frequency of the GPU. The first duration is less than or equal to the target duration, which means that the lowest frequency of the GPU can meet the task processing requirements. Adjusting the frequency of the GPU to the lowest value is beneficial to further save the energy consumption of electronic devices while meeting the task processing requirements.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, before increasing the GPU frequency, the method further includes: based on the target duration and the number of tasks to be processed by the GPU, obtaining the runtime of each task to be processed by the GPU under the second duration, wherein the second duration is less than or equal to the target duration; and based on the runtime, finding a second mapping relationship to obtain the increased target GPU frequency, wherein the second mapping relationship includes the mapping relationship between GPU frequency and runtime obtained by the GPU running the same task at different GPU frequencies.

[0015] In this embodiment, the second mapping relationship can be constructed based on the GPU frequency and runtime obtained from experiments of running the same task at different GPU frequencies. When it is necessary to query the GPU frequency corresponding to a certain runtime, the lookup method based on the mapping relationship requires almost no computation time, has an extremely fast response speed, and consumes low computational resources, which is conducive to obtaining results more quickly. Furthermore, obtaining results more quickly is conducive to electronic devices completing the frequency adjustment process quickly, and is conducive to timely adjustment of GPU frequency. For example, when the GPU is performing tasks related to image processing, it is conducive to avoiding electronic device lag caused by a sudden increase in GPU load, failure to adjust GPU frequency in time, or failure of GPU frequency to meet the screen refresh rate requirements, which is conducive to improving the user experience.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the first task is an image compositing task, at least one second task includes an image compositing task whose task acquisition time is prior to the first task in the task to be processed by the GPU, and the electronic device includes a display compositing system SurfaceFlinger, which is used to perform image compositing using the GPU, and the target duration is the working duration of SurfaceFlinger.

[0017] In some implementations, image compositing tasks have higher priority than image rendering tasks. If the first task is image compositing, then its priority is higher than that of image rendering tasks in the GPU's processing list. The time required for image rendering does not affect the time taken by the first task from its inception to completion, and can be disregarded. Furthermore, the SurfaceFlinger's runtime defines the time from the generation of the image compositing task to its display on the screen. When the first task is image compositing, the SurfaceFlinger's runtime, which specifies the time limit for image compositing, can be used as the target runtime. This allows for accurate assessment of whether the first runtime meets the task's processing requirements, facilitating precise adjustments to the GPU frequency.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the first task is an image rendering task, the image compositing task has a higher priority than the image rendering task, and at least one second task includes an image compositing task to be processed by the GPU and a task in the image rendering task to be processed by the GPU whose task acquisition time is before the first task, and the target duration is the working duration of the application.

[0019] It should be understood that the working time of the application is similar in meaning to the working time of the application described in method 200.

[0020] When the first task is an image rendering task, the processing order of tasks acquired before the first task in both the image compositing task to be processed by the GPU and the image rendering task to be processed by the GPU will affect the time from the generation to the completion of the first task. The working time of the application specifies the time requirement from the generation of the image rendering task to its display on the screen, including the time required to complete the first task and the working time of SurfaceFlinger. Using the working time of the application as the target time in this case to judge whether the first time can meet the task processing requirements is more reasonable, which is conducive to obtaining accurate judgment results and accurate adjustment of the GPU frequency.

[0021] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: determining whether the third duration is greater than the working duration of SurfaceFlinger, the third duration being the time required for the GPU to complete the image synthesis task to be processed by the GPU, the third duration being obtained using the image synthesis task to be processed by the GPU and the first mapping relationship; increasing the frequency of the GPU if the first duration is greater than the target duration, including: increasing the frequency of the GPU if the third duration is greater than the working duration of SurfaceFlinger, and / or if the first duration is greater than the working duration of the application.

[0022] The method provided in this application uses the working time of SurfaceFlinger to determine whether the third duration meets the time limit requirements of the image compositing task, and uses the working time of the application to determine whether the first duration meets the time limit requirements of the first task from generation of the first task, the compositing of the image corresponding to the first task, and then display on the screen. In cases where the third duration is longer than the working time of SurfaceFlinger, and / or the first duration is longer than the working time of the application, the GPU frequency is increased. This is beneficial for accurately adjusting the GPU frequency, and for ensuring that the adjusted GPU frequency simultaneously meets the time limit requirements of the image compositing task from generation to display on the screen as specified by the working time of SurfaceFlinger, and the time limit requirements of the image rendering task from generation to display on the screen as specified by the working time of the application.

[0023] In conjunction with the first aspect, in certain implementations of the first aspect, when the third duration is greater than the working duration of SurfaceFlinger, and / or the first duration is greater than the working duration of the application, increasing the GPU frequency includes: when the third duration is greater than the working duration of SurfaceFlinger and the first duration is greater than the working duration of the application, using the maximum value of the first GPU frequency and the second GPU frequency as the increased target GPU frequency; the first GPU frequency is obtained based on the working duration of SurfaceFlinger, the number of image synthesis tasks to be processed by the GPU, and a second mapping relationship, and the second GPU frequency is obtained based on the working duration of the application, the number of tasks to be processed by the GPU, and the second mapping relationship; or, when the third duration is less than or equal to the working duration of SurfaceFlinger and the first duration is greater than the working duration of the application, using the second GPU frequency as the increased target GPU frequency; or, when the third duration is greater than the working duration of SurfaceFlinger and the first duration is less than or equal to the working duration of the application, using the first GPU frequency as the increased target GPU frequency.

[0024] This approach effectively satisfies both the time requirements for image compositing (as defined by SurfaceFlinger's runtime) and image rendering (as defined by the application's runtime). Specific steps can be found in the descriptions and explanations of S204 to S213 of method 200 above. The first GPU frequency can be understood as the target GPU frequency value 1 in method 200, and the second GPU frequency can be understood as the target GPU frequency value 3 in method 200; these details will not be elaborated upon here.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, when the first duration is less than or equal to the target duration, the GPU frequency is adjusted to the minimum value, including: when the third duration is less than or equal to the working duration of SurfaceFlinger and the first duration is less than or equal to the working duration of the application, the GPU frequency is adjusted to the minimum value.

[0026] In this embodiment, the first mapping relationship is obtained based on the lowest value of the GPU frequency. When the third duration is less than or equal to the working duration of SurfaceFlinger and the first duration is less than or equal to the working duration of the application, it means that both the third duration and the first duration obtained based on the first mapping relationship can meet the task processing requirements. In other words, the electronic device can meet the task processing requirements by running at the lowest value of the GPU frequency. In this case, adjusting the GPU frequency to the lowest value is beneficial to saving the energy consumption of the electronic device.

[0027] In one possible implementation, if the third duration is less than or equal to the working duration of SurfaceFlinger and the first duration is less than or equal to the working duration of the application, the GPU frequency may not be adjusted to avoid the resource consumption caused by frequent GPU frequency adjustments.

[0028] Secondly, embodiments of this application provide a frequency adjustment device for an image processor. This frequency adjustment device can be an electronic device, or a chip or chip system within an electronic device. The frequency adjustment device can include a display unit and a processing unit. When the frequency adjustment device is an electronic device, the display unit can be a display screen. The display unit is used to perform display steps to enable the electronic device to implement the frequency adjustment method for an image processor described in the first aspect or any possible implementation of the first aspect. When the frequency adjustment device is an electronic device, the processing unit can be an image processor. The frequency adjustment device can also include a storage unit, which can be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to enable the electronic device to implement the frequency adjustment method for an image processor described in the first aspect or any possible implementation of the first aspect. When the frequency adjustment device is a chip or chip system within an electronic device, the processing unit can be a processor. The processing unit executes the instructions stored in the storage unit to enable the electronic device to implement the frequency adjustment method for an image processor described in the first aspect or any possible implementation of the first aspect. The storage unit can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0029] Thirdly, embodiments of this application provide an electronic device including one or more processors and a memory, the memory being coupled to one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, and one or more processors calling the computer instructions to cause the electronic device to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0030] Fourthly, embodiments of this application provide a computer-readable storage medium, which includes a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0031] Fifthly, embodiments of this application provide a computer program product, which includes computer program code. When the computer program code is run on a computer, it causes the computer to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0032] Sixthly, this application provides a chip or chip system including one or more processors and a communication interface. The communication interface and the one or more processors are interconnected via a circuit. The one or more processors are used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0033] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0034] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0035] Figure 1 A schematic diagram of the software structure of the electronic device provided in the embodiments of this application;

[0036] Figure 2 An exemplary flowchart of a frequency adjustment method for an image processor provided in an embodiment of this application;

[0037] Figure 3 A schematic diagram of the mapping relationships included in the first mapping table provided for embodiments of this application;

[0038] Figure 4 A schematic diagram of the mapping relationships included in the third mapping table provided for embodiments of this application;

[0039] Figure 5 An exemplary flowchart of a frequency adjustment method for an image processor provided in an embodiment of this application;

[0040] Figure 6 A schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application;

[0041] Figure 7 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0042] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0043] 1. Terminology

[0044] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0045] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0046] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0047] 2. Electronic equipment

[0048] The electronic devices in this application embodiment may include handheld devices with display functions, vehicle-mounted devices, etc. For example, some electronic devices include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving vehicles, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, in-vehicle devices, wearable devices, electronic devices in 5G networks, or future evolution of public land mobile communication networks. The embodiments of this application do not limit the scope of electronic devices in a network (PLMN).

[0049] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific type of application function and require the use of other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0050] Furthermore, in this application embodiment, the electronic device can also be an electronic device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0051] The electronic device in this application embodiment may also be referred to as: electronic device, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0052] A graphics processing unit (GPU) is a microprocessor in an electronic device that can perform graphics-related computations. In some implementations, electronic devices can use the GPU to render and / or composite layers before displaying the rendered and composited image on a screen. Therefore, the GPU's capabilities can affect the images displayed by an electronic device.

[0053] GPU frequency refers to the frequency at which the GPU performs graphics rendering and compositing tasks. It affects the amount of work the GPU can complete per unit of time. A higher GPU frequency means more tasks can be completed per unit of time, while a lower frequency means fewer tasks can be completed per unit of time. In some implementations, to improve the smoothness of electronic device displays, the screen refresh rate can be increased. However, if the GPU's efficiency in completing rendering and compositing tasks is not as high as the screen refresh rate, meaning the GPU frequency is insufficient to meet the screen refresh rate requirements, users may experience screen stuttering, resulting in a poor user experience.

[0054] To address the aforementioned issues, relevant technologies have proposed GPU scheduling mechanisms, primarily including two types. The first scheduling mechanism works by determining the appropriate GPU frequency level (GPU_freq) based on the GPU utilization rate (GPU_busy) over a period of time. If GPU_busy remains above a preset threshold (e.g., 90%) for a sustained period, it can be assumed that the current GPU frequency is insufficient to complete the current task, thus allowing for gradual frequency increases based on the preset frequency level. The second scheduling mechanism provides an interface U for adjusting the GPU frequency, enabling targeted frequency increases for specific scenarios.

[0055] The researchers in this application discovered that the GPU scheduling mechanism of the relevant technology may have the following problems.

[0056] Question 1: Regarding the first scheduling mechanism, electronic devices may not schedule GPU frequency in a timely manner. This delay in GPU frequency scheduling may occur in the following situations:

[0057] Scenario 1: Based on the GPU scheduling mechanism in related technologies, electronic devices will only gradually increase the GPU frequency after the GPU_busy level has been above a preset threshold for a period of time. Therefore, in some scenarios where the GPU load suddenly increases, such as when an electronic device switches to a scenario with higher image processing requirements, the electronic device will not increase the GPU frequency because the GPU_busy level has not been above the preset threshold for a period of time. This means that the situation where the GPU load has increased but the GPU frequency has not increased will continue for a period of time before the GPU frequency can be increased. However, during this period, the processed image frames may not be delivered to the screen in time, and the user may experience screen lag.

[0058] Scenario 2: GPU_busy refers to the GPU utilization rate within the target time interval. A sudden increase in GPU load may prevent some subsequent images from being sent to the screen in time, but the GPU is unaware of this. The calculation of GPU_busy may utilize the GPU utilization rate under this situation, which may create the illusion that GPU_busy is below the preset threshold. In this case, since GPU_busy is always below the preset threshold, electronic devices will not increase the GPU frequency, and the display interface of electronic devices may experience continuous stuttering.

[0059] Scenario 3: For the first frame that can cause an increase in GPU load, since the GPU frequency can only be increased after GPU_busy has been above the preset threshold for a period of time, by the time the GPU frequency is obtained, the frame may have already entered the display process, which may cause the frame to stutter due to the limited GPU capabilities.

[0060] Question 2: Regarding the second scheduling mechanism, electronic device manufacturers typically use the aforementioned interface U to increase the GPU frequency in known scenarios that would increase GPU load. However, for unknown scenarios, they can only uniformly adjust the GPU frequency to a safe value as a fallback. This approach may cover scenarios where frequency increases are unnecessary, leading to power consumption issues for electronic devices.

[0061] In summary, the GPU scheduling mechanisms proposed by related technologies have problems such as untimely GPU frequency increases and high device power consumption.

[0062] In view of this, this application provides a method for adjusting the frequency of an image processor, an electronic device, a chip system, a storage medium, and a program product. The electronic device can respond to a first task requiring GPU processing, and obtain a first duration required to complete the first task and at least one second task. The at least one second task includes tasks with higher priority than the first task among those to be processed by the GPU. This means that the first duration is the time required for the GPU to complete the first task from receiving it. If the first duration is longer than a target duration, it indicates that the GPU cannot complete the task within the set time, meaning the GPU frequency does not meet the task processing requirements. Therefore, when the first duration is longer than the target duration, increasing the GPU frequency improves GPU processing efficiency, which helps ensure the first task can be completed within the target duration. The method in this application's embodiments does not require waiting for the GPU utilization rate to remain at a preset threshold (e.g., 90%) for a period of time, facilitating timely adjustment of the GPU frequency. When the electronic device switches to a scenario with higher image processing requirements, it helps avoid screen stuttering. Furthermore, the electronic device can execute the method of this application for each task acquired, which facilitates targeted GPU frequency adjustment for scenarios requiring frequency increases, avoiding the energy waste caused by blindly and uniformly increasing the frequency.

[0063] Optionally, the first task can be an image rendering task (which can be understood as similar to the rendering drawing task indicated by drawing instruction 1 described below), an image compositing task (which can be understood as similar to the compositing drawing task indicated by drawing instruction 2 described below), or a task used to accelerate the analysis and processing of data streams, such as deep learning, artificial intelligence (AI) computing, or any other task supported by the GPU. This application does not limit this.

[0064] The software structure of the electronic device involved in the embodiments of this application will be described first below.

[0065] The software architecture of an electronic device can be a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. The operating system can be Linux, Unix, Android, iOS, or Windows, etc. This application uses a layered Android system as an example to illustrate the software architecture of the electronic device.

[0066] Figure 1 This is a software structure block diagram of an electronic device provided in an embodiment of this application. For example... Figure 1As shown, the layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the system runtime library layer, and the kernel layer.

[0067] The application layer can include a series of application packages. For example... Figure 1 As shown, the application package may include a first application, which may be any system application or third-party application that can be displayed on the screen of an electronic device. The first application may be, for example, an application for camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, SMS, etc.

[0068] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 2 As shown, the application framework layer can include at least a window management service and a view system.

[0069] The Window Manager Service (WMS) can be used to manage window programs, obtain screen size, determine the presence of a status bar, lock the screen, and capture the screen. The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system can be used to build and manage the application's target user interface (UI).

[0070] The system runtime library layer, also known as the native layer, can include at least a rendering module (RenderThread), a compositing module (RenderEngine), a layer compositing service (SurfaceFlinger), and a GPU scheduling center. The rendering module transmits drawing instructions from the view system to the GPU via the GPU driver and sends the buffer addresses of the layers corresponding to the target interface to SurfaceFlinger. The compositing module responds to SurfaceFlinger's instructions and uses the GPU to composite parts of the target interface. The layer compositing service (SurfaceFlinger) completes the drawing task based on the GPU and the Hardware Composer (HWC). The GPU scheduling center adjusts the GPU frequency in response to the GPU latency of the rendering module and / or the compositing module.

[0071] Alternatively, the rendering module can be included in the process of each application and run as a thread of the application. The rendering modules of different applications can be independent of each other.

[0072] The kernel layer is the layer between hardware and software. The kernel layer includes at least the GPU driver, HWC driver, and display driver.

[0073] The electronic device may also include a hardware layer. In this embodiment, the hardware layer may include at least a GPU, HWC, and memory. In this embodiment, the memory includes a buffer for caching layers. The GPU can store the rendered layer in the buffer. SurfaceFlinger can retrieve the GPU-rendered layer from the buffer and then instruct the HWC, or the HWC and the compositing module, to composite the GPU-rendered layer. This embodiment does not specifically limit which layer of the electronic device's software architecture the buffer operates at.

[0074] The following is combined with Figure 1 A brief description of the image processing workflow for electronic devices is provided.

[0075] As one possible scenario, the first application can respond to an operation that changes the display interface by sending instruction 1 to the window management service at the application layer. Instruction 1 can be used to indicate that the first application's layers have been updated, and instruction 1 can contain the first application's layer information. For example, the layer information can include layer identifiers, layer content, etc.

[0076] In some implementations, the window management service can transmit instruction 1 from the first application to the view system, and can also allocate windows for the first application based on instruction 1. The view system can then send drawing instruction 1 to the rendering module of the system runtime layer based on instruction 1. This drawing instruction 1 can include the identifier of the layer of the first application and the identifier of each drawing operation in at least one drawing operation of drawing instruction 1. The view system can also forward the drawing instructions of other applications to their rendering modules when the layers of other applications are updated in the entire user interface (UI) displayed on the screen.

[0077] The rendering module in the first application process can transmit the drawing instructions 1 from the view system to the GPU through the graphics processing unit (GPU) driver. The GPU then renders the layers and stores the rendered layers in a buffer.

[0078] The display compositing system, based on the identifiers of layers in the UI that need updating as indicated by the view system and the buffer addresses obtained from the rendering compositing module, retrieves the layers corresponding to the target interface after GPU rendering (which may include the rendered layers of the first application). The rendered layers are then transmitted to the hardware compositor driver via the hardware compositor driver, which performs compositing on them. However, if the hardware compositor driver determines that its capacity is insufficient to handle the compositing of all layers, it can transmit the layers it cannot handle to the layer compositing service. The layer compositing service transmits these layers, along with the drawing instructions 2 for these layers, to the compositing module. The compositing module then transmits these layers and their drawing instructions 2 to the GPU via the GPU driver, where the GPU combines these layers, for example, to composite interface 1. Further, after obtaining interface 1 from the GPU driver, the hardware compositor driver then drives the hardware compositor to composite interface 1 and the remaining layers to obtain the image frame of the target interface. The hardware compositor driver then transmits this image frame to the display screen via the display driver for display.

[0079] In this embodiment, the rendering module and / or compositing module can identify drawing instructions, obtain the GPU time corresponding to the drawing operation that meets preset conditions, and transmit it to the GPU scheduling center. The GPU scheduling center can adjust the GPU frequency based on the GPU time from the rendering module and / or compositing module.

[0080] The following is in conjunction with the above. Figure 1 The rendering module, compositing module, and GPU scheduling center in this application are discussed in detail, focusing on the module interaction between the image processor frequency adjustment method 200 provided in this embodiment. The process can be described as follows: Figure 2 As shown, the method 200 includes the following steps:

[0081] The rendering module receives drawing instruction 1 from the first application, forwarded by the window management service and the view system, and executes S201. Drawing instruction 1 includes at least one drawing operation of the first type, and includes a correspondence between information of at least one drawing operation and the layer identifier of the first application to indicate the rendering task. The target interface includes layers of the first application.

[0082] The first type of drawing operation can be understood as the basic drawing operation. The first type of drawing operation can include one or more of the following: first blur, dirty area, draw picture, draw arc, draw points, or draw rectangle, etc. The dirty area can be understood as the area that needs to be redrawn.

[0083] S201. The rendering module calls the renderFrameImpl function to traverse at least one drawing operation contained in drawing instruction 1. If there is a first target drawing operation in at least one drawing operation, the first GPU time consumption of the rendering drawing task corresponding to drawing instruction 1 is obtained based on the first mapping table.

[0084] The first GPU time consumption can be understood as the sum of the time of one or more drawing operations with a larger load among at least one drawing operation contained in drawing instruction 1, as predicted by the electronic device. It can also be understood as the time required for the rendering drawing task corresponding to drawing instruction 1, as predicted by the electronic device.

[0085] In a first possible implementation, the first mapping table includes information about the first target drawing operation and its GPU time at a set GPU frequency. The first mapping table can be obtained by setting the GPU frequency of the electronic device to a value, such as a minimum GPU frequency. The electronic device executes all drawing operations of the first type of drawing operation one by one at this GPU frequency, records the GPU time corresponding to each drawing operation, and forms a first mapping table by including the information of the first target drawing operations whose GPU time is greater than or equal to a threshold of 1, and stores it in the electronic device. In this way, the rendering module does not need to send a message to the GPU scheduling center every time. For example, in S202 below, it can send a message to the GPU scheduling center only when the first target drawing operation included in the first mapping table exists in the drawing instruction 1, which helps reduce resource consumption for inter-module transmission.

[0086] Optionally, the information of the first target drawing operation may include the identifier of the first target drawing operation and the parameters of the first target drawing operation. It should be understood that the identifier of the first target drawing operation may be the name, number, etc. of the first target drawing operation, and the parameters of the first target drawing operation may be parameters related to the drawing method, such as the blur radius in a blur operation.

[0087] In some implementations, the mapping relationships in the first mapping table can be mapping curves. Taking the first blur operation in the first type of drawing operation as an example, the electronic device sets the GPU frequency to the minimum value in the frequency range supported by the GPU. The electronic device uses the first blur operation with different blur radii to perform rendering tasks and records the GPU time consumed for each drawing, thus obtaining results such as... Figure 3 The relationship between different blur radii and GPU time for the first blur operation in the rendering environment is shown in the figure. Figure 3 It can be seen that during the rendering process, as the blur radius increases, the GPU rendering time first increases and then decreases. The first mapping table can contain, for example,... Figure 3The mapping curve shown indicates that when the rendering module traverses drawing instruction 1, it finds that the first blur operation in at least one of the drawing operations included in drawing instruction 1 is recorded in the first mapping table. It can further obtain the blur radius corresponding to the blur operation, and based on the mapping curve, it can obtain the GPU time consumption corresponding to the first blur operation.

[0088] Optionally, the number of first target drawing operations can be one or more, and this application does not limit this. When there are multiple first target drawing operations, the first GPU time consumption can be the sum of the GPU times consumption corresponding to the multiple first target drawing operations.

[0089] In a second possible implementation, the first mapping table may contain information about each drawing operation included in the first type of drawing operation and its mapping relationship with the GPU time consumption under a set GPU frequency value. Another possible implementation of S201 may be: the rendering module calls the renderFrameImpl function, iterates through at least one drawing operation included in drawing instruction 1, obtains the GPU time consumption corresponding to each drawing operation included in at least one drawing operation included in drawing instruction 1 based on the first mapping table, and obtains the sum of the GPU time consumption corresponding to each drawing operation included in at least one drawing operation included in drawing instruction 1 as the first GPU time consumption.

[0090] S202, The rendering module transmits the first GPU time consumption and the corresponding identifier 1 to the GPU scheduling center. The identifier 1 is used to indicate the drawing task type corresponding to the first GPU time consumption. Correspondingly, the GPU scheduling center receives the first GPU time consumption and the identifier 1.

[0091] It is worth noting that in the first possible implementation mentioned above, if the rendering module finds that there is no first target drawing operation in at least one of the drawing operations contained in drawing instruction 1, it may not send a message to the GPU scheduling center. This helps to reduce the workload of the GPU scheduling center and improve the efficiency of the entire GPU time prediction process.

[0092] It should also be noted that the GPU scheduling center can maintain three queues, such as queue 1, queue 2, and queue 3. Queue 1 contains the GPU time for each of the unfinished composite rendering tasks; queue 2 contains the GPU time for unfinished desktop launcher rendering tasks; and queue 3 contains the GPU time for each of the unfinished rendering tasks from applications other than the launcher. The task times in queues 1, 2, and 3 can be sorted according to the order in which the GPU receives the tasks.

[0093] It is understandable that the timing and method of obtaining the GPU time consumption of each task in queues 1, 2 and 3 above can be similar to the method of obtaining the first GPU time consumption, and will not be repeated here.

[0094] In some implementations, queue 2 and queue 3 can be merged, but this application does not limit this.

[0095] After receiving the first GPU timeout and identifier 1 from the GPU scheduling center, S203 to S213 are executed.

[0096] The GPU scheduling center receives the first GPU timeout and, based on identifier 1, executes S203 to refresh the queue maintained by the GPU scheduling center.

[0097] S203, the GPU scheduling center identifies the target rendering task type corresponding to the first GPU time consumption indicated by identifier 1, and stores the first GPU time consumption into the queue of the target rendering task type.

[0098] Optionally, the drawing task type may include composite drawing tasks, launcher rendering drawing tasks, and rendering drawing tasks of other applications besides the launcher; the identifier of the composite drawing task may be 0, the identifier of the launcher rendering drawing task may be 1, and the identifier of the rendering drawing task of other applications besides the launcher may be 2, but this application does not limit this.

[0099] If identifier 1 indicates that the first GPU time consumption is for a composite rendering task, the GPU scheduling center stores the first GPU time consumption in queue 1; if identifier 1 indicates that the first GPU time consumption is for a rendering task corresponding to the desktop launcher, the GPU scheduling center stores the first GPU time consumption in queue 2; if identifier 1 indicates that the first GPU time consumption is not for a rendering task corresponding to the desktop launcher, the GPU scheduling center stores the first GPU time consumption in queue 3.

[0100] In this embodiment of the application, for example, when the GPU scheduling center identifies that the specific value of the identifier 1 is 1, it stores the first GPU time consumption in queue 2; when the GPU scheduling center identifies that the specific value of the identifier 1 is 2, it stores the first GPU time consumption in queue 3.

[0101] S204. The GPU scheduling center determines whether the sum of GPU time consumption X in queue 1 is greater than the working time of SurfaceFlinger. If yes, then execute S205-S206 and then execute S207. If no, then execute S207.

[0102] S205, the GPU scheduling center obtains the adjusted target GPU frequency value 1 based on the second mapping table, the sum of GPU time consumption X in queue 1, and the working time of SurfaceFlinger.

[0103] S206. The GPU scheduling center adjusts the GPU frequency parameters in the GPU driver, adjusting the GPU frequency value to the target GPU frequency value of 1.

[0104] Optionally, before executing S206, the GPU scheduling center can also determine whether the current GPU frequency is the same as the target GPU frequency value 1. If they are the same, S206 is skipped and S207 is executed. If they are not the same, S207 is executed after S206.

[0105] S207. The GPU scheduling center determines whether the sum of X and Y is greater than the application's working time, where Y is the sum of GPU time consumed in queue 2; if yes, then execute S208-S209 and then execute S210; if no, then execute S210.

[0106] S208, the GPU scheduling center obtains the adjusted target GPU frequency value 2 based on the second mapping table, the sum of X and Y, and the application's working time.

[0107] If the GPU scheduling center has executed S205 to S206, and the target GPU frequency value 2 obtained in S208 is greater than the target GPU frequency value 1 obtained in S205, the GPU scheduling center can execute S209.

[0108] If the GPU scheduling center executes S204 and the result of S204 is negative, and further executes S207 to S208, then optionally, the GPU scheduling center can determine whether the current GPU frequency is the same as the target GPU frequency value 2. If they are the same, skip S209 and execute S210. If they are not the same, execute S209 and then execute S210. Alternatively, the GPU scheduling center can skip this determination and directly execute S209 and then execute S210.

[0109] S209. The GPU scheduling center modifies the GPU frequency parameters in the GPU driver, adjusting the GPU frequency value to the target GPU frequency value of 2.

[0110] S210: The GPU scheduling center determines whether the sum of X, Y, and Z is greater than the application's working time, where Z is the sum of GPU time consumed in queue 3; if yes, then execute S211 to S212; otherwise, execute S213.

[0111] S211, The GPU scheduling center obtains the adjusted target GPU frequency value 3 based on the second mapping table, the sum of X, Y, and Z, and the application's working time.

[0112] In one implementation, if the GPU scheduling center executes S205-S206 but not S208-S209, and the target GPU frequency value 3 obtained in S211 is greater than the target GPU frequency value 1 obtained in S205, the GPU scheduling center executes S212; otherwise, the GPU frequency is not modified.

[0113] In another implementation, after the GPU scheduling center executes S205-S206 and S208-S209, if the target GPU frequency value 3 is greater than the maximum value of the target GPU frequency value 1 and the target GPU frequency value 2, the GPU scheduling center executes S212; otherwise, the GPU frequency is not modified.

[0114] In another implementation, if the GPU scheduling center does not execute S205-S206, but executes S208-S209, and the target GPU frequency value 3 obtained in S211 is greater than the target GPU frequency value 2 obtained in S208, the GPU scheduling center executes S212; otherwise, the GPU frequency will not be modified.

[0115] S212. The GPU scheduling center modifies the GPU frequency parameters in the GPU driver, adjusting the GPU frequency value to the target GPU frequency value of 3.

[0116] Optionally, steps S206, S209, and S212 can all be optional steps. In one possible implementation, the GPU scheduling center can adjust the GPU frequency to the maximum value among the target GPU frequency value 1, target GPU frequency value 2, or target GPU frequency value 3 obtained from the preceding steps after executing one or more of steps S205, S208, or S211.

[0117] It is worth noting that if the GPU scheduling center determines "no" in both S204 and S207, it executes S213. If the GPU scheduling center determines "no" in some cases, such as in S204 and S207, there are three possible implementations.

[0118] In one implementation, if the GPU scheduling center determines "yes" in S204 and executes S205 to obtain the target GPU frequency value 1, and further determines "yes" in S207 and executes S208 to obtain the target GPU frequency value 2, then if the determination at S210 is "no," the GPU scheduling center adjusts the GPU frequency to the maximum value between the target GPU frequency value 1 and the target GPU frequency value 2. Optionally, if the GPU scheduling center has already modified the GPU frequency in the previous steps, it returns here without making any modifications.

[0119] In another implementation, if the GPU scheduling center determines "yes" in S204 and executes S205 to obtain the target GPU frequency value of 1, and further, if the GPU scheduling center determines "no" in S207, then if the determination in S210 is also "no," the GPU scheduling center will adjust the GPU frequency to the target GPU frequency value of 1. Optionally, if the GPU frequency has already been modified in the previous steps, the process returns here without modification.

[0120] In another implementation, if the GPU scheduling center determines "no" in S204, and further determines "yes" in S207 and executes S208 to obtain the target GPU frequency value of 2, then if the determination at S210 is "no," the GPU scheduling center will adjust the GPU frequency to the target GPU frequency value of 2. Optionally, if the GPU scheduling center has already modified the GPU frequency in previous steps, it will return here without making any modifications.

[0121] S213, the GPU scheduling center adjusts the GPU frequency to the minimum value. This helps save energy consumption of electronic devices.

[0122] In one possible implementation, the rendering module can execute S214 after iterating through drawing instructions 1.

[0123] S214. The rendering module calls the GPU driver's flush commands API to submit drawing commands to the GPU driver.

[0124] S215. The GPU driver uses the GPU to draw the first application layer included in drawing instruction 1 according to drawing instruction 1, and obtains the rendered layer.

[0125] The work-duration of SurfaceFlinger can be understood as the time it takes for SurfaceFlinger to process one frame of image. It can include the time from the start of the compositing and rendering task of SurfaceFlinger to the time it sends the image to the screen. It can also be understood as the time it takes for SurfaceFlinger to use HWC, or use HWC and GPU to composite and render the rendered layers.

[0126] The application's work-duration can be understood as the time from when the application issues a drawing command to when the application's layer is sent to the screen. It can be understood as including the rendering time of the GPU rendering the application's layer, as well as the time SurfaceFlinger takes to composite and draw the rendered layer using HWC, or using HWC and GPU.

[0127] Furthermore, after the GPU obtains the rendered layer, it can store the rendered layer in a buffer. SurfaceFlinger can then retrieve the rendered layer from the buffer and use HWC, or HWC and the GPU, to perform compositing processing on the rendered image. Simultaneously, the GPU driver may also receive drawing instructions from rendering modules of other applications. This means that the tasks to be processed by the GPU may include both rendering and compositing tasks, creating a possibility that rendering and compositing tasks may compete for GPU resources.

[0128] In this embodiment, GPU tasks can be executed serially, and the GPU can follow a certain priority order when executing tasks to ensure display effect. For example, the priority order of GPU execution of different tasks can be: priority of composite rendering tasks > priority of launcher rendering tasks > priority of rendering tasks of other applications besides the launcher, that is, priority of queue 1 > priority of queue 2 > priority of queue 3.

[0129] Therefore, in this embodiment, after the first GPU timeout refreshes the queue maintained by the GPU scheduling center, and the GPU scheduling center stores the first GPU timeout in queue 2, based on the above priority order, the GPU needs to complete all tasks in queue 1 before executing the task corresponding to queue 2. That is, the time required for the GPU to complete the drawing instruction 1 task is at least the sum of X and Y. By judging whether the sum of X and Y exceeds the application's working time, it is determined whether the GPU frequency can meet the performance requirements of the current task. If so, it means that the current GPU frequency does not meet the performance requirements of the current task. The GPU scheduling center determines the GPU frequency that can meet the performance requirements based on the second mapping table and modifies it through the GPU driver. In this case, S204 to S206 and S210 to S212 can all be optional steps. If the GPU scheduling center determines that it is not satisfied after executing S207, it can directly execute S213, but this application does not limit this.

[0130] When the GPU scheduling center stores the first GPU time consumption in queue 3, based on the aforementioned priority order, the GPU needs to execute all tasks corresponding to queue 1 first, then all tasks corresponding to queue 2, before executing the task corresponding to queue 3. That is, the minimum time required for the GPU to execute drawing instruction 1 is the sum of X, Y, and Z. The GPU frequency is determined to meet the performance requirements of the current task by checking if the sum of X, Y, and Z exceeds the application's working time. If so, it means the current GPU frequency does not meet the performance requirements of the current task. The GPU scheduling center determines the GPU frequency that meets the performance requirements based on the second mapping table and modifies it through the GPU driver. In this case, steps S204–S206 and S207–S209 are all optional, and this application does not limit them.

[0131] In one possible implementation, the second mapping table may include a mapping relationship between GPU frequency and GPU time consumption under standard rendering tasks. For example, it may include a mapping curve between GPU time consumption and GPU frequency obtained based on the time consumption of the electronic device when performing standard rendering tasks at different GPU frequencies within the GPU frequency range of the electronic device.

[0132] Optionally, the GPU time-GPU frequency curve for a standard rendering task can be a fitted curve of multiple GPU time-GPU frequency curves obtained from various rendering tasks. These multiple rendering tasks may include rendering tasks and / or compositing tasks, but this application does not limit this.

[0133] In one possible implementation, the GPU scheduling center can record the number of tasks in each of the three queues. As a possible example, the GPU scheduling center executes S204, determines that the sum of GPU execution time X in queue 1 is 12 milliseconds, but the working time of SurfaceFlinger is 10 milliseconds, so it continues to execute S205. The GPU scheduling center records that queue 1 includes 4 tasks. To reduce the sum of GPU execution time in queue 1 to 10 milliseconds, the maximum GPU execution time of each of the 4 tasks can be 2.5 milliseconds. It then looks up the second mapping table and obtains the GPU frequency corresponding to any GPU execution time less than or equal to 2.5 milliseconds as the target GPU frequency value 1.

[0134] As a possible example, the GPU scheduling center executes S207, determines that the sum of GPU time X in queue 1 is 12 milliseconds, the sum of GPU time X in queue 2 is 2 milliseconds, and their sum is 14 milliseconds. The application's working time is 16 milliseconds, so it continues to execute S210. The sum of GPU time X in queue 3 is 6 milliseconds, and the sum of GPU time X in the three queues is 20 milliseconds. However, the application's working time is 16 milliseconds, so the frequency needs to be adjusted. The GPU scheduling center records that queue 1 includes 4 tasks, queue 2 includes 1 task, and queue 3 includes 3 tasks, for a total of 8 drawing tasks. To reduce the sum of GPU time X in queues 1, 2, and 3 to 16 milliseconds, the maximum GPU time X in each of the 8 drawing tasks can be 2 milliseconds. The second mapping table is consulted to obtain the GPU frequency corresponding to any GPU time X less than or equal to 2 milliseconds as the target GPU frequency value 3.

[0135] In this embodiment, after receiving drawing instruction 1 from the application, the rendering module first traverses at least one drawing operation contained in drawing instruction 1. If a first target drawing operation exists among the at least one drawing operation, it obtains the first GPU time consumption of the rendering drawing task corresponding to drawing instruction 1 based on the first mapping table. Then, it transmits the first GPU time consumption to the GPU scheduling center so that the GPU scheduling center can adjust the GPU frequency based on the working time of SurfaceFlinger or the working time of the application, and the pre-stored second mapping relationship, provided that the corresponding GPU frequency adjustment conditions are met. In this process, the time consumption of the lookup method based on the mapping table is relatively short, and the overall execution is faster. Moreover, the rendering module can transmit drawing instruction 1 to the GPU driver after traversing the drawing operations, after executing the above S201, or after executing the above S202. In this way, when the GPU driver uses the GPU to draw the layer of the first application included in drawing instruction 1 according to drawing instruction 1, the GPU frequency may have already been adjusted, or the GPU frequency may be adjusted in the early stage of the GPU driver using the GPU to draw the layer of the first application. In other words, the method provided in this application embodiment has two advantages. First, it allows the GPU frequency to be adjusted before the GPU renders the layer of the first application or at the beginning of the GPU renders the layer of the target interface, without having to wait for the GPU utilization (GPU_busy) to remain at a preset threshold for a period of time, thus enabling timely adjustment of the GPU frequency. Second, in this application embodiment, for each rendering instruction, frequency adjustment can be performed only when the corresponding GPU frequency adjustment conditions are met, which helps reduce the energy consumption of electronic devices caused by GPU frequency adjustment covering scenarios that do not require frequency increase. Furthermore, since the relevant program code can be uniformly maintained by the rendering module and the GPU scheduling center, it also reduces the maintenance costs for electronic device manufacturers.

[0136] Combination Figure 1 The image processing flow of the electronic device shown in the diagram illustrates that after the GPU draws the layers on the target interface and obtains the rendered layers, SurfaceFlinger can use HWC, or HWC and GPU, to perform composite processing on the rendered image. The following describes the process involving a compositing module and the GPU in SurfaceFlinger's ability to use HWC and GPU to composite the rendered image.

[0137] Based on the description of the image processing flow of electronic devices above, it can be seen that if the HWC driver determines that its capacity is insufficient to handle the compositing of all layers of the target interface, the HWC driver can transfer the layers that it cannot handle to the layer compositing service. The layer compositing service then transfers these layers, along with the drawing instructions 2 for these layers, to the compositing module.

[0138] The drawing instruction 2 includes a second type of drawing operation, and a correspondence between the identifier of at least one of the second type of drawing operations and the identifier of the portion of the layer processed by the GPU in the layer of the target interface, to indicate the composite drawing task.

[0139] It should be understood that the second type of drawing instructions is not entirely the same as the first type of drawing operations. The second type of drawing operations can be understood as drawing operations used to optimize the basic drawing operations. The second type of drawing operations can include one or more of the following: second blur, rounded corners, or shading, etc. The difference between the second blur and the first blur may be in the execution method, the required GPU capabilities, etc.

[0140] S216. The compositing module calls the drawLayersInternal function to traverse at least one drawing operation in drawing instruction 2. If there is a second target drawing operation in at least one drawing operation in drawing instruction 2, the second GPU time consumption of the compositing drawing task corresponding to drawing instruction 2 is obtained based on the third mapping table.

[0141] In one possible implementation, the third mapping table includes information about the second target drawing operation and its mapping relationship with its GPU time consumption at a set GPU frequency value. The third mapping table can be obtained as follows: the GPU frequency of the electronic device is set to a value, such as a minimum GPU frequency value. The electronic device executes all drawing operations of the second type of drawing operation one by one at this GPU frequency value, records the GPU time consumption corresponding to each drawing operation, and forms a third mapping table with the information of drawing operations whose GPU time consumption is greater than or equal to a threshold of 2, storing it in the electronic device. The information about the second target drawing operation can include the identifier of the second target drawing operation and the parameters of the second target drawing operation. It should be understood that the identifier of the second target drawing operation can be the name of the second target drawing operation (e.g., a second type of blur operation), its number, etc., and the parameters of the second target drawing operation can be parameters related to the drawing method, such as the blur radius in a second type of blur operation.

[0142] In some implementations, the mapping relationships in the third mapping table can be mapping curves. Taking the second blur operation in the second type of rendering operation as an example, with the GPU frequency fixed at its minimum, the second blur operation is used for composite rendering tasks, and by changing the blur radius, the following can be obtained: Figure 4 The relationship between different blur radii and GPU time consumption for the second blur operation in the compositing environment is shown in the figure. Figure 4 It can be seen that during the composite rendering process, as the blur radius increases, the GPU time consumption first increases, then decreases, then increases again before gradually stabilizing. The third mapping table can contain, for example... Figure 4 As shown in the mapping relationship, when the synthesis module queries the second blur operation while traversing the drawing instruction 2, it obtains the blur radius corresponding to the second blur operation. Based on the third mapping table, it can obtain the GPU time consumption corresponding to the second blur operation.

[0143] Optionally, the number of second target rendering operations can be one or more, and this application does not limit this. When there are multiple second target rendering operations, the second GPU time consumption can be the sum of the GPU times consumption corresponding to the multiple second target rendering operations.

[0144] In another possible implementation, the third mapping table may contain information about each drawing operation included in the second type of drawing operation and its mapping relationship with the GPU time consumption at a set GPU frequency value. The electronic device may traverse at least one drawing operation included in drawing instruction 1 and obtain the GPU time consumption corresponding to each drawing operation in the at least one drawing operation included in drawing instruction 1 based on the second mapping table. In this implementation, the second target drawing operation includes at least one drawing operation included in drawing instruction 1, and the second GPU time consumption is the sum of the GPU time consumption corresponding to each drawing operation in the at least one drawing operation included in drawing instruction 1.

[0145] It is worth noting that the first and third mapping tables may differ due to changes in the rendering environment. One possible manifestation of this is that the same type of drawing operation, such as a blur operation, may have different GPU time consumption in the rendering environment and the compositing environment.

[0146] Optionally, the first mapping table, the second mapping table, and the third mapping table can all be stored in the electronic device. The electronic device manufacturer can update and iterate the first mapping table, the second mapping table, and the third mapping table through system updates or parameter updates to improve the accuracy of the electronic device in adjusting the GPU frequency.

[0147] S217. The synthesis module transmits the second GPU time consumption and the corresponding identifier 2 to the GPU driver; correspondingly, the GPU driver receives the second GPU time consumption and the corresponding identifier 2.

[0148] S218, the GPU scheduling center identifies the target rendering task type corresponding to the second GPU time consumption indicated by identifier 2, and stores the second GPU time consumption into the queue of the target rendering task type.

[0149] Specifically, identifier 2 can indicate that the second GPU time consumption is a compositing rendering task, and the GPU scheduling center can store the second GPU time consumption in queue 1.

[0150] After S218, the GPU scheduling center can continue to execute S219 to S228. S219 to S228 are similar to S204 to S213. The explanation of each step in S219 to S228 and other possible implementations of each step can be found in the descriptions of the steps in S204 to S213 that correspond to S219 to S228. They will not be repeated here.

[0151] In one possible implementation, the synthesis module may execute S229 after S216 or S217.

[0152] S229. The compositing module calls the GPU driver's flush surface API to submit drawing instructions to the GPU driver.

[0153] S230 and GPU driver are based on drawing instruction 2. The GPU is used to draw the layers of the target interface included in drawing instruction 2 and composite the interface 1.

[0154] S231, HWC driver obtains interface 1 from GPU driver.

[0155] Optionally, the HWC driver can obtain interface 1 by transmitting interface 1 from the GPU driver to the HWC driver, or by storing it in a buffer in the GPU driver and then obtaining interface 1 from the buffer based on the buffer address obtained from SurfaceFlinger and the identifiers of the layers included in interface 1. This application does not specifically limit the method by which the HWC driver obtains interface 1.

[0156] S232, the HWC driver combines interface 1 with the layers drawn by HWC in the layers of the target interface to obtain the target interface, and marks the first flag, which is used to indicate the drawing progress, corresponding to the flag of each layer in the target interface as the first value.

[0157] It should be understood that the first value can be any value that can indicate the completion of layer consumption. The first value can be true, false, 0, 1, or any other arbitrary value. This application does not limit this.

[0158] It should be understood that the first flag for any layer, used to indicate the drawing progress, can be accessed globally.

[0159] In one possible implementation, S202 also includes the identifier of the first application's layer. After the GPU driver completes S215, the identifier of the first application's layer can be sent back to the GPU scheduling center so that the GPU scheduling center can delete the first GPU time consumption stored in queue 2 or queue 3 based on the identifier of the first application's layer.

[0160] It should also be understood that the completion of the target interface composition also means that all layers included in the target interface have been consumed. At this time, SurfaceFlinger can transmit the identifiers of the consumed layers to the GPU scheduling center so that the GPU scheduling center can delete the second GPU time consumption stored in queue 1 based on the identifiers of the consumed layers.

[0161] In this embodiment of the application, the display driver can check whether the drawing of the target interface is completed at a set time. When the set time is reached, the display driver executes S226.

[0162] Optionally, this setting time can be related to the operating duration of SurfaceFlinger or the screen refresh rate. For example, if the screen of an electronic device refreshes every 16 milliseconds, then the display driver checks every 10 milliseconds after each display submission, i.e., executes S233.

[0163] S233: The display driver, based on the identifiers of all layers in the target interface from SurfaceFlinger, iterates through the first identifiers corresponding to the identifiers of all layers in the target interface, and determines whether the values ​​of all first identifiers are the first value; if yes, it returns; otherwise, it executes S234.

[0164] S234. The display driver adjusts the GPU frequency value to the maximum value through the GPU driver.

[0165] This allows for further adjustments to the GPU frequency if the GPU frequency adjusted by the GPU scheduling center is still insufficient to meet task requirements. Maximizing the GPU frequency helps to speed up GPU processing efficiency as much as possible and reduces the possibility of screen stuttering.

[0166] Furthermore, combined Figure 1 The image processing flow shown is such that, when the image frame of the target interface can be output, the HWC driver can execute S235, the display driver can receive the image frame of the target interface from the hardware synthesizer driver, and then execute S236.

[0167] The S235 and HWC drivers transmit image frames of the target interface to the display driver, and the display driver receives the image frames of the target interface accordingly.

[0168] S236, The display driver uses a display screen to show the target interface.

[0169] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.

[0170] The following is combined with Figure 5 This application describes a frequency adjustment method 500 for an image processor, which can be applied to electronic devices including an image processor (GPU). Its software structure can be as described above. Figure 1 As shown, the hardware structure can be as follows: Figure 6 As shown.

[0171] The method 500 may include the following steps:

[0172] S501, Obtain the first task, which is the task that needs to be processed by the GPU.

[0173] S502, in response to the first task, obtain the first duration required to complete the first task and at least one second task, wherein the at least one second task includes a task with a higher priority than the first task among the tasks to be processed by the GPU.

[0174] S503. If the first duration is greater than the target duration, increase the GPU frequency. The target duration includes the time required for the GPU to complete the first task and at least one second task while meeting the task processing requirements.

[0175] Optionally, the first task can be a task related to image processing, such as an image rendering task (which can be understood as similar to the rendering drawing task indicated by drawing instruction 1 described above), an image compositing task (which can be understood as similar to the compositing drawing task indicated by drawing instruction 2 described above), or a task used to accelerate the analysis and processing of data streams, such as deep learning, artificial intelligence (AI) computing, or any other task supported by the GPU. This application does not limit this.

[0176] In this embodiment, the electronic device responds to a first task by acquiring a first time required to complete the first task and at least one second task. The at least one second task includes tasks with higher priority than the first task among those to be processed by the GPU. This means that the time required for the GPU to complete the first task from acquisition is the first time. If the first time exceeds a target time, it indicates that the GPU cannot complete the task within the set time, meaning the GPU frequency does not meet the task processing requirements. Therefore, when the first time exceeds the target time, the GPU frequency is increased to ensure the first task can be completed within the target time, thus improving GPU processing efficiency. The method in this embodiment does not require waiting for the GPU utilization rate to remain at a preset threshold (e.g., 90%) for a period of time, facilitating timely adjustment of the GPU frequency. Furthermore, the electronic device can execute the method for each acquired task, allowing for targeted GPU frequency adjustments for scenarios requiring frequency increases, thus avoiding energy waste caused by blindly and uniformly increasing the frequency.

[0177] Furthermore, the target duration can be understood as the specified time from the inception to the completion of a task set by an electronic device. If the electronic device takes longer to process a task than this specified time, it may affect the device's performance. For example, if tasks related to image processing cannot be completed in time, they may not meet the screen refresh rate requirements, causing screen stuttering; if tasks related to data analysis and processing cannot be completed in time, they may prolong the user's waiting time for results, giving the user a poor user experience.

[0178] Optionally, the electronic device may include a rendering module, a compositing module, and a GPU scheduling center. If the first task is an image rendering task, the rendering module may be used to execute the above-mentioned S501 and S502. If the first task is an image compositing task, the compositing module may be used to execute the above-mentioned S501 and S502. The GPU scheduling center may execute S503 in response to a first duration from the rendering module or the compositing module.

[0179] In one possible implementation, the first processing time can be obtained from a pre-trained neural network. The training set of the pre-trained neural network model can be multiple GPU tasks and their processing times at a certain set frequency. By inputting the first task and at least one second task into the pre-trained neural network model, the processing time required by the GPU for the first task and at least one second task at that set frequency can be predicted. Optionally, the set frequency can be the lowest frequency of the GPU, but this application does not limit it.

[0180] In another possible implementation, the method for obtaining the first duration includes: based on the first task, at least one second task, and a first mapping relationship, obtaining the first predicted duration required by the GPU to execute the first task, and the predicted duration required by the GPU to execute each of the at least one second task, wherein the first mapping relationship includes the mapping relationship between different tasks and their predicted durations; and summing the predicted duration required by each of the at least one second task with the first predicted duration to obtain the first duration.

[0181] The first mapping relationship may include the first mapping table and the third mapping table mentioned above. The method of obtaining it can be similar to the method of obtaining the first mapping table or the third mapping table mentioned above, and will not be repeated here.

[0182] In this embodiment of the application, the first mapping relationship can be constructed based on known different tasks and their predicted durations. When it is necessary to query the predicted duration corresponding to any task, the search method based on the mapping relationship requires almost no computation time, has an extremely fast response speed, and consumes low computational resources, which is conducive to obtaining results more quickly.

[0183] As an optional embodiment, the first mapping relationship is obtained at the lowest frequency of the GPU, and the method further includes: adjusting the frequency of the GPU to the lowest value if the first duration is less than or equal to the target duration.

[0184] In this embodiment of the application, the first mapping relationship is obtained at the lowest frequency of the GPU. The first duration is less than or equal to the target duration, which means that the lowest frequency of the GPU can meet the task processing requirements. Adjusting the frequency of the GPU to the lowest value is beneficial to further save the energy consumption of electronic devices while meeting the task processing requirements.

[0185] As an optional embodiment, before increasing the GPU frequency, the method further includes: obtaining the runtime of each task in the tasks to be processed by the GPU under a second duration based on the target duration and the number of tasks to be processed by the GPU, wherein the second duration is less than or equal to the target duration; and finding a second mapping relationship based on the runtime to obtain the increased target GPU frequency, wherein the second mapping relationship includes the mapping relationship between GPU frequency and runtime obtained by the GPU running the same task at different GPU frequencies.

[0186] The second mapping relationship may include the second mapping table mentioned above, and its acquisition method can be similar to that of the second mapping table, which will not be elaborated here.

[0187] In some implementations, the runtime of each task in the tasks to be processed by the GPU can be the quotient of the second runtime and the number of tasks to be processed by the GPU. Based on this runtime, by looking up the second mapping table, the GPU frequency corresponding to this runtime can be obtained. In other words, if the GPU can complete each task in the tasks to be processed by the GPU at this GPU frequency within this runtime, then the tasks to be processed by the GPU can also be completed within the second runtime. This GPU frequency is the target GPU frequency.

[0188] In this embodiment of the application, the second mapping relationship can be constructed based on the GPU frequency and runtime obtained by the GPU running the same task at different GPU frequencies. When it is necessary to query the GPU frequency corresponding to a certain runtime, the lookup method based on the mapping relationship requires almost no computation time, has a very fast response speed, and consumes low computational resources, which is conducive to obtaining results more quickly.

[0189] Furthermore, obtaining results more quickly helps electronic devices complete the frequency adjustment process more quickly, and facilitates timely adjustment of GPU frequency. For example, when the GPU is performing tasks related to image processing, it helps avoid electronic device lag caused by a sudden increase in GPU load, failure to adjust GPU frequency in time, or failure of GPU frequency to meet screen refresh rate requirements, thus improving the user experience.

[0190] As an optional embodiment, the first task is an image compositing task, and at least one second task includes an image compositing task whose task acquisition time is prior to the first task among the tasks to be processed by the GPU. The electronic device includes a display compositing system SurfaceFlinger, which is used to perform image compositing using the GPU, and the target duration is the working duration of SurfaceFlinger.

[0191] In some implementations, image compositing tasks have higher priority than image rendering tasks. If the first task is image compositing, then its priority is higher than that of image rendering tasks in the GPU's processing list. The time required for image rendering does not affect the time taken by the first task from its inception to completion, and can be disregarded. Furthermore, the SurfaceFlinger's runtime defines the time from the generation of the image compositing task to its display on the screen. When the first task is image compositing, the SurfaceFlinger's runtime, which specifies the time limit for image compositing, can be used as the target runtime. This allows for accurate assessment of whether the first runtime meets the task's processing requirements, facilitating precise adjustments to the GPU frequency.

[0192] As an optional embodiment, the first task is an image rendering task, the image compositing task has a higher priority than the image rendering task, and at least one second task includes the image compositing task to be processed by the GPU and the task in the image rendering task to be processed by the GPU whose task acquisition time is before the first task, and the target duration is the working duration of the application.

[0193] It should be understood that the application's working time is similar in meaning to the application's working time described above.

[0194] In some implementations, when the first task is an image rendering task, the processing order of tasks whose acquisition time is before the first task, such as the image compositing task to be processed by the GPU and the image rendering task to be processed by the GPU, will affect the time from the generation to the completion of the first task. The working time of the application specifies the time requirement from the generation of the image rendering task to its display on the screen, including the time required to complete the first task and the working time of SurfaceFlinger. Using the working time of the application as the target time in this case to judge whether the first time can meet the task processing requirements is more reasonable, which is conducive to obtaining accurate judgment results and accurate adjustment of GPU frequency.

[0195] As an optional embodiment, the method further includes: determining whether a third duration is greater than the working duration of SurfaceFlinger, the third duration being the time required for the GPU to complete the image synthesis task to be processed by the GPU, the third duration being obtained using the image synthesis task to be processed by the GPU and a first mapping relationship; increasing the frequency of the GPU if the first duration is greater than the target duration, including: increasing the frequency of the GPU if the third duration is greater than the working duration of SurfaceFlinger, and / or if the first duration is greater than the working duration of the application.

[0196] Optionally, the time required for the GPU image synthesis task to be processed as described herein can be the sum of GPU time consumption X in queue 1 as described in method 200. The first time described herein can be the sum of X, Y, and Z as described in method 200, where Y is the sum of GPU time consumption in queue 2 and Z is the sum of GPU time consumption in queue 3.

[0197] It should be understood that the method for obtaining the third duration can be similar to that for obtaining the first duration, and will not be elaborated further.

[0198] The method provided in this application uses the working time of SurfaceFlinger to determine whether the third duration meets the time limit requirements of the image compositing task, and uses the working time of the application to determine whether the first duration meets the time limit requirements of the first task from generation of the first task, the compositing of the image corresponding to the first task, and then display on the screen. In cases where the third duration is longer than the working time of SurfaceFlinger, and / or the first duration is longer than the working time of the application, the GPU frequency is increased. This is beneficial for accurately adjusting the GPU frequency, and for ensuring that the adjusted GPU frequency simultaneously meets the time limit requirements of the image compositing task from generation to display on the screen as specified by the working time of SurfaceFlinger, and the time limit requirements of the image rendering task from generation to display on the screen as specified by the working time of the application.

[0199] Optionally, when the electronic device obtains a fourth task of image compositing type, the GPU adjustment process in response to the fourth task can be similar to the implementation process when the electronic device obtains a first task of rendering compositing type. In this way, the electronic device can use the same program code to adjust the GPU frequency regardless of the type of task it obtains, which helps to simplify the execution process of the electronic device.

[0200] In one possible implementation, the first task is an image rendering task. When the electronic device receives a fourth task of type image compositing, the GPU adjustment process in response to the fourth task may be as follows: In response to the fourth task, determine whether the fourth duration is greater than the working duration of SurfaceFlinger. The fourth duration is the time required for the GPU to complete the image compositing task to be processed by the GPU. The fourth duration is obtained using the image compositing task to be processed by the GPU and the first mapping relationship. The fourth duration includes the time required to complete the fourth task but does not include the time required to complete the first task. Determine whether the first duration is greater than the target duration. The first duration includes the time required to complete the first task. If the fourth duration is greater than the working duration of SurfaceFlinger and / or the first duration is greater than the working duration of the application, increase the GPU frequency.

[0201] As an optional embodiment, increasing the GPU frequency when the third duration is greater than the SurfaceFlinger's working duration and / or the first duration is greater than the application's working duration includes: when the third duration is greater than the SurfaceFlinger's working duration and the first duration is greater than the application's working duration, using the maximum value of the first GPU frequency and the second GPU frequency as the increased target GPU frequency; the first GPU frequency is obtained based on the SurfaceFlinger's working duration, the number of image synthesis tasks to be processed by the GPU, and a second mapping relationship, and the second GPU frequency is obtained based on the application's working duration, the number of tasks to be processed by the GPU, and the second mapping relationship; or, when the third duration is less than or equal to the SurfaceFlinger's working duration and the first duration is greater than the application's working duration, using the second GPU frequency as the increased target GPU frequency; or, when the third duration is greater than the SurfaceFlinger's working duration and the first duration is less than or equal to the application's working duration, using the first GPU frequency as the increased target GPU frequency.

[0202] This approach effectively satisfies both the time requirements for image compositing (as defined by SurfaceFlinger's runtime) and image rendering (as defined by the application's runtime). Specific steps can be found in the descriptions and explanations of S204 to S213 of method 200 above. The first GPU frequency can be understood as the target GPU frequency value 1 in method 200, and the second GPU frequency can be understood as the target GPU frequency value 3 in method 200; these details will not be elaborated upon here.

[0203] As an optional embodiment, when the first duration is less than or equal to the target duration, the GPU frequency is adjusted to the minimum value, including: when the third duration is less than or equal to the working duration of SurfaceFlinger and the first duration is less than or equal to the working duration of the application, the GPU frequency is adjusted to the minimum value.

[0204] The first mapping relationship is obtained based on the lowest value of the GPU frequency. If the third duration is less than or equal to the working duration of SurfaceFlinger and the first duration is less than or equal to the working duration of the application, it means that both the third duration and the first duration obtained based on the first mapping relationship can meet the task processing requirements. In other words, the electronic device can meet the task processing requirements by running at the lowest value of the GPU frequency. In this case, adjusting the GPU frequency to the lowest value is beneficial to saving the energy consumption of the electronic device.

[0205] In one possible implementation, if the third duration is less than or equal to the working duration of SurfaceFlinger and the first duration is less than or equal to the working duration of the application, the GPU frequency may not be adjusted to avoid the resource consumption caused by frequent GPU frequency adjustments.

[0206] As an optional embodiment, the electronic device includes a display driver. During image processing by the GPU, if the display driver determines that the next image frame to be displayed on the screen (or the current image frame being processed by the GPU) has not yet been completed after a preset time has elapsed, it adjusts the GPU frequency to its maximum value to accelerate GPU processing efficiency. The specific implementation can be found in the description of S233 to S234 of method 200 above.

[0207] The frequency adjustment method for an image processor according to an embodiment of this application has been described above. The apparatus for performing the above method, provided in the embodiments of this application, is described below. Those skilled in the art will understand that the methods and apparatus can be combined with and referenced in each other, and the related apparatus provided in the embodiments of this application can perform the steps in the above method.

[0208] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application is shown. For example... Figure 6 As shown, the electronic device may include a processor 610, a memory 620, a display screen 630, a sensor module 640, a universal serial bus (USB) interface 650, a charging management module 660, a power management module 661, a battery 662, an image processor 670, and a hardware synthesizer 680, etc.

[0209] Processor 610 may include one or more processing units, such as a central processing unit (CPU), application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). These different processing units may be independent devices or integrated into one or more processors.

[0210] The memory 620 can be used to store computer executable program code, which includes instructions, such as the program code corresponding to the method described in the embodiments of this application.

[0211] Display screen 630 is used to display images, videos, etc. Display screen 630 includes a display panel. In some embodiments, the electronic device may include one or N displays screens 630, where N is a positive integer greater than 1. The electronic device implements display functions through a GPU, display screen 630, and application processor, etc. In some implementations, the display screen can communicate with a touch sensor to collect signals generated by the user's touch operations.

[0212] The image processor 670 is a processor used to handle graphics and image-related tasks, accelerating computational tasks such as graphics rendering, image processing, computer vision, and graphics effects. In this embodiment, the image processor is used to perform rendering tasks and to cooperate with a hardware compositor to perform image compositing tasks.

[0213] The hardware compositor 680 can be used to composite multiple surfaces onto a key component of a screen. In this embodiment, the hardware compositor can be used to perform image compositing and rendering tasks.

[0214] The image processor frequency adjustment method provided in this application embodiment can be applied to electronic devices containing an image processor, and the software structure of the electronic device can be as follows: Figure 1 As shown, the hardware structure can be as follows Figure 6 As shown, the specific form of the electronic device can be referred to the above description, and will not be repeated here.

[0215] Figure 7 An exemplary schematic diagram of a chip structure provided in an embodiment is shown. The chip 700 includes one or more (including two) processors 701, communication lines 702, communication interfaces 703, and memory 704.

[0216] In some implementations, memory 704 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof.

[0217] The methods described in the embodiments of this application can be applied to processor 701, or implemented by processor 701. Processor 701 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuit in the hardware of processor 701 or by instructions in software form. The processor 701 may be a general-purpose processor (e.g., a microprocessor or conventional processor), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components. Processor 701 can implement or execute the various processing-related methods, steps, and logic block diagrams disclosed in the embodiments of this application.

[0218] The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in mature storage media in the art, such as random access memory, read-only memory, programmable read-only memory, or electrically erasable programmable read-only memory (EEPROM). This storage medium is located in memory 704, and processor 701 reads the information in memory 704 and, in conjunction with its hardware, completes the steps of the above method.

[0219] The processor 701, memory 704 and communication interface 703 can communicate with each other through communication line 702.

[0220] In the above embodiments, the instructions stored in the memory for execution by the processor can be implemented in the form of a computer program product. This computer program product can be pre-written into the memory, or it can be downloaded and installed into the memory as software.

[0221] In the embodiments of this application, the chip 700 may also be a chip system, such as a system on chip (SOC), and this application does not limit it in this way.

[0222] This application provides an electronic device, which includes: one or more processors and a memory; the memory is coupled to one or more processors, and stores computer program code, including computer instructions; one or more processors invoke the computer instructions to cause the electronic device to perform the methods described in the above embodiments.

[0223] This application provides a chip or chip system. The chip or chip system is applied to an electronic device and includes one or more processors. These processors invoke computer instructions to cause the electronic device to execute the methods described in the above embodiments. Its implementation principle and technical effects are similar to the related embodiments described above, and will not be repeated here.

[0224] This application also provides a computer-readable storage medium. The computer-readable storage medium includes computer instructions. When these computer instructions are executed on an electronic device, they cause the electronic device to perform the methods described in the above embodiments. The methods described in the above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted on the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0225] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data using laser optics. Combinations of the above should also be included within the scope of computer-readable media.

[0226] This application provides a computer program product, which includes computer program code. When the computer program code is run on an electronic device, the electronic device performs the method described in the above embodiments.

[0227] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0228] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A frequency adjustment method for an image processor, characterized in that, Electronic devices that include graphics processing units (GPUs) include: Obtain the first task, which is the task that needs to be processed by the GPU; In response to the first task, a first duration required to complete the first task and at least one second task is obtained, wherein the at least one second task includes a task with a higher priority than the first task among the tasks to be processed by the GPU. If the first duration exceeds the target duration, the frequency of the GPU is increased. The target duration includes the time required for the GPU to complete the first task and at least one second task while meeting the task processing requirements.

2. The method according to claim 1, characterized in that, The method for obtaining the first duration includes: Based on the first task, the at least one second task, and the first mapping relationship, the first prediction time required for the GPU to execute the first task and the prediction time required for the GPU to execute each of the at least one second task are obtained. The first mapping relationship includes the mapping relationship between different tasks and their prediction times. The first duration is obtained by summing the prediction duration required for each of the at least one second task with the first prediction duration.

3. The method according to claim 2, characterized in that, The first mapping relationship is obtained at the lowest frequency of the GPU, and the method further includes: If the first duration is less than or equal to the target duration, the frequency of the GPU is adjusted to the lowest value.

4. The method according to any one of claims 1 to 3, characterized in that, Before increasing the frequency of the GPU, the method further includes: Based on the target duration and the number of tasks to be processed by the GPU, the runtime of each task in the tasks to be processed by the GPU under the second duration is obtained, wherein the second duration is less than or equal to the target duration; Based on the runtime, a second mapping relationship is found to obtain the increased target GPU frequency. The second mapping relationship includes the mapping relationship between GPU frequency and runtime obtained by running the same task at different GPU frequencies.

5. The method according to any one of claims 1 to 4, characterized in that, The first task is an image compositing task, and the at least one second task includes an image compositing task whose task acquisition time is before the first task among the tasks to be processed by the GPU. The electronic device includes a display compositing system SurfaceFlinger, which is used to perform image compositing using the GPU. The target duration is the working duration of SurfaceFlinger.

6. The method according to any one of claims 1 to 4, characterized in that, The first task is an image rendering task, and the image compositing task has a higher priority than the image rendering task. The at least one second task includes the image compositing task to be processed by the GPU and the task whose task acquisition time is before the first task in the image rendering task to be processed by the GPU. The target duration is the working duration of the application.

7. The method according to claim 6, characterized in that, The method further includes: Determine whether the third duration is greater than the working duration of SurfaceFlinger. The third duration is the time required for the GPU to complete the image synthesis task to be processed by the GPU. The third duration is obtained by using the image synthesis task to be processed by the GPU and the first mapping relationship. The step of increasing the GPU frequency when the first duration exceeds the target duration includes: If the third duration exceeds the working duration of SurfaceFlinger, and / or the first duration exceeds the working duration of the application, increase the frequency of the GPU.

8. The method according to claim 7, characterized in that, In cases where the third duration exceeds the SurfaceFlinger's runtime, and / or the first duration exceeds the application's runtime, increasing the GPU frequency includes: If the third duration is greater than the working duration of SurfaceFlinger and the first duration is greater than the working duration of the application, the maximum value of the first GPU frequency and the second GPU frequency is taken as the target GPU frequency after adjustment. The first GPU frequency is obtained based on the working duration of SurfaceFlinger, the number of image synthesis tasks to be processed by the GPU, and the second mapping relationship. The second GPU frequency is obtained based on the working duration of the application, the number of tasks to be processed by the GPU, and the second mapping relationship. or, If the third duration is less than or equal to the working duration of SurfaceFlinger and the first duration is greater than the working duration of the application, the second GPU frequency will be used as the target GPU frequency after adjustment. or, If the third duration is greater than the working duration of SurfaceFlinger and the first duration is less than or equal to the working duration of the application, the first GPU frequency is used as the target GPU frequency after adjustment.

9. The method according to claim 7 or 8, characterized in that, The step of adjusting the GPU frequency to the lowest value when the first duration is less than or equal to the target duration includes: If the third duration is less than or equal to the working duration of the SurfaceFlinger and the first duration is less than or equal to the working duration of the application, the GPU frequency is adjusted to the lowest value.

10. An electronic device, characterized in that, The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 9.

11. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes computer program code that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 9.

Images