Image processing method, electronic device, storage medium and chip system

Abstract

The image processing method, the electronic device, the storage medium and the chip system provided by the embodiments of the present application relate to the technical field of terminals. The method comprises the following steps: the electronic device can provide a plurality of different optimization rendering schemes, and the quality and smoothness of different optimization rendering schemes are evaluated to select the most suitable or relatively suitable optimization scheme to dynamically adjust the rendering data. In this way, the electronic device can optimize the rendering data in a timely manner according to the actual application scene and the load state of the electronic device, so as to better balance the quality and smoothness, thereby improving the user experience.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image processing method, electronic device, storage medium and chip system. Background Technology

[0002] As the hardware performance of electronic devices continues to improve, along with the increasing screen resolution and refresh rate, the demands on their computing power are also rising. To meet user needs, electronic devices require both enhanced image clarity and a smooth, lag-free interface. This can lead to high power consumption and heat generation, making the interface prone to stuttering and other issues. Summary of the Invention

[0003] This application provides an image processing method, an electronic device, a storage medium, and a chip system. The electronic device can provide a variety of different optimized rendering schemes. By evaluating the image quality and smoothness of different optimized rendering schemes, the most suitable or relatively suitable optimization scheme is selected to dynamically adjust the rendering data, so as to achieve a better balance between image quality and smoothness, thereby improving the user experience.

[0004] In a first aspect, embodiments of this application provide an image processing method applied to an electronic device, the method comprising:

[0005] Based on the first rendering data of the Nth frame and system load information, one or more rendering schemes are determined, where N is a positive integer. Different rendering schemes correspond to different rendering data, which includes one or more of the following: resolution, frame rate, and rendering methods for post-processing effects. The image quality and smoothness of the one or more rendering schemes are evaluated. According to the target rendering scheme, the first rendering data is adjusted to the second rendering data, where the target rendering scheme is the one or more rendering schemes with the best image quality and / or the best smoothness, and the predicted power consumption of the target rendering scheme is less than a preset power consumption threshold. In this way, the electronic device can optimize the rendering data in a timely manner according to the actual application scenario and the load status of the electronic device, so as to achieve a better balance between image quality and smoothness, thereby improving the user experience.

[0006] One possible implementation involves evaluating the image quality and smoothness of one or more rendering schemes. This includes evaluating the image quality and smoothness of M rendering schemes out of the one or more rendering schemes, where the predicted power consumption of all M rendering schemes is less than a preset power consumption threshold, M is a positive integer, and the M rendering schemes include the target rendering scheme. This eliminates the need to evaluate the image quality and smoothness of each rendering scheme individually. While maintaining stable power consumption, it also reduces the computational load of image quality and smoothness evaluation, thereby improving the performance of electronic devices.

[0007] In one possible implementation, the target rendering scheme includes one or more of the following: adjusting resolution, adjusting frame rate, and adjusting post-processing effects. Adjusting the frame rate includes controlling the frame rate within a preset range. Post-processing effects include one or more of the following: reflection effects, shadow map resolution, bloom, and anti-aliasing. Adjusting post-processing effects includes a rendering method that reuses post-processing effects every other frame. It is understood that reducing resolution reduces the number of pixels that an electronic device needs to process, resulting in a smoother display while reducing the load on the device. Frame control allows for dynamic adjustment of the rendering frame rate to adapt to different hardware performance and user needs. For example, on lower-performance electronic devices, the frame rate can be reduced to ensure stable game operation. On higher-performance electronic devices, the frame rate can be increased to achieve better visual effects. This provides a better user experience across different electronic devices and environments, ensuring that applications function correctly and provide good visual effects. The rendering method that reuses post-processing effects every other frame ensures that the rendering data on the frame buffer remains the data from the previous frame, allowing the current frame to reuse the post-processing rendering data from the previous frame. Frame reuse can reduce the amount of computation per frame, reduce the power consumption of electronic devices, and thus improve rendering performance.

[0008] In one possible implementation, before adjusting the first rendering data to the second rendering data according to the target rendering scheme, the method further includes: identifying one or more rendering stages of the Nth frame image; wherein the one or more rendering stages include one or more of the following: main scene, user interface, motion blur, and bloom; identifying association groups associated with the first rendering stage based on the reference relationship of the first rendering stage, wherein the one or more rendering stages include the first rendering stage, and the association group includes one or more rendering stages that share frame buffer attachments or have dependencies with the first rendering stage; before adjusting the first rendering data to the second rendering data according to the target rendering scheme, the method further includes: according to the target rendering scheme, one or more rendering stages in the rendering stage association group respectively use the second rendering data for rendering. In this embodiment, when there is a dependency relationship with the first rendering stage, the first rendering stage can be merged with other rendering stages to reduce the overhead of data synchronization and data transmission. In this way, by identifying and using appropriate rendering stage association groups, rendering performance can be optimized, unnecessary frame buffer switching can be reduced, rendering efficiency can be improved, and the workload and energy consumption of the GPU can be reduced.

[0009] In one possible implementation, before one or more rendering stages in the rendering stage association group render using the second rendering data, the method further includes: generating multiple frame buffer attachments with different resolutions based on the resolution and frame buffer attachments of the first rendering stage, wherein the attachment texture format contained in the multiple frame buffer attachments with different resolutions is the same as the attachment texture format contained in the frame buffer attachments of the first rendering stage; the rendering stage association group renders using the second rendering data, including: obtaining the frame buffer attachment corresponding to the resolution based on the resolution in the second rendering data; and the rendering stage association group renders based on the frame buffer attachments. In this way, during the graphics rendering process, the rendering optimization module can dynamically adjust the frame buffer attachments at different resolutions to achieve image rendering at different resolutions, thereby making reasonable use of rendering optimization resources.

[0010] In one possible implementation, identifying one or more rendering stages of the Nth frame image includes: if the resolution of the Nth frame image is greater than a resolution threshold, the frame buffer attachment is greater than an attachment threshold, and the number of drawing instructions is the largest among one or more rendering stages, then the main scene rendering stage is identified.

[0011] Understandably, to ensure image clarity and complete detail, the main scene is typically rendered at a higher resolution, thus providing better visual quality. Different rendering stages may correspond to different facet attachments. For example, the main scene's facet attachments may include color attachments, depth attachments, and stencil attachments. Furthermore, since the main scene usually contains many 3D models and complex geometric structures, such as buildings, terrain, vegetation, and characters, these elements require a large number of vertices and faces to describe. Therefore, the main scene often has the largest number of facets (DCs), meaning it is the rendering stage with the most DCs among all rendering stages. Based on this, in this embodiment, when determining the rendering stage of the main scene, it can be determined whether the resolution is greater than or equal to a resolution threshold, whether the facet attachments are greater than or equal to an facet attachment threshold, and whether the number of DCs is the largest among all rendering stages.

[0012] This application's embodiments can also identify other scenes such as user interfaces, motion blur, and floodlight. Thus, by identifying different rendering scenes, different rendering parameters can be flexibly applied for image rendering, resulting in better image quality and effects on the interface.

[0013] In one possible implementation, before determining one or more rendering schemes, the method further includes: determining one or more rendering schemes if the predicted power consumption of the Nth frame image is greater than or equal to a preset power consumption threshold. In this embodiment, it is not necessary to perform rendering optimization for all application scenarios, as this would result in a large amount of rendering computation and a high power load on the electronic device. Instead, rendering optimization is performed for some complex scenes, scenes with many rendering elements, and scenes with a large amount of data computation, such as main scenes, fighting scenes, team battle scenes, and skill release scenes. This allows for flexible adjustments to rendering optimization based on the actual rendering scene and the power consumption of the electronic device, thereby reducing the amount of rendering computation and lowering the power load on the electronic device.

[0014] In one possible implementation, before determining one or more rendering schemes, the method further includes: if the Nth frame image is an image from the target application, determining one or more rendering schemes, wherein the target application is an application from a preset list. Different applications have different power consumption. Embodiments of this application can optimize the rendering of some power-intensive applications using a preset list. This allows for better management and optimization of application operation, more efficient improvement of electronic device performance, and reduced power consumption.

[0015] Secondly, embodiments of this application provide an image processing apparatus, which may be an electronic device, a chip or chip system within an electronic device. The apparatus may include a processing unit. The processing unit is used to implement any processing-related method executed by the electronic device in the first aspect or any possible implementation of the first aspect. When the apparatus is an electronic device, the processing unit may be a processor. The apparatus may further include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement the method described in the first aspect or any possible implementation of the first aspect. When the apparatus is a chip or chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to cause the electronic device to implement the method described in the first aspect or any possible implementation of the first aspect. The storage unit may be a storage unit within the chip (e.g., a register, cache, etc.), or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0016] Thirdly, embodiments of this application provide an electronic device including one or more processors and a memory, the memory being coupled to one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, and one or more processors being used to invoke the computer instructions to perform the method described in the first aspect or any possible implementation of the first aspect.

[0017] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0018] Fifthly, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0019] Sixthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation thereof. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0020] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0021] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0022] Figure 1 A schematic diagram of an electronic device provided in an embodiment of this application;

[0023] Figure 2 A schematic diagram of the software architecture of an electronic device provided in an embodiment of this application;

[0024] Figure 3 This application provides a schematic diagram illustrating the interaction between functional modules in a software architecture.

[0025] Figure 4 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0026] Figure 5 A flowchart illustrating the process of obtaining rendering instruction stream and CPU frequency is provided in an embodiment of this application.

[0027] Figure 6 A schematic diagram illustrating a process for power consumption prediction and image quality prediction based on rendering data, provided in an embodiment of this application;

[0028] Figure 7 A flowchart illustrating a decision optimization rendering scheme provided in an embodiment of this application;

[0029] Figure 8 A schematic diagram illustrating the process of a rendering recognition module identifying different rendering stages, provided in an embodiment of this application;

[0030] Figure 9 A schematic diagram illustrating the process of a resource management module generating multiple resources, provided in an embodiment of this application;

[0031] Figure 10 A schematic diagram of a rendering optimization process provided for an embodiment of this application;

[0032] Figure 11 A schematic diagram illustrating an image processing method provided in an embodiment of this application;

[0033] Figure 12 This is a schematic diagram of the structure of a chip provided in an embodiment of this application. Detailed Implementation

[0034] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application are briefly introduced below. In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and role. For example, "first chip" and "second chip" are only used to distinguish different chips and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" do not necessarily imply that they are different.

[0035] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0036] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0037] As the hardware performance of electronic devices continues to improve, along with the increasing screen resolution and refresh rate, the demands on their computing power are also rising. To meet user needs, electronic devices require both enhanced image clarity and a smooth, lag-free interface. This can lead to high power consumption and heat generation, making the interface prone to stuttering and other issues.

[0038] Taking gaming applications as an example, advancements in game engine technology have led to a continuous increase in the quality and complexity of game graphics. However, in gaming scenarios, some users may prioritize visual quality (referred to as image quality), while others may place greater emphasis on smoothness (referred to as fluidity) and frame rate stability, or even desire both high image quality and smoothness. Therefore, electronic devices need to strike a balance between image quality and smoothness.

[0039] In view of this, the image processing method provided in this application allows the electronic device to offer a variety of different optimized rendering schemes. By evaluating the image quality and smoothness of different optimized rendering schemes, the most suitable or relatively suitable optimization scheme is selected to dynamically adjust the rendering data. In this way, the electronic device can optimize the rendering data in a timely manner according to the actual application scenario and the load status of the electronic device, achieving a better balance between image quality and smoothness, thereby improving the user experience.

[0040] The electronic device in this application embodiment can also be any form of terminal device. For example, the electronic device may include: mobile phone, tablet computer, handheld computer, laptop computer, mobile internet device (MID), wearable device, virtual reality (VR) device, augmented reality (AR) device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, cellular phone, cordless phone, session initiation protocol (SIP) phone, wireless local loop (WLL) station, personal digital assistant (PDA), handheld device, computing device or other processing device connected to a wireless modem, vehicle-mounted device, electronic device in 5G network, or future evolved public land mobile communication network. The embodiments of this application do not limit the scope of electronic devices in a network (PLMN).

[0041] Furthermore, in this application embodiment, the electronic device can also be an electronic device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0042] The electronic equipment in this application embodiment may also be referred to as: user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0043] In this embodiment, the electronic device includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and main memory. The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0044] For example, Figure 1 A schematic diagram of the hardware structure of the electronic device is shown.

[0045] The electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0046] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may include hardware, software, or a combination of software and hardware.

[0047] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0048] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or is reusing. If the processor 110 needs to use the instruction or data again, it can directly retrieve it from the aforementioned memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system. For example, in the embodiments of this application, the processor 110 can be used to acquire rendering instruction streams, perform power consumption and image quality prediction, and output rendering optimization schemes, etc.

[0049] Internal memory 121 can be used to store executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc. The data storage area may store data created during the use of the electronic device, etc. Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of the electronic device by running instructions stored in internal memory 121 and / or instructions stored in memory disposed in the processor. For example, in this embodiment, internal memory 121 may be used to store rendering instruction streams, prediction of power consumption, image quality, and smoothness-related code, etc.

[0050] Figure 2 This is a schematic diagram of the software architecture of an electronic device according to an embodiment of this application.

[0051] For ease of description, the software architecture of electronic devices will be illustrated below using the Harmony operating system as an example. It should be noted that the embodiments of this application can also be applied to other operating systems (e.g., system, system, Any system (or similar system) can achieve the solution of this application as long as the functions implemented by each functional module are similar to those in the embodiments of this application.

[0052] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, HarmonyOS may include an application layer, a framework layer, a system infrastructure service layer, and a kernel layer, etc.

[0053] The application layer can include system applications and third-party applications, such as games and video applications. In some scenarios, the application layer can also be called the application layer.

[0054] The framework layer provides HarmonyOS applications with multi-language user program frameworks (Java / C / C++ / JavaScript, etc.), meta-capability frameworks, and the ArkUI framework, as well as multi-language framework application programming interfaces for various software and hardware services. In some scenarios, the framework layer can also be referred to as the application framework layer.

[0055] The ArkUI framework provides a complete infrastructure for the development of user interfaces (UI) for system applications, including UI functions such as components, layouts, animations, and interactive events, as well as a real-time interface preview tool. For example, in this embodiment, a game application can use the ArkUI framework to implement the functions of components in the game interface, the display of game animations, and user interaction with the game interface.

[0056] The system foundation service layer is the core capability set of the HarmonyOS system, supporting HarmonyOS in providing application services through the framework layer in multi-device deployment scenarios. The system foundation service layer may include, but is not limited to, one or more of the following components: system basic capability subsystem set, basic software service subsystem set, enhanced software service subsystem set, hardware service subsystem set, proprietary hardware service subsystem set, instruction reassembly layer, HarmonyOS driver framework, and hardware abstraction adaptation layer.

[0057] Among them, the system basic capability subsystem set provides the basic capabilities for the operation and scheduling of distributed applications on multiple devices of the HarmonyOS system. It can be composed of distributed task scheduling, distributed storage and data management, distributed soft bus, Ark runtime, distributed security and privacy protection, etc.

[0058] The basic software service subsystem provides common and general software services for HarmonyOS, and may consist of subsystems such as graphics and image software services and distributed media software services. The enhanced software service subsystem provides differentiated capability-enhancing software services for different devices, and may consist of subsystems such as tablet business software services and smart screen business software services. The hardware service subsystem provides common and adaptable hardware services for HarmonyOS, and may include subsystems such as power services. The proprietary hardware service subsystem provides differentiated hardware services for different devices, and may include subsystems such as tablet-specific hardware services and wearable-specific hardware services.

[0059] It is understandable that the above service subsystems can be deployed in different device configurations, and each subsystem can be tailored to its functional granularity.

[0060] In this embodiment, the instruction reassembly layer can be referred to as the instruction acceleration layer or the rendering parameter optimization layer. The instruction reassembly layer can also be understood as an instruction reassembly component. The instruction reassembly layer may include a parameter configuration module, a rendering data management module, a model prediction module, a rendering recognition module, a resource management module, a decision guidance module, and a rendering optimization module, etc.

[0061] On one hand, during the graphics rendering process, the instruction reassembly layer can intercept the rendering instruction stream used for interface rendering. This rendering instruction stream can also be called a rendering instruction data stream, graphics instruction stream, or graphics instruction data stream. The instruction reassembly layer can also obtain rendering data and configuration data based on the parameter configuration module and the rendering data management module.

[0062] For example, the parameter configuration module can be used to store pre-configured rendering parameters. The rendering data management module can be used to acquire intercepted rendering command streams, obtain rendering data from them, and cache rendering data. The rendering data management module can also be used to obtain GPU frequency points. For specific pre-configured rendering parameters and rendering data, please refer to... Figure 3 The relevant descriptions in the corresponding embodiments.

[0063] On the other hand, the instruction reorganization layer can adaptively predict rendering parameters and output decision guidance based on the model prediction module and the decision guidance module. For example, the model prediction module can be used to predict the power consumption of the current range of selectable rendering parameters and filter rendering parameters that are below the power consumption threshold; the model prediction module can also predict image quality and smoothness and pass the prediction results to the decision guidance module.

[0064] The decision guidance module can be used to select the optimal or better rendering optimization scheme based on image quality and smoothness, and pass the rendering optimization scheme to the rendering optimization module for rendering optimization.

[0065] On the other hand, the instruction reassembly layer can identify and optimize rendering scenes based on the rendering recognition module, resource management module, and rendering optimization module. For example, the rendering recognition module can identify stages such as the main scene, motion blur, bloom, and UI based on the number of draw calls (DCs), the rendering resolution at each stage, and information such as framebuffer attachment (FB attachment).

[0066] Here, the number of drawing commands (DCs) refers to the number of drawing instructions sent by the CPU to the GPU, which instruct the GPU to draw objects on the screen. For example, in Unity, each drawing instruction represents one drawing operation.

[0067] Frame buffer attachments, also known as FB attachments, are buffers used to store rendered output data. FB attachments can include color attachments, depth attachments, and stencil attachments. Color attachments store the color information of the rendered output. The color data for each pixel can include red, green, and blue, and may also include alpha (transparency). Depth attachments store the depth information for each pixel, i.e., the distance from the camera to the object's surface, to determine which pixels are in front and which are behind, thus correctly rendering occlusion relationships. Stencil attachments store stencil information used to control pixel rendering, such as stencil testing, shadow volume, and reflections.

[0068] The resource management module can obtain the size and FB attachment format of the main scene rendering stage based on the recognition results of the rendering recognition module, and generate multiple FBs of different sizes. The format of the FB attachments is the same as that of the main scene, and they are provided to the rendering optimization module.

[0069] The rendering optimization module can adjust the rendering resolution, rendering frame rate, and whether post-processing effects are rendered based on the scene and stage identified by the rendering recognition module.

[0070] The HarmonyOS driver framework and hardware abstraction adaptation layer are the foundation for the open hardware ecosystem of the HarmonyOS system. They provide hardware capability abstraction to the hardware at the top and development framework and runtime environment for various peripheral drivers at the bottom.

[0071] HarmonyOS employs a multi-kernel design, which may include, for example, the Linux kernel, the Harmony microkernel, and LiteOS. This design allows electronic devices with varying capabilities to select the appropriate system kernel.

[0072] The methods of this application will be described in detail below through specific embodiments. The following embodiments can be combined with each other or implemented independently, and the same or similar concepts or processes may not be described again in some embodiments.

[0073] In combination with the above Figure 2 Descriptions of each functional module. Figure 3 This illustration shows a schematic diagram of the interaction between functional modules in the software architecture involved in the embodiments of this application.

[0074] In the instruction reassembly layer, the parameter configuration module 301 can be configured with pre-configured rendering parameters. These pre-configured rendering parameters may include, but are not limited to, one or more of the following: resolution range, frame rate range, post-processing effects, and power consumption threshold.

[0075] The rendering data management module 302 can obtain rendering data from the intercepted rendering command stream and cache the rendering data. The rendering data management module 302 can also obtain the system GPU frequency. The rendering data may include, but is not limited to, one or more of the following: the rendering stage of the main scene, the MVP matrix of the camera in the main scene, the position information of the camera in the main scene, the number of DCs, the rendering pass, the GPU frequency, the resolution, the frame rate, the UI rendering stage, and post-processing effects.

[0076] It is understandable that the main scene rendering stage can be understood as the rendering stage used to generate the final scene image.

[0077] The MVP matrix for a camera refers to the combination of three matrices: Model, View, and Projection. The Model matrix transforms the object from the local coordinate system to the world coordinate system, the View matrix transforms the world coordinate system to the camera coordinate system, and the Projection matrix transforms the camera coordinate system to the clipping coordinate system.

[0078] The camera's position determines how the scene is observed and presented, and which objects are within the field of view. For example, the camera's position allows objects or characters in a game scene to be rendered according to the camera's perspective, thus simulating how the human eye observes the world.

[0079] A rendering pass is a stage or step in the rendering process. It defines a set of rendering operations, their execution order, and dependencies. For example, a rendering pass defines how framebuffer attachments, such as color attachments, depth attachments, and stencil attachments, are handled, and how these attachments are read and written during rendering. In some scenarios, a rendering pass may also be called a rendering stage or rendering step. For ease of understanding, it will be referred to as a rendering stage from now on.

[0080] The UI rendering stage can also be understood as the user interface rendering stage. The UI rendering stage is used to draw elements from the user interface onto the screen so that users can interact with the game. Elements in the user interface can include, but are not limited to, one or more of the following: menus, buttons, text, status bars, and icons.

[0081] Post-processing effects can also be understood as post-processing effects or special effects. Post-processing effects include, but are not limited to, one or more of the following effects: bloom, motion blur, reflection effect, shadow mapping, and anti-aliasing.

[0082] In this embodiment, on the one hand, the model prediction module 303 can obtain pre-configured rendering parameters from the parameter configuration module 301 and rendering data from the rendering data management module 302, thereby predicting the power consumption of the current frame.

[0083] Understandably, different game applications have different power consumption requirements. Therefore, different game applications can have different power consumption thresholds. Furthermore, the power consumption of the same game application also varies depending on the game scenario. Therefore, different game scenarios for the same game application can also have correspondingly different power consumption thresholds.

[0084] In a possible implementation, the model prediction module 303 can predict the power consumption for different scenes based on real-time rendering data (such as the number of rendering controllers, rendering stage, GPU frequency, post-processing effects, etc.). For example, when the number of rendering controllers is large, there are many post-processing effects, and / or the GPU frequency is high, the predicted power consumption output by the model will be relatively high. When the model prediction module 303 predicts that the power consumption of a certain scene exceeds a power consumption threshold, the model prediction module 303 can provide one or more optimized rendering schemes. It can be understood that the power consumption threshold can be interpreted as the power consumption threshold corresponding to that scene. The rendering parameters corresponding to these one or more optimized rendering schemes are different.

[0085] It is understandable that, since some of the optimized rendering schemes may have predicted power consumption exceeding the power consumption threshold, the model prediction module 303 can predict the power consumption of the optimized rendering schemes.

[0086] If, among the one or more optimized rendering schemes, M optimized rendering schemes have predicted power consumption less than a power consumption threshold, the model prediction module 303 can select the optimal or relatively optimal scheme from these M optimized rendering schemes. Here, M is a positive integer. In a possible implementation, the model prediction module 303 can predict the image quality and smoothness of the M optimized rendering schemes and pass these predicted image quality and smoothness to the decision guidance module 304. Then, the decision guidance module 304 can select the optimized rendering scheme with relatively better image quality and smoothness based on the predicted image quality and smoothness to reduce the power consumption of the current game scene.

[0087] On the other hand, the rendering recognition module 305 can obtain information such as the resolution, number and type of frame buffer attachments, number of DCs, and rendering resolution of each rendering stage from the rendering data management module 302, and identify rendering stages such as the main scene, motion blur, bloom, and UI based on information such as the number of DCs, rendering resolution, and frame buffer attachments. For specific scene recognition, please refer to... Figure 4 The relevant descriptions of step S404 and the rendering recognition module in the rendering stage in the corresponding embodiment will not be repeated here.

[0088] The resource management module 306 can obtain the main scene rendering stage, its size, and the FB attachment format based on the recognition results of the rendering recognition module 305, generate multiple FBs of different sizes, wherein the format of the FB attachments is the same as that of the main scene, and provide them to the rendering optimization module 307.

[0089] The rendering optimization module 307 can obtain rendering data from the rendering data management module 302, guidance strategies output by the decision guidance module 304, determine each rendering stage from the rendering identification module 305, and obtain multiple FBs of different sizes from the resource management module 306, thereby performing different rendering optimizations and adjusting rendering frame rate, resolution, post-processing effects, etc.

[0090] Figure 4 A flowchart of an image processing method is shown. The flowchart includes steps S401 to S406. It is understood that... Figure 4 The processes or steps in the corresponding embodiments do not necessarily have to be executed in a specific order. This application does not limit the execution order of steps S401 to S406, as long as the image processing method can be implemented. For example, the execution order of steps S402 and S404, as well as steps S403 and S404, is not strictly distinguished.

[0091] S401, the rendering data management module obtains the cached rendering instruction stream and GPU frequency.

[0092] like Figure 5 The diagram shown is a flowchart for obtaining the rendering instruction stream and CPU frequency.

[0093] When a game application is running, it can call the graphics API through the game engine to communicate with the graphics driver interface, thereby realizing the rendering of game functions and game interface.

[0094] For example, the instruction reassembly layer can intercept calls to the graphics API, obtain the rendering instruction stream, and cache the rendering data. Within the instruction reassembly layer, it can determine whether the game application is a target optimization application. One possible implementation is to use a whitelist to determine whether the game application is a target optimization application. This whitelist includes applications that can be optimized.

[0095] If the whitelist includes a game application, it indicates that the game application is the target optimization application. In this case, the instruction reassembly layer can utilize the reserved interfaces of the graphics driver in the software system, such as the reserved interfaces of the OpenGL graphics driver and / or Vulkan graphics driver, to intercept the rendering instruction stream. If the whitelist does not include a game application, it indicates that the game application is not the target optimization application, and the instruction reassembly layer will not intercept the rendering instruction stream.

[0096] When optimizing an application for gaming purposes, the rendering data management module can analyze and cache rendering data based on intercepted rendering command streams. This rendering data includes information such as the main scene's rendering stage, UI rendering stage, the camera's MVP matrix and position information, GPU frequency, resolution, and frame rate. Specific rendering data can be found in [reference needed]. Figure 3 The relevant descriptions in the corresponding embodiments will not be repeated here.

[0097] S402, the model prediction module performs power consumption prediction and image quality prediction.

[0098] like Figure 6 The diagram shows a flowchart illustrating the process of power consumption prediction and image quality prediction based on rendering data.

[0099] The model prediction module can obtain rendering data from the rendering data management module and pre-configured rendering parameters from the parameter configuration module, and perform power consumption prediction and image quality prediction based on the obtained data.

[0100] In a possible implementation, the model prediction module may include a power prediction model and a quality prediction model. The power prediction model can also be called a power consumption model, and the quality prediction model can also be called a quality model. The model prediction module can predict the power consumption of an image based on the power prediction model, and predict image quality and smoothness based on the quality prediction model.

[0101] Optionally, the model prediction module can perform predictions periodically. For example, the prediction period can be preset in the model prediction module, such as a period of 1 second. This application embodiment does not limit the specific period setting.

[0102] Optionally, the model prediction module can also trigger predictions based on specific scenarios. Scenarios can include, for example, complex scenes, scenes with many rendered elements, scenes with high data computation requirements, main scenes, combat scenes, team battle scenes, and skill release scenes. Understandably, these scenarios typically consume high power. Thus, the model prediction module can predict power consumption for some complex, high-power game scenes.

[0103] When performing power consumption prediction, the power prediction model can predict the power consumption corresponding to different scenarios based on real-time rendering data (such as the number of DCs, rendering stage, post-processing effects, GPU frequency, etc.). For example, when there are many DCs, many post-processing effects, and / or a high GPU frequency, the predicted power consumption output by the model will be relatively high.

[0104] When the predicted power consumption of the previous frame exceeds a power consumption threshold, the power prediction model can provide one or more optimized rendering schemes. Understandably, these optimized rendering schemes correspond to different rendering parameters. The power prediction model can predict the power consumption of these optimized rendering schemes. If, among these optimized rendering schemes, there are M optimized rendering schemes whose predicted power consumption is less than the power consumption threshold, the power prediction model can pass the M optimized rendering schemes that meet the power consumption threshold to the quality prediction model. Here, M is a positive integer.

[0105] After acquiring relevant data for the M optimized rendering schemes, the quality prediction model can perform image quality and smoothness predictions for each scheme based on factors such as resolution, frame rate, and post-processing effects. Specifically, after predicting image quality, the model can output an image quality score for each of the M optimized rendering schemes. The model can also evaluate the smoothness of each of the M optimized rendering schemes based on the continuity between consecutive frames.

[0106] The quality prediction model can also pass the image quality score and smoothness evaluation results corresponding to each of the M optimized rendering schemes to the decision guidance module. The decision guidance module can execute step S403 to determine the best rendering optimization scheme.

[0107] S403, the decision guidance module determines the best rendering optimization scheme.

[0108] like Figure 7 The diagram shown is a flowchart illustrating the process of deciding on and optimizing the rendering scheme.

[0109] The decision guidance module can select an optimized rendering scheme with higher image quality score and better smoothness based on the predicted image quality score and predicted smoothness evaluation results output by the model prediction model. This allows the best or better rendering parameters to be passed to the rendering optimization module to reduce the power consumption of the current game scene. The optimized rendering scheme may include, but is not limited to, one or more of the following optimization methods: frame reuse, resolution adjustment, frame rate adjustment, and post-processing rendering methods.

[0110] Frame reuse can be understood as a rendering method that skips or performs post-processing on every other frame. Frame reuse ensures that the rendering data in the framebuffer remains the data from the previous frame, and the current frame can reuse the post-processed rendering data from the previous frame. Frame reuse reduces the computational load per frame, lowers the power consumption of electronic devices, and thus improves rendering performance.

[0111] Adjusting the resolution can be understood as setting different resolutions for different scenarios. For example, lowering the resolution reduces the number of pixels that an electronic device needs to process, resulting in a smoother display while reducing the load on the device.

[0112] Adjusting the frame rate can be understood as adjusting the actual rendering frame rate through frame control. Frame control refers to limiting the frame rate to a preset value. In essence, frame control allows for dynamic adjustment of the rendering frame rate to adapt to different hardware performance and user needs. For example, on lower-performance electronic devices, the frame rate can be reduced to ensure stable game operation. On higher-performance electronic devices, the frame rate can be increased to achieve better visual effects. This provides a better user experience across different electronic devices and environments, ensuring applications function correctly and offer excellent visuals.

[0113] The rendering methods for post-processing effects may include, but are not limited to, one or more of the following: rendering methods for reflection effects, rendering methods for shadow map resolution, rendering methods for bloom effects, and rendering methods for anti-aliasing.

[0114] Among them, the rendering method of reflection effect can simulate the characteristics of light reflection on the surface of an object by calculating the reflection path of light on the object surface, generating a realistic reflection effect and making the surface of the object look more realistic.

[0115] Shadow mapping is a rendering method used to generate texture formats for shadow effects. When rendering a scene, light sources cast shadows onto objects, and shadow maps store this shadow information. The resolution of the shadow map determines the detail and sharpness of the shadows. Higher resolutions provide finer shadow details but increase computational and storage costs.

[0116] The floodlight rendering technique can be used to simulate the halo effect around a strong light source. By identifying brighter areas in an image and adding a soft halo around these areas, the light source appears more visually impactful.

[0117] Anti-aliasing rendering techniques can be used to reduce the appearance of jagged edges in images. Jagged edges typically appear when rendering diagonal lines or curves because pixels are discrete. Anti-aliasing improves image quality by smoothing edges.

[0118] S404, the rendering recognition module identifies the rendering stage.

[0119] The rendering recognition module can identify different rendering stages.

[0120] like Figure 8 The diagram shown illustrates the process of the rendering recognition module identifying different rendering stages.

[0121] The rendering identification module can obtain information such as the resolution of each rendering stage, the number and type of FB attachments, the number of DCs in each rendering stage, and the rendering resolution of each rendering stage from the rendering data management module, and determine each rendering stage based on the obtained information.

[0122] In possible implementations, to ensure image clarity and complete detail, the main scene is typically rendered at a higher resolution, thus providing better visual quality. Therefore, in this embodiment, when determining the rendering stage of the main scene, it can be determined whether the resolution is greater than or equal to a resolution threshold. The electronic device can pre-set the resolution threshold; for example, the resolution threshold could be 1080×1920. This embodiment does not limit the specific value of the resolution threshold. For some post-processing effects, rendering at a lower resolution does not significantly degrade the visual quality of the image. For example, post-processing effects such as motion blur and bloom have inherent softening or blurring characteristics; therefore, they can be rendered at a lower resolution. This reduces the computational load of post-processing and improves the performance of the electronic device.

[0123] It is understandable that different rendering stages may correspond to different FB attachments. For example, the FB attachments for the main scene may include color attachments, depth attachments, and stencil attachments. Therefore, in this embodiment, when determining the rendering stage of the main scene, it can be determined whether the FB attachment is greater than or equal to the FB attachment threshold. The FB attachment threshold can be preset in the electronic device; for example, the FB attachment threshold can be 3. This embodiment does not limit the specific value of the FB attachment threshold.

[0124] Furthermore, since the main scene typically contains numerous 3D models and complex geometric structures, such as buildings, terrain, vegetation, and characters, these elements require a large number of vertices and faces for description. Therefore, the main scene often has the largest number of rendering DCs among all rendering stages. Thus, in this embodiment, when determining the rendering stage of the main scene, it can be determined whether the number of rendering DCs is the largest among all rendering stages.

[0125] Based on the above analysis, it can be seen that, under the conditions of resolution, FB attachments, number of DCs, etc., the rendering recognition module can identify the rendering stage as the main scene rendering stage.

[0126] When identifying the motion blur rendering stage, the rendering recognition module needs a velocity vector buffer because the motion blur frame attachments use velocity vector textures. Furthermore, the number of motion blur DCs is relatively small, and the resolution is relatively low, typically occurring after the main scene. Therefore, the rendering recognition module can identify the motion blur rendering stage based on the texture format, number of DCs, and / or resolution in the frame attachments.

[0127] When identifying the floodlight rendering stage, the rendering recognition module needs an additional luminance buffer because floodlight rendering typically involves multiple rendering stages, such as luminance extraction and / or blurring. Furthermore, the floodlight rendering stage includes downsampling and upsampling processes, resulting in resolution changes. Therefore, the rendering recognition module can identify the floodlight rendering stage based on these multiple rendering stages and the resolution variations.

[0128] When identifying the UI rendering stage, the rendering recognition module considers that UI rendering is usually the last rendering stage and involves a relatively small number of data centers (DCs). Therefore, the rendering recognition module can identify the UI rendering stage based on factors such as the number of DCs.

[0129] In this embodiment, the rendering identification module can also identify applicable rendering stage association groups based on the reference relationships between rendering stages, and record the identified frame buffer attachments and texture formats. The reference relationships between rendering stages may include multiple rendering stages sharing the same frame buffer attachments, or the output of one rendering stage being the input of another. Based on these rendering stage reference relationships, the rendering identification module can determine the execution order and dependencies between multiple rendering stages, and identify which rendering stages can be combined into a rendering stage association group.

[0130] For example, if multiple rendering stages share the same framebuffer attachments, these stages can be grouped together into a single rendering stage association group. Alternatively, if the output of one rendering stage serves as the input to another, these two stages can be merged to reduce the overhead of data synchronization and data transfer. In this way, by identifying and using appropriate rendering stage association groups, rendering performance can be optimized, unnecessary framebuffer switching reduced, rendering efficiency improved, and GPU workload and power consumption lowered.

[0131] Texture format can be understood as the way texture data is stored and formatted. Different texture formats can affect rendering performance, image quality, and memory usage. Texture formats can include color formats, depth formats, and stencil formats. Color formats can include RGBA8, RGB10_A2, RGBA16F, etc. Depth formats can include 16-bit depth buffers, 24-bit depth buffers, and combinations of 24-bit depth and 8-bit stencil buffers. Stencil formats can include 8-bit stencil buffers, etc.

[0132] In a possible implementation, the rendering identification module can record the rendering stages that share the same depth texture format as the main scene rendering stage as associated rendering stages and provide this information to the rendering optimization module. The rendering optimization module can then execute step S405 to perform rendering optimization.

[0133] In addition, such as Figure 9 The diagram shown illustrates the process of generating multiple resources using the resource management module.

[0134] The resource management module can obtain the main scene rendering stage, its resolution, and the texture format of the FB attachments through the rendering recognition module. This allows the resource management module to generate multiple FB attachments at different resolutions based on the texture format, where the texture format of these multiple FB attachments is the same as that of the main scene's FB attachments. The main scene includes three FB attachments, and there are also three generated FB attachments.

[0135] It is understood that, in this embodiment of the application, the resource management module can pre-generate multiple FB attachments and texture formats of different resolutions and provide them to the rendering optimization module. In this way, during the graphics rendering process, the rendering optimization module can dynamically adjust the FB attachments of different resolutions to achieve image rendering at different resolutions, thereby making reasonable use of rendering optimization resources.

[0136] S405, the rendering optimization module performs rendering optimization.

[0137] like Figure 10 The diagram shown is a flowchart illustrating the rendering optimization process.

[0138] The rendering optimization module can acquire rendering data based on the rendering data management module, determine each rendering stage based on the rendering identification module, acquire multiple FB attachments of different resolutions based on the resource management module, and perform different rendering optimizations according to the guidance strategies output by the decision guidance module. Furthermore, the rendering optimization module can execute step S405 to issue driver instruction data to adjust the rendering frame rate, resolution, post-processing effects, etc.

[0139] For example, taking the main scene as an example, because the main scene is relatively complex, has many rendering elements, and involves a large amount of data calculation, it is prone to stuttering. Therefore, the rendering optimization module can reduce the resolution of the main scene and / or reduce the rendering frame rate of the main scene, making the main scene smoother and less prone to stuttering.

[0140] When game scenes require post-processing, the rendering optimization module can employ a frame-by-frame reuse rendering method. This allows electronic devices to use post-processed data from the previous frame when rendering post-processed effects. This reduces computational load, improves the data processing speed of electronic devices, and minimizes interface stuttering.

[0141] S406, Send driver command data.

[0142] The rendering optimization module sends the optimized rendering instructions to the driver module. The rendering optimization scheme provided in this embodiment can reduce the consumption of hardware resources, reduce the power consumption load and heat generation of electronic devices, reduce unnecessary rendering work, and improve the operating efficiency and battery life of electronic devices.

[0143] Figure 11 An image processing method according to an embodiment of this application is illustrated. Applied to an electronic device, the method includes:

[0144] S1101. Based on the first rendering data of the Nth frame image and the system load information, determine one or more rendering schemes, where N is a positive integer, different rendering schemes correspond to different rendering data, and the rendering data includes one or more of the following: resolution, frame rate, and rendering methods of post-processing effects.

[0145] In this embodiment, the first rendering data can be understood as rendering data used to render the Nth frame image. The first rendering data may include, but is not limited to, one or more of the following: resolution, frame rate, rendering method of post-processing effects, rendering stage, MVP matrix, number of drawing instructions, and frame buffer attachments.

[0146] System load information may include GPU frequency.

[0147] S1102. Evaluate the image quality and smoothness of one or more rendering schemes.

[0148] In this embodiment, image quality and smoothness prediction can be performed on one or more rendering schemes based on data such as resolution, frame rate, and post-processing effects. The electronic device can output the image quality prediction score for each of the one or more rendering schemes, and can also evaluate and output the smoothness evaluation result for each of the one or more rendering schemes based on the continuity between two consecutive frames. For the specific implementation of evaluating the image quality and smoothness of one or more rendering schemes, please refer to... Figure 4 The relevant descriptions in step S402 of the corresponding embodiment will not be repeated here.

[0149] S1103. According to the target rendering scheme, adjust the first rendering data to the second rendering data, wherein the target rendering scheme is the scheme with the best image quality and / or the best smoothness among one or more rendering schemes, and the predicted power consumption of the target rendering scheme is less than the preset power consumption threshold.

[0150] In this embodiment, the electronic device can select a target rendering scheme with a higher image quality score and better smoothness based on the predicted image quality score and predicted smoothness evaluation results, and perform image rendering based on the second rendering parameters corresponding to the target rendering scheme, so as to reduce the power consumption of the current application (such as a game application) scenario. The first rendering data and the second rendering data differ in the rendering methods for resolution, frame rate, and / or post-processing effects.

[0151] Understandably, when determining the target rendering scheme, in addition to judging image quality and smoothness, it is also necessary to predict the power consumption of one or more rendering schemes. For example, power consumption, image quality, and smoothness can be predicted for M rendering schemes. Only the rendering scheme that meets the predicted power consumption requirements and has good or best image quality and / or smoothness can be selected as the target rendering scheme. For details on the specific prediction scheme and the process of predicting image quality and smoothness, please refer to... Figure 4 The relevant descriptions of steps S402 and S403 in the corresponding embodiments will not be repeated.

[0152] The image processing method provided in this application allows an electronic device to offer various rendering schemes. By evaluating the image quality and smoothness of different rendering schemes, the most suitable or relatively suitable scheme is selected to dynamically adjust the rendering data. In this way, the electronic device can optimize the rendering data in a timely manner according to the actual application scenario and the load status of the electronic device, achieving a better balance between image quality and smoothness, thereby improving the user experience.

[0153] Optional, in Figure 11Based on the corresponding embodiments, evaluating the image quality and smoothness of one or more rendering schemes may include: evaluating the image quality and smoothness of M rendering schemes out of one or more rendering schemes, wherein the predicted power consumption of the M rendering schemes is less than a preset power consumption threshold, M is a positive integer, and the M rendering schemes include the target rendering scheme.

[0154] In this embodiment, power consumption prediction can be performed on one or more rendering schemes first. For rendering schemes whose predicted power consumption is less than a preset power consumption threshold, such as including M rendering schemes, image quality and smoothness can be further evaluated.

[0155] In this way, it is not necessary to evaluate the image quality and smoothness of each rendering scheme. While maintaining stable power consumption, the amount of computation required for image quality and smoothness evaluation can be reduced, thereby improving the performance of electronic devices.

[0156] Optional, in Figure 11 Based on the corresponding embodiments, the target rendering scheme includes one or more of the following: adjusting resolution, adjusting frame rate, and adjusting post-processing effects; wherein, adjusting frame rate includes controlling the frame rate within a preset frame rate range, and post-processing effects include one or more of the following effects: reflection effect, shadow map resolution, bloom, and anti-aliasing, and adjusting the rendering method of post-processing effects includes the rendering method of reusing post-processing effects every other frame.

[0157] In this embodiment, the specific methods for adjusting resolution, frame rate, and post-processing effects can be referred to the above. Figure 4 The relevant descriptions in step S403 of the corresponding embodiment will not be repeated here.

[0158] Understandably, reducing the resolution reduces the number of pixels that electronic devices need to process, resulting in a smoother display while reducing the load on the device.

[0159] Frame rate control allows for dynamic adjustment of the rendering frame rate to adapt to different hardware performance and user needs. For example, on lower-performance electronic devices, the frame rate can be reduced to ensure stable game operation. On higher-performance electronic devices, the frame rate can be increased to achieve better visual effects. This provides a better user experience across different electronic devices and environments, ensuring that applications function correctly and deliver excellent visuals.

[0160] Frame-by-frame reuse ensures that the rendering data in the frame buffer is still from the previous frame, allowing the current frame to reuse post-processed rendering data from the previous frame. Frame-by-frame reuse reduces the computational load per frame, lowers the power consumption of electronic devices, and thus improves rendering performance.

[0161] Optional, in Figure 11Based on the corresponding embodiments, before adjusting the first rendering data to the second rendering data according to the target rendering scheme, it may further include: identifying one or more rendering stages of the Nth frame image; wherein, the one or more rendering stages include one or more of the following: main scene, user interface, motion blur, and bloom; identifying the association group associated with the first rendering stage based on the reference relationship of the first rendering stage, wherein, the one or more rendering stages include the first rendering stage, and the association group includes one or more rendering stages that share frame buffer attachments or have dependencies with the first rendering stage; before adjusting the first rendering data to the second rendering data according to the target rendering scheme, it may include: according to the target rendering scheme, one or more rendering stages in the rendering stage association group respectively use the second rendering data for rendering.

[0162] In this embodiment, the association group can also be understood as the rendering stage association group in the above embodiments. The reference relationship between rendering stages may include multiple rendering stages sharing the same framebuffer attachment, or the output of one rendering stage being the input of another rendering stage. Based on the reference relationship of the first rendering stage, the execution order and dependency relationship between the first rendering stage and one or more rendering stages can be determined, and which rendering stages can be combined into a rendering stage association group can be identified.

[0163] Dependencies with the first rendering stage can include the output of the first rendering stage serving as input to other rendering stages, or vice versa. In cases where dependencies exist with the first rendering stage, merging the first stage with other rendering stages can be considered to reduce data synchronization and transfer overhead. By identifying and using appropriate rendering stage associations, rendering performance can be optimized, unnecessary framebuffer switching reduced, rendering efficiency improved, and GPU workload and power consumption lowered.

[0164] Optional, in Figure 11 Based on the corresponding embodiments, before one or more rendering stages in the rendering stage association group render using the second rendering data, the process may further include: generating multiple frame buffer attachments with different resolutions based on the resolution and frame buffer attachments of the first rendering stage, wherein the attachment texture format contained in the multiple frame buffer attachments with different resolutions is the same as the attachment texture format contained in the frame buffer attachments of the first rendering stage; rendering using the second rendering data by one or more rendering stages in the rendering stage association group includes: obtaining the frame buffer attachment corresponding to the resolution based on the resolution in the second rendering data; and rendering based on the frame buffer attachments by one or more rendering stages in the rendering stage association group.

[0165] In this embodiment, the electronic device can pre-generate multiple FB attachments and texture formats of different resolutions and provide them to the rendering optimization module. For specific implementation details, please refer to... Figure 4 Step S404 of the corresponding embodiment, and Figure 9 The relevant descriptions in the corresponding embodiments will not be repeated. Thus, during the graphics rendering process, the rendering optimization module can dynamically adjust the FB attachments at different resolutions to achieve image rendering at different resolutions, thereby making reasonable use of rendering optimization resources.

[0166] Optional, in Figure 11 Based on the corresponding embodiments, identifying one or more rendering stages of the Nth frame image may include: if the resolution of the Nth frame image is greater than a resolution threshold, the frame buffer attachment is greater than an attachment threshold, and the number of drawing instructions is the largest among one or more rendering stages, then the main scene rendering stage is identified.

[0167] It is understandable that, in order to ensure image clarity and complete detail, the main scene is usually rendered at a higher resolution to provide better visual quality. Therefore, in this embodiment, when determining the rendering stage of the main scene, it can be determined whether the resolution is greater than or equal to a resolution threshold. The resolution threshold may include 1080×1920, and this embodiment does not limit the specific value of the resolution threshold.

[0168] Different rendering stages may correspond to different FB attachments. For example, the FB attachments for the main scene may include color attachments, depth attachments, and stencil attachments. Therefore, in this embodiment, when determining the rendering stage of the main scene, it can be determined whether the FB attachment is greater than or equal to the FB attachment threshold. The FB attachment threshold may include 3, and this embodiment does not limit the specific value of the FB attachment threshold.

[0169] Furthermore, since the main scene typically contains numerous 3D models and complex geometric structures, such as buildings, terrain, vegetation, and characters, these elements require a large number of vertices and faces for description. Therefore, the main scene often has the largest number of rendering DCs among all rendering stages. Thus, in this embodiment, when determining the rendering stage of the main scene, it can be determined whether the number of rendering DCs is the largest among all rendering stages.

[0170] Based on the above analysis, it can be seen that, under the conditions of resolution, FB attachments, and number of DCs, the rendering stage can be identified as the main scene rendering stage.

[0171] Understandably, electronic devices can also recognize other scenes such as user interfaces, motion blur, and flood glare. For specific recognition methods, please refer to... Figure 4The relevant descriptions in step S404 of the corresponding embodiment will not be repeated. In this way, by identifying different rendering scenarios, different rendering parameters can be flexibly adopted for image rendering, resulting in better image quality and effects on the interface.

[0172] Optional, in Figure 11 Based on the corresponding embodiments, before determining one or more rendering schemes, it may further include: determining one or more rendering schemes if the predicted power consumption of the Nth frame image is greater than or equal to a preset power consumption threshold.

[0173] In this embodiment, it is not necessary to perform rendering optimization for all application scenarios, as this would result in a large amount of rendering computation and a high power consumption load on the electronic device. This embodiment can optimize rendering for some complex scenes, scenes with many rendering elements, and scenes with a large amount of data computation, such as main scenes, combat scenes, team battle scenes, and skill release scenes. In a possible implementation, power consumption prediction can be performed on the current rendering scene to determine if it is a high-power scene, and then rendering optimization can be performed on high-power scenes. In this way, rendering optimization can be flexibly adjusted according to the actual rendering scene and the power consumption of the electronic device, thereby reducing the amount of rendering computation and lowering the power consumption load of the electronic device.

[0174] Optional, in Figure 11 Based on the corresponding embodiments, before determining one or more rendering schemes, it may further include: if the Nth frame image is an image in the target application, determining one or more rendering schemes, wherein the target application is an application in a preset list.

[0175] In this embodiment, the preset list can also be understood as the whitelist in the above embodiments. The preset list may include applications that can be optimized, such as some power-intensive game applications.

[0176] It is understandable that different applications have different power consumption requirements. This application embodiment can optimize the rendering of some power-intensive applications using a preset list. This allows for better management and optimization of application operation, more efficient improvement of electronic device performance, and reduced power consumption.

[0177] The above detailed embodiments have provided a detailed explanation of the purpose, technical solutions, and beneficial effects of the embodiments of this application. It should be understood that the above are merely specific implementations of the embodiments of this application and are not intended to limit the scope of protection of the embodiments of this application. Any modifications, equivalent substitutions, or improvements made based on the technical solutions of the embodiments of this application should be included within the scope of protection of the embodiments of this application. The module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module; no specific restrictions are placed on the module names.

[0178] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0179] The foregoing mainly describes the technical solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that the method steps described in conjunction with the embodiments of this application can be implemented in hardware or a combination of hardware and software. Whether a function is executed in a hardware-driven or software-driven manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0180] This application provides an electronic device that may include one or more processors and a memory. The memory is coupled to one or more processors. The memory may be used to store computer program code, which includes computer instructions. The one or more processors may invoke the computer instructions to cause the electronic device to execute the technical solutions described above.

[0181] This application provides a chip system. This chip system is applied to an electronic device and may include one or more processors. The one or more processors can be used to invoke instructions to cause the electronic device to execute the technical solutions described above.

[0182] Figure 12 This is a schematic diagram of a chip system provided in an embodiment of this application. The chip system 1200 includes one or more processors 1201, communication lines 1202, communication interfaces 1203, and memory 1204.

[0183] In some implementations, memory 1204 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof.

[0184] The methods described in the embodiments of this application can be applied to, or implemented by, processor 1201. Processor 1201 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 1201 or by instructions in software form. Processor 1201 may be a general-purpose processor (e.g., a microprocessor or conventional processor), a DSP, an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (OPG) or other programmable logic devices, discrete gates, transistor logic devices, or discrete hardware components. Processor 1201 can implement or execute the processing-related methods, steps, and logic block diagrams in the embodiments of this application.

[0185] The method described in this application embodiment can be executed using a hardware decoding processor, or by a combination of hardware and software modules within the decoding processor. The software modules can be located in the memory 1204, and the processor 1201 can read information from the memory 1204 and, in conjunction with its hardware, complete the steps of the above method. The processor 1201, the memory 1204, and the communication interface 1203 can communicate via the communication line 1202.

[0186] This application also provides a readable storage medium (also referred to as a computer-readable storage medium). The readable storage medium includes computer instructions. When these computer instructions are executed on an electronic device, they cause the electronic device to perform the technical solutions described in the above embodiments.

[0187] The methods described in the above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored or transmitted on a readable medium as one or more instructions or programs. The readable medium can include storage media and communication media, and can also include any medium that can transfer a program from one place to another. The storage medium can be any accessible target medium.

[0188] In possible implementations, the readable medium may include random access memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, optical disc read-only memory, or other optical disc storage. The readable medium may also include disk storage or other disk storage devices, or be directed to any other medium or store the required program in the form of instructions or data structures, and be accessible.

[0189] This application also provides a program product (also referred to as a computer program product). The program product includes computer program code. When the computer program code runs on an electronic device, it causes the electronic device to execute the technical solutions described in the above embodiments.

[0190] In the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions between the various embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0191] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, Applied to electronic devices, the method includes: Based on the first rendering data of the Nth frame image and the system load information, one or more rendering schemes are determined, wherein N is a positive integer, different rendering schemes correspond to different rendering data, and the rendering data includes one or more of the following: resolution, frame rate, and rendering methods of post-processing effects. The image quality and smoothness of the one or more rendering schemes are evaluated; According to the target rendering scheme, the first rendering data is adjusted to the second rendering data, wherein the target rendering scheme is the scheme with the best image quality and / or the best smoothness among the one or more rendering schemes, and the predicted power consumption of the target rendering scheme is less than a preset power consumption threshold.

2. The method as described in claim 1, characterized in that, The evaluation of the image quality and smoothness of the one or more rendering schemes includes: The image quality and smoothness of M rendering schemes out of the one or more rendering schemes are evaluated, wherein the predicted power consumption of the M rendering schemes is less than the preset power consumption threshold, M is a positive integer, and the target rendering scheme is included among the M rendering schemes.

3. The method as described in claim 1 or 2, characterized in that, The target rendering scheme includes one or more of the following: adjusting resolution, adjusting frame rate, and adjusting post-processing effects. The frame rate adjustment includes controlling the frame rate within a preset frame rate range. The post-processing effects include one or more of the following effects: reflection effect, shadow map resolution, bloom, and anti-aliasing. The rendering method of the adjusted post-processing effects includes the rendering method of reusing post-processing effects every other frame.

4. The method according to any one of claims 1-3, characterized in that, Before adjusting the first rendering data to the second rendering data according to the target rendering scheme, the method further includes: Identify one or more rendering stages of the Nth frame image; wherein the one or more rendering stages include one or more of the following: main scene, user interface, motion blur, and bloom; Based on the reference relationship of the first rendering stage, identify the association group associated with the first rendering stage, wherein the one or more rendering stages include the first rendering stage, and the association group includes one or more rendering stages that share framebuffer attachments or have dependencies with the first rendering stage. Before adjusting the first rendering data to the second rendering data according to the target rendering scheme, the process includes: According to the target rendering scheme, one or more rendering stages in the rendering stage association group respectively use the second rendering data for rendering.

5. The method as described in claim 4, characterized in that, Before the one or more rendering stages in the rendering stage association group render using the second rendering data, the process further includes: Based on the resolution and frame buffer attachments of the first rendering stage, multiple frame buffer attachments with different resolutions are generated. The attachment texture format contained in the multiple frame buffer attachments with different resolutions is the same as the attachment texture format contained in the frame buffer attachments of the first rendering stage. The one or more rendering stages in the rendering stage association group are rendered using the second rendering data, including: Based on the resolution in the second rendering data, obtain the frame buffer attachment corresponding to the resolution; The one or more rendering stages in the rendering stage association group render based on the frame buffer attachment.

6. The method as described in claim 4 or 5, characterized in that, The identification of one or more rendering stages of the Nth frame image includes: If the resolution of the Nth frame image is greater than the resolution threshold, the frame buffer attachment is greater than the attachment threshold, and the number of drawing instructions is the largest among the one or more rendering stages, then it is identified as the rendering stage of the main scene.

7. The method according to any one of claims 1-6, characterized in that, Before determining one or more rendering schemes, the method further includes: If the predicted power consumption of the Nth frame image is greater than or equal to the preset power consumption threshold, the one or more rendering schemes are determined.

8. The method according to any one of claims 1-7, characterized in that, Before determining one or more rendering schemes, the method further includes: If the Nth frame image is an image from the target application, determine one or more rendering schemes, wherein the target application is an application from a preset list.

9. An electronic device, characterized in that, The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-8.

10. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, The computer program product includes computer program code that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1-8.

Images