A GPU state-aware general mobile operating system dynamic rendering control method and system

Abstract

The application provides a GPU state-aware general mobile operating system dynamic rendering control method and system, and belongs to the technical field of operating system graphic processing. The application is applicable to the graphic rendering optimization of other mobile terminal operating systems and Internet of Things devices in addition to the graphic subsystem of the OpenHarmony operating system. The application dynamically adjusts the rendering frame rate by monitoring the utilization rate and temperature of the GPU in real time, automatically reduces the rendering of part of nodes according to a specific strategy when the utilization rate and temperature of the GPU increase, and automatically restores the rendering of each node when the resources recover, so that the stable operation of the system is realized and the user experience is improved.

Description

Technical Field

[0001] This invention belongs to the field of operating system graphics processing technology, and in particular relates to a dynamic rendering control method and system for a general mobile operating system based on GPU state awareness. Background Technology

[0002] In modern graphics processing, GPUs bear heavy graphics rendering tasks. As applications continue to demand higher graphics quality, the load on GPUs is also increasing. Existing graphics rendering technologies often struggle to achieve a good balance between performance and power consumption. Under high load, GPU utilization and temperature rise rapidly, which may lead to GPU performance bottlenecks, thereby affecting graphics rendering efficiency and frame rate, resulting in issues such as screen stuttering and frame drops, and severely degrading the user experience.

[0003] Currently, the methods for optimizing GPU performance have the following defects: (1) Fixed frame rate mechanism defects: Traditional solutions use a fixed frame rate rendering strategy, which can easily lead to frame rate fluctuations when the GPU is overloaded, causing stuttering or device overheating; (2) Global frame reduction strategy is coarse: Existing frame reduction solutions do not distinguish the priority of rendering nodes, resulting in frame reduction of key interactive elements and affecting user experience; (3) State awareness lag: There is a lack of fine-grained GPU state monitoring, which cannot respond to transient load fluctuations in real time.

[0004] Fixed and global strategies cannot dynamically and finely adjust rendering strategies based on real-time GPU utilization and temperature. Furthermore, mobile operating systems, as user-facing systems, require more intelligent and efficient GPU performance optimization solutions for their graphics subsystems to meet diverse application scenarios and user needs.

[0005] Existing technologies calculate CPU utilization, memory usage, and GPU utilization using a weighted average, and then continuously compare the results according to a strategy. If rendering stutters, the rendering task is terminated directly. While this approach intervenes and controls rendering based on resource availability, it does not actively reduce frame rates, still requires GPU participation in rendering, and fails to prioritize rendering nodes. Consequently, it cannot operate stably and effectively under high GPU load, thus failing to guarantee a good user experience.

[0006] The existing technology has the following technical problems: (1) Limitations of rendering frame rate adjustment: Most existing rendering technologies rely on fixed thresholds or simple strategies, which cannot be flexibly adjusted in real time according to the actual state of the GPU, resulting in a serious decline in visual experience; (2) Coarse-grained global node control: Existing solutions generally adopt a unified global node frame reduction strategy, which cannot distinguish the priority of rendering nodes. This coarse-grained control leads to the same degradation of key interactive nodes (such as video playback) and non-key nodes (such as status bar and background effects), which cannot improve the user experience of key nodes when resources are bottlenecked; (3) Single control strategy: The current strategy only relies on one of GPU utilization and temperature as the trigger condition. The dimension is single, and it cannot accurately predict the current running state of the entire system, which is prone to strategy misjudgment; (4) Lagging resource status perception: The presence of a bottleneck is triggered and the strategy is activated during the rendering process. The perception is lagging and the rendering strategy cannot be determined before rendering, resulting in the continued occupation of GPU resources that have already reached a bottleneck state. Summary of the Invention

[0007] To address the technical problems existing in the prior art, this invention provides a general mobile operating system dynamic rendering control method and system based on GPU state awareness. In addition to the graphics subsystem of the OpenHarmony operating system, it is also applicable to the graphics rendering optimization of other mobile operating systems and IoT devices.

[0008] The first aspect of this invention discloses a dynamic rendering control system, which includes a graphics subsystem and driver hardware; wherein: The graphics subsystem includes: The RenderService graphics rendering framework is used to provide the operating system with 2D and 3D drawing capabilities, as well as animation and display effects. The Strategy module is used to record the priority of applications or SurfaceNodes, and determine whether to perform dynamic frame reduction and the applications or SurfaceNodes that need to be reduced in frame rate based on the current resource status of the Monitor. The Monitor module displays the resource monitoring information. It calls the HDF interface at a fixed frequency to obtain the GPU utilization and temperature in real time and decides whether to update the current information to the Strategy. If the resource status is consistent with the last report, it will not be updated; otherwise, it will update the normal, warning, and emergency status values ​​in the Strategy. The Render component is used to provide APIs and perform rendering processes. The Composer module is used to merge multiple layers into a single unit, and to adjust the transparency, color, and position of elements to achieve seamless blending and transitions. The driver hardware includes: The Hardware Device Framework (HDF) is used to enable communication and interaction between hardware devices and the operating system. GPUs are processors used for parallel data processing, including graphics rendering, parallel computing, and machine learning. Temperature sensors are used to monitor the temperature changes of various components inside the equipment in real time, so that the equipment can operate within the normal operating temperature range. The rendering task is requested by multiple applications (APPs) installed on the device, and the composite graphic image is displayed by the display.

[0009] The second aspect of the present invention discloses a GPU state-aware dynamic rendering control method for a general mobile operating system, wherein the method utilizes a dynamic rendering control system as described in claim 1 to realize GPU state-aware dynamic rendering control for a general mobile operating system. The method specifically includes: Step 1: Set the app's own priority; Step 2: The Monitor module displays the GPU utilization and temperature in real time and decides whether to update the current information to the Strategy. Step 3: The Strategy decision module determines whether to perform dynamic frame reduction and identifies the application or SurfaceNode that needs frame reduction based on the current resource status of the Monitor; Step 4: The Render component adjusts the dynamic rendering strategy based on the node type and performs the rendering process.

[0010] In step 1: the priorityType property is extended in Rosen::RSSurfaceNodeConfig; RSSurfaceNodeConfig is passed in when RSSurfaceNode is created, and the priorityType property is extended in RSSurfaceNode; among them, the SetRenderPriority interface is provided to dynamically adjust the priorityType parameter, and the GetRenderPriority interface is provided to allow the Strategy to obtain the priorityType of the current node.

[0011] In step 2: When the RenderService service starts, a real-time resource monitoring thread is initialized. This thread obtains the current utilization and temperature of the GPU through HDF at a fixed frequency. The thread records the previous state of utilization and temperature respectively. Configure resource load levels, including normal state, warning state, and emergency state. Frame reduction is performed in the warning state.

[0012] In step 3: When the load value is normal, critical nodes, secondary critical nodes and non-critical nodes are rendered at the original frame rate; when the load value is warning, the resources are in a relatively saturated state, critical nodes and secondary critical nodes remain unchanged, non-critical nodes reduce the frame rate, and some resources are released for critical and secondary critical nodes; when the load value is emergency, resources become a bottleneck, secondary critical nodes and non-critical nodes reduce the frame rate, and resources are reserved for critical nodes.

[0013] In step 3, frame rate control includes: determining the rendering frequency through a counter; in the RENDER_LOAD_CRITICAL emergency state, all critical nodes render one frame every two frames, secondary critical nodes render one frame every two frames, and non-critical nodes render one frame every four frames.

[0014] In step 3, frame rate control includes: sending tokens at a uniform frame rate determined by the current resource status for each node type; the rendering node only enters the rendering stage after receiving the token, otherwise the rendering task is discarded; in the RENDER_LOAD_CRITICAL emergency state, for critical nodes, tokens are produced at 60 per second, and the critical node performs subsequent rendering after receiving the token, with the frame rate controlled at 60 FPS; for secondary critical nodes, tokens are produced at 30 per second, and the secondary critical node only performs the rendering task after receiving the token, with the frame rate controlled at 30 FPS.

[0015] In step 4: When RenderService starts, Strategy loads the frame rate reduction and threshold strategy configuration, and simultaneously starts Monitor; Monitor continuously monitors GPU utilization and temperature information, and compares it with previous information. If a change in the range is detected, it updates the current load level of Strategy, and Strategy dynamically adjusts the rendering strategy; APP starts, creates SurfaceNode and sets its type; APP submits rendering task, RenderService obtains the current load level from Strategy. If the load level is normal, it continues to submit rendering tasks, compositing and display; if the load level is warning or emergency, it limits the task according to the current level, strategy, and frame rate control method.

[0016] A third aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the second aspect of the present invention.

[0017] The fourth aspect of the present invention discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in the second aspect of the present invention.

[0018] In summary, this invention dynamically adjusts the rendering frame rate by monitoring GPU utilization and temperature in real time. When GPU utilization and temperature increase, the rendering of some nodes is automatically reduced according to a specific strategy. At the same time, the rendering of each node is automatically restored when resources are restored, so as to achieve stable system operation and improve user experience. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating a GPU-state-aware dynamic rendering control method for a general mobile operating system according to an embodiment of the present invention. Figure 2 This is a schematic diagram of a single capture process of the resource real-time monitoring module according to an embodiment of the present invention; Figure 3 This is a schematic diagram of a GPU-based state-aware dynamic rendering control process according to an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] This invention discloses a dynamic rendering control system, such as... Figure 1 As shown, the upper part consists of the key modules of the graphics subsystem, which are responsible for responding to application VSync, policy decisions, rendering, compositing, and display; the lower part consists of drivers and hardware-related components, which are responsible for GPU utilization and temperature status awareness.

[0023] APP(101): refers to the application installed on the smart device, which is the initiator of the request for rendering tasks.

[0024] RenderService(102): refers to the graphics rendering framework, which provides the operating system with 2D and 3D drawing capabilities, as well as rich animation and display effects.

[0025] Strategy(103): Strategy decision module (new module), records the priority of each application or SurfaceNode, and decides whether to dynamically reduce the frame rate based on the current resource status of Monitor(104), and which applications or SurfaceNodes need to reduce the frame rate.

[0026] Monitor(104): Displays the resource monitoring module (new module). It calls the HDF interface at a fixed frequency (e.g., 25Hz) to obtain the GPU utilization and temperature in real time, and decides whether to update the current information to the Strategy module. If the resource status is the same as the last report, it will not be updated. Instead, it updates the normal, warning and emergency status values ​​in the Strategy module to prevent updates from being too frequent and affecting system performance.

[0027] Render(105): The core component responsible for rendering the graphical interface, providing rich APIs and powerful rendering capabilities.

[0028] Composer(106): Responsible for merging multiple layers into a single unit. This process typically involves adjusting the transparency, color, and position of elements to achieve a seamless blend and transition.

[0029] Display(107): Responsible for displaying the synthesized graphic image on the screen.

[0030] HDF(108): HDF is a hardware device framework, mainly used to realize communication and interaction between hardware devices and the operating system.

[0031] Temperature Sensor (109): The temperature sensor is used to monitor the temperature changes of various components inside the equipment in real time, ensuring that the equipment operates within the normal operating temperature range.

[0032] GPU (110): A processor specifically designed for efficient parallel processing of large amounts of data. Its main functions include graphics rendering, parallel computing, and machine learning.

[0033] This invention discloses a GPU-state-aware dynamic rendering control method for a general-purpose mobile operating system. The specific steps of the method to achieve GPU-state-aware dynamic rendering control for a general-purpose mobile operating system are as follows.

[0034]

Step 1

[0035] [Step 101] Extend the priorityType property in Rosen::RSSurfaceNodeConfig, and the values ​​are shown in Table 1.

[0036] Table 1: Values ​​of the extended priorityType attribute

[0037] [Step 102] When creating RSSurfaceNode, RSSurfaceNodeConfig is passed to Create, and the priorityType property is extended in RSSurfaceNode. At the same time, the SetRenderPriority interface is provided to allow dynamic adjustment of the priorityType priority parameter, and GetRenderPriority allows the Strategy module to obtain the priorityType priority of this node.

[0038]

Step 2

[0039] [Step 201] When the RenderService starts, a real-time resource monitoring thread is initialized. This thread obtains the current utilization and temperature data of the GPU through HDF at a fixed frequency (e.g., configured 25Hz). The thread records the previous state of utilization and temperature respectively.

[0040] [Step 202] Design resource load levels as shown in Table 2.

[0041] Table 2: Resource Load Levels

[0042] When GPU utilization is less than 70% and temperature is less than 50°C, the load is normal. When GPU utilization is between 70% and 85% or temperature is between 50°C and 65°C, the load is in a warning state and frame rate reduction is required. When GPU utilization is greater than 85% or temperature is greater than 65°C, the load is in an emergency state, at which point the GPU is severely bottlenecked.

[0043] [Step 203] As follows Figure 2 As shown, the processing flow of the real-time resource monitoring module is as follows: periodically obtain the GPU utilization and temperature, determine whether it is the same as the previously stored state range, if the same, the processing ends, otherwise update the Strategy's load level. The advantage is that it does not need to update the Strategy state every time, but only when it is different from the previous state. The judgment logic is all placed in the real-time resource monitoring module to prevent read-write lock operations on the Strategy from causing a decrease in the performance of the graphics subsystem.

[0044]

Step 3

[0045] [Step 301] Design a dynamic rendering strategy that adjusts based on node type, as shown in Table 3.

[0046] Table 3: Dynamic rendering strategy adjusted based on node type

[0047] When the load value is RENDER_LOAD_NORMAL (normal), critical, secondary, and non-critical nodes are rendered at the original frame rate. When the load value is RENDER_LOAD_WARNING (warning state), resources are already relatively saturated. Critical and secondary critical nodes remain at 60 FPS, while non-critical nodes are reduced to 15 FPS, freeing up some resources for critical and secondary critical nodes. When the load value is RENDER_LOAD_CRITICAL (critical state), resources have become a bottleneck. In this case, secondary critical nodes are reduced to 30 FPS, and non-critical nodes are reduced to 15 FPS, reserving resources for critical nodes as much as possible.

[0048] [Step 302] The dynamic rendering strategy module records information on key nodes, secondary key nodes and non-key nodes, as well as the current rendering frequency of each node, through various special data structures. Among them, there are two frame rate control methods.

[0049] Frame skipping method: The rendering frequency is determined by a counter. For example, in the RENDER_LOAD_CRITICAL emergency state, all critical nodes are rendered every frame, secondary critical nodes are rendered every 2 frames, and non-critical nodes are rendered every 4 frames. The advantage is that the strategy is simple and easy to implement, but the disadvantage is that the frame rate cannot be precisely controlled.

[0050] The uniform-rate token method sends tokens at a uniform rate determined by the current resource status for each node type. A rendering node only enters the rendering phase after receiving a token; otherwise, the rendering task is discarded. For example, in the RENDER_LOAD_CRITICAL emergency state, critical nodes produce tokens at 60 per second, and proceed with rendering after receiving a token, maintaining a frame rate of 60 FPS. Secondary critical nodes produce tokens at 30 per second, and only begin rendering after receiving a token, maintaining a frame rate of 30 FPS. The same principle applies to non-critical nodes. The advantage is decoupling frame rate control, allowing for flexible and precise control of the frame rate for each node type. The disadvantage is its complexity in implementation.

[0051]

Step 4

[0052] [Step 401] When the rendering service RenderService starts, Strategy loads the configuration of strategies such as frame reduction and threshold, and starts the Monitor module at the same time.

[0053] [Step 402] The Monitor module continuously senses the GPU utilization and temperature information and compares it with the previous information. If a change in the range is detected, the current load level of the Strategy is updated and the Strategy dynamically adjusts the rendering strategy.

[0054] [Step 403] The APP starts, creates a SurfaceNode and sets its type.

[0055] [Step 404] The APP submits a rendering task. The RenderService obtains the current load level from the Strategy. If the load level is normal, the APP continues to submit rendering tasks, compositing, and display. If the load level is warning or emergency, the APP selectively limits the rate of the task according to the current level, strategy, and frame rate control method.

[0056] In one embodiment, the triggering condition can be a single condition, such as considering only one of the following: GPU utilization, GPU temperature, or GPU memory utilization. However, due to the single dimension, it is impossible to accurately predict the current operating status of the entire system, which can easily lead to policy misjudgment.

[0057] In one embodiment, in addition to frame skipping and average rate token methods, frame rate control methods can also use sliding window or leaky bucket methods, but these are not suitable for certain scenarios and are more complex to implement.

[0058] In one embodiment, the node control granularity may not consider the fine-grained frame rate control based on SurfaceNode, and a globally unified node control may be used. However, this may result in the inability to improve the user experience of critical nodes when resources are bottlenecked.

[0059] The beneficial effects of this invention include: (1) Improved system stability: By monitoring the GPU status in real time and dynamically adjusting the rendering frame rate, the GPU is prevented from being in a high-load state for a long time, reducing the risk of overall system crash due to overload and overheating, ensuring stable system operation, and improving hardware lifespan; (2) Enhanced user experience: The method provided by this invention dynamically adjusts the rendering frame rate according to the criticality of the nodes. When there is a resource bottleneck, the rendering tasks of non-critical nodes can be automatically reduced, effectively ensuring the rendering quality of critical nodes, avoiding problems such as screen stuttering and frame drops, and providing users with a smooth graphics interaction experience; (3) Enhanced system adaptability: This invention can dynamically adjust according to the real-time status of the GPU, enhancing the adaptability and flexibility of the system; (4) Universality of mobile operating systems: The method of this invention can be applied to multiple mobile operating systems (OpenHarmony / Android / IOS), providing graphics subsystem rendering optimization capabilities.

[0060] Please note that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be pointed out that for those skilled in the art, several modifications and improvements can be made without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A dynamic rendering control system, characterized in that, The dynamic rendering control system includes a graphics subsystem and driver hardware; wherein: The graphics subsystem includes: The RenderService graphics rendering framework is used to provide the operating system with 2D and 3D drawing capabilities, as well as animation and display effects. The Strategy module is used to record the priority of applications or SurfaceNodes, and determine whether to perform dynamic frame reduction and the applications or SurfaceNodes that need to be reduced in frame rate based on the current resource status of the Monitor. The Monitor module displays the resource monitoring information. It calls the HDF interface at a fixed frequency to obtain the GPU utilization and temperature in real time and decides whether to update the current information to the Strategy. If the resource status is consistent with the last report, it will not be updated; otherwise, it will update the normal, warning, and emergency status values ​​in the Strategy. The Render component is used to provide APIs and perform rendering processes. The Composer module is used to merge multiple layers into a single unit, and to adjust the transparency, color, and position of elements to achieve seamless blending and transitions. The driver hardware includes: The Hardware Device Framework (HDF) is used to enable communication and interaction between hardware devices and the operating system. GPUs are processors used for parallel data processing, including graphics rendering, parallel computing, and machine learning. Temperature sensors are used to monitor the temperature changes of various components inside the equipment in real time, so that the equipment can operate within the normal operating temperature range. The rendering task is requested by multiple applications (APPs) installed on the device, and the composite graphic image is displayed by the display.

2. A general-purpose mobile operating system dynamic rendering control method based on GPU state awareness, characterized in that, The method utilizes a dynamic rendering control system as described in claim 1 to implement GPU state-aware dynamic rendering control for a general mobile operating system. The method specifically includes: Step 1: Set the app's own priority; Step 2: The Monitor module displays the GPU utilization and temperature in real time and decides whether to update the current information to the Strategy. Step 3: The Strategy decision module determines whether to perform dynamic frame reduction and identifies the application or SurfaceNode that needs frame reduction based on the current resource status of the Monitor; Step 4: The Render component adjusts the dynamic rendering strategy based on the node type and performs the rendering process.

3. The GPU-state-aware dynamic rendering control method for a general-purpose mobile operating system according to claim 2, characterized in that, In step 1: Extend the priorityType property in Rosen::RSSurfaceNodeConfig; When creating an RSSurfaceNode, pass in RSSurfaceNodeConfig and extend the priorityType property in RSSurfaceNode; Among them, the SetRenderPriority interface is provided to dynamically adjust the priorityType parameter, and the GetRenderPriority interface is provided to allow Strategy to obtain the priorityType of the current node.

4. The GPU-state-aware dynamic rendering control method for a general mobile operating system according to claim 3, characterized in that, In step 2: When the RenderService service starts, a resource real-time monitoring thread is initialized. This thread obtains the current utilization and temperature of the GPU through HDF at a fixed frequency, and records the previous state of utilization and temperature in the thread. Configure resource load levels, including normal state, warning state, and emergency state. Frame reduction is performed under the warning state.

5. The GPU-state-aware dynamic rendering control method for a general mobile operating system according to claim 4, characterized in that, In step 3: When the load value is normal, critical nodes, secondary critical nodes, and non-critical nodes are all rendered at the original frame rate. When the load value is warning, the resources are in a relatively saturated state. The critical and secondary critical nodes remain unchanged, while the non-critical nodes reduce the frame rate to release some resources for critical and secondary critical nodes. When the load value is emergency, resources become a bottleneck. Secondary critical nodes and non-critical nodes reduce the frame rate to reserve resources for critical nodes.

6. The GPU-state-aware dynamic rendering control method for a general-purpose mobile operating system according to claim 5, characterized in that, In step 3, frame rate control includes: The rendering frequency is determined by a counter. In the RENDER_LOAD_CRITICAL emergency state, all critical nodes render every frame, secondary critical nodes render every 2 frames, and non-critical nodes render every 4 frames.

7. The GPU-state-aware dynamic rendering control method for a general mobile operating system according to claim 6, characterized in that, In step 3, frame rate control includes: For each node type, tokens are sent at a uniform frame rate determined by the current resource status. Rendering nodes only enter the rendering stage after receiving the token; otherwise, the rendering task is discarded. In the RENDER_LOAD_CRITICAL emergency state, critical nodes produce tokens at 60 per second. After receiving the token, critical nodes proceed with subsequent rendering, with the frame rate controlled at 60 FPS. For secondary critical nodes, tokens are produced at 30 per second. Secondary critical nodes only begin rendering after receiving the token, with the frame rate controlled at 30 FPS.

8. The GPU-state-aware dynamic rendering control method for a general mobile operating system according to claim 7, characterized in that, In step 4: When RenderService starts, Strategy loads the frame reduction and threshold strategy configuration and starts Monitor. Monitor continuously monitors the GPU utilization and temperature information and compares it with previous information. If a change in the range is detected, it updates the current load level of Strategy and Strategy dynamically adjusts the rendering strategy. The app starts by creating a SurfaceNode and setting its type; the app submits a rendering task, and the RenderService obtains the current load level from the Strategy. If the load level is normal, the app continues to submit rendering tasks, compositing, and displaying; if the load level is warning or emergency, the app limits the rate of the task according to the current level, strategy, and frame rate control method.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 2-8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 2-8.

Images