Image frame drawing method, device and terminal equipment

Abstract

The present invention discloses a method, apparatus, and terminal device for drawing an image frame, applicable to the field of image processing technology. The method comprises: obtaining a load event related to drawing before starting to draw an image frame; determining the computing resources required for the load event; and then processing the load event based on the determined computing resources and drawing the image frame. The present invention improves the speed of drawing image frames, thereby improving the smoothness of the terminal device's display.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image frame drawing method, apparatus and terminal device. Background Art

[0002] During operation, terminal devices often experience high-load scenarios (hereinafter referred to as "load scenarios"). For example, when a user slides the terminal interface, the terminal device may experience a sliding scenario. Under load scenarios, the terminal device's image frame rendering speed will slow down, and frame drops or freezes may occur, resulting in poor image smoothness on the terminal device and a reduced user experience.

[0003] Therefore, how to improve the screen smoothness of terminal devices is a problem that needs to be solved in practical applications. Summary of the Invention

[0004] In view of this, embodiments of the present application provide an image frame drawing method, apparatus, and terminal device, which can improve the picture smoothness of the terminal device.

[0005] A first aspect of an embodiment of the present application provides an image frame drawing method, comprising:

[0006] Before starting to draw an image frame, first obtain a load event related to drawing, then determine the computing resources required for the load event, and finally process the load event based on the determined computing resources and draw the image frame.

[0007] In an embodiment of the present application, the terminal device can match the required computing resources for the load events required to execute the current image frame drawing, and allocate appropriate computing resources to handle these load events. Therefore, the embodiment of the present application can adapt to the actual needs of the terminal device for the computing resources required for image frame drawing under various load conditions, and the matching accuracy of computing resources can reach the specific level of each load event, so as to achieve more reasonable utilization of computing resources. When a high load situation occurs, more computing resources can be dispatched in time to handle events related to image frame drawing, thereby improving the image frame drawing speed and reducing the occurrence of frame loss and freezes. Therefore, the embodiment of the present application can improve the screen response speed and smoothness of the terminal device.

[0008] In a first possible implementation manner of the first aspect, determining computing resources required for the load event includes:

[0009] The parameters of the event characteristics of each load event are extracted to obtain the load characteristic parameters of all load events, and then the computing resources required for the load event are determined based on the load characteristic parameters.

[0010] The embodiment of the present application can determine the computing resources required for the load event based on the complexity of the load event. Therefore, the embodiment of the present application has a high degree of computing resource matching for each load event, so that the load event related to the final image frame drawing can achieve high-precision computing resource matching. Therefore, the utilization rate of computing resources can be effectively improved, and while meeting the dynamic requirements of picture smoothness, the situation of excessive supply of computing resources can also be reduced or avoided. Among them, considering the type of load event, the variable parameters involved, and the corresponding scene and other event characteristics, the complexity of the load event can be reflected. Therefore, the complexity of a load event can be well measured by the load characteristic parameters, so that the embodiment of the present application can achieve an accurate assessment of the complexity of the load event.

[0011] In a second possible implementation manner of the first aspect, determining computing resources required for a load event according to load characteristic parameters includes:

[0012] When the acquired load characteristic parameters include parameters of preset characteristics, the computing resources required for the load event are determined according to the preset characteristics.

[0013] The embodiments of the present application can quickly dispatch computing resources when pre-set features are detected. This simplifies the logic for scheduling computing resources and effectively improves the efficiency of resource scheduling. Furthermore, it reduces the workload of scheduling computing resources for image frame rendering on terminal devices, lowers hardware requirements, and can adapt to a wider range of terminal devices with different hardware configurations.

[0014] In a third possible implementation manner of the first aspect, determining computing resources required for the load event includes:

[0015] Parameters of event features are extracted for each load event, and when parameters of preset features are extracted, the computing resources required for the load event are determined based on the preset features.

[0016] The embodiments of the present application have at least the following beneficial effects:

[0017] 1. Therefore, the embodiments of the present application can adapt to the actual computing resource requirements of terminal devices for image frame rendering under various load conditions, and the matching accuracy of computing resources can reach the specific load event level, achieving more rational utilization of computing resources. Therefore, the embodiments of the present application can improve the responsiveness and smoothness of terminal device screens.

[0018] 2. The dynamic scheduling accuracy of computing resources in the embodiment of the present application can reach the single-frame level, that is, it can achieve independent matching of computing resources for each image frame. Therefore, the utilization rate and flexibility of computing resources are extremely high.

[0019] 3. The embodiments of the present application can assess the complexity of a load event using preset features and quickly determine the computing resources required for the load event based on the preset features. This can effectively improve the utilization of computing resources and, while still meeting the requirements for dynamic image smoothness, reduce or avoid the oversupply of computing resources.

[0020] 4. The embodiments of the present application can quickly dispatch computing resources when a preset feature requiring high computing resource processing is determined. This simplifies the logic for scheduling computing resources and effectively improves the efficiency of resource scheduling. Furthermore, it can reduce the workload of scheduling computing resources for image frame rendering on terminal devices, lower hardware requirements, and adapt to a wider range of terminal devices with different hardware configurations.

[0021] In a fourth possible implementation of the first aspect, parameters of event features are extracted for each load event, including: first identifying the current usage scenario, and determining the event features of the required parameters to be extracted based on the usage scenario, the determined event features are the target features, and then parameters of the target features are extracted for each load event.

[0022] In an embodiment of the present application, the terminal device can differentiate and customize the specific event features required for reference based on the computing resource scheduling accuracy requirements of the actual usage scenario. Therefore, the embodiment of the present application can better meet the requirements of the actual usage scenario for image frame drawing during resource scheduling, realize differentiated customization of computing resource scheduling accuracy for different scenarios, enhance the adaptability of computing resource scheduling accuracy to the usage scenario, enhance the degree of optimization of drawing speed for specific scenarios, and enhance the utilization and flexibility of computing resources.

[0023] In a fifth possible implementation of the first aspect, the operation of identifying a current usage scenario and determining event feature extraction parameters required according to the usage scenario includes:

[0024] When the usage scenario is a target scenario, the target features include the first type of features and some or all of the second type of features, wherein the first type of features and the second type of features are two different types of event features.

[0025] When the usage scenario is a scenario other than the target scenario, the target features include the first category features.

[0026] In the embodiment of the present application, the event features required for reference can be distinguished for different scenarios, thereby achieving differentiated customization of the usage scenarios.

[0027] As an optional embodiment of the present application, the target scenario is a usage scenario with higher requirements for the accuracy of computing resource scheduling, or in other words, the accuracy requirements of the target scenario for computing resource scheduling are higher than the accuracy requirements of scenarios other than the target scenario.

[0028] As an optional embodiment of the present application, the impact of the first type of features on the computing resources required for processing load events is higher than the impact of the second type of features on the computing resources required for processing load events.

[0029] In the embodiment of the present application, for target scenarios with high requirements for the accuracy of computing resource scheduling, more event features can be selected for reference, so that when subsequently matching computing resources, the target scenario requirements can be adapted with high accuracy, and the picture smoothness in the target scenario can be optimized in a targeted manner, thereby improving the user's real experience in this usage scenario.

[0030] In a sixth possible implementation manner of the first aspect, determining computing resources required for the load event includes:

[0031] Determine the resource parameters required for the load event and the computing resources corresponding to the resource parameters.

[0032] In the embodiments of the present application, by first determining the computing power requirements of the load event and then determining the computing resources that actually correspond to the computing power requirements, the computing resources to be scheduled for the load event can be effectively quantitatively assessed for the actual computing power required by the load. This makes the assessment and scheduling of computing resources required for the load event more flexible.

[0033] In a seventh possible implementation of the first aspect, determining the computing resource corresponding to the resource parameter includes:

[0034] A first drawing duration is obtained, and resource parameters are processed based on the first drawing duration to obtain computing resources, where the first drawing duration is an estimated duration for drawing an image frame.

[0035] When matching computing resources appropriate for a load event, the embodiments of this application also consider the expected duration of image frame rendering (i.e., the first rendering duration). By limiting this expected duration, the image frame rendering duration can be controlled when scheduling computing resources. This makes the image smoothness of the terminal device more controllable, achieving the effect of improving image smoothness.

[0036] In an eighth possible implementation manner of the first aspect, the first drawing duration is less than or equal to the reciprocal of the maximum screen refresh rate of the terminal device.

[0037] In an embodiment of the present application, the expected duration for drawing an image frame is less than or equal to the minimum refresh interval of the terminal device screen, so that the terminal device draws the image frame faster than the speed at which the terminal device refreshes the screen content, thereby preventing frame loss or freezes.

[0038] In a ninth possible implementation of the first aspect, obtaining a load event related to drawing includes:

[0039] Get the load event from the drawing-related event callback queue. The event callback queue belongs to any of the following three categories of queues: the first category queue related to user operation event processing, the second category queue related to animation-related event processing, and the third category queue related to interface element display event processing.

[0040] In the embodiment of the present application, it is possible to process load events in at least three categories of event callback queues that are strongly related to the image frame drawing process, namely the first queue, the second queue, and the third queue, thereby making the load events referenced by the embodiment of the present application more comprehensive and reasonable. Therefore, the embodiment of the present application matches the computing resources required for image frame drawing more accurately and reliably, thereby making the final control of the image frame drawing speed more reasonable. On the basis of improving the image frame drawing speed and improving the smoothness of the terminal device screen, it can also effectively improve the utilization of computing resources.

[0041] In a tenth possible implementation manner of the first aspect, the operation of extracting event feature parameters for each load event to obtain load feature parameters for all load events includes:

[0042] When the load event belongs to the first type queue, the event feature includes at least one feature of the following first type features: an executable function name and an input displacement variable.

[0043] When the load event belongs to the second type queue, the event feature includes at least one feature of the following first type features: executable class name, control feature, executable function name, and input displacement variable.

[0044] When the load event belongs to the third type of queue, the event characteristics include the first type of characteristics: the drawing change value of the next image frame in the interface.

[0045] Considering that the importance of different event features may vary for each event callback queue. For example, considering that different event features have different impacts on the complexity of the load event, and thus have different impacts on the computing resources required for the load event. The embodiment of the present application can set more important event features for each type of event callback queue, so that the embodiment of the present application can evaluate the computing resources required for the load event more accurately and efficiently.

[0046] In an eleventh possible implementation manner of the first aspect, before starting to draw an image frame and before obtaining a load event related to drawing, the method further includes:

[0047] When a preset trigger condition is met, an operation of obtaining a load event related to drawing is performed before starting to draw an image frame.

[0048] In the embodiment of the present application, some trigger conditions for computing resource scheduling can be set, so that the scheduling operation of computing resources required for image frames can be more targeted.

[0049] In a twelfth possible implementation manner of the first aspect, the triggering condition includes:

[0050] The user's preset operation is detected, or the preset scene is recognized.

[0051] At this time, it can be set as follows: when a preset operation of the user is detected, or when a preset scene is recognized, an operation of obtaining a load event related to drawing is executed before starting to draw an image frame.

[0052] In the embodiment of the present application, the optimization of image frame rendering can be triggered according to specific user operations or scenarios, and the corresponding computing resources can be dispatched. Therefore, the optimization of image frame rendering is more targeted and can meet the needs of more actual users.

[0053] In a thirteenth possible implementation manner of the first aspect, the trigger condition includes: performing an operation of obtaining a load event related to drawing before starting to draw the image frame at a preset trigger frequency.

[0054] In an embodiment of the present application, the scheduling of image frame computing resources can be triggered periodically, so that the image frame drawing speed can be continuously optimized and the smoothness of the terminal device screen can be continuously improved.

[0055] In a fourteenth possible implementation manner of the first aspect, the trigger frequency is 1 / N times the current screen refresh rate of the terminal device, where N is a positive integer.

[0056] As an embodiment of the present application, N = 1. In this case, the terminal device can trigger the scheduling of image frame computing resources each time it draws an image frame. This maximizes the image frame drawing speed and improves the smoothness of the terminal device screen. It also maximizes the utilization and flexibility of computing resources.

[0057] A second aspect of an embodiment of the present application provides an image frame drawing device, including:

[0058] The event acquisition module is used to obtain load events related to drawing before starting to draw an image frame.

[0059] The resource determination module is used to determine the computing resources required for the load event.

[0060] The drawing module is used to process the load event and draw the image frame based on the determined computing resources.

[0061] As an embodiment of the present application, the image frame drawing device can also implement any method of the first aspect described above.

[0062] In a third aspect, an embodiment of the present application provides a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the methods of the first aspect described above when executing the computer program.

[0063] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method of any one of the above-mentioned first aspects.

[0064] In a fifth aspect, embodiments of the present application provide a chip system, comprising a processor coupled to a memory, the processor executing a computer program stored in the memory to implement any of the methods described in the first aspect. The chip system can be a single chip or a chip module consisting of multiple chips.

[0065] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any of the methods described in the first aspect above.

[0066] It can be understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1A A schematic diagram of a sliding scene provided in an embodiment of the present application;

[0068] Figure 1B A timing diagram of image frame drawing in a sliding scenario provided in an embodiment of the present application;

[0069] Figure 1C A schematic diagram comparing the actual computing resources used by a terminal device and the lower limit of the computing resources provided in an embodiment of the present application;

[0070] Figure 2A A schematic diagram of the structure of a mobile phone provided in an embodiment of the present application;

[0071] Figure 2BA software structure diagram of a terminal device provided in an embodiment of the present application;

[0072] Figure 3 A flowchart of an image frame drawing method provided in an embodiment of the present application;

[0073] Figure 4A A schematic diagram of an event callback queue, load events, and load characteristic parameters provided in an embodiment of the present application;

[0074] Figure 4B Schematic diagram of the division of key features and auxiliary features of the three types of event callback queues provided in the embodiment of the present application;

[0075] Figure 5 A schematic diagram of a scenario for obtaining a mapping relationship provided in an embodiment of the present application;

[0076] Figure 6A A flowchart of an image frame drawing method provided in an embodiment of the present application;

[0077] Figure 6B A schematic flow chart of another image frame drawing method provided in an embodiment of the present application;

[0078] Figure 7 A flowchart of an image frame drawing method provided in an embodiment of the present application;

[0079] Figure 8 A schematic diagram of the structure of an image frame drawing device provided in an embodiment of the present application;

[0080] Figure 9 A schematic diagram of the structure of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0082] The following describes some concepts that may be involved in the embodiments of this application:

[0083] Resource parameters: Resource parameters are quantified values ​​of the terminal device's computing power in a certain quantitative manner and can be used to characterize the terminal device's computing power. In the embodiments of this application, resource parameters can also be referred to as computing power requirements.

[0084] The embodiments of the present application do not impose too many restrictions on the quantification method of computing power, which can be set by technical personnel. For example, in some optional embodiments, resource parameters can be some parameters with actual physical meanings, such as the frequency of the processor, the number of floating-point operations per second (TFLOPS), the number of double-precision floating-point operations per second, or the number of bytes processed per second. In other optional embodiments, the resource parameters can also be the score values ​​obtained after the computing power is quantified and scored according to certain scoring rules. For example, assuming that a fixed computing power is a full score of 100, for a single terminal device, the total computing power of the terminal device can be compared with the computing power of 100 points to obtain the total computing power requirement score value of the terminal device.

[0085] It should be understood that in actual applications, resource parameters are parameters with a time dimension, that is, the computing power of a terminal device within a certain period of time. For a terminal device with a fixed hardware configuration, theoretically, the longer the duration, the more information it can process, and therefore the greater the computing power. Unless otherwise specified, resource parameters can be assumed to be the computing power of the terminal device per unit time, where the specific duration of the unit time can be determined by technical personnel, for example, it can be 1 second (s) or 1 millisecond (ms). The above descriptions of resource parameters all omit the definition of the unit time. For example, when a resource parameter is expressed as "processor frequency", its full content should be: the frequency used by the processor per unit time. Considering that in actual applications, terminal devices require a certain amount of time to execute events, and this duration is not necessarily related to the unit time, in actual applications, the duration corresponding to the resource parameter can be set according to actual needs.

[0086] Computing resources: In the embodiment of the present application, computing resources are a general term for the computing resources of the processors in the terminal device that can be used to draw image frames, where the processors include but are not limited to CPUs and GPUs. Since the number of cores contained in different processors and the performance of each core may vary, the amount of computing resources owned by each processor will also vary to a certain extent. When scheduling computing resources, the terminal device can determine the accuracy of the computing resources to be scheduled based on actual needs. For example, in some optional embodiments, the computing resource accuracy can be set to the processor level. In this case, the terminal device can set the scheduled processor according to actual needs, and set the operating frequency requirements for the scheduled processor. In other optional embodiments, the computing resource accuracy can be set to the kernel level. In this case, the terminal device can set the scheduled processor kernel according to actual needs, and set the operating frequency requirements for the scheduled kernel. Among them, for a single-core processor, the computing resource accuracy at the processor level is the same as the computing resource accuracy at the kernel level.

[0087] Load event: In the embodiments of this application, unless otherwise specified, load events refer to some or all events related to image frame drawing. In actual applications, events can be stored and executed in the form of functions, so load events also refer to some or all functions related to image frame drawing. That is, in the embodiments of this application, load events and functions have the same meaning. The specific events included in the load event can be set by the technician. For example, all events related to image frame drawing can be regarded as load events, or some events related to image frame drawing can be selected as load events in the embodiments of this application.

[0088] Load characteristic parameters: Taking into account that the complexity of different events may be different, these different complexities will affect the speed at which the terminal device executes the event. Therefore, when scheduling computing resources, the complexity of the load event is an important reference indicator. In order to measure the complexity of the load event, some event characteristics of the load event can be read in the embodiment of the present application, and the parameters of these event characteristics can be used to characterize the complexity of the load event. These parameters of the event characteristics of the selected load events are collectively referred to as load characteristic parameters in the embodiment of the present application. Among them, the event characteristics required to be referenced for different load events may be different, and the specific details can be determined by technical personnel based on actual needs.

[0089] To illustrate, for example, for load event a (i.e., function a), the referenced event features may include: executable class name, executable function name, core control state, and behavior parameters. For load event b (i.e., function b), the referenced event features may include: executable class name, executable function name, active window name, and user operation type.

[0090] During the operation of the terminal device, as the user uses the terminal device, the terminal device often encounters some high-load scenarios. For example, when the user slides the terminal device interface, the terminal device encounters a sliding scenario. Another example is when the user opens multiple applications on the terminal device at the same time, causing the terminal device to run many tasks at the same time. Under load scenarios, the terminal device often needs to process a large number of events, which may slow down the speed at which the terminal device processes various events, resulting in a slower speed at which the terminal device draws image frames. The slower speed of drawing image frames may cause the terminal device to experience frame loss or screen freezes, resulting in poor screen smoothness of the terminal device and a reduced user experience of the terminal device.

[0091] Taking sliding scenarios as an example, the following are some common sliding scenarios:

[0092] Sliding scenario 1: The user slides the news display interface in a news application.

[0093] Sliding scenario 2: The user performs a sliding operation on the dynamic sharing interface of a social application (such as the Moments interface).

[0094] In scrolling scenarios, users have high expectations for the smoothness of the terminal device interface, meaning they have high expectations for screen smoothness and low tolerance for delays, freezes, or frame drops. While the screen is on, the terminal device needs to continuously draw and display newly drawn frames according to the screen refresh rate to refresh the screen content. When a user scrolls the terminal device interface, the terminal device needs to process numerous scrolling-related events to determine the content to draw for the next frame. For example, when scrolling through content such as videos that require loading a large amount of content, the workload of drawing the next frame increases significantly. Another example is when the user scrolls quickly, resulting in extremely rapid screen content updates, which also significantly increases the workload of drawing frames. Therefore, in scrolling scenarios, the frame drawing process is often complex and time-consuming. When the drawing time for a single frame exceeds the maximum screen refresh time, the terminal device may experience frame drops, reducing the smoothness of the screen.

[0095] You can refer to Figure 1A , is a schematic diagram of a sliding scenario provided by an embodiment of the present application. Figure 1A In (1), the content displayed in the friend circle only contains text and pictures. At this time, the user swipes up, causing the terminal device to display the content containing the video. Figure 1A (2). Since the video content needs to be displayed, the terminal device needs to load the corresponding video controls and video resources. Figure 1A (2) The workload of drawing the image frame is large, and the image frame may not be drawn in time, resulting in the terminal device being unable to draw the image frame in time. Figure 1A (1) to Figure 1A (2) The lag occurs.

[0096] For example, you can refer to Figure 1B, is a timing diagram of image frame drawing in a sliding scenario provided by an embodiment of the present application, wherein image frame a, image frame b, and image frame c are three consecutive image frames, and image frame n is an image frame after image frame c. For image frame a, the terminal device needs to process drawing-related events A1, event B1, event C1, and event D1 when drawing. Processing these events and drawing image frame a takes a total of 1.3 milliseconds. For image frame b, the terminal device needs to process drawing-related events A2, event B2, event C2, and event D2 when drawing. Processing these events and drawing image frame a takes a total of 3.1 milliseconds. For image frame c, the terminal device needs to process drawing-related events A3, event B3, event C3, and event D3 when drawing. Processing these events and drawing image frame a takes a total of 18.2 milliseconds. For image frame n, the terminal device needs to process drawing-related events A4, event B4, event C4, and event D4 when drawing. Processing these events and drawing image frame a takes a total of 2.5 milliseconds. After the image frame b is drawn, a high-load event suddenly occurs in the terminal device, such as the user quickly sliding in the terminal device interface. At this time, the event complexity associated with the image frame c that needs to be drawn is higher. Due to the higher number and complexity of events, the terminal device takes more time to process events A3 to D3. At this time, it takes a total of 18.2 milliseconds for the terminal device to process these events and draw image frame c. Assuming that the refresh rate of the terminal device screen is 90 Hz at this time, the terminal device is required to generate at least 90 image frames per second for refresh display. Accordingly, under normal circumstances, the time it takes for the terminal device to generate a single image frame must be controlled within 11.1 milliseconds. And in Figure 1B In the example shown, the time taken by the terminal device to draw the image frame c is 18.2 milliseconds, which exceeds 11.1 milliseconds. Therefore, the terminal device loses frames.

[0097] In order to improve the smoothness of the screen in the load scenario, the terminal device can choose to dispatch more computing resources according to the load scenario, so that the terminal device can use stronger computing power to process various events in the load scenario, thereby improving the processing efficiency of various events and further improving the speed of the terminal device to draw image frames. Optionally, the following computing resource scheduling scheme can be adopted:

[0098] Solution 1: The terminal device uses the Energy Aware Scheduling (EAS) to divide its historical load into time windows, for example, every 6 to 8 milliseconds. The device then uses this historical load data to predict the load for the current window. Based on this predicted load, the terminal device allocates computing resources to the current window.

[0099] For Solution 1, the current window load predicted based on historical load is not very accurate and has a certain lag compared to the actual load of the current window. For example, sudden loads often occur in load scenarios (i.e., the load suddenly increases in a short period of time), such as in a sliding scenario where the user may suddenly and quickly slide the screen content. Therefore, in the case of sudden loads, the load of the current window predicted based on historical loads is often quite different from the actual load. For example, you can refer to Figure 1B In the embodiment shown, if the load condition of the next image frame is predicted based on the load condition of image frame a and image frame b, since the load of image frame a and image frame b is relatively small, that is, the number and complexity of events to be processed are relatively small, the load condition of the next image frame predicted at this time is also relatively small. However, a sudden high load condition actually occurs after image frame b, so the load of the actual image frame c is much higher than the load condition predicted based on image frame a and image frame b. Therefore, the computing resources allocated based on the load condition predicted by image frame a and image frame b still cannot meet the drawing requirements of image frame c, and the drawing of image frame c will still take a long time. Figure 1B As shown in , the terminal device may increase the corresponding processor frequency (from a low frequency point to a high frequency point) during the drawing process of image frame c (or after the drawing of image frame c is completed). At this time, it cannot cover all the drawing work of image frame c well, so the drawing speed of image frame c will still be slow. At the same time, after the sudden load situation, the terminal device will overestimate the load of the next window due to the sudden load situation. Therefore, the terminal device's prediction of the load situation will lag behind the actual load situation. The computing resources allocated based on these inaccurate and lagging load conditions are often less matched with the actual computing resource requirements of the load scenario. Therefore, the terminal device will still experience frame loss, freezes and other unsmooth screen responses, which will cause users to feel stuck in the operation when using the terminal device.

[0100] Solution 2: The terminal device identifies the load scenario and sets a preset lower limit for computing resources when it identifies the load scenario. During the load scenario, the terminal device processor uses computing resources at or above the lower limit to process various tasks in the load scenario to improve image smoothness.

[0101] For Solution 2, by setting a lower limit to prevent the situation where too few computing resources are allocated during the load scenario, although it can improve the smoothness of the terminal device screen to a certain extent, it is found in actual applications that the terminal device does not maintain a high load during the load scenario, so it is not necessary to use more computing resources all the time. Figure 1C , is a schematic diagram showing the comparison between the actual computing resources used by the terminal device and the lower limit of the computing resources during a load scenario provided by an embodiment of the present application. Figure 1C As can be seen, during most load scenarios, the terminal device's demand for computing resources is low, and the actual amount of computing resources used by the terminal device is below the lower limit. Therefore, in Solution 2, the terminal device will have low computing resource utilization during load scenarios, resulting in an oversupply of computing resources, which will lead to wasted power consumption and heat generation in the terminal device.

[0102] Therefore, how to reasonably allocate the computing resources of the terminal device to improve the smoothness of the terminal device's picture is a problem that needs to be solved in practical applications.

[0103] To more effectively utilize computing resources and improve the smoothness of terminal devices, in an embodiment of the present application, before drawing an image frame, the terminal device first determines the load events associated with the image being drawn, then obtains the computing resources required for drawing these load events. Finally, based on the determined computing resources, the terminal device processes these load events and draws the image frame.

[0104] In an embodiment of the present application, the terminal device can match the required computing resources for the load events required to execute the current image frame drawing, and allocate appropriate computing resources to handle these load events. Therefore, the embodiment of the present application can adapt to the actual needs of the terminal device for the computing resources required for image frame drawing under various load conditions, and the matching accuracy of computing resources can reach the specific level of each load event, so as to achieve more reasonable utilization of computing resources. When a high load situation occurs, the embodiment of the present application can timely dispatch more computing resources to handle events related to image frame drawing, thereby improving the image frame drawing speed and reducing the occurrence of frame loss and freezes. Therefore, the embodiment of the present application can improve the screen response speed and smoothness of the terminal device.

[0105] In addition, the scheduling accuracy of computing resources in the embodiment of the present application can reach the single-frame level, that is, the computing resources for each image frame can be independently matched at most. Therefore, the computing resource utilization and flexibility of the embodiment of the present application are extremely high. Under high load, the computing resources for image frame drawing can be increased as needed to speed up the image frame drawing speed, so that each image frame can be drawn efficiently and quickly, thereby greatly improving the smoothness of the picture. Under low load, the computing resources for image frame drawing can be reduced to save computing resource overhead and reduce terminal device power consumption and heat generation.

[0106] The following describes the scenarios to which the embodiments of this application are applicable:

[0107] Since the embodiments of the present application can adapt to the needs of actual scenarios with image frame level accuracy, the embodiments of the present application can be applied to any scenario that requires optimization of the smoothness of the terminal device screen, including but not limited to any high-load scenarios and low-load scenarios.

[0108] The image frame drawing method provided in the embodiment of the present application can be applied to terminal devices such as mobile phones, tablet computers and wearable devices. In this case, the terminal device is the executor of the image frame drawing method provided in the embodiment of the present application. The embodiment of the present application does not impose any restrictions on the specific type of the terminal device.

[0109] The following takes the mobile phone as an example. Figure 2A FIG. 1 shows a schematic structural diagram of the mobile phone 100 .

[0110] The mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a SIM card interface 195, etc. The sensor module 180 may include a gyroscope sensor 180A, an acceleration sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an ambient light sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, and a touch sensor 180K (of course, the mobile phone 100 may also include other sensors, such as a temperature sensor, a pressure sensor, an air pressure sensor, a bone conduction sensor, etc., which are not shown in the figure).

[0111] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processor (NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. Among them, the controller can be the nerve center and command center of the mobile phone 100. The controller can generate operation control signals based on the instruction opcode and timing signal to complete the control of instruction fetching and execution.

[0112] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0113] The processor 110 can run the image frame drawing method provided in the embodiment of the present application to improve the smoothness of the picture, improve the utilization and flexibility of computing resources, and enhance the user experience. The processor 110 may include different devices. For example, when a CPU and a GPU are integrated, the CPU and the GPU can cooperate to execute the image frame drawing method provided in the embodiment of the present application. For example, part of the algorithm in the image frame drawing method is executed by the CPU, and another part of the algorithm is executed by the GPU to obtain faster processing efficiency. For another example, in the image frame drawing method provided in the embodiment of the present application, all algorithms are executed by the CPU, or all algorithms are executed by the GPU.

[0114] Mobile phone 100 implements display functionality through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0115] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0116] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display 194 and the camera 193. MIPI interfaces include the camera serial interface (CSI) and the display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the camera function of the mobile phone 100. The processor 110 and the display 194 communicate via the DSI interface to implement the display function of the mobile phone 100.

[0117] The GPIO interface can be configured via software. The GPIO interface can be configured as either a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to the camera 193, display 194, wireless communication module 160, audio module 170, sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0118] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present application is merely an illustrative illustration and does not constitute a structural limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may also adopt a different interface connection method from the above embodiment, or a combination of multiple interface connection methods.

[0119] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), organic light-emitting diode (OLED), active-matrix organic light-emitting diode (AMOLED), flexible light-emitting diode (FLED), MiniLED, MicroLED, Micro-oLED, quantum dot light-emitting diode (QLED), etc. In some embodiments, mobile phone 100 may include one or N display screens 194, where N is a positive integer greater than 1. Display screen 194 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces (GUIs). For example, display screen 194 can display photos, videos, web pages, or files. For another example, display screen 194 can display a graphical user interface. The graphical user interface includes a status bar, a hideable navigation bar, a time and weather widget, and application icons, such as a browser icon. The status bar includes the operator name (such as China Mobile), the mobile network (such as 4G), the time, and the remaining power. The navigation bar includes a back key icon, a home key icon, and a forward key icon. In addition, it is understood that in some embodiments, the status bar may also include a Bluetooth icon, a Wi-Fi icon, an external device icon, etc. It is also understood that in other embodiments, the graphical user interface may also include a Dock bar, which may include commonly used application icons, etc. When the processor detects a touch event of a user's finger (or stylus, etc.) on an application icon, it opens the user interface of the application corresponding to the application icon in response to the touch event, and displays the user interface of the application on the display screen 194.

[0120] In the embodiment of the present application, the display screen 194 can be an integrated flexible display screen, or a spliced ​​display screen consisting of two rigid screens and a flexible screen located between the two rigid screens.

[0121] Camera 193 (either a front-facing camera or a rear-facing camera, or one camera serving as both) is used to capture still images or videos. Typically, camera 193 includes a photosensitive element, such as a lens assembly and an image sensor. The lens assembly includes multiple lenses (convex or concave) that capture light signals reflected from the object to be photographed and transmit the captured light signals to the image sensor. The image sensor generates an original image of the object to be photographed based on the light signals.

[0122] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store the code of the operating system, application programs (such as camera applications, WeChat applications, etc.). The data storage area can store data created during the use of the mobile phone 100 (such as images, videos, etc. collected by the camera application), etc.

[0123] The internal memory 121 may also store one or more computer programs corresponding to the image frame drawing method provided in the embodiment of the present application. The one or more computer programs are stored in the above-mentioned memory 121 and are configured to be executed by the one or more processors 110. The one or more computer programs include instructions, which can be used to execute the following instructions: Figures 3 to 7 In each step of the corresponding embodiment, the computer program may include an account verification module and a priority comparison module. The account verification module is used to authenticate the system authentication account of other terminal devices in the local area network; the priority comparison module can be used to compare the priority of the audio output request service and the priority of the current output service of the audio output device. The status synchronization module can be used to synchronize the device status of the audio output device currently connected to the terminal device to other terminal devices, or synchronize the device status of the audio output device currently connected to other devices to the local. When the code of the image frame drawing method stored in the internal memory 121 is executed by the processor 110, the processor 110 can control the terminal device to schedule computing resources for image frame-related events.

[0124] In addition, the internal memory 121 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0125] Of course, the code of the image frame drawing method provided in the embodiment of the present application can also be stored in an external memory. In this case, the processor 110 can execute the code of the image frame drawing method stored in the external memory through the external memory interface 120, and the processor 110 can control the terminal device to perform data processing of image frame related events.

[0126] The functions of the sensor module 180 are described below.

[0127] Gyroscope sensor 180A can be used to determine the motion posture of mobile phone 100. In some embodiments, gyroscope sensor 180A can be used to determine the angular velocity of mobile phone 100 around three axes (i.e., x, y, and z axes). In other words, gyroscope sensor 180A can be used to detect the current motion state of mobile phone 100, such as whether it is shaking or still.

[0128] When the display screen in the embodiment of the present application is a foldable screen, the gyroscope sensor 180A can be used to detect a folding or unfolding operation on the display screen 194. The gyroscope sensor 180A can report the detected folding or unfolding operation as an event to the processor 110 to determine the folded state or unfolded state of the display screen 194.

[0129] Acceleration sensor 180B can detect the magnitude of the acceleration of mobile phone 100 in various directions (generally three axes). That is, gyroscope sensor 180A can be used to detect the current motion state of mobile phone 100, such as whether it is shaking or stationary. When the display screen in the embodiment of the present application is a foldable screen, acceleration sensor 180B can be used to detect the folding or unfolding operation acting on display screen 194. Acceleration sensor 180B can report the detected folding or unfolding operation as an event to processor 110 to determine the folded state or unfolded state of display screen 194.

[0130] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The mobile phone emits infrared light outward through the light emitting diode. The mobile phone uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the mobile phone. When insufficient reflected light is detected, the mobile phone can determine that there is no object near the mobile phone. When the display screen in the embodiment of the present application is a foldable screen, the proximity light sensor 180G can be set on the first screen of the foldable display screen 194, and the proximity light sensor 180G can detect the size of the folding angle or unfolding angle between the first screen and the second screen based on the optical path difference of the infrared signal.

[0131] The gyroscope sensor 180A (or the acceleration sensor 180B) can send the detected motion state information (such as angular velocity) to the processor 110. The processor 110 determines whether the current state is handheld or tripod based on the motion state information (for example, when the angular velocity is not 0, it indicates that the mobile phone 100 is in the handheld state).

[0132] The fingerprint sensor 180H is used to collect fingerprints. The mobile phone 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0133] The touch sensor 180K is also called a "touch panel." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the mobile phone 100, in a location different from that of the display screen 194.

[0134] For example, the display screen 194 of the mobile phone 100 displays a main interface, which includes icons of multiple applications (such as a camera application, a chat application, etc.). The user clicks the icon of the camera application in the main interface through the touch sensor 180K, triggering the processor 110 to start the camera application and turn on the camera 193. The display screen 194 displays the interface of the camera application, such as the viewfinder interface.

[0135] The wireless communication function of the mobile phone 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0136] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile phone 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0137] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the mobile phone 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110. In an embodiment of the present application, the mobile communication module 150 can also be used to exchange information with other terminal devices.

[0138] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0139] The wireless communication module 160 can provide wireless communication solutions applied to the mobile phone 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive signals to be sent from the processor 110, frequency modulate them, amplify them, and convert them into electromagnetic waves for radiation through the antenna 2. In the embodiment of the present application, the wireless communication module 160 can be used to access an access point device and send and receive messages to other terminal devices.

[0140] In addition, the mobile phone 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor. For example, music playback, recording, etc. The mobile phone 100 can receive input from the key 190 and generate key signal input related to the user settings and function control of the mobile phone 100. The mobile phone 100 can use the motor 191 to generate a vibration prompt (such as a vibration prompt for an incoming call). The indicator 192 in the mobile phone 100 can be an indicator light, which can be used to indicate the charging status, power changes, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface 195 in the mobile phone 100 is used to connect the SIM card. The SIM card can be inserted into the SIM card interface 195 or pulled out from the SIM card interface 195 to achieve contact and separation with the mobile phone 100.

[0141] It should be understood that in actual applications, the mobile phone 100 may include Figure 2A The embodiments of the present application are not limited to more or fewer components shown. The illustrated mobile phone 100 is merely an example, and the mobile phone 100 may have more or fewer components than shown, may combine two or more components, or may have a different component configuration. The various components shown in the figures may be implemented in hardware, including one or more signal processing and / or application-specific integrated circuits, software, or a combination of hardware and software.

[0142] The software system of the terminal device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal device. Figure 2B It is a software structure block diagram of the terminal device in an embodiment of the present application.

[0143] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0144] The application layer can include a series of application packages.

[0145] like Figure 2B As shown, in an embodiment of the present application, the application layer includes an application package with an interactive interface, and the application includes a visualization component, an interface control module, an input operation module and a control drawing operation module.

[0146] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0147] In an embodiment of the present application, the application framework layer includes a choreographer mechanism for uniformly scheduling the drawing of image frames on the terminal device interface. The choreographer belongs to the image frame drawing management class and can also perform drawing-related load identification in the image frame drawing method provided in the embodiment of the present application. Specifically, the choreographer can be used to coordinate the triggering time of three events: animation, input, and drawing. Whenever a synchronization signal arrives, the choreographer can take out the corresponding tasks from the corresponding queue and call back in the order of user operation event processing, motion effect related event processing, and interface element display event processing, and finally trigger the execution of touch event distribution, animation execution, and drawing process related logic. Among them, corresponding feature reporting interfaces are provided for user operation event processing, motion effect related event processing, and interface element display event processing to achieve the acquisition and reporting of load feature parameters corresponding to the load event. After obtaining the load characteristic parameters for user operation event processing, motion effect-related event processing, and interface element display event processing, the choreographer can perform load characteristic calculations on the load characteristic parameters, determine the resource parameters required for these load characteristic parameters, and report these resource parameters to the system resource scheduling service (FPS SchedService) in the system library.

[0148] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0149] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0150] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0151] refer to Figure 2BIn this embodiment of the present application, the system library contains a system resource scheduling service, which interacts with the choreographer via the Java Native Interface (JNI). The system resource scheduling service can perform scene recognition on the terminal device and, after obtaining resource parameters of load characteristic parameters, write the resource parameters to the kernel layer node through inter-frame load writing for use by the kernel layer's energy-efficiency-aware scheduler EAS. The node written to can be / proc / fps / cur_draw_util.

[0152] The kernel layer is the layer between hardware and software. In an embodiment of the present application, the kernel layer includes an energy-efficiency-aware scheduler, which includes a drawing tool class (get_draw_util), a CPU tool class (CPU_util), and an entity tool class (entity_util), as well as a find_energy_efficient_cpu function and a cpu_util_next function. The find_energy_efficient_cpu function can be used to find a target CPU with the highest energy efficiency ratio for the awakened task, and the cpu_util_next function can be used to obtain the CPU status. The energy-efficiency-aware scheduler can obtain resource parameters from the node and schedule computing resources based on the resource parameters.

[0153] In an embodiment of the present application, the kernel layer may interact with hardware, for example, interact with a CPU or a GPU to schedule computing resources of the corresponding CPU or GPU.

[0154] The following describes the workflow of the software and hardware of the mobile phone 100 by way of example, with reference to the mobile phone 100 drawing a scene based on an image frame.

[0155] When the touch sensor 180K receives a touch operation, the choreographer of the framework layer can detect the load events related to the image frame drawing generated by the touch operation, and determine the event characteristics related to these load events. After determining the event characteristics, the choreographer then performs load characteristic calculations on the event characteristics, determines the load characteristic parameters of these event characteristics, and reports these load characteristic parameter loads to the system resource scheduling service in the system library. The system resource scheduling service then writes the load characteristic parameters to the kernel layer node: / proc / fps / cur_draw_util through inter-frame load writing. The energy-efficiency-aware scheduler obtains resource parameters from the node: / proc / fps / cur_draw_util, and then determines the computing resources required to draw the image frame based on the resource parameters, and schedules the corresponding CPU or GPU resources to draw the image frame, thereby completing the drawing of the image frame.

[0156] In order to illustrate the technical solution described in this application, specific embodiments are provided below.

[0157] Figure 3 The following is a flowchart of the image frame drawing method according to the first embodiment of the present application, which is described in detail as follows:

[0158] S101: Before starting to draw an image frame, the terminal device obtains a load event related to image frame drawing.

[0159] The normal process of a terminal device drawing an image frame generally includes the following steps: creating a canvas, measuring the layout, creating a view, rendering, compositing, exchanging buffers, and sending to the display. From creating the canvas to sending to the display is a series of steps for the terminal device to draw the image frame after calling the frame-related functions. During the screen-on period, the terminal device continuously draws and displays image frames at the set screen refresh rate to refresh the screen content. The embodiment of the present application can be regarded as a supplementary solution to the normal drawing of image frames and can be applied to the drawing operation of image frames. Specifically, after the image frame drawing operation is triggered, the embodiment of the present application can first determine the computing resources required for drawing the image frame before the normal drawing of the image frame begins, and then perform the above-mentioned drawing steps from creating the canvas to sending to the display based on these computing resources. Therefore, the execution timing of S101 can be: after the image frame drawing operation (On Vsync) is triggered, before the image frame drawing starts.

[0160] It should be understood that in actual applications, the embodiment of the present application can be flexibly applied to the drawing operation of any image frame. The specific triggering conditions of S101 are not limited here and can be set by technicians. As an optional embodiment of the present application, the triggering conditions can be set to any one or more of the following triggering conditions:

[0161] Trigger condition 1: A preset function for controlling the switch of the embodiment of the present application is set in the terminal device, and the operation of S101 is executed when the preset function is turned on. In this case, S101 can be replaced with: if the preset function is turned on, before starting to draw the image frame, obtain a drawing-related load event.

[0162] Trigger condition 2: The terminal device executes the operation of S101 at a preset trigger frequency. The trigger frequency can be 1 / N times the terminal device's refresh rate, where N is a positive integer. That is, the operation of S101 can be triggered once every N image frames are drawn. For example, N can be set to 1, in which case the terminal device triggers the operation of S101 every time an image frame is drawn.

[0163] Trigger condition 3: When the terminal device detects a preset operation of the user or identifies a preset scenario, it starts to execute the operation of S101. At this time, the terminal device can apply the embodiment of the present application for some specific operations or scenarios.

[0164] In actual applications, any one or more of the above-mentioned trigger conditions can be selected as the actual trigger conditions of S101. At the same time, other trigger conditions other than the above-mentioned trigger conditions can also be selected to trigger the operation of S101, which is not limited here. When multiple trigger conditions are selected at the same time, the trigger conditions can be combined in an "and" or "or" manner. "And" means that the parallel trigger conditions must be met at the same time before S101 is executed. "Or" means that S101 can be executed if any one or more of the parallel trigger conditions are met. For example, suppose trigger conditions 1 and 2 are selected at the same time and combined in an "and" manner. At this time, after the preset function is turned on, the terminal device executes the operation of S101 at a trigger frequency, where the trigger frequency is 1 / N times the refresh rate of the terminal device. For another example, trigger condition 2 and trigger condition 3 can be combined. At this time, the terminal device can turn on the preset function when detecting the user's preset operation or when recognizing the preset scene, so as to achieve targeted optimization of the preset operation and preset scene.

[0165] When the trigger conditions are met, the terminal device begins to obtain load events related to image frame drawing, that is, reads functions related to image frame drawing. Among them, when the load events are all events related to image frame drawing, the terminal device can read out all events related to image frame drawing one by one. When the load events are some events related to image frame drawing, the technical staff can pre-set the event list or filter range for these load events, and then filter out the load events from the events related to image frame drawing according to the event situation or filter range. Among them, events related to image frame drawing refer to events whose content is related to the image frame content to be drawn, such as events related to executing steps such as creating a canvas, measuring layout, creating a view, rendering, synthesis, exchanging buffers and sending to display.

[0166] The embodiments of this application do not impose any restrictions on the method for reading load events, which can be customized by technicians. For example, in some embodiments, a terminal device can obtain drawing-related load events from an event callback queue related to image frame drawing. This method can obtain all load events from the event callback queue at once, or it can read the load events in the event callback queue one by one in an erroneous manner.

[0167] S102: The terminal device extracts parameters of event characteristics of each load event to obtain load characteristic parameters of each load event.

[0168] After determining each load event, the terminal device can evaluate the complexity of each load event so that the computing resources required for each load event can be accurately determined based on the complexity data. In some embodiments, the complexity of a load event can also be referred to as the estimated load of the load event. The method for evaluating the complexity of a load event is not specifically defined herein and can be customized by the technician.

[0169] As an optional embodiment of the present application, taking into account the type of load event (i.e., function), the variable parameters involved, and the corresponding scenario and other event characteristics, the complexity of the load event can be reflected. In theory, for a single load event, the more event characteristic parameter content, the more computing resources are required when processing the load event. Therefore, in the embodiment of the present application, some or all of the event characteristic parameters of the load event can be used as load characteristic parameters, and the load characteristic parameters can be used as complexity data.

[0170] As an optional embodiment of the present application, when implementing S101 and S102 and obtaining load events and complexity data (load characteristic parameters), you can choose to obtain all load events at one time, and then obtain the load characteristic parameters of these load events. You can also choose to traverse and obtain each load event one by one, and after each load event is obtained, continue to obtain the load characteristic parameters of the corresponding load event. For example, when obtaining a load event from an event callback queue related to image frame drawing, you can choose to traverse the event callback queue. And each time a load event is determined from the event callback queue, continue to determine the load characteristic parameters corresponding to the load event until all event callback queues are traversed and all load events and corresponding load characteristic parameters are obtained.

[0171] As a specific example of obtaining load characteristic parameters in the embodiment of the present application, please refer to Figure 4A , is a schematic diagram of the event callback queue, load events, and load characteristic parameters provided in the embodiment of the present application. In the embodiment of the present application, based on the classification of the image frame drawing process, the event callback queue includes at least the following three categories:

[0172] Type 1: Queues related to user operation event processing, also known as first-class queues;

[0173] Type 2: Queues related to animation-related event processing, also known as the second type of queue;

[0174] Type 3: Queues related to interface element display event processing, also known as the third type of queue.

[0175] Each type can contain one or more specific event callback queues. Each event callback queue contains one or more functions to be executed (i.e. load events), and each function can extract some preset event features to obtain corresponding load feature parameters. For example, Figure 4A In the illustrated embodiment, the queue related to user operation event processing includes an event callback queue 1, and the event callback queue 1 includes function 1, function 2, function 3, function 4...function n, a total of n functions, where n is an arbitrary positive integer. For the functions in the queue related to user operation event processing, the event features referenced by them are set to include: executable class name, executable function name, member parameters of the key execution class, scenario, core control state and behavior parameters. At this time, the specific event callback queue 1 and the n functions contained in the queue can be determined first. For the specific function 1 in the queue, the feature parameters of the executable class name, executable function name, member parameters of the key execution class, scenario, core control state and behavior parameters can be extracted to obtain the corresponding load feature parameters of function 1.

[0176] It should be understood that Figure 4A This is just an example. In actual application, the event callback queue may contain more than Figure 4A The embodiments shown may have more or fewer types.

[0177] Take an example Figure 4A The embodiment shown is used as an example for illustration. Assume that the function TraversalRunnable in the queue related to the interface element display event processing extracts the load characteristic parameters. In the embodiment of the present application, the load characteristic parameters can be obtained as follows:

[0178] Executable class name: ViewRootImpl;

[0179] Executable function name: TraversalRunnable;

[0180] Key execution class member parameters: whether to perform measurement, whether to perform layout, drawing complexity, and key member parameters: mStoped;

[0181] Scenario: Moments sliding scenario, and the user is in the Fling state;

[0182] Core control status: number of loaded items;

[0183] Behavior parameters: The animation is in progress, and the displacement variable is 15 pixels.

[0184] As an optional embodiment of the present application, it is taken into account that for each event callback queue, the importance of different event features may vary to a certain extent. For example, different event features have different effects on the complexity of the load event, and thus have different degrees of influence on the computing resources required for the load event. Therefore, in the embodiment of the present application, technicians can divide the event features into key features and auxiliary features for each event callback queue or each type of event callback queue. In actual applications, the parameters of the key features can be selected as load feature parameters, and for the auxiliary features, any number of parameters of the auxiliary features can be selected as load feature parameters. Therefore, the evaluation of the computing resources required for the load event by the embodiment of the present application can be more accurate and efficient. Among them, the key features can also be called the first type of features, and the auxiliary features can also be called the second type of features. The key features are more important to the load event than the auxiliary features.

[0185] As an optional embodiment of the present application, the impact of key features on the complexity of load events is greater than the impact of auxiliary features on the complexity of load events. That is, when processing a load event, the impact of key features on the time required to process the load event is greater than the impact of auxiliary features on the time required to process the load event. Alternatively, it can be described as follows: when processing a load event, the impact of key features on the computing resources required to process the load event is greater than the impact of auxiliary features on the computing resources required to process the load event.

[0186] You can refer to Figure 4B , is a schematic diagram of the division of the key features and auxiliary features of the three types of event callback queues provided in the embodiment of this application. In the embodiment of this application:

[0187] For type 1 queues related to user action event processing, key features may include: executable function name, input displacement variable (e.g., the number of pixels displaced in a certain coordinate direction in a move event). Auxiliary features may include: executable class name, active window name, and user action type (e.g., click or slide).

[0188] For type 2 queues related to animation-related event processing, key features can include: executable class name, control features (such as list item type, list item width and height, number of list item caches, and list control width and height), executable function name, and input offset variables. Auxiliary features can include: active window name and user operation type.

[0189] For type 3 queues related to UI element display event processing, key features may include: the drawing change value of the next frame in the UI (including layout requirement variables and stop change variables). Auxiliary features may include: the active window name, user operation type, and input displacement variables.

[0190] As an optional embodiment of the present application, different requirements for the accuracy of computing resource scheduling are taken into account in different usage scenarios. For example, for some usage scenarios where the interface display content varies greatly and users have high expectations for the smoothness of the screen, such as the circle of friends and news reading interface, text, audio and video may appear and the interface display content changes frequently. The content of their image frames varies greatly. That is, the content displayed between different image frames is very different, resulting in a large difference in the workload of drawing different image frames. At the same time, users have high expectations for the smoothness of the screen and are less tolerant of delays, freezes or frame drops in the screen. Therefore, these usage scenarios often have greater requirements for the accuracy of computing resource scheduling. Therefore, in order to be able to better meet the requirements of the actual usage scenario for image frame drawing when scheduling resources, differentiation of computing resource scheduling accuracy for different scenarios is achieved. In the embodiment of the present application, S102 can be replaced by: S1021 to S1022.

[0191] S1021: The terminal device identifies the current usage scenario and determines event features of the required parameters to be extracted based on the usage scenario. The determined event features may also be referred to as target features.

[0192] The embodiments of the present application do not impose any restrictions on the method for identifying the usage scenario, which can be set by technicians and is not limited here. For example, in some embodiments, the usage scenario can be identified based on some scene features of the current scene. For example, the specific usage scenario can be identified based on the current window name or the control name or control type contained in the current window.

[0193] As an optional embodiment of the present application, "determining event features of required extraction parameters according to the usage scenario" in S1021 can be replaced by: S12011 and S12012.

[0194] S12011: If the usage scenario is a target scenario, the target features include key features and some or all of the auxiliary features.

[0195] S12012: If the usage scenario is a scenario other than the target scenario (also referred to as a non-target scenario), the target features include key features.

[0196] Target scenarios are scenarios with higher requirements for computing resource scheduling accuracy. Specifically, the target scenarios' accuracy requirements or demands for computing resource scheduling are higher than those of non-target scenarios. Specifically, technical personnel can classify scenarios based on the actual accuracy requirements for computing resource scheduling in each scenario, thereby determining the specific target scenarios and identifying the characteristics of target scenarios that can be used to distinguish them from other scenarios.

[0197] S1022: The terminal device extracts target feature parameters for each load event.

[0198] After determining the specific target features to be used, the terminal device extracts load feature parameters based on the target features.

[0199] In an embodiment of the present application, the terminal device can differentiate and customize the specific event features required for reference according to the computing resource scheduling accuracy requirements of the actual usage scenario. For example, for a target scenario with high accuracy requirements for computing resource scheduling, more event features can be selected for reference, so that when the computing resources are subsequently matched, the target scenario requirements can be adapted with high accuracy, and the picture smoothness under the target scenario can be optimized in a targeted manner, thereby improving the user's real experience in the usage scenario. Therefore, the embodiment of the present application can be more in line with the requirements of the actual usage scenario for image frame drawing during resource scheduling, realize differentiated customization of computing resource scheduling accuracy for different scenarios, enhance the adaptability of computing resource scheduling accuracy for usage scenarios, enhance the degree of optimization of drawing speed for specific scenarios, and enhance the utilization and flexibility of computing resources.

[0200] As an optional embodiment of the present application, the embodiment of the present application can be combined with Figure 2B At this time, after obtaining the load characteristic parameters, the terminal device can pass the load characteristic parameters to the resource scheduling service, which writes them to the file node to trigger a new scheduling window.

[0201] S103: The terminal device determines the total resource parameters required for all load events according to the load characteristic parameters.

[0202] After obtaining the complexity data of all load events through S102, the terminal device begins to determine the appropriate computing power requirements for processing the load events based on the complexity. Specifically, in the embodiment of the present application, the technical personnel can pre-set the mapping relationship between complexity and computing power requirements (also referred to as the first mapping relationship). On this basis, the terminal device can query the resource parameters corresponding to each load event one by one according to the mapping relationship, and then summarize these resource parameters to obtain the total resource parameters required for all load events. When load characteristic parameters are used to characterize the complexity data, the resource parameters required for each load event under different load characteristic parameters are recorded in the mapping relationship. This allows the terminal device to query the appropriate computing power requirements corresponding to the load event through the mapping relationship.

[0203] To illustrate the above mapping relationship with an example, please refer to Figure 5, is a scene diagram of obtaining a mapping relationship provided by an embodiment of the present application. In the embodiment of the present application, the "frequency point of the processor" is set to represent the computing power demand. In the embodiment of the present application, technicians can simulate the load events that may be involved in image frame drawing in advance. In the process of simulating each load event, various event characteristics of the load event can be designed to obtain the appropriate resource parameters for the load event under various load characteristic parameters. For example, Figure 5 In the embodiment of the present application, it is assumed that the load characteristic parameters of a load event include: executable class name, executable function name, member parameters of the key execution class, scenario, core control status and behavior parameters. At this time, in the embodiment of the present application, technicians can set a variety of different parameter value combinations for these load characteristic parameters, and perform execution time consumption tests of load events for each specific load characteristic parameter. Since the processor frequency used in the test is known, after obtaining the test execution time, the test execution time is multiplied by the test frequency to obtain the computing power required for the load event under the load characteristic parameters. By testing different load characteristic parameters separately, the appropriate computing power requirements for load events under different load characteristic parameters can be obtained.

[0204] Let's take an example to illustrate. Suppose that in a test, for the load event: TraversalRunnable, the load characteristic parameters are set as follows:

[0205] Executable class name: ViewRootImpl;

[0206] Executable function name: TraversalRunnable;

[0207] Member parameters of key execution classes: no measurement and no layout required;

[0208] Scenario: Moments sliding scenario, and the user is in the Fling state;

[0209] Core control status: the number of loaded entries is 2;

[0210] Behavior parameters: The animation is in progress, and the displacement variable is 15 pixels.

[0211] Assume that during the test, the total frequency of the processor used to process the load event is 2000Hz, and the test execution time is 1ms (that is, the load event is executed in 0.001s). At this time, it can be obtained that under this load characteristic parameter, the computing power required for the load event: TraversalRunnable is 2000×0.001=2. In actual operation, if after the load characteristic parameters are determined in S102, if it is found according to the mapping relationship query that the load characteristic parameters of the load event: TraversalRunnable are the load characteristic parameters in this instance, then it can be determined that the computing power required for the load event: TraversalRunnable is 2.

[0212] S104: The terminal device determines the computing resources corresponding to the resource parameters, processes the load event based on the determined computing resources, and draws the image frame.

[0213] After determining the appropriate computing power requirements for the load event, the master device first determines the computing resources corresponding to the computing power requirements. Specifically, if the processor used to draw image frames in the terminal device only includes a single-core processor, for example, only a single-core CPU is used to draw image frames, the terminal device can determine a corresponding processor frequency based on the computing power requirements and schedule the processor to handle the load event based on this frequency, thereby improving the processing speed of the load event. This increased processing speed also improves the speed of image frame drawing. If the terminal device includes multiple processors for drawing image frames, or only includes a multi-core processor, for example, both the CPU and GPU can be used to draw image frames, or a multi-core GPU or CPU can be used to draw image frames, the terminal device can determine the specific processor core to use and the specific core frequency requirements for the core to be used based on the computing power requirements. For example, only one core of the processor can be used and the corresponding core frequency can be set to process these load events and draw image frames. Alternatively, multiple cores can be used simultaneously, with the corresponding core frequency set for each core to process these load events and draw image frames. The embodiments of the present application do not impose too many restrictions on the specific method for determining computing resources, which can be set by technical personnel.

[0214] After determining the computing resources to be scheduled, the terminal device can schedule the corresponding computing resources to process load events and draw the corresponding image frames, thereby increasing the drawing speed of the image frames and improving the smoothness of the terminal device screen.

[0215] As an optional embodiment of the present application, it is considered that the processing of load events is an operation that requires a certain amount of time. Reducing the time spent on processing load events can increase the speed of drawing image frames, thereby improving the screen smoothness of the terminal device. Therefore, when determining computing resources, the time required for drawing image frames can be comprehensively considered. Specifically, a technician can pre-set an expected drawing time for a single image frame (also referred to as a first drawing time), which is the expected time required to draw the image frame. Among them, for a terminal device with a fixed screen refresh rate, the expected drawing time should be less than or equal to the interval between screen refreshes of the terminal device, that is, less than or equal to the reciprocal of the screen refresh rate. For a terminal device with multiple screen refresh rates, the expected drawing time should be less than or equal to the interval between the minimum screen refreshes of the terminal device, that is, less than or equal to the reciprocal of the maximum screen refresh rate of the terminal device. This ensures that the speed of drawing image frames is faster than the speed at which the terminal device refreshes the screen content, thereby preventing frame loss or freezes. For example, in some optional embodiments, assuming the terminal device screen refresh rate is 90Hz, the expected rendering time is less than or equal to 1 / 90 second. Assuming the terminal device screen refresh rate can be set to 60Hz, 90Hz, or 120Hz, the expected rendering time is less than or equal to 1 / 120 second. As an optional embodiment of the present application, the expected rendering time can be set to any value between 3ms and 8ms, such as 5ms.

[0216] On the basis of setting the expected drawing duration, S104 can convert the resource parameters when determining the computing resources, that is, calculate the minimum computing resources required for the terminal device to execute all load events within the expected drawing duration based on the resource parameters. Take an example to illustrate, set the "processor frequency" to represent the computing power demand, and assume that the total computing power demand determined is 10. At the same time, assuming that the expected drawing duration is 5ms (i.e. 0.005s), the minimum frequency required for scheduling is: 10÷0.005=2000Hz. That is, at this time, the terminal device is required to schedule at least a processor frequency of 2000Hz to process load events, so that the processing time for all load events is controlled within 5ms. Among them, the scheduling of the 2000Hz processor frequency can be scheduling a single core or scheduling multiple cores, which is not limited here.

[0217] As an optional embodiment of the present application, in order to improve the utilization of computing resources, a scheduling termination time for each scheduling of computing resources can be set based on S104. That is, when S104 schedules computing resources to process load events, the timing starts, and when the timing reaches the scheduling termination time, the occupation of the computing resources determined in the embodiment of the present application is terminated. Among them, as an embodiment of the present application, when an expected drawing time is set, the scheduling termination time should be greater than or equal to the expected drawing time to prevent interference with the image frame drawing operation.

[0218] In the embodiments of the present application, there are at least the following beneficial effects:

[0219] 1. The terminal device can match the required computing resources for the load events required to execute the current image frame drawing, and allocate appropriate computing resources to handle these load events. Therefore, the embodiment of the present application can adapt to the actual needs of the terminal device for the computing resources required for image frame drawing under various load conditions, and the matching accuracy of computing resources can reach the specific level of each load event, so as to achieve more reasonable use of computing resources. When a high load situation occurs, the embodiment of the present application can timely dispatch more computing resources to handle events related to image frame drawing, thereby improving the image frame drawing speed and reducing the occurrence of frame loss and freezes. Therefore, the embodiment of the present application can improve the screen response speed and smoothness of the terminal device.

[0220] 2. The dynamic scheduling accuracy of computing resources in the embodiment of the present application can reach the single-frame level, that is, the computing resources for each image frame can be independently matched. Under high load, the computing resources for image frame drawing can be increased as needed to speed up the image frame drawing speed, so that each image frame can be drawn efficiently and quickly, thereby greatly improving the smoothness of the picture. Under low load, the computing resources for image frame drawing can be reduced to save computing resource overhead and reduce terminal device power consumption and heat generation. Therefore, the embodiment of the present application has extremely high computing resource utilization and flexibility.

[0221] 3. The embodiment of the present application can evaluate the complexity of the load event through load characteristic parameters and other methods, and can determine the computing resources required for the load event based on the complexity of the load event. Therefore, the embodiment of the present application has a high degree of computing resource matching for each load event, so that the load event related to the final image frame drawing can achieve high-precision computing resource matching. Therefore, the utilization rate of computing resources can be effectively improved, and while meeting the dynamic requirements of the picture smoothness, it can also reduce or avoid the situation of excessive supply of computing resources.

[0222] As an optional embodiment of the present application, the above S102 to S104 can be replaced by: S201. Figure 6A, is an image frame drawing method provided by an embodiment of the present application, and in this case, the embodiment of the present application includes:

[0223] S101: Before starting to draw an image frame, the terminal device obtains a load event related to image frame drawing.

[0224] S201 : Determine the total computing resources required for the load event, process the load event based on the determined computing resources, and draw an image frame.

[0225] The principle and beneficial effects of S201 are substantially the same as those of S102 to S104 above. For details, please refer to the above description of S102 to S104 and will not be repeated here.

[0226] The embodiments of the present application do not place too many restrictions on the implementation details of S201. For example, in some optional embodiments, the operations of S102 to S104 may be used to implement S201. In other optional embodiments, the operation of S201 may be implemented in other ways. For example, a mapping relationship between load events and computing resources (also referred to as a second mapping relationship) may be preset. After the load event is acquired, the computing resources required for each load event may be determined by querying the mapping relationship. Finally, the computing resources required for all load events are summarized to determine the total computing resources required. Among them, the principle of the mapping relationship between load events and computing resources is similar to the principle of the mapping relationship between load events and resource parameters in the description of S104, so reference may be made to the relevant description in S104 and Figure 5 The description of the embodiments shown will not be repeated here.

[0227] The embodiments of the present application can realize dynamic and accurate scheduling of computing resources required for load events, thereby effectively improving the utilization of computing resources while improving the smoothness of the picture.

[0228] As an optional embodiment of the present application, the above S102 to S103 can be replaced by: S202. In this case, the present application embodiment includes:

[0229] S101: Before starting to draw an image frame, the terminal device obtains a load event related to image frame drawing.

[0230] S202: The terminal device determines total resource parameters required for the load event.

[0231] S104: The terminal device determines the computing resources corresponding to the resource parameters, processes the load event based on the determined computing resources, and draws the image frame.

[0232] In an embodiment of the present application, the load event can be used as a whole to match the appropriate resource parameters for each load event, and the required computing resources can be determined. Among them, the embodiment of the present application does not make too many restrictions on the implementation details of S202. For example, in some optional embodiments, the operations of S102 to S103 can be used to implement S202. In other optional embodiments, the operation of S201 can also be implemented in other ways. For example, the required resource parameters can be tested for each load event in advance, and the corresponding mapping relationship (also referred to as the third mapping relationship) can be stored. On this basis, the resource parameters required for each load event can be determined according to the mapping relationship, thereby determining the total resource parameters.

[0233] The embodiments of the present application can realize dynamic and accurate scheduling of computing resources required for load events, thereby effectively improving the utilization of computing resources while improving the smoothness of the picture.

[0234] As an optional embodiment of the present application, the above S103 to S104 can be replaced by: S203. In this case, the present application embodiment includes:

[0235] S101: Before starting to draw an image frame, the terminal device obtains a load event related to image frame drawing.

[0236] S102: The terminal device extracts parameters of event characteristics of each load event to obtain load characteristic parameters of each load event.

[0237] S203: The terminal device determines the computing resources required for the load event according to the load characteristic parameters, processes the load event based on the determined computing resources, and draws an image frame.

[0238] In an embodiment of the present application, after obtaining the complexity data (load characteristic parameters) of a load event, the terminal device can determine the computing resources required for the load event based on the complexity data. The embodiment of the present application does not impose excessive restrictions on the implementation details of S203. For example, in some optional embodiments, the operations of S103 to S104 can be used to implement S202. In other optional embodiments, the operation of S203 can also be implemented in other ways. For example, the required computing resources can be pre-tested for each load event under different load characteristic parameters, and the corresponding mapping relationship (also referred to as a fourth mapping relationship) can be stored. Based on this mapping relationship, the computing resources required for each load event under the current load characteristic parameters can be determined, thereby determining the total computing resources. For example, in some embodiments, each load event can be treated as an independent computing resource scheduling object. In this case, the load characteristic parameters of each load event can be obtained, and the corresponding computing resources required for each load event can be determined separately, such as which core to use and how many frequency points to allocate for processing load event A (when only a single core is used to process image frame drawing operations, the allocated frequency points can be determined). After allocating corresponding computing resources to all load events, computing resource scheduling for image frame drawing is achieved.

[0239] The embodiment of the present application can achieve computing resource matching at the load event level, so when a high-load event occurs, more computing resources can be dispatched in time to draw image frames, thereby improving the smoothness of the picture. At the same time, the embodiment of the present application can also achieve dynamic scheduling of computing resources at the highest single-frame level, with extremely high computing resource utilization and flexibility. Finally, the embodiment of the present application can match computing resources according to the complexity of the load event, so that the computing resource matching degree of the embodiment of the present application for each load event is very high, so that the load event related to the image frame drawing can achieve high-precision computing resource matching. Therefore, the utilization rate of computing resources can be effectively improved.

[0240] As an optional embodiment of the present application, considering that some event features in actual applications require high computing resources when processing, the processing is very time-consuming. Therefore, when encountering these event features, it can be theoretically explained that the image frame drawing at this time already requires more computing resources. Based on this, in order to improve the scheduling efficiency of computing resources while improving the smoothness of the picture, it is possible to Figure 3 The embodiment shown is partially replaced to obtain another image frame drawing method. Figure 6B , is an image frame drawing method provided by an embodiment of the present application, and in this case, the embodiment of the present application includes:

[0241] S101: Before starting to draw an image frame, the terminal device obtains a load event related to image frame drawing.

[0242] The S101 step is Figure 3 The step S101 in the illustrated embodiment is the same, so please refer to the description of S101 for details, which will not be repeated here.

[0243] S301: The terminal device extracts parameters of event features of each load event.

[0244] Steps of S301 and Figure 3 The content of "the terminal device extracts event characteristic parameters for each load event" in S102 of the illustrated embodiment is the same, so please refer to the relevant description of S102 for details and will not be repeated here. Among them, the method of extracting the load characteristic parameters includes at least the following three situations:

[0245] Case 1: Taking event features as units, each time a parameter of an event feature (ie, load feature parameter) is extracted, S302 is executed.

[0246] Case 2: Taking a single load event as a unit, after all required reference load characteristic parameters of a load event are extracted each time, S302 is executed.

[0247] Case 3: Multiple load events are used as units, and S302 is executed after all required reference load characteristics of each load event are extracted. For example, all load events can be taken as a whole, and S302 is executed after all required reference load characteristic parameters of all load events are extracted.

[0248] S302: When the parameters of the preset characteristics are extracted, the terminal device determines the computing resources required for the load event according to the preset characteristics.

[0249] Among them, the preset features refer to event features that have a high demand for computing resources when processed. Specifically, the preset features can be selected and set by technical personnel based on the actual demand for computing resources when processing each event feature. For example, in some optional embodiments, some class member variables can be selected as preset features. For example, considering that when drawing an image frame, re-layout requires more computing resources and takes a long time, layout requirement variables and / or stop change variables can be selected as preset features. In other optional embodiments, considering that some controls are more difficult to draw, require more computing resources, and take a long time. Therefore, some control names that require more computing resources for drawing can be selected as preset features. For example, list view (ListView) and / or recycling view (RecyclerView) can be preset features. In some other optional embodiments, class member variables, control names that require more computing resources for drawing, and other event features that require more computing resources can also be used as preset features in the embodiments of this application.

[0250] As an optional embodiment of the present application, in S301, the terminal device can extract the load characteristic parameters while determining whether there are parameters of preset characteristics, that is, whether there are preset characteristics that need to be processed (such as the above-mentioned case 1 or case 2). If there are preset characteristics at this time, S302 is executed. For example, a traversal method can be used to traverse the load characteristic parameters of the load event, and each time a load characteristic parameter is obtained, it is synchronously determined whether the load characteristic parameter is a parameter of the preset characteristic. At this time, the efficiency of identifying the preset characteristics can be improved.

[0251] As another optional embodiment of the present application, S301 may also extract all load characteristic parameters at one time, and then determine whether there are parameters of preset characteristics, that is, whether there are preset characteristics that need to be processed (for example, in the above-mentioned situation 3, all load events are taken as a whole). At this time, if there are preset characteristics, the terminal device can determine the computing resources required for the load event based on the preset characteristics. Accordingly, S301 at this time can be replaced by: S3011, the terminal device extracts the parameters of the event characteristics of each load event, and obtains the load characteristic parameters of all load events. S302 can be replaced by: S3021, when the acquired load characteristic parameters contain parameters of preset characteristics, the computing resources required for the load event are determined based on the preset characteristics.

[0252] After determining that there are preset features, the terminal device can start to quickly determine the computing resources required for drawing and scheduling based on the preset features. Specifically, the embodiment of the present application does not impose too many restrictions on the method for determining the computing resources corresponding to the preset features, and can be set by technical personnel. For example, in some optional embodiments, the demand for computing resources of these preset features can be tested in advance, so as to pre-establish a mapping relationship between the preset features and the computing resources. On this basis, S302 can determine the computing resources suitable for the current preset features by querying the mapping relationship. Among them, the mapping relationship between the preset features and the computing resources can also be called the fifth mapping relationship.

[0253] As an optional embodiment of the present application, when determining computing resources based on preset features, it is also possible to first determine the computing power requirements corresponding to the preset features, and then determine the corresponding computing resources based on the computing power requirements. In this case, S302 can be replaced by: S3022 and S3023.

[0254] S3022: When the parameters of the preset characteristics are extracted, the resource parameters required for the load event are determined according to the preset characteristics.

[0255] In an embodiment of the present application, the terminal device may first determine the resource parameters (i.e., computing power requirements) required for the load event based on the preset features, and then determine the specific scheduled computing resources based on the computing power requirements. Since the computing power requirement is a quantitative value of the required computing power, it is convenient for the terminal device to perform quantitative calculations when matching computing resources, so that the current actual allocated computing resources can be matched more flexibly. Among them, the embodiment of the present application does not make too many restrictions on the method for determining the computing power requirements corresponding to the preset features, and can be set by technical personnel. For example, in some optional embodiments, the computing power requirements corresponding to these preset features can be pre-tested, so as to pre-establish a mapping relationship between the preset features and the computing power requirements. On this basis, S3022 can determine the computing power requirements suitable for the current preset features by querying the mapping relationship. Among them, the mapping relationship between the preset features and the computing power requirements can also be called the sixth mapping relationship.

[0256] S3023: The terminal device determines the computing resources corresponding to the resource parameters, processes the load event based on the determined computing resources, and draws the image frame.

[0257] The operation details, principles and beneficial effects of S3023 are the same as those of S104, so please refer to the description of S104 and will not be repeated here.

[0258] In the embodiment of the present application, considering that some event feature processing in actual applications has high requirements for computing resources and is very time-consuming, the embodiment of the present application can quickly determine whether the parameters of the preset features are extracted while extracting the load feature parameters, that is, quickly determine whether the preset features need to be processed. At the same time, when it is determined that the preset features need to be processed, the computing resources required for image frame drawing can be quickly determined based on the preset features, and the computing resources can be scheduled. Therefore, the embodiment of the present application has at least the following beneficial effects:

[0259] 1. The terminal device can match the required computing resources for the load events required to execute the current image frame drawing, and allocate appropriate computing resources to handle these load events. Therefore, the embodiment of the present application can adapt to the actual needs of the terminal device for the computing resources required for image frame drawing under various load conditions, and the matching accuracy of computing resources can reach the specific level of each load event, so as to achieve more reasonable use of computing resources. When a high load situation occurs, the embodiment of the present application can timely dispatch more computing resources to handle events related to image frame drawing, thereby improving the image frame drawing speed and reducing the occurrence of frame loss and freezes. Therefore, the embodiment of the present application can improve the screen response speed and smoothness of the terminal device.

[0260] 2. The dynamic scheduling accuracy of computing resources in the embodiment of the present application can reach the single-frame level, that is, the computing resources for each image frame can be independently matched. Under high load, the computing resources for image frame drawing can be increased as needed to speed up the image frame drawing speed, so that each image frame can be drawn efficiently and quickly, thereby greatly improving the smoothness of the picture. Under low load, the computing resources for image frame drawing can be reduced to save computing resource overhead and reduce terminal device power consumption and heat generation. Therefore, the embodiment of the present application has extremely high computing resource utilization and flexibility.

[0261] 3. The embodiments of the present application can evaluate the complexity of load events through preset features and quickly determine the computing resources required for the load events based on the preset features. Therefore, the embodiments of the present application have a high degree of matching of computing resources for load events, so that the load events related to the final image frame drawing can achieve high-precision matching of computing resources. Therefore, the utilization rate of computing resources can be effectively improved. While meeting the requirements of dynamic image smoothness, it can also reduce or avoid the situation of excessive supply of computing resources.

[0262] 4. The embodiments of the present application can quickly dispatch computing resources when a preset feature requiring high computing resource processing is determined. This simplifies the logic for scheduling computing resources and effectively improves the efficiency of resource scheduling. Furthermore, it can reduce the workload of scheduling computing resources for image frame rendering on terminal devices, lower hardware requirements, and adapt to a wider range of terminal devices with different hardware configurations.

[0263] As a specific embodiment of this application, please refer to Figure 7 , is a flowchart of an implementation method of an image frame drawing method provided in an embodiment of the present application. Detailed description is as follows:

[0264] S401: When the terminal device needs to draw an image frame, before starting to draw the image frame, it first determines whether a preset function is enabled.

[0265] If the preset function is enabled, then S402 is executed; if the preset function is not enabled, then S412 is executed.

[0266] The preset function can serve as a trigger condition for scheduling computing resources when drawing image frames in the embodiment of the present application. In the embodiment of the present application, the terminal device can be set to trigger the operation of S401 every time it needs to draw an image frame. When the preset function is enabled, the embodiment of the present application can achieve precise scheduling of computing resources for image frame drawing at the single-frame level.

[0267] S402: The terminal device traverses an event callback queue related to image frame drawing to obtain a load event related to image frame drawing.

[0268] S403: Each time a load event is acquired, the terminal device determines the load characteristic parameters of the load event according to the load model statistics table.

[0269] The load model statistics table records event characteristics required for reference of each load event, so the load characteristic parameters required for the load event can be determined based on the load model statistics table.

[0270] S404: The terminal device determines whether the traversal of the event callback queue is complete. If the traversal is complete, S405 is executed; if not, the process returns to S403.

[0271] S405: The terminal device generates load characteristic parameters for all load events.

[0272] S406: The terminal device transmits the load characteristic parameters to the resource scheduling service.

[0273] S407: The resource scheduling service writes the load characteristic parameters to the temporary file node.

[0274] The temporary file node written therein may be / proc / fps / cur_draw_util.

[0275] S408: Trigger a new scheduling window.

[0276] S409: The terminal device determines the total computing power required for all load events based on the load characteristic parameters.

[0277] At this time, you can query the mapping relationship between the load characteristic parameters of the preset load event and the computing power requirements to determine the total computing power requirements.

[0278] S410: The terminal device determines the computing resources to be scheduled based on the computing power requirements.

[0279] S411: The terminal device processes the load event based on the determined computing resources and draws the image frame.

[0280] S412, drawing the image frame normally.

[0281] The implementation details, principles and beneficial effects of each step of the embodiment of this application can be referred to Figures 2A to 7 The illustrated embodiments and other method embodiments are not described in detail here.

[0282] It should be understood that, in the absence of logical conflicts, the above-mentioned embodiments of the present application can be combined with each other to obtain new implementations. These new implementations obtained by combination are also within the scope of protection of this application. For example, Figures 3 to 6A The embodiment shown can be partially combined with Figure 6B The illustrated embodiments are applied in combination.

[0283] Corresponding to the image frame drawing method described in the above embodiment, Figure 8 A structural schematic diagram of an image frame drawing device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0284] Reference Figure 8 , the image frame drawing device includes:

[0285] The event acquisition module 81 is used to acquire a load event related to drawing before starting to draw an image frame.

[0286] The resource determination module 82 is used to determine the computing resources required for the load event.

[0287] The drawing module 83 is configured to process load events based on the determined computing resources and draw image frames.

[0288] The process of each module in the image frame drawing device provided in the embodiment of the present application realizing its own function can be specifically referred to the aforementioned Figures 3 to 7 The description of the illustrated embodiment and other related method embodiments will not be repeated here.

[0289] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0290] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0291] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0292] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0293] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0294] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish descriptions and should not be understood as indicating or implying relative importance. It should also be understood that although the terms "first", "second", etc. are used in the text to describe various elements in some embodiments of the present application, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first table can be named a second table, and similarly, a second table can be named a first table without departing from the scope of the various described embodiments. Both the first table and the second table are tables, but they are not the same table.

[0295] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0296] The image frame drawing method provided in the embodiments of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). The embodiments of the present application do not impose any restrictions on the specific type of terminal device.

[0297] For example, the terminal device can be a station (STAION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with wireless communication capabilities, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a vehicle networking terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite wireless device, a wireless modem card, a TV set-top box (STB), customer premise equipment (CPE) and / or other devices for communicating on a wireless system and a next-generation communication system, such as a terminal device in a 5G network or a terminal device in a future evolved Public Land Mobile Network (PLMN) network.

[0298] As an example and not a limitation, when the terminal device is a wearable device, the wearable device can also be a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are full-featured, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0299] Figure 9 This is a schematic diagram of the structure of a terminal device provided by an embodiment of the present application. Figure 9 As shown, the terminal device 9 of this embodiment includes: at least one processor 90 ( Figure 9 Only one is shown), a memory 91, wherein the memory 91 stores a computer program 92 that can be run on the processor 90. When the processor 90 executes the computer program 92, the steps in the above-mentioned various image frame drawing method embodiments are implemented, such as Figure 3 Alternatively, when the processor 90 executes the computer program 92, the functions of the modules / units in the above-mentioned device embodiments are realized, for example, Figure 8Functions of modules 81 to 83 are shown.

[0300] The terminal device 9 can be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device can include, but is not limited to, a processor 90 and a memory 91. It can be understood by those skilled in the art that Figure 9 It is only an example of the terminal device 9 and does not constitute a limitation on the terminal device 9. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include an input and sending device, a network access device, a bus, etc.

[0301] The processor 90 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0302] In some embodiments, the memory 91 may be an internal storage unit of the terminal device 9, such as a hard disk or memory of the terminal device 9. The memory 91 may also be an external storage device of the terminal device 9, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 9. Furthermore, the memory 91 may include both an internal storage unit of the terminal device 9 and an external storage device. The memory 91 is used to store an operating system, application programs, a boot loader, data, and other programs, such as the program code of the computer program. The memory 91 may also be used to temporarily store data that has been sent or is about to be sent.

[0303] In addition, those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0304] An embodiment of the present application also provides a terminal device, which includes at least one memory, at least one processor, and a computer program stored in the at least one memory and executable on the at least one processor. When the processor executes the computer program, the terminal device implements the steps of any of the above-mentioned method embodiments.

[0305] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0306] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0307] An embodiment of the present application further provides a chip system, which includes a processor coupled to a memory, and the processor executes a computer program stored in the memory to implement the steps in the above-mentioned various method embodiments.

[0308] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program, when executed by the processor, can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0309] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0310] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0311] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0312] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for drawing an image frame, characterized in that: include: Before starting to draw the image frame, obtaining a load event from an event callback queue related to drawing; Extracting parameters of event characteristics of each load event to obtain load characteristic parameters of each load event, wherein each load characteristic parameter is used to characterize the complexity of each load event, and the event characteristics include: the type of the load event, the variable parameters involved in the load event, and the scenario corresponding to the load event; Determining computing resources required for the load event according to the load characteristic parameters; processing the load event and drawing the image frame based on the determined computing resources; Among them, the event callback queue belongs to any one of the following three types of queues: a first type of queue related to user operation event processing, a second type of queue related to motion effect related event processing, and a third type of queue related to interface element display event processing.

2. The image frame drawing method according to claim 1, wherein: The determining the computing resources required for the load event according to the load characteristic parameters includes: When the acquired load characteristic parameters include parameters of preset characteristics, the computing resources required for the load event are determined according to the preset characteristics.

3. The image frame drawing method according to claim 1, wherein: The determining of the computing resources required for the load event includes: Extracting parameters of event features for each of the load events; When the parameters of the preset characteristics are extracted, the computing resources required for the load event are determined according to the preset characteristics.

4. The image frame drawing method according to claim 2 or 3, characterized in that: The extracting parameters of event features of each load event includes: Identify the current usage scenario, and determine the event feature of the required extraction parameters based on the usage scenario, the determined event feature being the target feature; Parameters of the target features are extracted for each of the load events.

5. The image frame drawing method according to claim 4, characterized in that: The event feature operation of identifying the current usage scenario and determining the required extraction parameters according to the usage scenario includes: When the usage scenario is a target scenario, the target features include first-category features and part or all of second-category features; wherein the first-category features and the second-category features are two different types of event features; When the usage scenario is a scenario other than the target scenario, the target features include the first category of features.

6. The image frame drawing method according to any one of claims 1 to 3, characterized in that: The determining of the computing resources required for the load event includes: Determine resource parameters required for the load event and the computing resources corresponding to the resource parameters.

7. The image frame drawing method according to claim 6, characterized in that: The determining the computing resource corresponding to the resource parameter includes: A first drawing duration is acquired, and the resource parameter is processed based on the first drawing duration to obtain the computing resource, where the first drawing duration is an estimated duration for drawing the image frame.

8. The image frame drawing method according to claim 7, characterized in that: The first drawing duration is less than or equal to the inverse of the maximum screen refresh rate of the terminal device.

9. The image frame drawing method according to claim 1, wherein: The operation of extracting event feature parameters of each load event to obtain the load feature parameters of all the load events includes: When the load event belongs to the first type of queue, the event feature includes at least one feature of the following first type features: an executable function name and an input displacement variable; When the load event belongs to the second-category queue, the event feature includes at least one feature of the following first-category features: executable class name, control feature, executable function name, and input displacement variable; When the load event belongs to the third type of queue, the event feature includes, among the first type of features: a drawing change value of the next image frame in the interface.

10. A terminal device, characterized in that: The terminal device includes a memory and a processor, the memory stores a computer program that can be run on the processor, and the processor implements the image frame drawing method according to any one of claims 1 to 9 when executing the computer program.

11. A chip system, characterized in that: The chip system includes a processor coupled to a memory, and the processor executes a computer program stored in the memory to implement the image frame drawing method according to any one of claims 1 to 9.

Images