Processing method of terminal device and related apparatus

Abstract

The application provides a processing method of a terminal device and a related device. The terminal device comprises a display. The processing method comprises the following steps: when it is detected that a first thread in the terminal device calls a first system function and the first thread is not a system frame drawing thread, determining that the first thread is a third-party frame drawing thread; the first system function is a function called by the frame drawing thread, and the first system function is used for submitting a buffer drawn by the frame drawing thread to a buffer queue BufferQueue; wherein the terminal device displays the buffer drawn by the frame drawing thread on the display by consuming the BufferQueue. The method can identify the third-party frame drawing thread in the terminal device.

Description

Technical Field

[0001] This application relates to the field of terminal equipment, and in particular to processing methods and related apparatus for terminal equipment. Background Technology

[0002] When a user uses a terminal device, each frame displayed on the device needs to be drawn by at least one drawing thread. If a frame cannot be drawn within the allotted time, the user will experience a noticeable lag, which significantly impacts the user experience.

[0003] Currently, in order to reduce lag and improve user experience, terminal devices will identify whether it is a frame drawing thread. If it is identified as a frame drawing thread, it will be assigned a higher priority so that more computing resources can be allocated to the frame drawing thread during system scheduling, thereby reducing the occurrence of lag and unsmoothness.

[0004] However, currently, terminal devices primarily recognize the system rendering thread (also known as the System Render thread). For some apps on these devices, in addition to the System Render thread, they may also rely on rendering threads from third-party rendering frameworks (such as Flutter and WebView) to draw each frame. Currently, there is no solution for identifying rendering threads from these third-party frameworks. Summary of the Invention

[0005] This application provides a processing method and related apparatus for a terminal device, used to identify the drawing thread in a third-party drawing frame framework.

[0006] In a first aspect, this application provides a processing method for a terminal device, the terminal device including a display, the method including: when it is detected that a first thread in the terminal device calls a first system function and the first thread is not a system drawing thread, determining that the first thread is a third-party drawing thread;

[0007] The first system function is called by the drawing thread after filling the drawn data into the requested buffer. The first system function is used to submit the buffer filled by the drawing thread to the BufferQueue. The terminal device displays the data in the buffer filled by the drawing thread on the display by consuming the BufferQueue.

[0008] Because the drawing thread allocates a buffer before drawing and then fills the buffer with the drawn data, the buffer data filled by the drawing thread is also referred to as the drawing buffer in this application.

[0009] In other words, in this application, the buffer drawn by the frame drawing thread is the data drawn by the frame drawing thread.

[0010] Because the frame drawing threads share a common characteristic: after completing the drawing, the frame drawing thread submits the drawn buffer to the BufferQueue through a system function (also referred to as the first system function in this application), so that the terminal device can consume the buffer in the BufferQueue to display the data in the buffer drawn by the frame drawing thread on the display.

[0011] Therefore, in this embodiment, the third-party frame drawing thread in the terminal device is identified based on this feature.

[0012] It should be understood that the technical solution provided in this application enables the terminal device to fully identify third-party frame drawing threads within the terminal device. In other words, the technical solution provided in this application will not overlook any frame drawing threads within the terminal device.

[0013] Furthermore, once a third-party rendering thread is identified, the terminal device can, for example, allocate a higher priority to the third-party rendering thread to allocate more computing resources, thereby reducing the time the third-party rendering thread spends drawing the buffer and further reducing the impact of the third-party rendering thread on the overall frame duration. This will improve the smoothness of the APP that is rendered by the rendering thread in the third-party rendering framework during runtime.

[0014] It should be understood that when a frame is drawn by at least one drawing frame thread, some drawing frame threads may draw buffers of small size, while others may draw buffers of large size.

[0015] The definition of large and small sizes can be determined based on the actual scenario. For example, a threshold can be set, and buffers with sizes larger than the threshold are considered large, while buffers with sizes smaller than or equal to the threshold are considered small.

[0016] It should be understood that the larger the size of the drawing buffer, the longer the drawing time. Therefore, compared to drawing smaller-sized drawing frames, drawing larger-sized drawing frames is crucial to the ability to complete the drawing of a frame. In view of this, as an optional embodiment, after identifying a third-party drawing thread, this application first determines whether the third-party drawing thread is a critical drawing thread. Only when the third-party drawing thread is a critical drawing thread is the first thread assigned a high priority; whereby a critical drawing thread is one that can affect the latency of a frame.

[0017] Specifically, in this application, a drawing thread whose size of the drawn buffer is greater than a preset threshold is considered a drawing thread that affects the latency of a frame.

[0018] For example, determining whether the first thread is a critical frame thread includes:

[0019] Get the monitor's screen size;

[0020] Determine whether the size of the buffer drawn by the first thread is greater than the product of the screen size and a first preset percentage;

[0021] If the size of the buffer drawn by the first thread is greater than the product of the screen size and the first preset percentage, the first thread is determined to be the critical drawing frame thread.

[0022] For example, determining whether the first thread is a critical frame drawing thread includes:

[0023] The target size is obtained, and the terminal device displays one frame within the target size, which is smaller than the screen size of the display.

[0024] Determine whether the size of the buffer drawn by the first thread is greater than the product of the target size and the second preset percentage;

[0025] If the size of the buffer drawn by the first thread is greater than the product of the target size and the second preset percentage, the first thread is determined to be the critical drawing frame thread.

[0026] For example, determining whether the first thread is a critical drawing frame thread includes:

[0027] Get the monitor's screen size;

[0028] Determine whether the size of the buffer drawn by the first thread is greater than the product of the screen size and a first preset percentage;

[0029] If the size of the buffer drawn by the first thread is greater than the product of the screen size and the first preset percentage, it is determined whether the size of the buffer drawn by the first thread is greater than the product of the target size and the second preset percentage. The terminal device displays one frame within the target size, and the target size is smaller than the screen size of the display.

[0030] If the size of the buffer drawn by the first thread is greater than the product of the target size and the second preset percentage, the first thread is determined to be the critical drawing frame thread.

[0031] In conjunction with the first aspect, in one possible implementation, the first system function includes any one of the following: the queuebuffer function and the onFrameAvailable function.

[0032] In conjunction with the first aspect, in one possible implementation, the method further includes: when it is detected that a second thread in the terminal device calls a first system function and the second thread is a system drawing thread, assigning a high priority to the second thread.

[0033] Secondly, this application provides a processing apparatus that may include various functional modules for implementing the method in the first aspect. For example, the apparatus includes a detection module and an identification module.

[0034] For example, the detection module can be used to detect whether the first thread in the terminal device calls the first system function;

[0035] The identification module is used to identify whether the first thread is a system drawing function when the detection module detects that the first thread in the terminal device calls the first system function. If not, the first thread is determined to be a third-party drawing thread.

[0036] The first system function is called by the drawing thread after it fills the drawn data into the requested buffer. The first system function is used to submit the buffer drawn by the drawing thread to the BufferQueue. The terminal device displays the data in the buffer filled by the drawing thread on the display by consuming the BufferQueue.

[0037] In conjunction with the second aspect, in one possible implementation, the terminal device further includes a scheduling module, which is used to allocate a high priority to the first thread;

[0038] The higher the priority assigned to a thread by the terminal device, the more computing resources the terminal device provides to the thread.

[0039] In conjunction with the second aspect, in one possible implementation, the identification module is further configured to determine whether the first thread is a critical drawing frame thread;

[0040] The scheduling module is specifically used to: assign high priority to the first thread when the first thread is a critical drawing frame thread; whereby a critical drawing frame thread is a drawing frame thread that can affect the latency of drawing a frame.

[0041] In conjunction with the second aspect, in one possible implementation, the identification module identifies whether the first thread is a key drawing frame thread, including: obtaining the screen size of the display; determining whether the size of the buffer filled by the first thread is greater than the product of the screen size and a first preset percentage; if the size of the buffer filled by the first thread is greater than the product of the screen size and the first preset percentage, determining that the first thread is a key drawing frame thread.

[0042] In conjunction with the second aspect, in one possible implementation, the identification module identifies whether the first thread is a key rendering frame thread, including: obtaining the target size, the terminal device displays a frame within the target size, and the target size is smaller than the screen size of the display; determining whether the size of the buffer filled by the first thread is greater than the product of the target size and a second preset percentage; if the size of the buffer filled by the first thread is greater than the product of the target size and the second preset percentage, determining that the first thread is a key rendering frame thread.

[0043] In conjunction with the second aspect, in one possible implementation, the identification module identifies whether the first thread is a key rendering frame thread, including: obtaining the screen size of the display; determining whether the size of the buffer filled by the first thread is greater than the product of the screen size and a first preset percentage; if the size of the buffer filled by the first thread is greater than the product of the screen size and the first preset percentage, determining whether the size of the buffer filled by the first thread is greater than the product of the target size and a second preset percentage, and the terminal device displays one frame within the target size, where the target size is smaller than the screen size of the display; if the size of the buffer filled by the first thread is greater than the product of the target size and the second preset percentage, determining that the first thread is a key rendering frame thread.

[0044] In conjunction with the second aspect, in one possible implementation, the first system function includes any one of the following: the queuebuffer function and the onFrameAvailable function.

[0045] In conjunction with the second aspect, in one possible implementation, when the detection module detects that the second thread in the terminal device calls the first system function and the identification module identifies the second thread as the system drawing thread, the scheduling module is also used to allocate a high priority to the second thread.

[0046] In some implementations, these modules can be implemented in software and / or hardware. In such implementations, the device may include a processor and a transceiver. For example, the detection module and the identification module can be implemented by the processor executing program code stored in memory, and the acquisition module can be implemented by the transceiver. Optionally, the device may also include memory.

[0047] Thirdly, this application provides a terminal device that may include various functional modules for implementing the method in the first aspect. In some implementations, these modules may be implemented in software and / or hardware.

[0048] Fourthly, this application provides a computer-readable storage medium that stores program code for execution by a communication device, the program code including instructions for implementing the method in the first aspect.

[0049] Fifthly, this application provides a computer program product containing instructions that, when run on a communication device, causes the communication device to implement the method in the first aspect.

[0050] It is understandable that the effects that can be obtained from the second, third, fourth, and fifth aspects can be referred to the description in the first aspect, and will not be repeated here. Attached Figure Description

[0051] Figure 1 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0052] Figure 2 This is a software structure block diagram of a terminal device to which this application embodiment applies;

[0053] Figure 3 This is a schematic diagram of a frame displayed by a terminal device according to an embodiment of this application;

[0054] Figure 4 An architecture for a method applicable to this application, provided as an embodiment of this application;

[0055] Figure 5 An architecture applicable to the method of this application is provided for another embodiment of this application;

[0056] Figure 6 This is a flowchart illustrating the process of identifying whether a first thread is a third-party drawing thread, as provided in one embodiment of this application.

[0057] Figure 7 A flowchart illustrating the process of identifying whether a first thread is a third-party drawing thread, provided for another embodiment of this application;

[0058] Figure 8 A schematic diagram of the target dimensions provided for another embodiment of this application;

[0059] Figure 9 A flowchart illustrating the process of identifying whether a first thread is a third-party drawing thread, provided as another embodiment of this application;

[0060] Figure 10 This is a schematic diagram of the structure of a processing apparatus provided in one embodiment of this application;

[0061] Figure 11 This is a schematic diagram of the structure of a processing apparatus provided in another embodiment of this application. Detailed Implementation

[0062] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0063] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" are not necessarily different.

[0064] It should be noted that, in this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0065] Furthermore, "at least one" refers to one or more, while "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can mean: a, or b, or c, or a and b, or a and c, or b and c, or a, b, and c, where a, b, and c can be single or multiple.

[0066] The processing method for terminal devices provided in this application, also known as a method for identifying third-party frame-drawing threads, can be applied to terminal devices with display capabilities. Terminal devices can be wearable devices such as mobile phones, tablets, personal computers (PCs), smart screens, in-vehicle systems, and smartwatches; they can also be various teaching aids (e.g., learning machines, early education machines), smart toys, portable robots, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, etc.; and they can also be devices with mobile office functions, smart home functions, audio-visual entertainment functions, or devices supporting smart travel. It should be understood that the embodiments of this application do not limit the specific technology or device form used in the terminal device.

[0067] To better understand the embodiments of this application, the hardware structure of the terminal device of this application embodiment is described below. For example, Figure 1 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application.

[0068] Figure 1 A schematic diagram of the terminal device is shown. The terminal device may include a processor 110, internal memory 121, charging management module 140, power management module 141, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, sensor module 180, buttons 190, display screen 194, timer 196, etc.

[0069] Optionally, the sensor module 180 may include a pressure sensor 180A and a touch sensor 180B, etc.

[0070] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the terminal device. In other embodiments of this application, the terminal device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0071] Processor 110 may include one or more processing units. For example, processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, etc. Different processing units may be independent devices or integrated into one or more processors. Processor 110 may also include memory for storing instructions and data.

[0072] The charging management module 140 receives charging input from the charger. The charger can be a wireless charger or a wired charger. The power management module 141 connects the charging management module 140 to the processor 110.

[0073] The wireless communication function of the terminal device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0074] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Antennas in terminal equipment can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0075] The mobile communication module 150 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G on terminal devices. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation.

[0076] The wireless communication module 160 can provide solutions for wireless communication applications on terminal devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), etc.

[0077] In some embodiments, antenna 1 of the terminal device is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling the mobile phone to communicate with the network and other devices through wireless communication technology.

[0078] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the terminal device may include one or N display screens 194, where N is a positive integer greater than 1.

[0079] As an example, each frame displayed on the display screen 194 needs to be drawn by a frame drawing process in the terminal device.

[0080] Terminal devices can implement display functions through graphics processing units (GPUs), displays (194), and application processors. A GPU is a microprocessor for image processing, connecting the display (194) and the application processor. GPUs are used to perform mathematical and geometric calculations and for graphics rendering. GPUs can also be called display cores, visual processors, and display chips.

[0081] Internal memory 121 can be used to store executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area can store application programs required for at least one function of the operating system (such as sound playback, image playback, etc.). The data storage area can store data created during the use of the terminal device (such as audio data, phonebook, etc.).

[0082] The terminal device can implement audio functions such as music playback and recording through an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, and an application processor.

[0083] Audio module 170 is used to convert digital audio information into analog audio signal output, and also to convert analog audio input into digital audio signal. Speaker 170A, also called a "loudspeaker," is used to convert audio electrical signals into sound signals. Terminal devices can listen to music or hands-free calls through speaker 170A. Receiver 170B, also called a "handpiece," is used to convert audio electrical signals into sound signals. When the terminal device answers a phone call or voice message, it can listen to the voice by bringing the receiver 170B close to the user's ear. Microphone 170C, also called a "microphone" or "voice transducer," is used to convert sound signals into electrical signals.

[0084] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A may be disposed on display screen 194. When a touch operation is applied to display screen 194, the terminal device can detect the intensity of the touch operation based on pressure sensor 180A. The terminal device can also calculate the touch position based on the detection signal from pressure sensor 180A. Touch sensor 180B may be disposed on display screen 194, and the touch sensor 180B and display screen 194 together form a touch screen, also known as a "touchscreen". Touch sensor 180B is used to detect touch operations applied to or near it. Touch sensor 180B can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180B may also be disposed on the surface of the mobile phone, in a different location than display screen 194.

[0085] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. The terminal device can receive button input and generate key signal inputs related to user settings and function control. Indicator 192 can be an indicator light, used to indicate charging status, battery level changes, messages, missed calls, notifications, etc. Timer 196 can be used to record time information. For example, when the terminal device is recording video, it can record the start and end times of the video, as well as the duration of the recorded video.

[0086] Figure 2This is a software architecture block diagram of a terminal device applicable to embodiments of this application. The layered architecture divides the terminal device's software system into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into four layers, from top to bottom: applications, application framework, Android runtime, system libraries, and kernel.

[0087] The application layer can include a series of application packages. The application layer runs applications by calling the application programming interface (API) provided by the application framework layer.

[0088] For example, the application package may include a desktop, calendar, notes, map, navigation, Bluetooth, music, JD.com, and SMS applications.

[0089] The application framework layer provides APIs and a programming framework for applications within the application layer. The application framework layer includes some predefined functions.

[0090] For example, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0091] The window manager manages windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots. The content provider stores and retrieves data, making it accessible to applications. This data can include video, images, audio, made and received calls, browsing history and bookmarks, and phone books. The view system includes visual controls, such as controls for displaying text and controls for displaying images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon can include views for displaying text and views for displaying images. The phone manager provides communication functionality for terminal devices, such as managing call status (including connection and disconnection). The resource manager provides applications with various resources, such as localized strings, icons, images, layout files, and video files. The notification manager allows applications to display notifications in the status bar. It can be used to convey informational messages and can disappear automatically after a short pause without user interaction. For example, the notification manager is used to notify of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the system's top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting alert sounds, causing the device to vibrate, and flashing indicator lights.

[0092] The Android runtime consists of the core libraries and the virtual machine. The Android runtime is responsible for scheduling and managing the Android system. The core libraries comprise two parts: one part contains the functionalities that Java needs to call, and the other part is the Android core library itself. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection. System libraries can contain modules with multiple functionalities, such as surface managers, media libraries, and 3D graphics processing libraries.

[0093] The Surface Manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The Media Library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as JPG and PNG. The 3D Graphics Processing Library implements 3D graphics drawing, image rendering, compositing, and layer processing.

[0094] The kernel layer is the layer between hardware and software. It drives the hardware, enabling it to function. The kernel layer includes at least display drivers, screen drivers, camera drivers, and sensor drivers, but this embodiment does not limit this.

[0095] for Figure 1 and Figure 2 The terminal device shown requires at least one drawing thread to render each frame it displays. For example, assuming the terminal device displays 40 frames per second, the terminal device might display something like this at the current moment: Figure 3 The image shown is a single frame, which is drawn using at least one frame-drawing thread.

[0096] Specifically, the process from drawing data by each frame drawing thread to displaying the drawn frame data on the terminal device's display is as follows: each frame drawing thread fills the drawn data into a pre-allocated buffer, and then calls a system function (also referred to as the first system function in this application) to submit the filled buffer to the buffer queue BufferQueue. After that, the terminal device calls the frame consumption module to consume the BufferQueue to display the data drawn by each frame drawing thread on the display.

[0097] For example, after each drawing thread calls the system function to submit the buffer to the BufferQueue, the terminal device calls the frame consumption module surfacelinger to display the drawn data on the display. The surfacelinger module is used to receive the data drawn by each drawing thread in the BufferQueue, combine these data into a frame, and then send it to the display.

[0098] As mentioned above, since each drawing frame thread first requests a buffer and then fills the buffer with the drawn data when drawing, in this application, the drawing by each drawing frame thread is also referred to as drawing buffer.

[0099] Additionally, it should be noted that in this application, the terminal device calling the frame consumption module to consume BufferQueue can also be referred to as consuming the bufffer in BufferQueue, or consuming the data in the bufffer in BufferQueue, which does not constitute a limitation of this application.

[0100] Understandably, if a frame cannot be drawn within the allotted time, the user will experience a noticeable lag, thus impacting the user experience.

[0101] Therefore, in order to reduce lag and improve user experience, terminal devices currently identify the frame drawing thread and assign a higher priority to the identified frame drawing thread so that more computing resources can be allocated to the frame drawing thread during system scheduling. This results in a shorter time for the frame drawing thread to complete the drawing buffer, thereby reducing the occurrence of lag and other issues.

[0102] However, the frame rendering threads that current terminal devices can recognize are mainly the system frame rendering thread (also known as System Render). For some apps on terminal devices, in addition to System Render, they may also rely on the frame rendering threads of some third-party frame rendering frameworks (such as Flutter, WebView, etc.) when rendering each frame.

[0103] For example, for shopping apps, the frames drawn nowadays are becoming increasingly complex. In addition to drawing information and images of the products to be displayed, they may also need to draw video information. This video information may be drawn by a third-party frame drawing thread.

[0104] For example, some mini-programs within the WeChat app may display content based on third-party rendering threads. This could be a game-type mini-program. However, there is currently no solution to identify the rendering threads within these third-party frameworks. Because the terminal device cannot recognize these threads, users may experience lag and stuttering when using apps rendered using such frameworks, thus impacting the user experience.

[0105] by Figure 3 Taking a single frame as an example, for instance, Figure 3 The live stream bar 301 shown in the frame example is rendered using a third-party rendering thread. If the terminal device cannot recognize this third-party rendering thread, it may allocate insufficient computing resources for it, resulting in a longer rendering time. This can lead to frame drops and stuttering due to the extended rendering time.

[0106] In view of this, this application provides a processing method and processing device for a terminal device, used to identify the drawing thread in a third-party drawing frame framework, further improving the smoothness of the APP that is drawn through the drawing thread in the third-party drawing frame framework during operation, and further improving the user experience.

[0107] The processing method of the terminal device provided in this application will now be described with reference to the accompanying drawings.

[0108] Before introducing the processing method of the terminal device provided in this application, the system architecture of this application will be introduced first. For example... Figure 4 The system architecture includes a default drawing framework 401, a third-party drawing framework 402, and a third-party drawing framework 403.

[0109] The default rendering framework 401 includes the system UI thread and the system Render thread. Detailed descriptions of the UI thread and the system Render thread can be found in relevant technical documents and will not be repeated here.

[0110] The third-party drawing framework 402 and the third-party drawing framework 403 include a third-party drawing thread for drawing frames and / or at least one wake-up thread, wherein the at least one wake-up thread is used to wake up the third-party drawing thread.

[0111] For example, such as Figure 4 As shown, threads 1 to 5 included in the third-party rendering framework 402 can be considered as the wake-up threads of the third-party rendering frame thread 1 in the third-party rendering framework 402, and threads 1' to 5' included in the third-party rendering framework 403 can be considered as the wake-up threads of the third-party rendering frame thread 2 in the third-party rendering framework 403.

[0112] Specifically, in the third-party rendering framework 402, the meaning of thread 1 pointing to thread 2 is that thread 1 wakes up thread 2; the meaning of thread 2 pointing to thread 3 is that thread 2 wakes up thread 3; the meaning of thread 3 pointing to the third-party rendering frame thread 1 is that thread 3 wakes up the third-party rendering frame thread 1; the meaning of thread 4 pointing to thread 5 is that thread 4 wakes up thread 5; and the meaning of thread 5 pointing to the third-party rendering frame thread 1 is that thread 5 wakes up the third-party rendering frame thread 1. It can be understood that threads 1, 2, and 4 can be considered as indirect wake-up threads of the third-party rendering frame thread 1, while threads 3 and 5 can be considered as direct wake-up threads of the third-party rendering frame thread 1.

[0113] Similarly, threads 1', 2', and 4' in the third-party rendering framework 403 can be considered as indirect wake-up threads of the third-party rendering thread 2, while threads 3' and 5' can be considered as direct wake-up threads of the third-party rendering thread 2.

[0114] Specifically, the system's Render thread, third-party frame drawing thread 1, and third-party frame drawing thread 2 work together to complete the drawing of a frame.

[0115] It should be noted that this embodiment does not limit the specific form between at least one third-party drawing framework.

[0116] For example, at least one third-party rendering framework may be the same or different. Figure 4 The third-party drawing frameworks 402 and 403 can both be Flutter drawing frameworks, or both can be WebView frameworks, or one can be a Flutter drawing framework and the other can be a WebView framework.

[0117] It should also be noted that, Figure 4 This example only includes two third-party drawing threads, but it does not constitute a limitation of this application.

[0118] For example, it may also include only one third-party frame drawing thread. In this case, it can be considered that the system render thread and the third-party frame drawing thread jointly complete the drawing of a frame.

[0119] For example, it could include more third-party rendering threads, such as a total of N third-party rendering threads. In this case, it can be considered that the system's Render thread and the N third-party rendering threads jointly complete the rendering of a frame.

[0120] It should be understood that the size of the buffer drawn by the frame-drawing thread may vary. The buffer size can be understood as follows: for example, if the frame-drawing thread requests a buffer with a length of M and a width of P, then the size of this buffer is M*N. Furthermore, it should be understood that the larger the size of the buffer drawn by the frame-drawing thread, the more data the thread draws, and correspondingly, the longer it will take.

[0121] For buffers drawn by third-party rendering threads within at least one third-party rendering framework, some buffers drawn by third-party rendering threads may have relatively large sizes, while others may have relatively small sizes. It should be noted that this embodiment does not impose restrictions on how the large or small size is defined.

[0122] In implementation, a threshold can be set, and buffers with a size greater than the threshold are considered large buffers, while buffers with a size less than the threshold are considered small buffers.

[0123] For example, such as Figure 5 As shown, the buffer drawn by third-party rendering thread 1 is a large-sized buffer, while the buffer drawn by third-party rendering thread 2 is a small-sized buffer. It should be noted that this is merely an example and does not constitute a limitation. For instance, third-party rendering thread 1 and third-party rendering thread 2 may both draw large-sized buffers, or they may both draw small-sized buffers.

[0124] like Figure 4 and Figure 5As shown, all frame drawing threads have the following characteristics when drawing: after completing the drawing, the frame drawing thread will submit the drawn buffer to the BufferQueue through the system function (i.e., the first system function), so that the frame consumption module in the terminal device can consume the buffer in the BufferQueue to display the buffer drawn by the frame drawing thread on the display.

[0125] As an example, the first system functions are the queuebuffer function and the onFrameAvailable function.

[0126] It should be noted that when there are multiple third-party drawing threads, the system functions used by these threads when submitting the buffer can be the same or different, which does not constitute a limitation of this application. Figure 4 or Figure 5 For example, third-party frame drawing thread 1 might use the `queuebuffer` system function, while third-party frame drawing thread 2 might use the `onFrameAvailable` system function. Alternatively, third-party frame drawing thread 1 could use the `onFrameAvailable` system function, while third-party frame drawing thread 2 might use the `queuebuffer` system function.

[0127] As an example, the frame consumption module is surfaceflinger.

[0128] Therefore, the method for identifying a third-party frame drawing thread provided in this application includes: when it is detected that a first thread in a terminal device calls a first system function and the first thread is not a system frame drawing thread, the first thread is determined to be a third-party frame drawing thread; wherein, the first system function is a function called by the frame drawing thread, and the first system function is used to submit the buffer drawn by the frame drawing thread to a buffer queue BufferQueue, wherein the terminal device displays the buffer drawn by the frame drawing thread on the display by consuming BufferQueue.

[0129] Or, to put it another way, in this application, the terminal device determines the thread that submits the buffer using the first system function as the frame drawing thread, and then compares it with the system frame drawing thread to identify whether it is a third-party frame drawing thread.

[0130] It should be understood that the technical solution provided in this application enables the terminal device to fully identify third-party rendering threads within the terminal device. In other words, the technical solution provided in this application will not overlook any third-party rendering threads within the terminal device.

[0131] Furthermore, once a third-party rendering thread is identified, the terminal device can, for example, allocate a higher priority to the third-party rendering thread to allocate more computing resources, thereby reducing the time the third-party rendering thread spends drawing the buffer and further reducing the impact of the third-party rendering thread on the overall frame duration. This will improve the smoothness of the APP that is rendered by the rendering thread in the third-party rendering framework during runtime.

[0132] As mentioned earlier, when drawing a frame using at least one drawing thread, some of these threads may draw small-sized frames, while others may draw large-sized frames. Since larger frames require more time to draw, the thread drawing large-sized frames is more crucial in determining whether a frame can be drawn successfully and efficiently. In other words, the thread drawing small-sized frames may have a less significant impact on the time required to draw a single frame compared to the thread drawing large-sized frames.

[0133] In view of this, in some implementations provided in this application, after identifying a third-party frame drawing thread, when assigning a high priority to the third-party frame drawing thread, the first thread is only assigned a high priority when the third-party frame drawing thread is a critical frame drawing thread; wherein, a critical frame drawing thread is a frame drawing thread that can affect the latency of a frame.

[0134] Below are three examples of identifying the first thread as the critical drawing frame thread.

[0135] Example 1

[0136] The method includes: when the terminal device calls the first system function to submit the drawn buffer in the first thread, the recognition module is invoked.

[0137] For example, the first system functions are the queuebuffer function and the onFrameAvailable function.

[0138] In practical implementation, when the first frame drawing thread submits the buffer through the first system function, a function call stub can be inserted at the first system function, or the identification module interface can be inserted, thereby entering the identification process of the third-party frame drawing thread.

[0139] When the terminal device calls the recognition module, such as Figure 6 As shown, execute:

[0140] S601, determine whether the first thread is a third-party drawing thread; if so, execute S602.

[0141] S602, Obtain the screen size of the display.

[0142] For example, obtaining the physical dimensions of the display of the terminal device's hardware.

[0143] S603, determine whether the size of the buffer drawn by the first thread is greater than the product of the screen size and the first preset percentage. If it is greater, execute S604.

[0144] For example, the first preset percentage can be represented as x%, which means determining whether the size of the buffer drawn by the first thread is greater than the screen size * x%.

[0145] In this embodiment, a buffer whose size is greater than x% of the screen size is considered a large buffer, and a buffer whose size is less than or equal to x% of the screen physical size is considered a small buffer.

[0146] It should be noted that this embodiment does not limit the specific value of the first preset percentage.

[0147] S604, determine the first thread as the critical drawing frame thread.

[0148] As can be seen, in this embodiment, the size of the buffer drawn by the first thread is compared with the screen size * x% to determine whether the buffer drawn by the first thread is large or small. Then, when it is determined that the buffer drawn by the first thread is large, the first thread is identified as the key drawing frame thread that affects the drawing of a frame.

[0149] Furthermore, when the first thread is identified as the critical third-party rendering thread that affects the rendering of a frame, more computing power is allocated to the first thread to reduce the rendering buffer time of the critical third-party rendering thread, so as to ensure that the rendering of a frame can be completed within the rated time and improve the smoothness of the APP that is rendered by the first rendering thread of the third party during operation.

[0150] In this embodiment, the identification process can be executed within the application process, thus being independent of the operating system. In other words, when executed within the application process, the identification process can be completed regardless of the type of operating system used by the terminal device, such as Android or iOS.

[0151] Example 2

[0152] The method includes: when the terminal device calls the first system function to submit the drawn buffer in the first thread, the recognition module is invoked.

[0153] For example, the first system functions are the queuebuffer function and the onFrameAvailable function.

[0154] Similarly, in practical implementation, when the first frame drawing thread submits the buffer through the first system function, a function call stub can be inserted at the first system function, or the identification module interface can be inserted, so as to enter the identification process of the third-party frame drawing thread.

[0155] When the terminal device calls the recognition module, such as Figure 7 As shown, execute:

[0156] S701, determine whether the first thread is a third-party drawing thread; if so, execute S702.

[0157] S702 notifies the frame consumption module of the size of the buffer drawn by the first thread.

[0158] S703, obtain the target size.

[0159] It should be understood that when each drawing thread used to draw a frame completes its drawing, the target size for displaying the drawn frame when the terminal device displays it may be different from the screen size of the terminal device's display. That is, when the terminal device displays a frame on the screen of the display, it may only use a part of the area to display it.

[0160] For example, Figure 8 As shown, the terminal device needs to display a drawn frame in the area located in the dashed box 801, instead of filling the entire display.

[0161] In this embodiment, the target size is also referred to as the display size. It is important to emphasize that the display size in this embodiment is different from... Figure 6 The screen sizes of the displays in the embodiments are different. Figure 6 The screen size of the display in this embodiment is determined by hardware. For example, the screen size can be understood as the total screen size, including, for example, the status bar, etc., while the display size is determined by the system process.

[0162] S704: Determine whether the size of the buffer drawn by the first thread is greater than the product of the target size and the second preset percentage. If it is greater, execute S704.

[0163] For example, the second preset percentage can be represented as m%, which means determining whether the size of the buffer drawn by the first thread is greater than the target size * m%.

[0164] In this embodiment, a buffer whose size is greater than the target size * m% is considered a large buffer, and a buffer whose size is less than or equal to the target size * m% is considered a small buffer.

[0165] It should be noted that this embodiment does not limit the specific value of the second preset percentage.

[0166] S705, determine the first thread as the critical drawing frame thread.

[0167] As can be seen, in this embodiment, the size of the buffer drawn by the first thread is compared with the target size * m% to determine whether the buffer drawn by the first thread is large or small. Then, when it is determined that the buffer drawn by the first thread is large, the first thread is identified as the key drawing frame thread that affects the drawing of a frame.

[0168] Furthermore, when the first thread is identified as the critical third-party rendering thread that affects the rendering of a frame, more computing power is allocated to the first thread to reduce the rendering buffer time of the critical third-party rendering thread, so as to ensure that the rendering of a frame can be completed within the rated time and improve the smoothness of the APP that is rendered by the first rendering thread of the third party during operation.

[0169] This embodiment and Figure 6 The difference in the illustrated embodiment is that, Figure 6 The illustrated embodiment determines whether the first frame drawing thread is a critical frame drawing thread by comparing the buffer size drawn by the first frame drawing thread with the screen size of the hardware display. This embodiment, however, determines whether the first frame drawing thread is a critical frame drawing thread by comparing the buffer size drawn by the first frame drawing thread with the target size used when actually displaying a frame. Therefore, this... Figure 7 Compared to the example Figure 6 The implementation example can improve the accuracy of identifying key drawing frame threads.

[0170] Example 3

[0171] The method includes: when the terminal device calls the first system function to submit the drawn buffer in the first thread, the recognition module is invoked.

[0172] For example, the first system functions are the queuebuffer function and the onFrameAvailable function.

[0173] Similarly, in practical implementation, when the first frame drawing thread submits the buffer through the first system function, a function call stub can be inserted at the first system function, or the identification module interface can be inserted, so as to enter the identification process of the third-party frame drawing thread.

[0174] When the terminal device calls the recognition module, such as Figure 9 As shown, execute:

[0175] S901, determine whether the first thread is a third-party drawing thread; if so, execute S902.

[0176] S902, obtain the screen size of the display.

[0177] S903, determine whether the size of the buffer drawn by the first thread is greater than the product of the screen size and the first preset percentage. If it is greater, execute S904.

[0178] The above process can be executed in the application process space.

[0179] S904 notifies the frame consumption module of the size of the buffer drawn by the first thread.

[0180] S905, the frame consumption module determines whether the size of the buffer drawn by the first thread is greater than the product of the target size and the second preset percentage. If it is greater, S906 is executed.

[0181] S906, determine the first thread as the critical drawing frame thread.

[0182] It can be seen that this embodiment is similar to... Figure 9 The illustrated embodiments and Figure 7 The difference in the illustrated embodiment is that this embodiment first compares the size of the buffer drawn by the first thread with the product of the screen size and a first preset percentage (which can be considered the first layer of screening). After satisfying that the size of the buffer drawn by the first rendering thread is greater than the product of the screen size and the first preset percentage, it is further required that the size of the buffer drawn by the first thread is greater than the product of the target size and a second preset percentage (which can be considered the second layer of screening) before the first thread is identified as a critical rendering thread affecting the rendering of a frame. That is, the terminal device determines whether the first thread is a critical rendering thread through a two-layer screening. It should be understood that, through this two-layer screening method, when there are multiple third-party rendering threads, compared to... Figure 6 The implementation example can improve the accuracy of identifying key drawing frame threads compared to... Figure 7 In this example, since the frame consumption module on the system side no longer identifies all third-party frame drawing threads, but only a portion of them, the overhead on the system side can be reduced.

[0183] Furthermore, when the first thread is identified as a critical third-party rendering thread that affects the rendering of a frame, more computing power is allocated to this first thread to reduce the rendering buffer duration of the critical third-party rendering thread. This ensures that the rendering of a frame can be completed within the specified time, thereby improving the smoothness of the APP that is rendered through the first third-party rendering thread during operation.

[0184] The above describes three implementation methods for identifying critical rendering frame threads. In one example, if the goal is only to identify them through the application, implementation method 1 can be chosen. If the goal is to identify them through a system-application approach, implementation methods 2 and 3 can be used. However, implementation method 3 is more efficient than implementation method 2: when there are multiple rendering frame threads, having the system identify all of them at once is less resource-intensive than first filtering through the application and then partially filtering through the system. Figure 10 This is a schematic diagram of the structure of a processing device for a terminal device provided in one embodiment of this application. Figure 10 As shown, the processing device 1000 includes a detection module 1001 and an identification module 1002.

[0185] As an example, the detection module 1001 can be used to perform the operation of detecting whether the first thread calls the first system function, or the detection module 1001 can be used to perform the operation of detecting whether the system drawing thread calls the first system function.

[0186] As an example, the identification module 1002 can be used to perform... Figure 6 The process for identifying whether the first thread is a critical drawing frame thread in the illustrated embodiment.

[0187] As another example, the identification module 1002 can be used to perform... Figure 7 The process for identifying whether the first thread is a critical drawing frame thread in the illustrated embodiment.

[0188] As yet another example, the identification module 1002 can be used to perform... Figure 9 The process for identifying whether the first thread is a critical drawing frame thread in the illustrated embodiment.

[0189] Figure 11 This is a schematic diagram of the structure of a processing device for a terminal device provided in another embodiment of this application. (See diagram below.) Figure 11 As shown, the processing device 1100 may include a processor 1101 and an interface circuit 1102. The processor 1101 and the interface circuit 1102 are coupled to each other. It is understood that the interface circuit 1102 may be a transceiver or an input / output interface. Optionally, the processing device 1100 may also include a memory 1103 for storing instructions executed by the processor 1101, or storing input data required by the processor 1101 to execute instructions, or storing data generated after the processor 1101 executes instructions.

[0190] It is understood that the processor in the embodiments of this application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor may be a microprocessor or any conventional processor.

[0191] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory, flash memory, read-only memory, programmable read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Additionally, the ASIC can reside in a network device or terminal device. Alternatively, the processor and storage medium can exist as discrete components in the network device or terminal device.

[0192] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video optical disc; or it can be a semiconductor medium, such as a solid-state drive.

[0193] In the various embodiments of this application, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of different embodiments are consistent and can be referenced by each other. The technical features of different embodiments can be combined to form new embodiments according to their inherent logical relationship.

[0194] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers described above does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

Claims

1. A processing method of a terminal device, characterized by, The terminal device comprises a display, and the method comprises: when it is detected that a first thread in the terminal device calls a first system function and the first thread is not a system frame drawing thread, determining that the first thread is a third-party frame drawing thread; determining whether the first thread is a critical frame drawing thread, the critical frame drawing thread being a frame drawing thread that can affect the time delay of drawing a frame of picture; when the first thread is a critical frame drawing thread, assigning a high priority to the first thread; wherein the higher the priority assigned to a thread by the terminal device, the more computing resource the terminal device provides to the thread; the first system function being a function called by a frame drawing thread after filling data to be drawn to an applied buffer, the first system function being used to submit the buffer filled by the frame drawing thread to a buffer queue, wherein the terminal device displays the data in the buffer filled by the frame drawing thread on the display by consuming the BufferQueue.

2. The method of claim 1, wherein, The determination of whether the first thread is a critical frame drawing thread comprises: obtaining the screen size of the display; determining whether the size of the buffer filled by the first thread is greater than the product of the screen size and a first preset percentage; if the size of the buffer filled by the first thread is greater than the product of the screen size and the first preset percentage, determining that the first thread is a critical frame drawing thread.

3. The method of claim 1, wherein, The determination of whether the first thread is a critical frame drawing thread comprises: obtaining a target size, the terminal device displaying a frame of picture within the target size, the target size being smaller than the screen size of the display; determining whether the size of the buffer filled by the first thread is greater than the product of the target size and a second preset percentage; if the size of the buffer filled by the first thread is greater than the product of the target size and the second preset percentage, determining that the first thread is a critical frame drawing thread.

4. The method of claim 1, wherein, The determination of whether the first thread is a critical frame drawing thread comprises: obtaining the screen size of the display; determining whether the size of the buffer filled by the first thread is greater than the product of the screen size and a first preset percentage; if the size of the buffer filled by the first thread is greater than the product of the screen size and the first preset percentage, determining whether the size of the buffer filled by the first thread is greater than the product of a target size and a second preset percentage, the terminal device displaying a frame of picture within the target size, the target size being smaller than the screen size of the display; if the size of the buffer filled by the first thread is greater than the product of the target size and the second preset percentage, determining that the first thread is a critical frame drawing thread.

5. The method according to any one of claims 1 to 4, characterized in that, The first system function comprises any one of the following: a queue buffer function, an onFrameAvailable function.

6. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: when it is detected that a second thread in the terminal device calls a first system function and the second thread is a system frame drawing thread, assigning a high priority to the second thread.

7. A processing device, characterized by comprising functional means for implementing the method according to any one of claims 1 to 6.

8. A processing device, characterized by comprising: a memory and a processor; said memory for storing program instructions; said processor for executing the program instructions in the memory to implement the method according to any one of claims 1 to 6.

9. A terminal device, comprising: a computer program product comprising instructions for implementing the method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, said computer-readable storage medium stores program code for computer execution, the program code comprising instructions for implementing the method according to any one of claims 1 to 6.

11. A computer program product, characterised in that, said computer program product comprises instructions for implementing the method according to any one of claims 1 to 6.

Images