Processing method for terminal device, and related apparatus

Abstract

Provided in the present application are a processing method for a terminal device, and a related apparatus, wherein the terminal device comprises a display. The processing method comprises: when it is detected that a first thread in a terminal device calls a first system function and the first thread is not a system render thread, determining that the first thread is a third-party render thread, wherein the first system function is a function called by a render thread, the first system function is configured to submit a buffer, which is drawn by the render thread, to BufferQueue, and by means of consuming BufferQueue, the terminal device displays the buffer, which is drawn by the render thread, onto a display. The method can identify a third-party render thread in a terminal device.

Description

Processing method and related device of terminal equipment

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 14, 2023, with application number 202311024831.8 and application name “Processing Method and Related Device of Terminal Equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal equipment, and in particular to a processing method and related apparatus for terminal equipment. Background Art

[0003] When a user uses a terminal device, each frame displayed on the terminal device must be drawn by at least one frame drawing thread. If a frame cannot be drawn within the specified time, the user will intuitively feel lag, which greatly affects the user experience.

[0004] Currently, in order to reduce lag and improve user experience, the terminal device will identify whether it is a frame drawing thread. If it is identified as a frame drawing thread, it will be assigned a higher priority so that more computing resources can be allocated to the frame drawing thread during system scheduling, thereby reducing the occurrence of lag and unsmoothness.

[0005] However, the frame drawing threads currently recognized by terminal devices are mainly concentrated in the system frame drawing thread (also known as System Render). For some apps on terminal devices, when drawing each frame, in addition to the system frame drawing thread (also known as System Render), they may also be based on the frame drawing threads of some third-party frame drawing frameworks (such as Flutter and WebView). Currently, there is no solution to identify the frame drawing threads of third-party frame drawing frameworks.

[0006] Summary of the Invention

[0007] The present application provides a processing method and related apparatus for a terminal device, which are used to identify a frame drawing thread in a third-party frame drawing framework.

[0008] In a first aspect, the present application provides a processing method for a terminal device, wherein the terminal device includes a display, the method comprising: upon detecting that a first thread in the terminal device calls a first system function and the first thread is not a system frame drawing thread, determining that the first thread is a third-party frame drawing thread;

[0009] Among them, the first system function is a function called after the frame drawing thread fills the drawn data into the applied buffer buffer. The first system function is used to submit the buffer filled by the frame drawing thread to the BufferQueue. Among them, the terminal device displays the data in the buffer filled by the frame drawing thread on the display by consuming the BufferQueue.

[0010] Since the frame drawing thread first applies for a buffer when drawing, and then fills the drawn data into the buffer, in this application, the buffer data filled by the frame drawing thread is also referred to as the drawn buffer.

[0011] Or in other words, in this application, the buffer drawn by the frame drawing thread is the data drawn by the frame drawing thread.

[0012] Because the frame drawing threads have a common feature: after completing the drawing, the frame drawing thread will submit the drawn buffer to the BufferQueue through the system function (also referred to as the first system function in this application), so that the terminal device can display the data in the buffer drawn by the frame drawing thread on the display by consuming the buffer in the BufferQueue.

[0013] Therefore, in this embodiment, the third-party frame drawing thread in the terminal device is identified based on this feature.

[0014] It should be understood that the technical solution provided by this application can enable the terminal device to fully identify the third-party frame drawing thread in the terminal device. Or to put it another way, the technical solution provided by this application will not miss the frame drawing thread in the terminal device.

[0015] Furthermore, after identifying the third-party frame drawing thread, the terminal device can, for example, assign a high priority to the third-party frame drawing thread so as to allocate more computing resources to the third-party frame drawing thread, thereby reducing the time it takes for the third-party frame drawing thread to draw the buffer, and further reducing the impact of the third-party frame drawing thread on the duration of the entire frame, thereby achieving smoothness in the operation of the APP that is drawn by the frame drawing thread in the third-party frame framework.

[0016] It should be understood that when a frame is drawn by at least one frame drawing thread, some of the at least one frame drawing thread may draw buffers of small sizes, while some of the frame drawing threads may draw buffers of large sizes.

[0017] How to define large size and small size may depend on the actual scenario. For example, a threshold may be set, and a buffer size larger than the threshold is considered large, while a buffer size smaller than or equal to the threshold is considered small.

[0018] It should be understood that the larger the drawing buffer size, the longer the drawing time required. Therefore, compared to the drawing thread with a smaller drawing size, the drawing thread with a larger drawing size is more critical to the ability to draw a frame and complete it. In view of this, as an optional embodiment, after identifying a third-party drawing thread, this application first determines whether the third-party drawing thread is a key drawing thread. Only if the third-party drawing thread is a key drawing thread, a high priority is assigned to the first thread; wherein, a key drawing thread is a drawing thread that can affect the latency of a frame.

[0019] Specifically, in the present application, a frame drawing thread whose drawing buffer size is larger than a preset threshold is considered to be a frame drawing thread that affects the latency of a frame.

[0020] For example, determining whether the first thread is a key frame thread includes:

[0021] Get the screen size of the monitor;

[0022] Determine whether the size of the buffer drawn by the first thread is greater than the product of the screen size and a first preset percentage;

[0023] If the size of the buffer drawn by the first thread is larger than the product of the screen size and a first predetermined percentage, the first thread is determined to be a key frame thread.

[0024] For another example, determining whether the first thread is a key frame thread includes:

[0025] Get the target size, and the terminal device displays a frame within the target size. The target size is smaller than the screen size of the display;

[0026] Determine whether the size of the buffer drawn by the first thread is greater than the product of the target size and a second preset percentage;

[0027] If the size of the buffer drawn by the first thread is larger than the product of the target size and the second predetermined percentage, the first thread is determined to be a key frame thread.

[0028] For another example, determining whether the first thread is a key frame thread includes:

[0029] Get the screen size of the monitor;

[0030] Determine whether the size of the buffer drawn by the first thread is greater than the product of the screen size and a first preset percentage;

[0031] If the size of the buffer drawn by the first thread is larger than the product of the screen size and the first preset percentage, determining whether the size of the buffer drawn by the first thread is larger than the product of the target size and the second preset percentage, the terminal device displays a frame within the target size, and the target size is smaller than the screen size of the display;

[0032] If the size of the buffer drawn by the first thread is larger than the product of the target size and the second predetermined percentage, the first thread is determined to be a key frame thread.

[0033] In combination with the first aspect, in a possible implementation manner, the first system function includes any one of the following: a queue buffer queuebuffer function and an onFrameAvailable function.

[0034] In combination with the first aspect, in a possible implementation, the method further includes: when it is detected that a second thread in the terminal device calls the first system function and the second thread is a system frame drawing thread, allocating a high priority to the second thread.

[0035] In a second aspect, the present application provides a processing device, which may include various functional modules for implementing the method in the first aspect. For example, the device includes a detection module and an identification module.

[0036] For example, the detection module may be used to detect whether a first thread in the terminal device calls a first system function;

[0037] an identification module, configured to, when the detection module detects that the first thread in the terminal device calls the first system function, identify whether the first thread is a system frame drawing function, and if not, determine that the first thread is a third-party frame drawing thread;

[0038] Among them, the first system function is a function called after the frame drawing thread fills the drawn data into the applied buffer buffer. The first system function is used to submit the buffer buffer drawn by the frame drawing thread to the BufferQueue. Among them, the terminal device displays the data in the buffer filled by the frame drawing thread on the display by consuming the BufferQueue.

[0039] In conjunction with the second aspect, in a possible implementation, the terminal device further includes a scheduling module, the scheduling module being configured to assign a high priority to the first thread;

[0040] Among them, the higher the priority assigned by the terminal device to the thread, the more computing resources the terminal device provides to the thread.

[0041] In conjunction with the second aspect, in a possible implementation, the identification module is further configured to determine whether the first thread is a key frame thread;

[0042] The scheduling module is specifically configured to: assign a high priority to the first thread when the first thread is a key frame drawing thread; wherein the key frame drawing thread is a frame drawing thread that can affect the delay of drawing a frame of picture.

[0043] In combination with the second aspect, in one possible implementation, the identification module identifies whether the first thread is a key frame thread, including: obtaining the screen size of the display; determining whether the size of the buffer filled by the first thread is greater than the product of the screen size and a first preset percentage; if the size of the buffer filled by the first thread is greater than the product of the screen size and the first preset percentage, determining that the first thread is a key frame thread.

[0044] In combination with the second aspect, in one possible implementation, the identification module identifies whether the first thread is a key frame thread, including: obtaining a target size, the terminal device displays a frame of picture within the target size, and the target size is smaller than the screen size of the display; determining whether the size of the buffer filled by the first thread is greater than the product of the target size and a second preset percentage; if the size of the buffer filled by the first thread is greater than the product of the target size and the second preset percentage, determining that the first thread is a key frame thread.

[0045] In combination with the second aspect, in a possible implementation, the identification module identifies whether the first thread is a key frame thread, including: obtaining the screen size of the display; determining whether the size of the buffer filled by the first thread is greater than the product of the screen size and a first preset percentage; if the size of the buffer filled by the first thread is greater than the product of the screen size and the first preset percentage, determining whether the size of the buffer filled by the first thread is greater than the product of the target size and a second preset percentage, and the terminal device displays a frame within the target size, and the target size is smaller than the screen size of the display; if the size of the buffer filled by the first thread is greater than the product of the target size and the second preset percentage, determining that the first thread is a key frame thread.

[0046] In combination with the second aspect, in a possible implementation, the first system function includes any one of the following: a queue buffer queuebuffer function and an onFrameAvailable function.

[0047] In combination with the second aspect, in a possible implementation, when the detection module detects that the second thread in the terminal device calls the first system function and the identification module identifies that the second thread is a system frame drawing thread, the scheduling module is also used to assign a high priority to the second thread.

[0048] In some implementations, these modules may be implemented in software and / or hardware. In this implementation, the device may include a processor and a transceiver. For example, the detection module and the identification module may be implemented by the processor executing program code stored in a memory, and the acquisition module may be implemented by a transceiver. Optionally, the device may also include a memory.

[0049] In a third aspect, the present application provides a terminal device, which may include various functional modules for implementing the method in the first aspect. In some implementations, these modules may be implemented in software and / or hardware.

[0050] In a fourth aspect, the present application provides a computer-readable storage medium storing a program code for execution by a communication device, wherein the program code includes instructions for implementing the method in the first aspect.

[0051] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a communication device, enables the communication device to implement the method in the first aspect.

[0052] It can be understood that the effects that can be obtained in the second, third, fourth and fifth aspects can be referred to the description in the first aspect and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] FIG1 is a schematic structural diagram of a terminal device provided in an embodiment of the present application;

[0054] FIG2 is a software structure block diagram of a terminal device applicable to an embodiment of the present application;

[0055] FIG3 is a schematic diagram of a frame of image displayed by a terminal device according to an embodiment of the present application;

[0056] FIG4 is an architecture of a method applicable to the present application provided in one embodiment of the present application;

[0057] FIG5 is a diagram illustrating an architecture of a method applicable to the present application provided in another embodiment of the present application;

[0058] FIG6 is a schematic diagram of a process for identifying whether a first thread is a third-party frame drawing thread according to an embodiment of the present application;

[0059] FIG7 is a schematic diagram of a process for identifying whether a first thread is a third-party frame drawing thread according to another embodiment of the present application;

[0060] FIG8 is a schematic diagram of target size provided by another embodiment of the present application;

[0061] FIG9 is a schematic diagram of a process for identifying whether a first thread is a third-party frame drawing thread according to another embodiment of the present application;

[0062] FIG10 is a schematic structural diagram of a processing device provided in one embodiment of the present application;

[0063] FIG11 is a schematic structural diagram of a processing device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0064] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0065] To facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or execution order, and the words "first" and "second" do not necessarily mean different.

[0066] It should be noted that, in this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0067] In addition, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b and c can mean: a, or b, or c, or a and b, or a and c, or b and c, or a, b and c, where a, b, c can be single or multiple.

[0068] The processing method of the terminal device provided in the embodiment of the present application can also be called a method for identifying a third-party frame drawing thread, and can be applied to a terminal device with a display function. The terminal device can be a mobile phone, a tablet computer, a personal computer (PC), a smart screen, a car device, a wearable terminal device such as a smart watch, and can also be various teaching auxiliary tools (such as learning machines, early education machines), smart toys, portable robots, personal digital assistants (PDAs), augmented reality technology (AR) devices, virtual reality (VR) devices, etc., and can also be a device with mobile office functions, a device with smart home functions, a device with audio and video entertainment functions, a device that supports smart travel, etc. It should be understood that the embodiments of the present application do not limit the specific technology and specific device form adopted by the terminal device.

[0069] In order to better understand the embodiment of the present application, the hardware structure of the terminal device of the embodiment of the present application is introduced below. For example, Figure 1 is a schematic diagram of the structure of a terminal device provided by the embodiment of the present application.

[0070] Figure 1 shows a schematic diagram of the structure of a terminal device. The terminal device may include a processor 110, an internal memory 121, a charging management module 140, a power management module 141, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a sensor module 180, a button 190, a display 194, a timer 196, and the like.

[0071] Optionally, the sensor module 180 may include a pressure sensor 180A, a touch sensor 180B, and the like.

[0072] It is understood that the structures illustrated in the embodiments of the present application do not constitute specific limitations on the terminal device. In other embodiments of the present application, the terminal device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0073] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, etc. The different processing units may be independent devices or integrated into one or more processors. The processor 110 may also include a memory for storing instructions and data.

[0074] The charging management module 140 is used to receive charging input from a charger. The charger can be a wireless charger or a wired charger. The power management module 141 is used to connect the charging management module 140 to the processor 110.

[0075] The wireless communication function of the terminal device can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0076] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. The antennas in a terminal device can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization.

[0077] Mobile communication module 150 can provide wireless communication solutions for terminal devices, including 2G / 3G / 4G / 5G. It can include at least one filter, switch, power amplifier, and low-noise amplifier (LNA). Mobile communication module 150 receives electromagnetic waves from antenna 1, filters and amplifies the received electromagnetic waves, and transmits them to the modem processor for demodulation.

[0078] The wireless communication module 160 can provide wireless communication solutions applied to terminal devices, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), etc.

[0079] In some embodiments, antenna 1 of the terminal device is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that the mobile phone can communicate with the network and other devices through wireless communication technology.

[0080] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the terminal device may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0081] As an example, each frame displayed on the display screen 194 needs to be drawn by a frame drawing process in the terminal device.

[0082] The terminal device can implement display functions through a graphics processing unit (GPU), display screen 194, and application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The GPU can also be called a display core, a visual processor, or a display chip.

[0083] The internal memory 121 can be used to store computer-executable program code, which includes instructions. The internal memory 121 can include a program storage area and a data storage area. The program storage area can store applications required for at least one function of the operating system (such as a sound playback function, an image playback function, etc.). The data storage area can store data created during the use of the terminal device (such as audio data, a phone book, etc.).

[0084] The terminal device can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, and the application processor.

[0085] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The speaker 170A, also known as the "speaker", is used to convert audio electrical signals into sound signals. The terminal device can listen to music or listen to hands-free calls through the speaker 170A. The receiver 170B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. When the terminal device answers a call or voice message, the voice can be heard by placing the receiver 170B close to the human ear. The microphone 170C, also known as the "microphone" or "microphone", is used to convert sound signals into electrical signals.

[0086] The pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor 180A can be set on the display screen 194. When a touch operation is applied to the display screen 194, the terminal device can detect the intensity of the touch operation based on the pressure sensor 180A. The terminal device can also calculate the position of the touch based on the detection signal of the pressure sensor 180A. The touch sensor 180B can be set on the display screen 194, and the touch sensor 180B and the display screen 194 form a touch screen, also known as a "touch screen". The touch sensor 180B is used to detect touch operations applied thereto or nearby. The touch sensor 180B can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180B can also be set on the surface of the mobile phone, at a different location from the display screen 194.

[0087] Keys 190 include a power button, a volume button, and the like. Keys 190 can be mechanical or touch-sensitive. The terminal device can receive key inputs and generate key signal inputs related to user settings and function control of the terminal device. Indicator 192 can be an indicator light that can be used to indicate charging status, battery level changes, messages, missed calls, notifications, and the like. Timer 196 can be used to record time information. For example, when the terminal device is recording a video, the start and end times of the video can be recorded, as well as the duration of the recorded video.

[0088] Figure 2 is a block diagram of the software structure of a terminal device applicable to embodiments of the present application. The layered architecture divides the terminal device's software system into several layers, each with clear roles and divisions of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into four layers: from top to bottom, the application layer (applications), the application framework layer (application framework), the Android runtime (Android runtime) and system libraries, and the kernel layer (kernel).

[0089] The application layer may include a series of application packages, and the application layer runs applications by calling the application programming interface (API) provided by the application framework layer.

[0090] For example, an application package may include desktop, calendar, memo, map, navigation, Bluetooth, music, JD.com, and short message applications.

[0091] The application framework layer provides APIs and programming frameworks for applications in the application layer. The application framework layer includes some predefined functions.

[0092] For example, the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

[0093] The window manager manages windowed applications. It can obtain the display size, determine whether a status bar is present, lock the screen, and take screenshots. Content providers store and retrieve data and make it accessible to applications. This data can include video, images, audio, incoming and outgoing calls, browsing history and bookmarks, and the phone book. The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images. The phone manager provides communication functions for the terminal device, such as managing call states (including connected and ended calls). The resource manager provides applications with various resources, such as localized strings, icons, images, layout files, and video files. The notification manager enables applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically, without requiring user interaction. For example, the notification manager is used to notify users of download completions and message reminders. The notification manager can also be a notification that appears in the system's top status bar in the form of an icon or scrolling text bar, such as a notification from an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message may be displayed in the status bar, a notification sound may be emitted, the terminal device may vibrate, or an indicator light may flash.

[0094] The Android system runtime consists of core libraries and a virtual machine. The Android system runtime is responsible for scheduling and management of the Android system. The core library consists of two parts: one for the Java language's callable functions and the other for the Android core library. The application layer and the application framework layer run in the virtual machine. The virtual machine executes the Java files in the application layer and application framework layer as binary files. The virtual machine is responsible for performing functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection. The system library can include modules for multiple functions, such as the surface manager, media library, and 3D graphics processing library.

[0095] The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as static image files. It also supports various audio and video encoding formats, such as JPG and PNG. The 3D graphics processing library implements 3D graphics drawing, image rendering, compositing, and layer processing.

[0096] The kernel layer is the layer between hardware and software. The kernel layer is used to drive the hardware, making it work. The kernel layer includes at least display drivers, screen drivers, camera drivers, and sensor drivers, etc., which are not limited in this embodiment of the application.

[0097] For the terminal devices shown in Figures 1 and 2, each frame displayed by the terminal device is drawn by at least one frame drawing thread. For example, assuming the terminal device displays 40 frames per second, the terminal device currently displays a frame as shown in Figure 3, where this frame is drawn by at least one frame drawing thread.

[0098] Specifically, the process from drawing data by each frame drawing thread to displaying the drawn frame of data on the display of the terminal device is as follows: each frame drawing thread fills the drawn data into a buffer applied for in advance, and then calls the system function (also referred to as the first system function in this application) to submit the filled buffer to the buffer queue BufferQueue. After that, the terminal device calls the frame consumption module to consume the BufferQueue to display the data drawn by each frame drawing thread on the display.

[0099] For example, after each frame drawing thread calls the system function to submit the buffer to the BufferQueue, the terminal device calls the frame consumption module SurfaceLinger to display the drawn data on the display. The SurfaceLinger module is used to receive the data drawn by each frame drawing thread in the BufferQueue, synthesize these data into a frame and then send it to the display.

[0100] As described above, since each frame drawing thread first applies for a buffer when drawing, and then fills the drawn data into the buffer, in this application, the drawing performed by each frame drawing thread is also referred to as drawing a buffer.

[0101] In addition, it should be noted that in this application, the terminal device calling the frame consumption module to consume the BufferQueue can also be called consuming the bufffer in the BufferQueue, or it can also be called consuming the data in the bufffer in the BufferQueue, which does not constitute a limitation of this application.

[0102] It is understandable that if a frame cannot be drawn within the rated time, the user will intuitively feel a lag, which will affect the user experience.

[0103] Therefore, in order to reduce lag and improve user experience, the terminal device will identify the frame drawing thread and then assign a higher priority to the identified frame drawing thread so that more computing resources can be scheduled for the frame drawing thread during system scheduling, thereby shortening the time it takes for the frame drawing thread to complete the drawing buffer, thereby reducing the occurrence of lag and unsmoothness.

[0104] However, the frame drawing threads currently recognized by terminal devices are mainly concentrated in the system frame drawing thread (also known as System Render). For some apps on terminal devices, when drawing each frame, in addition to the System Render, they may also be based on the frame drawing threads of some third-party frame drawing frameworks (such as Flutter, webview, etc.).

[0105] For example, for shopping apps, the graphics of each frame are becoming more and more complex. In addition to the information and pictures of the displayed products, video information may also need to be drawn. In this case, the drawing of this video information may be done by a third-party frame drawing thread.

[0106] For example, some mini-programs within the WeChat app may display screens based on third-party drawing threads. For example, a mini-program might be a game. However, there is currently no solution for identifying drawing threads within third-party drawing frameworks. Consequently, because terminal devices cannot identify third-party drawing threads, users may experience lag and stuttering when using apps that rely on third-party drawing frameworks, impacting the user experience.

[0107] Taking the frame shown in Figure 3 as an example, for example, the live bar 301 displayed in the frame shown in Figure 3 needs to be drawn based on a third-party frame drawing thread. At this time, if the terminal device cannot recognize the third-party frame drawing thread, then there may be a situation where the terminal device has less computing power resources scheduled for the third-party frame drawing thread, causing the third-party frame drawing thread to take a long time to draw. In this case, there may be frame drops due to the long drawing time of the third-party frame drawing thread, resulting in lag problems.

[0108] In view of this, the present application provides a processing method and a processing device for a terminal device, which is used to identify the frame drawing thread in a third-party frame drawing framework, further improve the smoothness of the APP drawn by the frame drawing thread in the third-party frame drawing framework during operation, and further improve the user experience.

[0109] The processing method of the terminal device provided by this application is described below with reference to the accompanying drawings.

[0110] Before introducing the processing method of the terminal device provided by the present application, the system architecture of the present application is first introduced. As shown in FIG4 , the system architecture includes a default rendering framework 401 , a third-party rendering framework 402 , and a third-party rendering framework 403 .

[0111] The default drawing framework 401 includes a system UI thread and a system Render thread (ie, a System Render thread). Detailed descriptions of the UI thread and the System Render thread can be found in the related art and will not be repeated here.

[0112] The third-party drawing framework 402 and the third-party drawing framework 403 include a third-party drawing thread for drawing frames and / or at least one wake-up thread, wherein the at least one wake-up thread is used to wake up the third-party drawing thread.

[0113] For example, as shown in FIG4 , threads 1 to 5 included in the third-party drawing framework 402 can be considered as wake-up threads of the third-party drawing frame thread 1 in the third-party drawing framework 402 , and threads 1 ′ to 5 ′ included in the third-party drawing framework 403 can be considered as wake-up threads of the third-party drawing frame thread 2 in the third-party drawing framework 403 .

[0114] Specifically, in third-party drawing frame 402, thread 1 pointing to thread 2 means that thread 1 wakes up thread 2, thread 2 pointing to thread 3 means that thread 2 wakes up thread 3, thread 3 pointing to third-party drawing frame thread 1 means that thread 3 wakes up third-party drawing frame thread 1, thread 4 pointing to thread 5 means that thread 4 wakes up thread 5, and thread 5 pointing to third-party drawing frame thread 1 means that thread 5 wakes up third-party drawing frame thread 1. It can be understood that threads 1, 2, and 4 can be considered as indirectly waking up threads of third-party drawing frame thread 1, while threads 3 and 5 can be considered as directly waking up threads of third-party drawing frame thread 1.

[0115] Similarly, thread 1 ′, thread 2 ′, and thread 4 ′ in the third-party drawing frame 403 can be considered as indirectly woken threads of the third-party drawing frame thread 2 , and thread 3 ′ and thread 5 ′ can be considered as directly woken threads of the third-party drawing frame thread 2 .

[0116] Specifically, the system Render thread, the third-party frame drawing thread 1, and the third-party frame drawing thread 2 jointly complete the drawing of one frame.

[0117] It should be noted that this embodiment does not limit the specific form of at least one third-party drawing frame.

[0118] For example, at least one third-party rendering framework may be the same or different. For example, third-party rendering framework 402 and third-party rendering framework 403 in FIG4 may both be Flutter rendering frameworks, or both may be webview frameworks, or one may be a Flutter rendering framework and the other may be a webview framework.

[0119] It should also be noted that FIG4 only takes two third-party frame drawing threads as an example, but does not constitute a limitation of the present application.

[0120] For example, only one third-party frame drawing thread may be included. In this case, it can be considered that the system Render thread and the third-party frame drawing thread jointly complete the drawing of one frame.

[0121] For another example, more third-party frame drawing threads may be included, for example, there are a total of N third-party frame drawing threads. In this case, it can be considered that the system Render thread and the N third-party frame drawing threads jointly complete the drawing of one frame.

[0122] It should be understood that the buffer size used by the frame drawing thread may vary. The buffer size can be understood as follows: for example, if the frame drawing thread requests a buffer with a length of M and a width of P, the buffer size is M*N. Furthermore, it should be understood that a larger buffer size means more data for the frame drawing thread to draw, and accordingly, a longer drawing time.

[0123] For the buffers drawn by the third-party drawing frame threads in at least one third-party drawing framework, some of the buffers drawn by the third-party drawing frame threads may be relatively large, while some of the buffers drawn by the third-party drawing frame threads may be relatively small. It should be noted that this embodiment does not limit how to define large or small size.

[0124] During implementation, a threshold may be set, and then a buffer with a size larger than the threshold is considered a large-size buffer, and a buffer with a size smaller than the threshold is considered a small-size buffer.

[0125] For example, as shown in Figure 5, third-party frame drawing thread 1 draws into a large buffer, while third-party frame drawing thread 2 draws into a small buffer. It should be noted that this is merely an example and does not constitute a limitation. For example, third-party frame drawing thread 1 and third-party frame drawing thread 2 may both draw into large buffers, or they may both draw into small buffers.

[0126] As shown in Figures 4 and 5, all frame drawing threads have the following characteristics when drawing: after completing the drawing, the frame drawing thread will submit the drawn buffer to the BufferQueue through the system function (i.e., the first system function), so that the frame consumption module in the terminal device consumes the buffer in the BufferQueue to display the buffer drawn by the frame drawing thread on the display.

[0127] As an example, the first system function is the queue buffer queuebuffer function and the onFrameAvailable function.

[0128] It should be noted that when there are multiple third-party frame drawing threads, the system functions used by these multiple third-party frame drawing threads when submitting buffers can be the same or different, and this does not constitute a limitation of this application. Taking Figure 4 or Figure 5 as an example, for example, the system function used by third-party frame drawing thread 1 is the queuebuffer function, while the system function used by third-party frame drawing thread 2 is the onFrameAvailable function. Alternatively, the system function used by third-party frame drawing thread 1 can be the onFrameAvailable function, while the system function used by third-party frame drawing thread 2 is the queuebuffer function.

[0129] As an example, the frame consumption module is surfaceflinger.

[0130] Therefore, the method for identifying a third-party frame drawing thread provided by the present application includes: when it is detected that the first thread in the terminal device calls the first system function and the first thread is not the system frame drawing thread, determining that the first thread is a third-party frame drawing thread; wherein, the first system function is a function called by the frame drawing thread, and the first system function is used to submit the buffer drawn by the frame drawing thread to the buffer queue BufferQueue, wherein the terminal device displays the buffer drawn by the frame drawing thread on the display by consuming the BufferQueue.

[0131] Or to put it another way, in this application, the terminal device determines the thread that submits the buffer using the first system function as the frame drawing thread, and then compares it with the system frame drawing thread to identify whether it is a third-party frame drawing thread.

[0132] It should be understood that the technical solution provided by this application can enable the terminal device to fully identify the third-party frame drawing thread in the terminal device. Or to put it another way, the technical solution provided by this application will not miss the third-party frame drawing thread in the terminal device.

[0133] Furthermore, after identifying the third-party frame drawing thread, the terminal device can, for example, assign a high priority to the third-party frame drawing thread so as to allocate more computing resources to the third-party frame drawing thread, thereby reducing the time it takes for the third-party frame drawing thread to draw the buffer, and further reducing the impact of the third-party frame drawing thread on the duration of the entire frame, thereby achieving smoothness in the operation of the APP that is drawn by the frame drawing thread in the third-party frame framework.

[0134] As previously mentioned, when drawing a frame using at least one frame drawing thread, some of these threads may draw small-sized images, while others may draw large-sized images. Because larger images require longer rendering time, threads drawing large-sized images are more critical to the ability to draw a frame than threads drawing small-sized images. In other words, threads drawing small-sized images may have less impact on the time it takes to draw a frame than threads drawing large-sized images.

[0135] In view of this, in some implementations provided in this application, after identifying a third-party frame drawing thread, when assigning a high priority to the third-party frame drawing thread, the high priority is assigned to the first thread only when the third-party frame drawing thread is a key frame drawing thread; wherein the key frame drawing thread is a frame drawing thread that can affect the latency of a frame of the image.

[0136] Three embodiments of identifying the first thread as a key frame thread are given below.

[0137] Example 1

[0138] The method comprises: when a first thread of a terminal device calls a first system function to submit a drawn buffer, the terminal device calls an identification module.

[0139] For example, the first system function is the queuebuffer function and the onFrameAvailable function.

[0140] In specific implementation, when the first frame drawing thread submits the buffer through the first system function, a function call stub, or also called an identification module interface, can be inserted into the first system function to enter the identification process of the third-party frame drawing thread.

[0141] When the terminal device calls the identification module, as shown in FIG6 , it executes:

[0142] S601, determine whether the first thread is a third-party frame drawing thread, if so, execute S602.

[0143] S602: Acquire the screen size of the display.

[0144] For example, obtain the physical size of the display of the terminal device hardware.

[0145] S603: Determine whether the size of the buffer drawn by the first thread is larger than the product of the screen size and a first preset percentage. If so, execute S604.

[0146] For example, the first preset percentage is expressed as x%, that is, it is determined whether the size of the buffer drawn by the first thread is larger than the screen size * x%.

[0147] In this embodiment, a buffer whose size is larger than the screen size*x% is considered a large-size buffer, and a buffer whose size is smaller than or equal to the screen physical size*x% is considered a small-size buffer.

[0148] It should be noted that this embodiment does not limit the specific value of the first preset percentage.

[0149] S604: Determine that the first thread is a key frame thread.

[0150] It can be seen that in this embodiment, by comparing the size of the buffer drawn by the first thread with the screen size * x%, it is determined whether the buffer drawn by the first thread is large or small. Then, when it is determined that the buffer drawn by the first thread is large, the first thread is determined as a key frame drawing thread that affects the drawing of one frame.

[0151] Furthermore, when it is determined that the first thread is a third-party key frame thread that affects the rendering of a frame, more computing power is allocated to the first thread, thereby reducing the time it takes for the third-party key frame thread to draw the buffer, to ensure that the rendering of a frame can be completed within the rated time, and to improve the smoothness of the APP that is drawn by the third-party first frame thread during operation.

[0152] In this embodiment, the identification process can be executed in the application process so as not to rely on the operating system. In other words, when executed in the application process, the identification process can be completed regardless of the type of operating system used by the terminal device, such as Android system or iOS system.

[0153] Example 2

[0154] The method comprises: when a first thread of a terminal device calls a first system function to submit a drawn buffer, the terminal device calls an identification module.

[0155] For example, the first system function is the queuebuffer function and the onFrameAvailable function.

[0156] Similarly, in specific implementation, when the first frame drawing thread submits the buffer through the first system function, a function call stub can be inserted at the first system function, or also called an identification module interface, to enter the identification process of the third-party frame drawing thread.

[0157] When the terminal device calls the identification module, as shown in FIG7 , it executes:

[0158] S701, determine whether the first thread is a third-party frame drawing thread, if so, execute S702.

[0159] S702: Notify the frame consumption module of the size of the buffer drawn by the first thread.

[0160] S703: Obtain target size.

[0161] It should be understood that after each frame drawing thread used to draw a frame completes drawing, when the terminal device displays it, the target size used to display the drawn frame may be different from the screen size of the terminal device's display, that is, when the terminal device displays a frame on the screen of the display, it may only use a partial area for display.

[0162] For example, as shown in FIG8 , the terminal device needs to display a drawn frame in the area within the dotted box 801 rather than covering the entire display.

[0163] In this embodiment, the target size is also referred to as the display size. It should be emphasized that the display size in this embodiment is different from the screen size of the display in the embodiment of FIG. 6 . The screen size of the display in the embodiment of FIG. 6 is determined by the hardware. For example, the screen size can be understood as the size of the entire screen, including the status bar, etc., while the display size is determined by the system process.

[0164] S704 , determining whether the size of the buffer drawn by the first thread is larger than the product of the target size and the second preset percentage; if so, executing S704 .

[0165] For example, the second preset percentage is expressed as m%, that is, it is determined whether the size of the buffer drawn by the first thread is larger than the target size*m%.

[0166] In this embodiment, a buffer whose size is larger than the target size*m% is considered a large-size buffer, and a buffer whose size is smaller than or equal to the target size*m% is considered a small-size buffer.

[0167] It should be noted that this embodiment does not limit the specific value of the second preset percentage.

[0168] S705: Determine that the first thread is a key frame thread.

[0169] It can be seen that in this embodiment, by comparing the size of the buffer drawn by the first thread with the target size * m%, it is determined whether the buffer drawn by the first thread is large or small. Then, when it is determined that the buffer drawn by the first thread is large, the first thread is determined as a key frame thread that affects the drawing of one frame.

[0170] Furthermore, when it is determined that the first thread is a third-party key frame thread that affects the rendering of a frame, more computing power is allocated to the first thread, thereby reducing the time it takes for the third-party key frame thread to draw the buffer, to ensure that the rendering of a frame can be completed within the rated time, and to improve the smoothness of the APP that is drawn by the third-party first frame thread during operation.

[0171] This embodiment differs from the embodiment shown in FIG6 in that, whereas FIG6 uses the buffer size drawn by the first drawing thread and the screen size of the hardware display to determine whether the first drawing thread is a key drawing thread, this embodiment uses the buffer size drawn by the first drawing thread and the target size used when actually displaying a frame to determine whether the first drawing thread is a key drawing thread. Therefore, compared to the embodiment shown in FIG6, the embodiment shown in FIG7 can improve the accuracy of identifying key drawing threads.

[0172] Example 3

[0173] The method comprises: when a first thread of a terminal device calls a first system function to submit a drawn buffer, the terminal device calls an identification module.

[0174] For example, the first system function is the queuebuffer function and the onFrameAvailable function.

[0175] Similarly, in specific implementation, when the first frame drawing thread submits the buffer through the first system function, a function call stub can be inserted at the first system function, or also called an identification module interface, to enter the identification process of the third-party frame drawing thread.

[0176] When the terminal device calls the identification module, as shown in FIG9 , it executes:

[0177] S901, determine whether the first thread is a third-party frame drawing thread, if so, execute S902.

[0178] S902: Obtain the screen size of the display.

[0179] S903: Determine whether the size of the buffer drawn by the first thread is larger than the product of the screen size and a first preset percentage. If so, execute S904.

[0180] The above process can be executed in the application process space.

[0181] S904: Notify the frame consumption module of the size of the buffer drawn by the first thread.

[0182] S905 , the frame consumption module determines whether the size of the buffer drawn by the first thread is larger than the product of the target size and the second preset percentage. If so, execute S906 .

[0183] S906: Determine that the first thread is a key frame thread.

[0184] As can be seen, this embodiment differs from the embodiments shown in Figures 9 and 7 in that this embodiment first compares the size of the buffer drawn by the first thread with the product of the screen size and a first preset percentage (which can be considered a first-level screening). After the first frame drawing thread's buffer size is greater than the product of the screen size and the first preset percentage, the first thread is then identified as a key frame drawing thread affecting the drawing of a frame only if the size of the buffer drawn by the first thread is greater than the product of the target size and a second preset percentage (which can be considered a second-level screening). In other words, the terminal device uses a double-level screening process to determine whether the first thread is a key frame drawing thread. It should be understood that this double-level screening approach can improve the accuracy of identifying key frame threads when there are multiple third-party frame drawing threads, compared to the embodiment of Figure 6. Compared to the embodiment of Figure 7, since the system-side frame consumption module no longer identifies all third-party frame drawing threads, but only a subset, system-side overhead can be reduced.

[0185] Furthermore, when it is determined that the first thread is a third-party key frame thread that affects the rendering of a frame, more computing power is allocated to the first thread, thereby reducing the time it takes for the third-party key frame thread to draw the buffer, to ensure that the rendering of a frame can be completed within the rated time, and to improve the smoothness of the APP that is drawn by the third-party first frame thread during operation.

[0186] Above, three embodiments for determining key frame drawing threads are introduced. During implementation, in one example, if you only want to identify through the application, you can choose embodiment 1. If you want to identify through the system-application, you can go through embodiment 2 and embodiment 3. However, embodiment 3 is more efficient than embodiment 2: because when there are multiple frame drawing threads, handing them all over to the system to identify which ones are key frame drawing threads at one time will be lower than first screening through the application and then handing some over to the system for screening. Figure 10 is a structural diagram of a processing device of a terminal device provided by an embodiment of the present application. As shown in Figure 10, the processing device 1000 includes a detection module 1001 and an identification module 1002.

[0187] As an example, the detection module 1001 may be used to detect whether the first thread calls the first system function, or the detection module 1001 may be used to detect whether the system frame drawing thread calls the first system function.

[0188] As an example, the identification module 1002 may be used to execute the process of identifying whether the first thread is a key frame thread in the embodiment shown in FIG. 6 .

[0189] As another example, the identification module 1002 may be used to execute the process of identifying whether the first thread is a key frame thread in the embodiment shown in FIG. 7 .

[0190] As another example, the identification module 1002 may be used to execute the process of identifying whether the first thread is a key frame thread in the embodiment shown in FIG. 9 .

[0191] Figure 11 is a schematic diagram of the structure of a processing device of a terminal device provided in another embodiment of the present application. As shown in Figure 11, the processing device 1100 may include a processor 1101 and an interface circuit 1102. The processor 1101 and the interface circuit 1102 are coupled to each other. It is understood that the interface circuit 1102 may be a transceiver or an input / output interface. Optionally, the processing device 1100 may also include a memory 1103 for storing instructions executed by the processor 1101, or storing input data required by the processor 1101 to execute instructions, or storing data generated after the processor 1101 executes instructions.

[0192] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0193] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, a register, a hard disk, a mobile hard disk, a CD-ROM or any other form of storage medium well known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a network device or a terminal device. Of course, the processor and the storage medium can also be present in a network device or a terminal device as discrete components.

[0194] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are performed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, or other programmable device. The computer program or instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions may be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video disk; or a semiconductor medium, such as a solid-state drive.

[0195] In the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between different embodiments are consistent and can be referenced by each other. The technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

[0196] It is understood that the various numbers used in the embodiments of this application are merely for ease of description and are not intended to limit the scope of the embodiments of this application. The order of the sequence numbers of the above-mentioned processes does not necessarily imply a specific order of execution; the order of execution of the processes should be determined by their functions and inherent logic.

Claims

1. A processing method of a terminal device, characterized in that: The terminal device includes a display, and the method includes: When it is detected that a first thread in the terminal device calls a first system function and the first thread is not a system frame drawing thread, determining that the first thread is a third-party frame drawing thread; The first system function is a function called after the frame drawing thread fills the drawn data into the applied buffer buffer, and the first system function is used to submit the buffer filled by the frame drawing thread to the buffer queue BufferQueue, wherein the terminal device displays the data in the buffer filled by the frame drawing thread on the display by consuming the BufferQueue.

2. The method according to claim 1, characterized in that After identifying the first thread as a third-party frame drawing thread, the method further includes: assigning a high priority to the first thread; Among them, the higher the priority assigned by the terminal device to the thread, the more computing resources the terminal device provides for the thread.

3. The method according to claim 2, characterized in that Before assigning a high priority to the first thread, the method further includes: Determine whether the first thread is a key frame thread; The allocating a high priority to the first thread specifically includes: When the first thread is a key frame drawing thread, assigning a high priority to the first thread; The key frame drawing thread is a frame drawing thread that can affect the delay of drawing a frame of picture.

4. The method according to claim 3, characterized in that The determining whether the first thread is a key drawing frame thread includes: Obtaining the screen size of the display; Determine whether the size of the buffer filled by the first thread is greater than the product of the screen size and a first preset percentage; If the size of the buffer filled by the first thread is larger than the product of the screen size and a first preset percentage, the first thread is determined to be a key frame thread.

5. The method according to claim 3, characterized in that: The determining whether the first thread is a key drawing frame thread includes: Acquire a target size, the terminal device displays a frame of picture within the target size, and the target size is smaller than the screen size of the display; Determine whether the size of the buffer filled by the first thread is greater than the product of the target size and a second preset percentage; If the size of the buffer filled by the first thread is larger than the product of the target size and a second preset percentage, the first thread is determined to be a key frame thread.

6. The method according to claim 3, characterized in that: The determining whether the first thread is a key drawing frame thread includes: Obtaining the screen size of the display; Determine whether the size of the buffer filled by the first thread is greater than the product of the screen size and a first preset percentage; If the size of the buffer filled by the first thread is larger than the product of the screen size and a first preset percentage, determining whether the size of the buffer filled by the first thread is larger than the product of a target size and a second preset percentage, the terminal device displays a frame within the target size, and the target size is smaller than the screen size of the display; If the size of the buffer filled by the first thread is larger than the product of the target size and a second preset percentage, the first thread is determined to be a key frame thread.

7. The method according to any one of claims 1 to 6, characterized in that The first system function includes any one of the following: a queue buffer queuebuffer function and an onFrameAvailable function.

8. The method according to any one of claims 1 to 6, characterized in that The method further comprises: When it is detected that a second thread in the terminal device calls a first system function and the second thread is a system frame drawing thread, a high priority is assigned to the second thread.

9. A processing device, characterized in that: The method comprises a functional module for implementing the method according to any one of claims 1 to 8.

10. A processing device, characterized in that: include: Memory and processor; The memory is used to store program instructions; The processor is configured to execute program instructions in the memory to implement the method according to any one of claims 1 to 8.

11. A terminal device, characterized in that: Comprising the device as claimed in claim 9 or 10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code for computer execution, wherein the program code includes instructions for implementing the method according to any one of claims 1 to 8.

13. A computer program product, characterized in that The computer program product comprises instructions for implementing the method according to any one of claims 1 to 8.

Images