Data processing method and device and storage medium

Abstract

The invention discloses a data processing method and device and a storage medium. According to the method, in the sliding operation process, whether frame insertion is carried out or not is determined according to the real-time drawing and rendering condition, when frame insertion is determined to be carried out, one frame is inserted in advance or the lost frame is supplemented back to one frame or multiple frames, so that the situation of no frame synthesis caused by subsequent drawing and rendering timeout is reduced, display lagging is reduced, and the display efficiency is improved. And meanwhile, the display fluency and the jump of the displayed content can be improved, and the user experience is improved.

Description

Data processing method, device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on November 7, 2022, with application number 202211382250.7 and invention name “Data processing method, device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image display technology, and in particular to a data processing method, device, and storage medium. Background Art

[0003] Currently, users can access a wide range of content through the display screens of terminal devices. When there is a large amount of content, the display screen cannot display all of it at once. The terminal device can respond to user swipes on the display screen, controlling the displayed content to follow or not follow the user's swipe, making it easier for the user to browse relevant content. This content must undergo rendering, synthesis, and display processing before it can be presented on the display screen.

[0004] However, when there is a lot of content to be displayed, the drawing rendering may time out and cause frame drops, making the image synthesis and display cycle unstable, and the image is not obtained based on continuous frames, which may cause the content displayed on the display to experience abnormal phenomena such as display freezes and jumps.

[0005] Summary of the Invention

[0006] In order to solve the above technical problems, the present application provides a data processing method, device and storage medium, which aim to solve the jamming and jumping phenomena caused by drawing rendering timeout and frame loss.

[0007] In a first aspect, the present application provides a data processing method. The method includes: displaying a first interface of a first application; responding to a sliding operation on the first interface, obtaining an input event corresponding to the sliding operation; obtaining a first VSync signal, and rendering the Nth frame based on a first MOVE event, the first MOVE event being extracted from the input event corresponding to the sliding operation based on the timestamp of the first VSync signal; when the rendering time of the Nth frame is longer than one VSync signal cycle, after the rendering of the Nth frame is completed, obtaining the number of lost frames and displaying the Nth frame; selecting a minimum value from the number of lost frames and a set maximum number of interpolated frames as the number of interpolated frames M; before a second VSync signal arrives, rendering M frames based on the second MOVE event, and displaying the M frames; wherein the second VSync signal is the first VSync signal received after the rendering of the Nth frame is completed, and the second MOVE event is extracted from the input event corresponding to the sliding operation based on the timestamps of all or part of the VSync signals received during the rendering process of the Nth frame.

[0008] Therefore, when the drawing rendering times out, the lost frames can be made up by one or more frames through frame filling, reducing the frame loss caused by missing the VSync signal due to the drawing rendering timeout, so that the content displayed on the display can change smoothly, increase the smoothness of the display, reduce jumps, and further improve the user experience.

[0009] According to the first aspect, obtaining the number of lost frames includes: determining a first time for starting drawing rendering and a second time for ending drawing rendering of the Nth frame; calculating the number of lost frames based on the first time, the second time, and a set drawing rendering duration corresponding to the Nth frame, where the set drawing rendering duration is one VSync signal cycle.

[0010] Exemplarily, the first time is, for example, Tbegin mentioned below, the second time is, for example, Tend mentioned below, the set drawing rendering duration is, for example, one VSync signal cycle, namely, VSync mentioned below, and the number of lost frames is, for example, M mentioned below.

[0011] According to the first aspect, or any implementation of the first aspect above, based on the following formula, the number of lost frames is calculated according to the set drawing rendering duration corresponding to the first time, the second time, and the Nth frame: number of lost frames = floor[(second time - first time) / VSync signal period] - 1.

[0012] According to the first aspect, or any implementation of the first aspect above, selecting a minimum value from the number of lost frames and the set maximum number of insertable frames as the number of insertable frames M includes: determining the reception time of the second VSync signal according to the VSync signal period; determining the average rendering time of each frame in the N frames that have completed rendering according to the rendering time of the N frames that have completed rendering; calculating the predicted number of insertable frames according to the receiving time, the second time, and the average rendering time; and selecting a minimum value from the predicted number of insertable frames, the number of lost frames, and the set maximum number of insertable frames as the number of insertable frames M.

[0013] For example, the receiving time may be TnextVsyn mentioned below, the average drawing rendering time may be Taverage mentioned below, and the predicted number of insertable frames may be countAllow mentioned below.

[0014] According to the first aspect, or any implementation of the first aspect above, based on the following formula, the predicted number of interpolated frames is calculated according to the receiving time, the second time, and the average drawing and rendering time: predicted number of interpolated frames ≤ (receiving time - second time) / average drawing and rendering time.

[0015] Exemplarily, the predicted number of insertable frames is an integer.

[0016] According to the first aspect, or any implementation of the first aspect above, the method also includes: when the first application is cold started, obtaining the package name of the first application; determining the application category of the first application based on the package name; when the application category of the first application matches the set application type that supports interpolation, when the drawing and rendering time of the Nth frame is greater than one VSync signal cycle, after the drawing and rendering of the Nth frame is completed, executing the steps of obtaining the number of lost frames, selecting a minimum value from the number of lost frames and the set maximum number of interpolable frames as the number of interpolated frames, and drawing and rendering M frames based on the second MOVE event before the second VSync signal arrives.

[0017] This can speed up the data processing process and reduce unnecessary frame insertion processing.

[0018] According to the first aspect, or any implementation of the first aspect above, the method also includes: determining that the input event acts on a control in the first interface based on the reporting point information corresponding to the input event; when the control acted on is a RecyclerView control or a ListView control, when the drawing and rendering time of the Nth frame is greater than one VSync signal cycle, after the drawing and rendering of the Nth frame is completed, executing the step of obtaining the number of lost frames, selecting a minimum value from the number of lost frames and the set maximum number of insertable frames as the number of insertable frames, and before the second VSync signal arrives, drawing and rendering M frames based on the second MOVE event.

[0019] This can speed up the data processing process and reduce unnecessary frame insertion processing.

[0020] According to the first aspect, or any implementation of the first aspect above, the method further includes: determining the number of layers to be rendered during the drawing and rendering of the Nth frame; when the number of layers is one, and the drawing and rendering time of the Nth frame is greater than one VSync signal cycle, after the drawing and rendering of the Nth frame is completed, executing the steps of obtaining the number of lost frames, selecting a minimum value from the number of lost frames and the set maximum number of insertable frames as the number of insertable frames, and drawing and rendering M frames based on the second MOVE event before the second VSync signal arrives.

[0021] This can speed up the data processing process and reduce unnecessary frame insertion processing.

[0022] According to the first aspect, or any implementation of the first aspect above, the method further includes: when a sliding distance corresponding to a MOVE event obtained by extracting from an input event corresponding to a sliding operation based on timestamps of two adjacent VSync signals is greater than a minimum sliding distance threshold, and when a drawing rendering duration of the Nth frame is greater than one VSync signal cycle, after the drawing rendering of the Nth frame is completed, executing the steps of obtaining the number of lost frames, selecting a minimum value from the number of lost frames and a set maximum number of insertable frames as the number of insertable frames, and drawing and rendering M frames based on the second MOVE event before a second VSync signal arrives.

[0023] This can speed up the data processing process and reduce unnecessary frame insertion processing.

[0024] According to the first aspect, or any implementation of the first aspect above, the method further includes: when the drawing rendering time of the Nth frame is no more than one VSync signal cycle, and the Nth frame is the first frame of the drawing rendering operation, after the drawing rendering of the Nth frame is completed, offsetting the set offset on the basis of the Nth frame to obtain the N+1th frame; before the second VSync signal arrives, drawing and rendering the N+1th frame, and displaying the N+1th frame.

[0025] For example, when the Nth frame is the first frame, such as the frame 1 described below, the N+1th frame is the frame 1' described below.

[0026] Therefore, a frame is pre-inserted for drawing and rendering at the beginning of the sliding operation, and then one more frame is cached in the cache queue, reducing the situation of frameless synthesis caused by subsequent drawing and rendering timeout, and reducing display jamming. For example, if only one frame is lost, a smooth transition can be achieved through the inserted frame without jamming, thereby improving the user experience.

[0027] According to the first aspect, or any implementation of the first aspect above, the method further includes: when an event extracted from an input event corresponding to a sliding operation based on a timestamp of a third VSync signal is a DOWN event, determining that the Nth frame is the first frame of a drawing and rendering operation; wherein the third VSync signal is a VSync signal received before the first VSync signal and is adjacent to the first VSync signal.

[0028] In a second aspect, the present application provides a terminal device. The terminal device includes: a memory and a processor, the memory and the processor being coupled; the memory storing program instructions, which, when executed by the processor, cause the terminal device to execute instructions of the method of the first aspect or any possible implementation of the first aspect.

[0029] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0030] In a third aspect, the present application provides a computer-readable medium for storing a computer program, wherein the computer program includes instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0031] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0032] In a fourth aspect, the present application provides a computer program comprising instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0033] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0034] In a fifth aspect, the present application provides a chip comprising a processing circuit and a transceiver pin. The transceiver pin and the processing circuit communicate with each other via an internal connection path, and the processing circuit executes the method of the first aspect or any possible implementation of the first aspect to control the receive pin to receive a signal and to control the transmit pin to send a signal.

[0035] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] FIG1 is a schematic diagram showing an exemplary hardware structure of a terminal device;

[0037] FIG2 is a schematic diagram showing an exemplary software structure of a terminal device;

[0038] FIG3 is a schematic diagram illustrating an exemplary application scenario;

[0039] FIG4 is a schematic diagram illustrating an exemplary application scenario;

[0040] FIG5 is a schematic diagram illustrating an exemplary data processing flow;

[0041] FIG6 is a schematic diagram showing an exemplary trend of data frames during data processing;

[0042] FIG7 is a schematic diagram showing exemplary changes in interface display content without frame loss;

[0043] FIG8 is a schematic diagram illustrating an exemplary data processing flow when one frame of data is lost;

[0044] FIG9 is a schematic diagram showing exemplary changes in interface display content when a frame of data is lost;

[0045] FIG10 is a schematic diagram illustrating an exemplary data processing flow when multiple frames of data are lost continuously;

[0046] FIG11 is a schematic diagram showing exemplary changes in interface display content when multiple frames of data are lost continuously;

[0047] FIG12 is a schematic diagram illustrating exemplary functional modules involved in a data processing method provided in an embodiment of the present application;

[0048] FIG13 is a timing diagram exemplarily illustrating the interaction process between functional modules involved in the data processing method provided in an embodiment of the present application;

[0049] FIG14 is a schematic diagram illustrating an exemplary data processing flow for performing first frame insertion;

[0050] FIG15 is a schematic diagram illustrating an exemplary data processing flow for performing frame supplementation when one frame is lost;

[0051] FIG16 is a schematic diagram illustrating an exemplary data processing flow for performing first frame insertion and frame supplementation when one frame is lost;

[0052] FIG17 is a schematic diagram illustrating an exemplary data processing flow for performing first frame insertion and frame supplementation when multiple frames are lost continuously;

[0053] FIG18 is a schematic diagram illustrating another data processing flow for exemplarily performing first frame insertion and frame supplementation when multiple frames are lost continuously;

[0054] FIG19 is a flowchart illustrating a data processing method provided by an embodiment of the present application;

[0055] FIG20 is a schematic diagram illustrating specific processing operations included in the drawing, rendering, and synthesis stages;

[0056] FIG21 is a flowchart illustrating another data processing method provided by an embodiment of the present application;

[0057] FIG22 is a schematic diagram illustrating an exemplary first interface of a first application;

[0058] FIG23 is another schematic diagram illustrating an exemplary first interface of a first application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0060] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0061] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0062] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0063] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0064] In order to better understand the technical solution provided by the embodiments of the present application, before describing the technical solution of the embodiments of the present application, the hardware structure of the terminal device (such as a mobile phone, a tablet computer, a touch-screen PC, etc.) to which the embodiments of the present application are applicable is first described in conjunction with the accompanying drawings.

[0065] 1 , the terminal device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0066] Exemplarily, in some implementations, the sensor module 180 may include a pressure sensor, a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc., which are not listed one by one here and are not limited in this application.

[0067] Based on the above-mentioned sensor, when the user operates the display screen 194 during the use of the terminal device, such as single-clicking, double-clicking, sliding, or sliding without letting go of the hand, the current operation can be accurately determined, as well as the location of the operation, the reported point information of the location, etc. Specifically, in the technical solution provided by this application, taking the sliding operation without letting go of the hand as an example, the data processing flow during the operation is specifically described. The specific details of the data processing method generated by the sliding operation without letting go of the hand are detailed below and will not be repeated here.

[0068] It should be noted that the so-called hands-free sliding refers to the behavior in which, within a certain application, the user moves his finger or stylus on the display screen to change the content displayed in the current interface.

[0069] In addition, it should be noted that the processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0070] Furthermore, it should be noted that processor 110 may also include a memory for storing instructions and data. In some implementations, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0071] Continuing with FIG1 , illustratively, the charge management module 140 is configured to receive charging input from a charger. The power management module 141 is configured to connect the battery 142, the charge management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charge management module 140 and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 may also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance).

[0072] Continuing with FIG1 , illustratively, the wireless communication function of the terminal device 100 can be implemented by antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor. It should be noted that antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals.

[0073] Continuing to refer to Figure 1, illustratively, the mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some implementations, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some implementations, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0074] In addition, it should be noted that the modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some implementations, the modem processor may be an independent device. In other implementations, the modem processor may be independent of the processor 110 and be provided in the same device as the mobile communication module 150 or other functional modules.

[0075] Continuing to refer to Figure 1, illustratively, the wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., applied to the terminal device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0076] It should also be noted that the terminal device 100 implements its display functionality through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0077] 1 , illustratively, the display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some implementations, the terminal device 100 may include one or N display screens 194 , where N is a positive integer greater than one.

[0078] In addition, it should be noted that the terminal device 100 can realize the shooting function through the ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0079] Continuing with FIG. 1 , illustratively, the external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage. For example, files such as music and videos can be stored on the external memory card.

[0080] Continuing to refer to Figure 1, illustratively, the internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the terminal device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications of the terminal device and the data processing method provided in this application by running the instructions stored in the internal memory 121 and / or the instructions stored in the memory provided in the processor 110.

[0081] In addition, it should be noted that the terminal device 100 can implement audio functions such as music playback and recording through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0082] Continuing with FIG1 , illustratively, buttons 190 include a power button, a volume button, and the like. Buttons 190 may be mechanical buttons or touch buttons. Terminal device 100 may receive key inputs and generate key signal inputs related to user settings and function control of terminal device 100. Motor 191 may generate vibration prompts. Indicator 192 may be an indicator light that can be used to indicate charging status, battery level changes, messages, missed calls, notifications, and the like.

[0083] This concludes the introduction to the hardware structure of the terminal device 100. It should be understood that the terminal device 100 shown in FIG1 is merely an example. In a specific implementation, the terminal device 100 may have more or fewer components than shown in the figure, may combine two or more components, or may have a different component configuration. The various components shown in FIG1 may be implemented in hardware, including one or more signal processing and / or application-specific integrated circuits, software, or a combination of hardware and software.

[0084] 1 , the software structure of the terminal device 100 is described below. Before describing the software structure of the terminal device 100, the architecture that the software system of the terminal device 100 can adopt is first described.

[0085] Specifically, in actual applications, the software system of the terminal device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture.

[0086] In addition, it is understandable that the software systems currently used by mainstream terminal devices include but are not limited to Windows systems, Android systems, and iOS systems. For ease of explanation, the embodiment of the present application takes the layered architecture Android system as an example to exemplify the software structure of the terminal device 100.

[0087] In addition, the data processing solutions provided in the subsequent embodiments of this application are also applicable to other systems in specific implementations.

[0088] See Figure 2, which is a software structure block diagram of the terminal device 100 according to an embodiment of the present application.

[0089] As shown in Figure 2, the layered architecture of the terminal device 100 divides the software into several layers, each with clear roles and divisions of labor. Layers communicate with each other via software interfaces. In some implementations, the Android system is divided into four layers: from top to bottom, the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0090] The application layer may include a series of application packages. As shown in FIG2 , the application package may include applications such as gallery, settings, text messages, mailboxes, browsers, and videos. These applications are not listed here one by one and are not limited in this application.

[0091] The application framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. In some implementations, these programming interfaces and programming frameworks can be described as functions / services / frameworks, etc.

[0092] In addition, it should be noted that the application framework layer can be divided into the system service framework layer (commonly known as the Framework layer, this layer is implemented based on the Java language) and the local service framework layer (commonly known as the native layer, this layer is implemented based on the C or C++ language) based on the programming interface it manages and the implementation language of the programming framework.

[0093] Continuing to refer to FIG3 , illustratively, the Framework layer may include a window manager, an input manager, a content provider, a view system, an activity manager, etc., which are not listed here one by one and are not limited in this application.

[0094] The window manager is used to manage window programs. It can obtain the display screen size, determine whether there is a status bar, lock the screen, etc. Specifically, in the technical solution provided by this application, it is also used to determine the focus window and obtain information such as the focus window layer and the corresponding application package name.

[0095] The InputManagerService is used to manage input device programs. For example, the InputManager can determine input operations such as mouse clicks, keyboard inputs, and hands-free swipes. Specifically, in the technical solution provided by this application, it is primarily used to determine hands-free swipe operations.

[0096] The content provider is used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc., which are not listed here and are not limited by this application.

[0097] The view system includes visual controls, such as controls for displaying text and images. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon can include a view for displaying text and a view for displaying images.

[0098] The Activity Manager is used to manage the lifecycle of each application and the navigation back function. It is responsible for creating the Android main thread and maintaining the lifecycle of each application.

[0099] Continuing to refer to FIG3 , illustratively, the native layer may include an input reader, an input dispatcher, an image synthesis system, etc., which are not listed here one by one and are not limited in this application.

[0100] It should be noted that when a terminal device is in use, the display screen will call the event listening port / function (EventHub) every few milliseconds according to a set period. If it detects a user operation on the display screen, such as a swipe without releasing the user's hand (which can be considered an event), it will report the event to the input reader (InputReader). In other words, the InputReader is used to read events from the EventHub or directly receive events reported by the EventHub.

[0101] In addition, after receiving the event, InputReader will send the event to the input dispatcher (InputDispatcher).

[0102] Among them, after receiving the event sent by InputReader, InputDispatcher will distribute the event to the corresponding application (hereinafter referred to as: application) through the input manager.

[0103] The image synthesis system, namely the surface flinger (hereinafter referred to as the SF thread), is used to control image synthesis and generate a vertical synchronization (VSync) signal.

[0104] As you can understand, the SF thread includes: a composition thread, a VSync thread, and a cache thread (e.g., a queue buffer). The composition thread is used to be awakened by the VSync signal for composition; the VSync thread is used to generate the next VSync signal based on the VSync signal request; the cache thread has one or more cache queues, each of which is used to store cached data of its corresponding application, such as image data rendered from a data frame.

[0105] The Android Runtime includes the core library and the virtual machine. The Android Runtime is responsible for scheduling and management of the Android system.

[0106] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0107] The system library can include multiple functional modules, such as image rendering library, image synthesis library, input processing library, media library, etc.

[0108] Among them, the image rendering library is used for rendering two-dimensional or three-dimensional images; the image synthesis library is used for synthesizing two-dimensional or three-dimensional images.

[0109] In a possible implementation, the application renders the image using the image rendering library, and then sends the rendered image to the SF thread's cache queue. Whenever a VSync signal arrives, the SF thread sequentially retrieves a frame of image to be synthesized from the cache queue and then synthesizes the image using the image synthesis library.

[0110] Among them, the input processing library is a library for processing input devices, which can realize mouse, keyboard and hand-free sliding input processing, etc.

[0111] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0112] In addition, it is understood that the kernel layer in the Android system is the layer between hardware and software. The kernel layer includes at least sensor drivers, display drivers, audio drivers, Bluetooth drivers, GPS drivers, etc.

[0113] This concludes the introduction to the software structure of the terminal device 100. It should be understood that the layers in the software structure shown in FIG2 and the components contained in each layer do not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, and this application does not impose any limitations thereon.

[0114] To facilitate understanding, some descriptions of concepts related to the embodiments of the present application are given as examples for reference.

[0115] 1. Frame: This refers to the smallest unit of a single image displayed on an interface. A frame can be understood as a still image. Displaying multiple frames in rapid succession can create the illusion of motion.

[0116] 2. Frame rate: This refers to the number of frames that refresh an image in one second. It can also be understood as the number of times per second that the graphics processor in the terminal device refreshes the screen. A higher frame rate produces smoother and more realistic animation. The more frames per second, the smoother the displayed motion.

[0117] It should be noted that before the interface displays a frame, it usually needs to go through processes such as drawing, rendering, and synthesis.

[0118] 3. Frame drawing: refers to the image drawing of the display interface. The display interface can be composed of one or more views. Each view can be drawn by the visual control of the view system. Each view is composed of subviews. A subview corresponds to a small widget in the view. For example, a subview corresponds to a symbol in the image view.

[0119] 4. Frame rendering: This is to shade the drawn view or add 3D effects, etc. For example, 3D effects can be lighting effects, shadow effects, and texture effects.

[0120] 5. Frame synthesis: It is the process of synthesizing multiple or more rendered views into a display interface.

[0121] 6. Vertical synchronization (VSync) signal: a signal used to control the start of frame rendering, synthesis, and display processes.

[0122] It should be noted that in order to ensure the smoothness of display and avoid display freezes and other phenomena, terminal devices generally display based on VSync signals to synchronize the processes of image drawing, rendering, synthesis and screen refresh display.

[0123] It is understood that the VSync signal is a periodic signal, and the VSync signal period can be set according to the screen refresh rate. For example, when the screen refresh rate is 60Hz, the VSync signal period can be 16.6ms, that is, the terminal device generates a control signal every 16.6ms to trigger the VSync signal period. For another example, when the screen refresh rate is 90Hz, the VSync signal period can be 11.1ms, that is, the terminal device generates a control signal every 11.1ms to trigger the VSync signal period.

[0124] In addition, it should be noted that VSync signals include software VSync (VSync-APP or VSync-SF) and hardware VSync (VSync-HW). Among them, VSync-APP is used to trigger the drawing and rendering process; VSync-SF is used to trigger the synthesis process. The hardware VSync signal (VSync-HW) is used to trigger the screen display refresh process. Under normal circumstances, software VSync and hardware VSync maintain cycle synchronization. Taking the change between 60Hz and 120Hz as an example, if VSync-HW switches from 60Hz to 120Hz, VSync-APP and VSync-SF will change synchronously, switching from 60Hz to 120Hz.

[0125] The following describes the workflow of the software and hardware of the terminal device 100 in combination with the scenario of application startup or interface switching within an application.

[0126] For example, when the touch sensor in the touch panel receives a touch operation, the kernel layer processes the touch operation into an original input event (including touch coordinates, touch force, timestamp of the touch operation, and other information). EventHub monitors the original input event and stores it in the kernel layer. InputReader reads the original input event from the kernel layer through the input processing library and hands it over to InputDispatcher, which packages the original input event, such as encapsulating it into a set data format, and determines the application corresponding to the original input event, and then reports the original input event to the input manager. The input manager parses the information of the original input event (including: operation type and reporting point location, etc.) and determines the focus application based on the current focus, that is, the application corresponding to the original input event, and sends the parsed information to the focus application.

[0127] It is understood that the focus can be the finger touch point in a touch operation or the click location in a stylus or mouse click operation. The focus application is the application running in the foreground of the terminal device or the application corresponding to the touch location in a touch operation. The focus application determines the control corresponding to the original input event based on the information of the parsed original input event (for example, the reported point location).

[0128] For example, if the touch operation is a continuous swipe operation, and the control corresponding to this continuous swipe operation is a list control in the WeChat application, the WeChat application uses the view system to call the image rendering library in the system library to draw and render the image. The WeChat application sends the rendered image to the cache queue of the SF thread. The image synthesis library in the system library synthesizes the rendered image into the WeChat interface. The SF thread uses the display driver in the kernel layer to display the corresponding WeChat application interface on the screen.

[0129] The following describes the application scenarios to which the technical solution provided by this application is applicable, in conjunction with the accompanying drawings. For example, (1) and (2) in FIG3 and (1) and (2) in FIG4 are schematic diagrams of interfaces of a terminal device under different applications in a possible implementation.

[0130] Exemplarily, the terminal device can receive an upward sliding operation or a downward sliding operation by the user in the direction of the arrow in the interface of the social application shown in (1) in 3, or in the setting-related interface shown in (2) in 3, or in the document interface shown in (1) in FIG4, or in the product browsing interface shown in (2) in FIG4. When the terminal device receives the sliding operation made by the user, the terminal device performs frame drawing, rendering, synthesis and other processes based on the sliding operation, and finally displays the synthesized picture, so that the content displayed on the display screen changes as the user's finger moves on the display screen, that is, follows the movement of the finger to update the content of the current interface.

[0131] It is understood that the interface display of a terminal device generally requires processes such as drawing, rendering, and compositing. For example, the interface drawing process of the terminal device may include processes such as background drawing, subview drawing, and scroll bar drawing. The interface compositing process of the terminal device may include processes such as vertex processing and pixel processing.

[0132] However, if the terminal device times out when drawing the rendered image (frame) (for example, exceeding one VSync cycle), the image frames to be drawn and rendered in other cycles occupied by it will be lost, which will cause the terminal device to experience abnormal phenomena such as display freeze and jump.

[0133] The following describes the data processing flow involved in the interface display of the terminal device, the routing of data frames during data processing, and the data processing flow and interface content changes when frame loss occurs, in conjunction with Figures 5 to 11.

[0134] Exemplarily, after the focus application obtains the original input event dispatched by InputDispatcher through InputManagerService, it will distribute the original input event. The distribution process is, for example, that the drawing rendering thread (such as Choreographer) used for interface drawing in the UI thread corresponding to the focus application initiates a VSync signal request, and draws one frame after receiving the VSync signal. Based on this premise, referring to Figure 5, exemplarily, t1 to t7 are the time points when the Choreographer in the UI thread receives the VSync signal. Among them, the VSync signal is sent to the Choreographer by the SF thread, specifically the VSync thread in the SF thread, in each VSync signal cycle after receiving the VSync signal request. Since the VSync signal has a fixed period, the duration between any two time points from t1 to t7 is fixed, that is, one VSync signal cycle.

[0135] Continuing to refer to FIG. 5 , illustratively, after receiving the VSync signal at time point t1 , Choreographer renders frame 1 .

[0136] As you can understand, in actual applications, each frame of rendered image data will be sent by the Choreographer in the UI thread, such as the image data for frame M in Figure 6, to the cache thread of the SF thread. The cache thread will then cache the frame of image data in the cache queue corresponding to the focus application, allowing the compositing thread in the SF thread to retrieve the frame from the cache queue for compositing at the corresponding compositing time point. Here, N = M-1, where M is an integer greater than 1.

[0137] Based on the flow of frame image data shown in Figure 6, and continuing with Figure 5, illustratively, after frame 1 is rendered, Choreographer sends the rendered frame 1 image data to the SF thread's cache thread, which caches the frame 1 image data into the cache queue corresponding to the focus application. Accordingly, at the corresponding synthesis time point, the synthesis thread retrieves the frame 1 image data from the cache queue for synthesis. After synthesis is complete, the terminal device can display the content corresponding to frame 1 on the display screen by calling the kernel layer's display driver.

[0138] It should be noted that in Figure 5, and subsequent Figures 8, 10, 13, 15 and 16, the frame data appearing in the SF thread, such as frame 1, frame 2, frame 3, etc., all refer to the rendered image data corresponding to the frame data; the frame data appearing in the display driver, such as frame 1, frame 2, frame 3, etc., all refer to the synthesized pictures corresponding to the frame data.

[0139] Continuing to refer to FIG. 5 , illustratively, the drawing, rendering, synthesis, and display of frames 2 to 7 received at time points t2 to t7 are similar to those of frame 1 and are not described again here.

[0140] In addition, it should be noted that each frame has a certain delay from rendering to synthesis, and another delay from synthesis to display. These two delays can be the same or different. Therefore, as shown in Figure 5, at time t1, frame 1 begins to be rendered. During the rendering cycle of frame 1, such as the time from t1 to t2 in Figure 5, the SF thread synthesizes the frame image data before frame 1 (not shown in Figure 5), and the display driver drives the display screen to display the content before the original input event is triggered.

[0141] For example, after the rendering of frame 1 is completed, Choreographer will send a request for the next VSync signal to the VSync thread. Even if there is still time available in the current cycle, it will not render the next frame. Instead, it will start rendering frame 2 after receiving the next VSync signal, such as at time point t2 in Figure 5. In this embodiment, the lag time from rendering to synthesis of each frame is one VSync information cycle. At time point t2, when Choreographe starts rendering frame 2, the synthesis thread will retrieve the image data of frame 1 from the cache queue and merge the image data of frame 1.

[0142] Exemplarily, for the duration of the lag from synthesis to display, still taking one VSync information cycle as an example, since the synthesis of the frame 1 image data is performed at time point t2, the display driver drives the display screen to display the content corresponding to frame 1 starting at time t3, and displays the content corresponding to frame 1 during the entire time from t3 to t4, until the next VSync signal cycle takes out the newly synthesized content for display.

[0143] It is understandable that each frame in Figure 5 is rendered within the corresponding cycle. In this case, the content displayed on the display screen will refresh the interface according to a fixed cycle as the finger / stylus moves. Taking the focus application as a social application as an example, as shown in Figure 7 (1), when the user slides from point P1 to point P4 along the arrow direction of the current interface without letting go of the hand, the touch sensor in the touch panel receives the operation, and the kernel layer processes the operation as a raw input event, and its EventHub listens to the raw input event and stores it in the kernel layer. InputReader reads the raw input event from the kernel layer through the input processing library and hands it over to InputDispatcher. InputDispatcher packages the raw input event, such as encapsulating it into a set data format, and determines the application corresponding to the raw input event, and then reports the raw input event to the input manager. The input manager parses the information of the raw input event (including: operation type and reporting point location, etc.) and determines the focus application based on the current focus, and sends the parsed information to the focus application. The Choreographer in the focus application will request a VSync signal. After receiving the VSync signal, it will start rendering the frame data corresponding to the current reported point. The synthesis thread will synthesize it and the display driver will send it to the display. When the finger slides to P2, after the frame data of P2 is rendered and synthesized, the interface content shown in Figure 7 (2) will be displayed.

[0144] It is understandable that since each frame goes from drawing and rendering to synthesis and then to display, taking the lag time of each stage as one VSync signal cycle as an example, the first frame has only two VSync signal cycles from drawing and rendering to final display. Subsequently, under the data processing process shown in Figure 5, the interface content will be automatically switched in each VSync signal cycle, that is, when sliding from point P1 to point P2, it will immediately switch to the content of (2) in Figure 7 without the user's perception. When sliding from point P2 to point P3, it will immediately switch to the content of (3) in Figure 7 without the user's perception. When sliding from point P3 to point P4, it will immediately switch to the content of (4) in Figure 7 without the user's perception. That is, during the entire sliding process, the change of interface content is relatively smooth, without any freeze or jump.

[0145] However, this is only true under ideal conditions, where the rendering process does not time out or drop frames. However, as user demands for applications continue to increase, the application interface displays more and more content. This results in an increasing number of controls and icons to be rendered for each frame of data during the scrolling process, which in turn makes it impossible to complete the rendering of a frame of data within a single cycle.

[0146] As shown in Figure 8, at time point t3, Choreographer starts drawing and rendering frame 3. For example, in one possible case, if Choreographer has not completed the drawing of frame 3 within the period from t3 to t4, that is, the drawing and rendering of frame 3 times out, when a new VSysn signal is received at time point t4, frame 3 has not been drawn and rendered yet, so frame 4 is not drawn and rendered. At the same time, since frame 3 starting at time point t3 has not been drawn and rendered yet, there is no frame 3 image data in the cache queue at time point t4. Therefore, the synthesis thread cannot obtain frame 3 image data from the cache queue at time point t4.

[0147] Continuing to refer to Figure 8, illustratively, in the interval from t4 to t5, Choreographer completes the rendering of frame 3. Since it has not received a new VSync signal, that is, the VSync signal corresponding to time point t5, Choreographer enters a short blank period. After receiving the VSync signal at time point t5, it starts rendering frame 5. At the same time, since the rendering of frame 3 is completed before time point t5, the image data of frame 3 is cached in the cache queue. Therefore, at time point t5, the synthesis thread can take out the image data of frame 3 from the cache queue for merging, so that the display driver can display the content corresponding to frame 3 synthesized by the synthesis thread within the lag time of one VSync signal cycle, that is, time point t6 in Figure 8.

[0148] Continuing to refer to Figure 8, for example, since the content corresponding to frame 2 begins to be displayed at time point t4, the display driver does not obtain a new frame synthesized by the synthesis thread until time point t6, specifically the content corresponding to frame 3. Therefore, the content corresponding to frame 2 will continue to be displayed from time point t2 to time point t6.

[0149] It can be understood that the drawing, rendering, synthesis, and display logic of each frame in Figure 8 are similar to those shown in Figure 5. For parts not fully described in this embodiment, please refer to the part of the embodiment shown in Figure 5, and no further details will be given here.

[0150] Regarding the data processing flow shown in FIG8 , specifically in actual applications, we still take the case of a user sliding the finger in the interface of a social application and changing the interface content as an example. As shown in FIG9 (1), the user still slides the finger from point P1 to point P4 along the arrow direction. Among them, the drawing and rendering of the frames corresponding to points P1 and P2 have not timed out. Therefore, when the finger is at point P1, the corresponding interface content is displayed normally as shown in FIG9 (1). When the finger is at point P2, the corresponding interface content is also displayed normally as shown in FIG9 (2). However, when sliding from point P2 to point P3, the drawing and rendering of the frame corresponding to point P3 timed out, resulting in no change in the corresponding interface content when the finger is at point P3, as shown in FIG9 (3), which is still P2. The content corresponding to the point is drawn and rendered, and then when the finger slides from point P3 to point P4, the frame corresponding to point P3 completes the drawing and rendering. When the finger slides to point P4, the interface will be updated to the content of the frame corresponding to point P3, that is, the interface content shown in (4) in Figure 9 is the same as the normal drawing and rendering, and the interface content of point P3 shown in (3) in Figure 7 without timeout. In the case of a frame loss, the interface will not change for at least two VSync signal cycles. If the number of frame losses is greater, the duration will be longer, causing the user to perceive the lag, affecting the user experience.

[0151] Continuing to refer to Figure 8, for example, since frame 4 is lost, at time point t7, the image displayed on the display screen will directly jump from the content of frame 3 to the content of frame 5. When only one frame is lost, the interface jump may not be obvious and the user will not perceive it. However, if several frames are lost continuously, the interface jump will be very obvious, and the user will perceive the interface jump, affecting the user experience.

[0152] As shown in Figure 10, at time point t3, Choreographer starts rendering frame 3. During the period from t3 to t4, Choreographer has not completed the rendering of frame 3. When a new VSysn signal is received at time point t4, frame 3 has not been rendered yet, so frame 4 is not rendered. At the same time, since frame 3 starting at time point t3 has not been rendered yet, there is no frame 3 image data in the cache queue at time point t4. Therefore, the synthesis thread cannot obtain frame 3 image data from the cache queue at time point t4.

[0153] Continuing to refer to Figure 10, for example, during the period from t4 to t5, the Choreographer has not completed the drawing of frame 3. When a new VSysn signal is received at time t5, frame 3 has not been drawn and rendered yet, so frame 5 is not drawn and rendered. At the same time, since frame 3 starting at time t3 has not been drawn and rendered yet, there is no frame 3 image data in the cache queue at time t5. Therefore, at time t6, the synthesis thread cannot obtain frame 3 image data from the cache queue.

[0154] Continuing to refer to Figure 10, for example, during the period from t5 to t6, the Choreographer has not completed the drawing of frame 3. When a new VSysn signal is received at time t6, frame 3 has not been drawn and rendered yet, so frame 6 is not drawn and rendered. At the same time, since frame 3 starting at time t3 has not been drawn and rendered yet, there is no frame 3 image data in the cache queue at time t6. Therefore, the synthesis thread cannot obtain frame 3 image data from the cache queue at time t7.

[0155] Continuing to refer to FIG10 , illustratively, in the interval from t6 to t7, Choreographer completes the drawing and rendering of frame 3. Since no new VSync signal, i.e., the VSync signal corresponding to time point t7, has been received, Choreographer enters a short blank period. After receiving the VSync signal at time point t7, Choreographer starts drawing and rendering frame 7. At the same time, since the drawing and rendering of frame 3 is completed before time point t7, the image data of frame 3 is cached in the cache queue. Therefore, at time point t7, the synthesis thread can take out the image data of frame 3 from the cache queue for merging, so that the display driver can display the content corresponding to frame 3 synthesized by the synthesis thread within the lag time of one VSync signal cycle, i.e., time point t8 in FIG10 .

[0156] Continuing to refer to Figure 10, for example, since the content corresponding to frame 2 starts to be displayed at time t4, the display driver does not obtain a new frame synthesized by the synthesis thread until time t8, specifically the content corresponding to frame 3. Therefore, the content corresponding to frame 2 will be continuously displayed from time t2 to time t8. If more frames are lost during the drawing and rendering phase, the same picture will be displayed on the display screen for a longer time when the user keeps sliding the screen. Moreover, since multiple frames are lost, such as frames 4, 5, and 6, after the content corresponding to frame 3 is displayed, in the next display cycle, the display driver will directly drive the display screen to display the content of frame 7. The lost frames in the middle will cause the picture to jump, affecting the user experience.

[0157] It can be understood that the drawing, rendering, synthesis, and display logic of each frame in Figure 10 are similar to those shown in Figure 5. For parts not fully described in this embodiment, please refer to the part of the embodiment shown in Figure 5, and no further details will be given here.

[0158] Regarding the data processing flow shown in FIG10, specifically in actual application, the interface content changes when the user slides the finger in the interface of a social application. As shown in FIG11 (1), the user still slides the finger from point P1 to point P4 along the arrow direction. The drawing and rendering of the frame corresponding to point P1 has not timed out. Therefore, the interface content corresponding to the finger at point P1 is displayed normally as shown in FIG11 (1). If there is a displacement point between point P1 and point P2, and the drawing and rendering of the frame corresponding to this point has timed out, assuming that the drawing and rendering of the frame corresponding to this point continues until point P4, then when the finger moves from this point to point P2, and from point P2 to point P3, the interface content does not change. As shown in FIG11 (2) and (3), the content of the frame drawing, rendering, synthesis, and display corresponding to point P1 is the same, that is, the interface content of FIG11 (1), (2) and (3) is the same. After the finger moves to point P4, since the frames corresponding to points P2 and P3 have been lost, after the frame corresponding to point P4 is rendered and merged, in the next cycle, the picture displayed on the display screen will directly jump from (3) in Figure 11, that is, the content of the frame corresponding to point P1, to (4) in Figure 11, the content of the frame corresponding to point P4, so that the user can perceive the jump, affecting the user experience.

[0159] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0160] Ultimately, within an application, when a user moves on the display screen with their fingers or a stylus (fingers will be used as an example later), as the content of the interface increases, more and more controls and icons need to be drawn and rendered in each frame of data. The drawing and rendering process may time out and drop frames, which will cause the picture displayed on the display screen to freeze and jump.

[0161] In view of this, the present application provides a data processing method to solve the above-mentioned stuttering and jumping phenomena caused by drawing rendering timeout and frame loss.

[0162] The data processing method provided by this application is described in detail below through specific embodiments. The following embodiments can be combined with each other or implemented independently, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0163] FIG12 is a schematic diagram of the functional modules involved in a data processing method provided in an embodiment of the present application, and the locations of these functional modules.

[0164] As shown in Figure 12, during the hands-free sliding operation performed in any application, based on the data processing method provided in the embodiment of the present application, the functional modules involved may include an application located in the application layer, which is referred to as the focus application in this embodiment, a window manager and input manager located in the Framework layer, SF threads, input readers, and input dispatchers located in the native layer, a hardware synthesizer located in the hardware abstraction layer (HAL layer), a display driver and sensor driver located in the kernel layer, and hardware such as display screens and sensors.

[0165] In addition, from the above description of the software architecture of the terminal device described in FIG. 2 , it can be seen that the implementation of the data processing method will also involve the image rendering library, image synthesis library, and input processing library in the system library.

[0166] Continuing to refer to FIG12 , for example, for each focus application, when drawing and rendering the frame data, it specifically involves the input thread (ViewRootImplement, which can be expressed as ViewRootImpl) implemented / called in the UI thread of the application, the drawing and rendering thread (Choreographer), and the interpolation module (FirstInputManager).

[0167] Continuing to refer to FIG. 12 , illustratively, the SF thread includes a VSync thread, a cache thread, and a composition thread.

[0168] For example, when performing a hand-free sliding operation, the VSync thread will send a VSync signal to the Choreographer in each VSync signal cycle, so that the Choreographer starts rendering a frame after receiving the VSync signal.

[0169] Specifically, for each frame of drawing and rendering, Choreographer needs to first read the original input event (hereinafter referred to as: Input event) from ViewRootImpl and process it according to the timestamp corresponding to the VSync signal (during the sliding operation without letting go of the hand, each reporting point will correspond to a timestamp). Specifically, the acquisition of Input events is, for example, sending a request for obtaining reporting point information (carrying the timestamp corresponding to the above-mentioned VSync signal) to InputManagerService through ViewRootImpl, and then InputManagerService transmits the reporting point information acquisition request to InputDispatcher, and InputDispatcher transmits the reporting point information acquisition request to InputReader, and finally InputReader obtains the Input event corresponding to the timestamp carried in the reporting point information acquisition request from the kernel layer, and obtains and saves the reporting point information in the kernel layer when the Input event occurs.

[0170] Accordingly, after obtaining the corresponding reporting information, it is passed up step by step to the InputDispatcher, which calls the callBack callback registered by the focus application in it, and returns the reporting information to the ViewRootImpl of the focus application through the InputManagerService. The point information is then passed from ViewRootImpl to Choreographer to complete the acquisition of a frame of data.

[0171] In addition, it should be noted that in this embodiment, in order to solve the problem of screen freeze and jump on the display screen caused by timeout and frame loss during the drawing and rendering process of frame data, Choreographer will pass the currently read Input event to FirstInputManager before obtaining the frame data and starting drawing and rendering.

[0172] It is understandable that the input events for the sliding operation without lifting the hand usually include a DOWN event (finger drops), a MOVE event (finger moves), and an UP event (finger lifts).

[0173] For example, after Choreographer passes the currently read Input event to FirstInputManager, FirstInputManager will detect the serial number of the continuously input Input event changes. When FirstInputManager detects that the DOWN event changes to the serial number of the MOVE event and the content displayed on the display starts to move for the first time, a new event Event can be generated and the new Event can be notified to Choreographer, so that after drawing and rendering the currently received frame data, Choreographer continues to draw and render one frame in the same cycle, that is, after the first frame is drawn and rendered, a frame is inserted into the cycle where the first frame is located.

[0174] In addition, when FirstInputManage detects frame loss, it also calculates the number of frames to be supplemented and the specific frames that need to be supplemented, and then re-triggers the above frame acquisition process to obtain the lost frames for supplementation.

[0175] Continuing to refer to Figure 12, for example, after the Choreographer completes the drawing and rendering of each frame, it will pass the drawn and rendered frame image data to the cache thread, and then the cache thread will cache it into the corresponding cache queue. When it reaches the processing cycle of the synthesis thread, the synthesis thread will take out the frame image data from the cache queue for synthesis, and finally transmit the synthesized content to the hardware synthesizer, which calls the display driver to drive the display screen for display, thereby realizing the update of the picture displayed on the display screen.

[0176] Regarding the specific interaction process of the functional modules shown in FIG12 during the data processing process, the following is described with reference to the accompanying drawings.

[0177] FIG13 is a timing diagram of the interaction of functional modules involved in the implementation of a data processing method provided in an embodiment of the present application.

[0178] Referring to FIG13 , some functional modules are omitted for example, such as the display screen, sensor, input reader, input dispatcher, input manager, window manager, and hardware compositor in FIG12 . FIG13 directly uses the example of a user performing a hands-free swipe (Input event) in a focused application. The sensor detects the swipe operation, the input reader obtains the Input event, obtains the corresponding reporting point information for the Input event, and reports the Input event to the input dispatcher. The input dispatcher obtains the focused application corresponding to the Input event from the window manager, and then, based on the package name and other information corresponding to the focused application, dispatches the Input event to the input thread of the focused application via the input manager for recording and management through the CallBack callback registered by the focused application in the input dispatcher. Furthermore, in response to the user's hands-free swipe operation, the focused application initiates a request for a VSync-APP signal to the VSync thread. This example specifically illustrates the interaction between the input thread, rendering thread, interpolation module, cache thread, compositor thread, and display driver during the hands-free swipe process.

[0179] In addition, before explaining in conjunction with Figure 13, the VSync-APP signal and VSync-SF signal involved in Figure 13 are first explained. Specifically, in this embodiment, the time point of receiving the VSync-APP signal is the time point that triggers the drawing and rendering thread to start drawing and rendering, such as each time point from t1 to t7 in Figure 5, and the VSync-SF signal is the time point that triggers the synthesis thread to start synthesis. In this embodiment, taking any VSync-APP signal that lags behind by one VSync signal cycle as an example, the synthesis processing time point for the drawing and rendering image corresponding to frame 1 can be the time point t2 in Figure 5.

[0180] In addition, it should be noted that the request for the VSync-APP signal is initiated by the thread corresponding to the focus application, such as the application main thread (UI thread) to the VSync thread (the UI application is not shown in Figure 13, and the request is directly taken from the drawing rendering thread in the UI thread as an example), and the request for the VSync-SF signal can be initiated by the cache thread to the VSync thread, for example.

[0181] In addition, it should be noted that since the sending period corresponding to the VSync-APP signal and the VSync-SF signal is fixed, it is related to the frame rate, and the direct lag time between the VSync-APP signal and the VSync-SF signal is also fixed, such as the one VSync signal period mentioned above, the VSync thread generates the VSync-APP signal and the VSync-SF signal according to the VSync signal period corresponding to the current frame rate, and sends the generated VSync-APP signal to the drawing and rendering thread and the generated VSync-SF signal to the synthesis thread in each VSync signal period.

[0182] S101 , when receiving VSync-APP signal 1 , the drawing rendering thread reads recorded Input event 1 from the input thread according to the timestamp of VSync-APP signal 1 .

[0183] As can be seen from the above description, the Input events corresponding to the sliding operation can include DOWN events, MOVE events, and UP events. For example, during a DOWN event, the rendering thread sends a request for a VSync-SF signal to the VSync thread and responds. At this time, when VSync-APP signal 1 is received, if the finger moves, the read Input event 1 may be a MOVE event, and if the finger is lifted, it is an UP event. This embodiment takes the case where Input event 1 is a MOVE event as an example.

[0184] As you can understand, during a hands-free scrolling session, the input dispatcher continuously dispatches received input events to the input thread via the input manager. The reporting frequency depends on the display's sampling rate. For example, if the sampling rate is 120Hz, the display driver dispatches data to the input thread approximately every 8ms, storing it in the event queue for consumption. The rendering thread consumes the input events in the event queue based on the VSync signal timestamp.

[0185] S102 , after receiving Input event 1 (MOVE event), the drawing rendering thread starts processing Input event 1 (MOVE event).

[0186] Specifically, the drawing and rendering thread will obtain the frame data required for drawing and rendering through the input thread, input manager, input dispatcher and input reader through the method of obtaining point information mentioned above, such as frame 1 mentioned above, and then draw and render frame 1, and cache the drawn and rendered image 1 into the cache queue allocated by the cache thread for the focus application.

[0187] It is understandable that in actual applications, the drawing and rendering thread needs to first initiate a request to the cache thread to allocate a cache queue. In response to the request, the cache thread allocates the corresponding cache queue for the focus application and informs the drawing and rendering thread of the address information of the allocated cache queue. In this way, after the drawing and rendering thread completes the drawing and rendering of each frame, it can cache the rendered image into the cache queue according to the address information.

[0188] Continuing to refer to FIG13 , for example, in this embodiment, when the rendering thread draws the rendered image 1 according to the Input event 1, it also sends the Input event 1 to the interpolation module, which determines whether to perform an interpolation operation or a frame supplementation operation.

[0189] S103, the interpolation module detects the types of two adjacent Input events (in this embodiment, Input event 1 is a MOVE event type, and is adjacent to Input event 1, and the last received Input event 0 is a DOWN event as an example), determines that the two adjacent Input events are a DOWN event and a MOVE event, respectively, and generates an interpolation event.

[0190] Specifically, when two adjacent Input events are DOWN and MOVE, it indicates that the first frame is currently being rendered. That is, the rendering operation performed after receiving VSync-APP signal 1 is performed on the first frame data. To prevent the display screen from freezing when a frame is subsequently lost, the data processing method provided in this embodiment uses the interpolation module to generate a new event, namely, an interpolation event, and notifies the rendering thread to process the interpolation event, that is, to perform another rendering operation within the current processing cycle.

[0191] Understandably, since the cycle for sending a new VSync-APP signal, such as VSync-APP signal 2, has not yet arrived, the drawing and rendering thread has not received VSync-APP signal 2 and will not obtain Input Event 2 corresponding to VSync-APP signal 2 from the input thread, that is, the frame data for drawing and rendering. To ensure this drawing and rendering operation, the interpolation module can, when generating an interpolation event, for example, add a corresponding offset to the position of the previous Input event to generate a new Input event. This new Input event is the interpolation event mentioned above, so that the drawing and rendering thread can perform another drawing and rendering based on the new Input event (interpolation event).

[0192] In addition, it should be noted that in order to speed up the data processing process and reduce unnecessary interpolation processing, whether the interpolation module should generate an interpolation event during the rendering of each frame, as well as the frame filling operation mentioned below, can be determined according to the following judgment logic.

[0193] For example, in some implementations, when each application is cold started, the package name of the application can be obtained first, and then the application category of the application can be determined based on the package name. Finally, it is determined whether the application category of the application matches the application type that supports first-frame interpolation or lost frame compensation, such as whether it is in the set whitelist. When it is determined that the application category of the application matches the application type in the set whitelist, when the Input event corresponding to the focus application is received and drawing rendering is performed, the drawing rendering thread sends the currently processed Input event to the interpolation module, so that the interpolation module executes the operations in steps S103, S108, S112, etc.

[0194] It is understandable that the cold start mentioned above refers to when the application is launched and there is no background process for the application. At this time, the system will recreate a new process and assign it to the application. This startup method is called a cold start (the application process does not exist in the background). During a cold start, the system will recreate a new process and assign it to the application. Therefore, it will first create and initialize the application class (Application class), then create and initialize the corresponding MainActivity class that provides the interface for user interaction (including a series of measurement, layout, and drawing). Finally, the application interface, such as the default home screen displayed after launching the application, will be displayed on the screen.

[0195] In addition, regarding the aforementioned operation of obtaining the package name of the application when each application is cold started, for example, the security protection program (which may be called IAware APK) in the terminal device specifically obtains the package name.

[0196] In addition, regarding the application types set in the whitelist mentioned above, they can be, for example, news, instant messaging, shopping, browsers, videos, short videos, forums and other application types. They are not listed one by one here, and this embodiment does not limit this.

[0197] In addition, the method of determining the application type according to the package name may be based on the classification of application types according to the package names of different applications in an application market installed in the terminal device for providing downloadable applications.

[0198] Therefore, when the application type of each application is determined to match the application type in the set whitelist through the above method, the enable identifier corresponding to the interpolation module can be set to "True". In this way, when drawing and rendering each frame, when the drawing and rendering thread recognizes that the enable identifier corresponding to the interpolation module is "True", it can transmit the relevant information of the currently processed Input event to the interpolation module, thereby triggering the interpolation module to perform judgment and processing. Conversely, if the enable identifier corresponding to the interpolation module is recognized as "False", the interpolation module will not participate in the entire sliding operation, that is, the first frame interpolation operation will not be performed, and no frame supplementation will be performed after the frame is lost.

[0199] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0200] Furthermore, in some other implementations, when the drawing and rendering thread recognizes that the enable identifier corresponding to the interpolation module is "True," it can also determine, based on the reporting point information corresponding to the currently processed Input event, which control in the interface displayed on the display screen the Input event targets. Accordingly, when the current Input event targets a RecyclerView or ListView control, the drawing and rendering thread will only send the currently processed Input event to the interpolation module when it performs drawing and rendering based on the Input event, causing the interpolation module to execute the operations in steps S103, S108, and S112.

[0201] It is understandable that since the RecyclerView control and ListView control usually involve drawing and rendering of a large amount of content, the above judgment can further determine whether the interpolation module needs to perform corresponding processing when drawing and rendering each frame, so as to avoid the involvement of the interpolation module in scenarios where drawing timeout and frame loss do not usually occur, thereby reducing the occupancy of terminal device resources and improving data processing speed.

[0202] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0203] Furthermore, in some implementations, when the current Input event targets a RecyclerView or ListView control, the rendering thread can determine the number of layers to render for each frame. When there is only one layer, i.e., in a single-layer rendering scenario, the rendering thread sends information about the currently processed Input event to the interpolation module for processing. For multi-layer scenarios, to avoid delays in subsequent frames caused by the interpolation module and increased processing difficulty, information about the current Input event is not sent to the interpolation module when rendering multiple layers.

[0204] For example, the multi-layer scenario mentioned above is a situation where other controls are covered on a RecyclerView control or a ListView control. Specifically in actual applications, for example, when using a shopping app to view product details, the live content is also displayed in a small window on the details interface.

[0205] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0206] In addition, in some other implementations, when the rendering thread is currently rendering a single layer of the interface, it can also be determined whether the sliding distance corresponding to two adjacent Input events is greater than the minimum sliding distance threshold TouchSlop corresponding to the sliding operation.

[0207] Accordingly, when the sliding distance corresponding to two adjacent Input events is greater than the minimum sliding distance threshold, when rendering is performed according to the Input, the rendering thread sends the currently processed Input event to the interpolation module, so that the interpolation module executes the operations in steps S103, S108, S112, etc.

[0208] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0209] S104 , the rendering thread renders an interpolation frame image according to the interpolation frame event, and caches the rendered interpolation frame image into a cache queue corresponding to the focus application.

[0210] It is understandable that the drawing and rendering operations performed based on the frame data extracted from the interpolation event are similar to the drawing and rendering operations performed based on the frame data in the normally received Input event. The specific implementation details can be found above and will not be repeated here.

[0211] Taking frame 1 in Figure 14 as an example, if VSync-APP signal 1 is received at time t1 and VSync-APP signal 2 is received between t1 and t2, the rendering thread renders frame 1. After rendering frame 1, it also renders frame 1' in the interpolated frame event during this period. That is, during the period from t1 to t2, two rendered images are obtained: the image corresponding to frame 1 and the image corresponding to frame 1'.

[0212] Continuing to refer to FIG. 13 , illustratively, after the inserted frame is rendered and the interpolated frame image is cached in the cache queue, if the current cycle has not ended, the drawing and rendering thread enters a short blank period. If the current cycle ends, that is, the VSync-APP signal 2 is received, step S106 is executed.

[0213] Continuing to refer to FIG13 , for example, if the time point at which the synthesis thread starts synthesis lags behind the time point at which the Input event 1 is drawn and rendered by one VSync signal cycle, the VSync thread will also send a VSync-SF signal 1 to the synthesis thread when sending a VSync-APP signal 2 to the drawing and rendering thread.

[0214] Correspondingly, after receiving the VSync-SF signal 1, the composition thread takes out a frame of rendered image from the cache queue and executes step S105.

[0215] For example, after receiving the VSync-SF signal, the synthesis thread will take out a frame of rendered image that was previously put into the cache queue from the cache queue for merging. If there is no frame in the queue, no processing will be done.

[0216] S105 , after receiving the VSync-SF signal 1 , the synthesis thread takes out the rendered image 1 corresponding to the Input event 1 previously put into the cache queue from the cache queue, and performs synthesis processing on the rendered image 1 .

[0217] Continuing to refer to FIG13 , illustratively, after obtaining the synthesized image 1 , the synthesis thread will send the synthesized image 1 to the display driver, which will then display it, that is, drive the display screen to display the content of FIG1 , thereby updating the screen.

[0218] It can be understood from the above description that when the synthesis thread sends the synthesized image 1 to the display driver, it specifically goes through the hardware synthesizer located at the HAL layer shown in Figure 12, and then the hardware synthesizer transmits the synthesized image 1 to the display driver.

[0219] As shown in Figure 14, at time t2, when VSync-SF signal 1 is received, and between t2 and t3, that is, before receiving VSync-SF signal 2, the rendering thread merges the image corresponding to frame 1. Thus, after obtaining the composite image corresponding to frame 1, after a certain delay (as mentioned above, when the synthesis to display also takes one VSync signal cycle), at time t3, the display driver receives the image corresponding to frame 1 synthesized by the synthesis thread and can drive the display to display the image corresponding to frame 1.

[0220] S106 , after receiving the VSync-APP signal 2 , the drawing rendering thread reads the recorded Input event 2 from the input thread according to the timestamp of the VSync-APP signal 2 (taking the Input event as a MOVE event as an example).

[0221] S107 , after receiving Input event 2 (MOVE event), the drawing and rendering thread starts to process Input event 2 (MOVE event), that is, draws and renders image 2 according to Input event 2 .

[0222] Regarding the rendering of Input event 2, it is similar to that of Input event 1. Please refer to the above for details and will not be repeated here.

[0223] Continuing with Figure 13 , for example, after rendering the frame data in Input Event 2, the rendering thread caches Image 2 in the cache queue. While rendering Image 2 based on Input Event 2, the rendering thread also sends Input Event 2 to the interpolation module, which determines whether to perform a frame insertion operation.

[0224] It should be noted that the frame insertion operation / frame insertion event mentioned in this embodiment is specifically a frame insertion operation performed on the first frame. The frame inserted in this operation is part of the content of the first frame, and the drawing and rendering thread does not need to read a new Input event from the input thread. The frame supplement operation / frame supplement event is specifically an operation performed when a frame is lost due to a drawing and rendering timeout of a certain frame during finger movement. The frame inserted in this operation is a lost frame, that is, the drawing and rendering thread needs to read an Input event from the input thread. In actual applications, both operations can be called frame insertion operations, and this embodiment does not limit this name.

[0225] S108: The interpolation module detects the types of two adjacent Input events (Input event 1 and Input event 2), determines that the two adjacent Input events are MOVE events and MOVE events, and that the drawing and rendering of Image 2 has not timed out, and does not trigger a frame interpolation operation.

[0226] In this case, that is, normal drawing and rendering, there is no drawing and rendering timeout or frame loss. After the drawing and rendering thread completes the drawing and rendering of Input event 2, if the current cycle has not ended, the drawing and rendering thread enters a short blank period. If the current cycle ends, that is, VSync-APP signal 3 is received, step S110 is executed.

[0227] Continuing with FIG13 , for example, during the drawing and rendering process of the drawing and rendering thread, the VSync thread will continue to send the corresponding VSync-SF signal to the compositing thread during each VSync signal cycle. Accordingly, upon receiving a new VSync-SF signal, the compositing thread will also retrieve the rendered image at the head of the cache queue for compositing, and then send the composited image to the display driver for display. For example, upon receiving VSync-SF signal 2, the compositing thread will execute step S109, and upon receiving VSync-SF signal 3, the compositing thread will execute step S115, and so on.

[0228] S109 , after receiving the VSync-SF signal 2 , the synthesis thread takes out the rendered interpolated frame image at the head of the queue from the cache queue, and performs synthesis processing on the rendered interpolated frame image.

[0229] As shown in FIG14 , the composition thread receives the VSync-SF signal 2 at time t3 , at which time the composition thread takes out the rendered image corresponding to frame 1 ′ from the cache queue for composition processing.

[0230] Accordingly, after a delay of one VSync signal cycle, that is, at time point t4, the display driver receives the synthesized image corresponding to frame 1' synthesized by the synthesis thread, and then drives the display screen to display the synthesized image corresponding to frame 1'.

[0231] S110 , after receiving the VSync-APP signal 3 , the drawing rendering thread reads the recorded Input event 3 from the input thread according to the timestamp of the VSync-APP signal 3 (taking the Input event as a MOVE event as an example).

[0232] S111 , after receiving Input event 3 (MOVE event), the drawing and rendering thread starts to process Input event 3 (MOVE event), that is, draws the rendering image 3 according to Input event 3 .

[0233] Regarding the rendering of Input event 3, it is similar to that of Input event 1. Please refer to the above for details and will not be repeated here.

[0234] Continuing with Figure 13 , for example, after rendering the frame data in Input event 3, the rendering thread caches Image 3 in the cache queue. While rendering Image 3 based on Input event 3, the rendering thread also sends Input event 3 to the interpolation module, which determines whether to perform a frame insertion operation.

[0235] S112, the interpolation module detects the types of two adjacent Input events (Input event 3 and Input event 4), determines that the two adjacent Input events are MOVE events and MOVE events, and the drawing and rendering of image 3 has timed out (when the drawing and rendering thread receives VSync-APP signal 4, it is still drawing and rendering image 3), triggering the frame insertion operation.

[0236] For example, in some implementations, for a scenario in which a VSync-APP signal 4 is received during the drawing and rendering process of an image 3, and a VSync-APP signal 5 is not received after the drawing and rendering of the image 3 is completed, the interpolation operation may be completed before the VSync thread sends the VSync-APP signal 5, that is, after the drawing and rendering thread receives the interpolation instruction sent by the interpolation module, it directly reads the Input event 4 from the event thread according to the timestamp of the received VSync-APP signal 4, executes step S113, and then executes step S114.

[0237] In addition, it should be noted that the Input events that appear in this embodiment, such as Input Event 0, Input Event 1, Input Event 2, Input Event 3, Input Event 4, etc., are generated in chronological order. For example, when the sliding operation process without removing the hand includes DOWN, MOVE1, MOVE2, MOVE3, and UP, as the user's finger slides on the display screen, the Input Event 0, Input Event 1, Input Event 2, Input Event 3, and Input Event 4 generated in sequence are DOWN event, MOVE1 event, MOVE2 event, MOVE3 event, and UP event, respectively.

[0238] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0239] S113, the drawing rendering thread reads the recorded Input event 4 from the input thread according to the timestamp of the VSync-APP signal 4 (the Input event is a MOVE event, such as the MOVE3 event mentioned above as an example).

[0240] S114 , after receiving Input event 4 (MOVE event), the drawing and rendering thread starts to process Input event 4 (MOVE event), that is, draws rendering image 4 according to Input event 4 .

[0241] Regarding the rendering of Input event 4, it is similar to that of Input event 1. Please refer to the above for details and will not be repeated here.

[0242] Correspondingly, after the drawing rendering thread obtains the image 4 through drawing rendering, it will also cache the drawn rendered image 4 in the cache queue, so that when the synthesis thread receives the VSync-SF signal 4, it can take out the drawn rendered image 4 from the cache queue for synthesis processing.

[0243] As shown in Figure 15, when the drawing and rendering thread has not completed the drawing and rendering of frame 3 between t3 and t4, that is, before receiving VSync-APP signal 4, after receiving VSync-APP signal 4, the drawing and rendering thread does not read Input event 4 from the event thread according to the timestamp of VSync-APP signal 4, and continues to draw and render frame 3. When the drawing and rendering thread completes the drawing and rendering of frame 3 before time point t5, the interpolation module determines that a frame supplement is required. When the remaining time meets the duration required for drawing and rendering frame 4, the drawing and rendering thread can read frame 4 corresponding to time point t4 from the input thread and then perform drawing and rendering.

[0244] Continuing to refer to Figure 15, for example, since frame 3 has not been drawn and rendered at time point t4, there is no image of frame 3 after drawing and rendering in the cache queue. The synthesis thread does not perform the merge operation at time point t4, and reads the image of frame 3 after drawing and rendering from the cache queue at time point t5 before starting the synthesis process. Accordingly, since the drawing and rendering thread makes up for the lost frame 4, at each subsequent synthesis time point, that is, after receiving the VSync-SF signal, if there is no drawing and rendering timeout or frame loss, the synthesis thread can take out the image corresponding to frame 4, the image corresponding to frame 5, the image corresponding to frame 6, the image corresponding to frame 7, and the image corresponding to frame 8 from the cache queue in sequence.

[0245] Accordingly, the display driver only fails to properly retrieve the composite image from the compositing thread at time t5. It will continue to display the content corresponding to frame 2 during the VSync signal cycle from t5 to Figure 6. Subsequently, it can normally update and display the content corresponding to the next frame data with each VSync signal cycle. In this way, because the content corresponding to frame 2 is only displayed for an extra VSync signal cycle, which is only a few milliseconds, the user will not notice any noticeable lag. Furthermore, the content corresponding to the sequentially changing frame data will be updated normally with each subsequent VSync signal cycle, so the content displayed on the screen will not jump.

[0246] In addition, it should be noted that if in Figure 15, frame 1' is inserted after frame 1 (first frame) is drawn and rendered at time point t1, then when frame 3 is subsequently drawn and rendered, if only one VSync information cycle is exceeded and the lost frame 4 is supplemented, the final processing process of merging the threads and the display driver will be as shown in Figure 16, that is, the content displayed on the display will be updated in sequence for each VSync signal cycle, and since there is no lost frame, the content displayed on the display will not jump.

[0247] S115 , after receiving the VSync-SF signal 3 , the synthesis thread takes out the rendered image 2 at the head of the cache queue and performs synthesis processing on the rendered image 2 .

[0248] As shown in FIG14 , the composition thread receives the VSync-SF signal 3 at time t4 . At this time, the composition thread takes out the rendered image corresponding to frame 2 from the cache queue for composition processing.

[0249] Accordingly, after a delay of one VSync signal cycle, that is, at time point t5, the display driver receives the synthesized image corresponding to frame 2 synthesized by the synthesis thread, and then drives the display screen to display the synthesized image corresponding to frame 2.

[0250] Furthermore, it should be noted that, based on the content shown in Figure 13, if the rendering thread has not yet completed rendering of Image 3 upon receiving VSync-APP signal 5, as shown in Figure 17, the rendering thread is still rendering frame 3 at time t5. It only completes rendering frame 3 at time t6, i.e., upon receiving VSync-APP signal 6, because the rendering of frame 3 takes up three VSync signal cycles, t3 to t6, it is not rendered during the time from t4 to t5, when frame 4 should have been rendered, resulting in loss of frame 4. Similarly, it is not rendered during the time from t5 to t6, when frame 5 should have been rendered, resulting in loss of frame 5. In other words, as shown in Figure 17, frames 4 and 5 were lost while rendering frame 3. To avoid noticeable lag, the rendering thread can first compensate for the loss of frame 6 upon receiving VSync-APP signal 6 at time t6.

[0251] For example, when performing frame interpolation, you can first determine whether the drawing and rendering time of frame 4 + frame 6 can be completed within the time period from t6 to t7. If so, you can first read the frame 4 corresponding to the t4 time point from the input event in the above manner for drawing and rendering. After the drawing and rendering of frame 4 is completed, the frame 6 originally corresponding to the timestamp of the VSync-APP signal 6 is normally read for drawing and rendering. After interpolating the frames in this way, only one frame (frame 5) is lost in the end, and the content corresponding to the new frame data can be updated and displayed normally in each VSync signal cycle. In this way, since the content corresponding to frame 2 is only displayed for one more VSync signal cycle, which is only a few milliseconds, the user will not feel obvious lag. Since one frame (frame 4) is interpolated, the display screen will not directly jump from the content of frame 3 to the content of frame 6. The content of frame 4 will be too large in the middle, so the jump will not be very obvious.

[0252] For example, if the rendering of frame 4+frame 5+frame 6 can be completed within the time period from t6 to t7, that is, two frames can be supplemented, then the frame 4 corresponding to the time point t4 can be read from the input event for rendering in the above manner. After frame 4 is rendered, the frame 5 corresponding to the time point t5 can be read from the input event for rendering. After the rendering of frame 5 is completed, the frame 6 originally corresponding to the timestamp of the VSync-APP signal 6 is normally read for rendering. In this way, when the VSync-APP signal 7 is received at the time point t7, before the rendering of frame 7 is performed, while ensuring that frame 6 is not lost, the lost frames 4 and 5 can be supplemented. Finally, the processing process of merging the thread and the display driver will be as shown in Figure 18, that is, the content displayed on the display will be updated in sequence for each VSync signal cycle, and since there is no lost frame, the content displayed on the display will not jump.

[0253] In this way, a frame is pre-inserted for drawing and rendering at the beginning of the sliding operation, and then one more frame is cached in the cache queue, reducing the situation of frameless synthesis caused by subsequent drawing and rendering timeout, and reducing display jamming. For example, if only one frame is lost, a smooth transition can be achieved through the inserted frame without jamming, thereby improving the user experience.

[0254] In addition, when the drawing rendering times out, the lost frames are made up by one or more frames through frame filling, reducing the frame loss caused by missing the VSync signal due to the drawing rendering timeout, so that the content displayed on the display can change smoothly, increase the display smoothness, reduce jumps, and further improve the user experience.

[0255] In order to better understand the data processing method provided in the embodiment of the present application, the specific processing logic between the drawing rendering thread and the interpolation module in Figure 13, as well as the specific implementation details of determining whether to insert frames, supplement frames, and how many frames to supplement are described in detail below.

[0256] FIG19 is a flow chart of a data processing method provided in an embodiment of the present application. As shown in FIG19 , the method specifically includes:

[0257] S201, displaying a first interface of a first application.

[0258] For example, the first application is the focus application currently running in the foreground. The first interface is the interface currently displayed by the first application. For example, if the first application is an instant messaging application, the first interface can be the circle of friends interface shown in Figure 7 above. The content displayed in the first interface is, for example, the screen shown in (1) in Figure 7.

[0259] S202: In response to a sliding operation on the first interface, obtain an input event corresponding to the sliding operation.

[0260] Exemplarily, the input events corresponding to the sliding operation may include a DOWN event, a MOVE event, and an UP event.

[0261] In addition, regarding the acquisition of input events, please refer to the above description of the interaction between the display screen, sensor, sensor driver, EventHub, input reader, input dispatcher, window manager, input manager and input thread, which will not be repeated here.

[0262] S203: Obtain a first VSync signal, and render the Nth frame based on the first MOVE event.

[0263] Specifically, the first MOVE event is extracted from the input event corresponding to the sliding operation based on the timestamp of the first VSync signal, and the Nth frame is an image data frame corresponding to the first MOVE event that needs to be drawn and rendered.

[0264] It is understandable that the first VSync signal mentioned in this embodiment can be, for example, any time point mentioned above, such as the VSync signal received at any time point from t1 to t7 in Figure 18. Correspondingly, the first MOVE event is the event corresponding to the time point of the VSync signal, which may be a DOWN event, or a MOVE event, or an UP event in actual applications. The interpolation (supplementation) scenario targeted by the data processing method provided in this embodiment is specifically targeted at MOVE events. Therefore, here, the event extracted from the input event corresponding to the sliding operation based on the timestamp of the first VSync signal is taken as an example of a MOVE event. In order to facilitate the distinction between the MOVE time corresponding to the timestamp of the VSync signal received at other times, the MOVE event extracted from the input event corresponding to the sliding operation based on the timestamp of the first VSync signal is referred to as the first MOVE event.

[0265] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0266] In addition, it should be noted that in the process of rendering, compositing, and displaying, each step has a corresponding VSync signal. Among them, the first VSync signal mentioned above, as well as the subsequent second and third VSync signals, all refer to the vertical synchronization signals used to trigger the rendering process. The vertical synchronization signal used to trigger the rendering process is specifically the VSync-APP signal mentioned above.

[0267] Correspondingly, the vertical synchronization signal for triggering the synthesis process is specifically the VSync-SF signal mentioned above, and the vertical synchronization signal for triggering the display screen refresh process is specifically the VSync-HW signal mentioned above.

[0268] In addition, it should be noted that in actual applications, the interval between two adjacent VSync-APP signals is a first time length, the interval between two adjacent VSync-SF signals is a second time length, and the interval between two adjacent VSync-HW signals is a third time length. That is, every first time length, the first application, such as the application main thread (UI thread) mentioned above or directly the drawing and rendering thread, will receive a VSync-APP signal, such as receiving VSync-APP signal 1 at time point t1 mentioned above, receiving VSync-APP signal 2 at time point t2, and so on.

[0269] Correspondingly, every second duration, the synthesis thread for performing synthesis processing will receive a VSync-SF signal; every third duration, the display driver will receive a VSync-HW signal.

[0270] In addition, it should be noted that in some implementations, the first duration, the second duration, and the third duration may be the same duration, such as the VSync signal period mentioned above. This is related to the frame rate of the terminal device when processing data. For example, for a 60Hz scene, the VSync signal period is 16.6ms, that is, every 16.6ms the VSync thread will generate a corresponding VSync-APP signal to the drawing and rendering thread, and similarly every 16.6ms the VSync thread will generate a corresponding VSync-SF signal to the compositing thread, and similarly every 16.6ms the VSync thread will generate a corresponding VSync-HW signal to the display driver.

[0271] In addition, from the above description, it can be seen that the time for the display driver to obtain the content synthesized by the synthesis thread and drive the display screen to display it lags behind the time when the synthesis thread starts to perform the synthesis operation, and the time when the synthesis thread starts to perform the synthesis operation lags behind the time when the drawing and rendering thread starts to perform the drawing and rendering operation, that is, the time when the VSync thread sends the VSync-HW signal lags behind the time when the VSync-SF signal is sent, and the time when the VSync-SF signal is sent lags behind the time when the VSync-APP signal is sent. For example, taking the sending time of the VSync-HW signal as one VSync signal cycle later than the sending time of the VSync-SF signal, and the sending time of the VSync-SF signal as one VSync signal cycle later than the sending time of the VSync-APP signal, the VSync thread can send VSync-APP signal 1 to the drawing and rendering thread at time point t1 above, send VSync-APP signal 2 to the drawing and rendering thread at time point t2, and send VSync-SF signal 1 to the synthesis thread at the same time, and send VSync-APP signal 3 to the drawing and rendering thread at time point t3, send VSync-SF signal 2 to the synthesis thread, and send VSync-HW signal 1 to the display driver.

[0272] Furthermore, it should be noted that in some implementations, the first duration, the second duration, and the third duration can be different, specifically satisfying the following: third duration > second duration > first duration. This ensures that the next stage can begin only after the previous stage is completed, ensuring that the next stage can obtain the data processed by the previous stage. For example, the display driver can obtain the content synthesized by the synthesis thread, and the synthesis thread can obtain the content rendered by the drawing and rendering thread.

[0273] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0274] In addition, the premise for triggering the VSync thread to generate VSync-APP signal, VSync-SF signal, and VSync-HW signal according to the VSync signal cycle and sending them according to the VSync signal cycle is, as mentioned above, to initiate a request to the VSync thread when the input event is a DOWN event.

[0275] For example, when the event extracted from the input event corresponding to the sliding operation based on the timestamp of the VSync-APP signal is a DOWN event, the first application, such as the drawing and rendering thread described above, sends a first message (a request for a VSync-APP signal) to the VSync thread in the SF thread. In this way, the VSync thread can be generated according to the first duration (such as a VSync signal cycle) and send a corresponding VSync-APP signal to the drawing and rendering thread of the first application in each VSync signal cycle.

[0276] Accordingly, when the event extracted from the input event corresponding to the sliding operation based on the timestamp of the VSync-APP signal is a DOWN event, the cache thread in the SF thread can send a second message (a request for a VSync-SF signal) to the VSync thread. In this way, the VSync thread can generate a VSync-SF signal according to the second duration (e.g., a VSync signal cycle) and send the corresponding VSync-SF signal to the composition thread in each VSync signal cycle.

[0277] Accordingly, when the event extracted from the input event corresponding to the sliding operation based on the timestamp of the VSync-APP signal is a DOWN event, the display driver can send a third message (a request for a VSync-HW signal) to the VSync thread. In this way, the VSync thread can be generated according to the third duration (such as one VSync signal cycle) and send a corresponding VSync-HW signal to the display driver in each VSync signal cycle.

[0278] The relationship between each VSync-APP signal's timestamp, reporting point, and input event can be found above. Based on this relationship, it is possible to determine which reporting point corresponds to the input data, and thus obtain the image frame corresponding to that reporting point. Regarding image frame acquisition, please refer to the above section and will not be repeated here.

[0279] As shown in Figure 20, drawing and rendering can be divided into the drawing phase and the rendering phase. The drawing phase specifically includes: input (for passing input events to the corresponding object for processing), animation (for calculating the position of each frame of animation), measurement (for obtaining and maintaining the size of each view (View) and view group (ViewGrop) according to the control property settings in the XML layout file and code), layout (for determining the display position of the control based on the information obtained by the strategy), and drawing (after the user determines the display position of the control, all layers in the application window are drawn on the canvas (canvas) to construct drawing instructions).

[0280] Continuing with FIG. 20 , the rendering phase illustratively includes synchronization (for synchronizing drawing instructions from the CPU), rendering (for adjusting brightness, contrast, saturation, etc. of the drawn layer), and storing in a cache queue (for storing the rendered execution results in the cache queue). The specific drawing and rendering steps are not further described in this embodiment.

[0281] S204 , when the rendering time of the Nth frame is longer than one VSync signal cycle, after the rendering of the Nth frame is completed, obtaining the number of lost frames and displaying the Nth frame.

[0282] With reference to FIG18 , for example, when the Nth frame is frame 3 in FIG18 , since the drawing and rendering time of frame 3 occupies 3 VSync signal cycles, after the drawing and rendering of frame 3 is completed, the interpolation module mentioned above will obtain the number of lost frames. At the same time, the SF thread will synthesize the content of frame 3 after receiving the VSync-SF signal, and the display driver will display the content of frame 3 synthesized by the SF thread after receiving the VSync-HW signal.

[0283] S205 : Select a minimum value from the number of lost frames and the set maximum number of insertable frames as the number M of insertable frames.

[0284] Regarding the implementation of obtaining the number of lost frames given in step S204 and finally determining the number of inserted frames M in step S205, for example, it can be:

[0285] (1) Determine the first time Tbegin for starting rendering and the second time Tend for ending rendering of the Nth frame.

[0286] Tbegin is the time when the rendering thread calls the doFrame interface (the interface used to start rendering), such as the time point when each VSync-APP signal is received above, such as t1, t2, t3, etc. Tend is the actual time point when rendering is completed.

[0287] (2) Calculate the number of dropped frames count based on Tbegin, Tend, and the set rendering duration corresponding to the Nth frame (such as a VSync signal cycle, hereinafter referred to as VSync).

[0288] Exemplarily, in one implementation, count=floor[(Tend−Tbegin) / VSync]−1.

[0289] It should be noted that the set drawing rendering duration refers to the maximum drawing rendering duration corresponding to each frame of data under ideal conditions, such as one VSync signal cycle.

[0290] (3) Select a minimum value from the number of lost frames count and the set maximum number of interpolated frames as the number of frames to be inserted after the current drawing and rendering is completed, that is, the number of interpolated frames M.

[0291] For example, in order to avoid the impact of frame insertion on subsequent data processing, the maximum number of insertable frames can be set to 2.

[0292] That is, M=min(count, 2).

[0293] For example, if the calculated current frame is lost, that is, count=1, then the number of frames to be inserted after the current drawing and rendering is completed is M=1; if the calculated count is greater than 2, then the number of frames to be inserted after the current drawing and rendering is completed is M=2.

[0294] Furthermore, in actual applications, the actual remaining time available for frame insertion may be insufficient. To avoid an inappropriate M determined by the above method, it is also possible to estimate the number of frames that can be inserted in the remaining time, and then select a minimum value between the predicted number of insertable frames and the M determined by the above method as the number of frames to be inserted after the current drawing and rendering is completed. The specific implementation method can be as follows:

[0295] (4) According to the sending cycle of the VSync-APP signal, the receiving time TnextVsyn of the next VSync-APP signal is determined.

[0296] For example, taking the sending period of the VSync-APP signal as one VSync signal period, such as the duration between any two time points (t1 and t2) mentioned above as an example, as shown in FIG15 , when the rendering of frame 3 starts at time point t3 and times out, resulting in the loss of frame 4, the receiving time TnextVsyn of the next VSync-APP signal can be time point t5 in FIG15 .

[0297] (5) According to the actual rendering time of the multiple frames of image data that have completed rendering, determine the average rendering time (T average ) of each frame in the N frames that have completed rendering.

[0298] For example, the rendering time set for each frame is one VSync signal cycle, such as 16.6ms. In actual operations, the time required to complete the rendering of frames 1, 2, and 3 is 4.6ms, 5.6ms, and 16.8ms respectively. Based on the actual rendering time of these three frames, the calculated T average = (4.6ms + 5.6ms + 16.8ms) / 3 = 9ms.

[0299] (6) Calculate the predicted number of insertable frames countAllow based on TnextVsyn, Tend and Taverage.

[0300] Exemplarily, in one implementation, countAllow≤(TnextVsyn-Tend) / Tavg.

[0301] Taking TnextVsyn-Tend=16.3ms as an example, countAllow≤1.45, and rounding it up shows that countAllow=1.

[0302] (7), M = min(M (obtained in the above step (3)), countAllow), that is, a minimum value is selected from countAllow, count and the set maximum number of insertable frames (such as 2) as the number of frames to be inserted after the current drawing and rendering is completed.

[0303] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0304] In addition, regarding the above-mentioned determination of the relationship between the image data frame to be inserted and the lost image data frame, it specifically refers to which frame of image data is lost, and then according to the time point at which the lost image data frame should theoretically be drawn and rendered (the timestamp of the VSync-APP signal received at this time point), the corresponding input event is found from the input thread, and then the frame in the input event is used as the frame to be inserted, and it is drawn and rendered.

[0305] The timestamp corresponding to the frame to be inserted can satisfy Tend - (Tend – Tbegin) % VSync. For example, if VSync = 10ms, Tend = 45ms, and Tbegin = 30ms, the final calculated timestamp corresponding to the frame to be inserted is 40ms. Still using Figure 15 as an example, if frame 3 is rendered and frame 4 is lost due to a rendering timeout, the timestamp corresponding to the frame to be inserted calculated using this method is 40ms, which theoretically corresponds to frame 4 at time t4.

[0306] It should be understood that the above description is only an example for better understanding the technical solution of this embodiment and is not intended to be the sole limitation of this embodiment. For the scenario of losing multiple frames, the same can be handled as above and will not be repeated here.

[0307] S206 , before the second VSync signal arrives, render M frames based on the second MOVE event, and display the M frames.

[0308] It should be noted that the second VSync signal is the first VSync signal received after the Nth frame is rendered, and the second MOVE event is extracted from the input event corresponding to the sliding operation based on the timestamps of all or part of the VSync signals received during the Nth frame rendering process.

[0309] 18 , for example, still taking the Nth frame as frame 3, frame 3 is rendered before time point t6, and the first VSync signal received after frame 3 is rendered is the VSync signal received at time point t6.

[0310] Correspondingly, the second MOVE event is obtained by extracting the timestamps of all or part of the VSync signals received during the rendering process of frame 3, such as from t2 to t6, from the input event corresponding to the sliding operation.

[0311] Still using the example shown in Figure 18, for example, in a scenario where a second MOVE event is extracted from the input event corresponding to a slide operation using the timestamps of all VSync signals received during the rendering process of frame N, the second MOVE event specifically comprises a MOVE event extracted from the input event corresponding to the slide operation using the timestamp of the VSync signal received at time t4, and a MOVE event extracted from the input event corresponding to the slide operation using the timestamp of the VSync signal received at time t5. Accordingly, the M frames ultimately inserted after frame N (frame 3) are frames 4 and 5.

[0312] The processing of drawing, rendering, and displaying the inserted M frames is similar to that of the Nth frame, and will not be described in detail here.

[0313] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0314] Therefore, the data processing method provided in this embodiment, when the drawing rendering times out, makes up for the lost frames by one or more frames through frame filling, thereby reducing the frame loss caused by missing the VSync signal due to the drawing rendering timeout, so that the content displayed on the display screen can change smoothly, increase the smoothness of the display, reduce jumps, and further improve the user experience.

[0315] FIG21 is a flow chart of a data processing method provided in an embodiment of the present application. As shown in FIG21 , the method specifically includes:

[0316] S301, displaying a first interface of a first application, wherein the first interface displays a first screen, the first screen including first content and second content, the first content being displayed in a first area of ​​the first interface, and the second content being displayed in a second area of ​​the first interface.

[0317] For example, the first screen is shown in FIG22 , where the first content is, for example, content related to friend A displayed in the first area shown in FIG22 , and the second content is, for example, content related to friend B displayed in the second area shown in FIG22 .

[0318] S302, when receiving a sliding operation of the user on the first interface, obtaining an input event corresponding to each reported point during the sliding operation, where the input event is any one of a DOWN event, a MOVE event, and an UP event.

[0319] Regarding the acquisition of input events, please refer to the above description of the interaction between the display screen, sensor, sensor driver, EventHub, input reader, input dispatcher, window manager, input manager and input thread, which will not be repeated here.

[0320] S303 : upon receiving the vertical synchronization signal that triggers the drawing and rendering process, determining an input event corresponding to the timestamp of the vertical synchronization signal that triggers the drawing and rendering process.

[0321] For detailed descriptions of the vertical synchronization signal (VSync-APP signal) that triggers the drawing and rendering process, as well as the vertical synchronization signal (VSync-SF signal) that triggers the synthesis process and the vertical synchronization signal (VSync-HW signal) that triggers the display refresh process that appear below, please refer to the above embodiments and will not be repeated here.

[0322] S304: Obtain the first image frame in the input event corresponding to the timestamp, render the first image frame, and determine the interpolation strategy according to the rendering time of two adjacent input events and the current input event during the rendering process.

[0323] The specific implementation details of drawing and rendering are detailed above and will not be repeated here.

[0324] In addition, regarding the judgment and processing of whether the interpolation module needs to intervene in the drawing and rendering process to perform interpolation (first frame interpolation, frame supplementation after frame loss (interpolation)), please refer to the description after S103 in the above embodiment for details, which will not be repeated here.

[0325] In addition, as can be seen from the above description, based on the data processing method provided by this application, the case of frame insertion during the drawing and rendering phase can be divided into the following: inserting a frame of image data after the first frame at the beginning of the sliding operation; and inserting one or more frames of image data when the drawing and rendering timeout occurs during the sliding operation, resulting in frame loss. Therefore, in this embodiment, the frame insertion strategy is determined based on the drawing and rendering time of two adjacent input events and the current input event, which may include:

[0326] When two adjacent input events are a DOWN event and a MOVE event respectively, an offset is added to the previous Input event to generate a new Input event as an insertion event, thereby obtaining an insertion strategy.

[0327] or,

[0328] When two adjacent input events are different MOVE events and the rendering time of the current input event exceeds the set rendering time, the number of frames to be inserted after the current rendering is completed and the relationship between the image data frames to be inserted and the lost image data frames are determined. Frame insertion instructions are generated based on the number of frames and the relationship to obtain a frame insertion strategy.

[0329] It can be understood that the interpolation strategy mentioned in this embodiment is an implementation scenario of the interpolation event, that is, the solution of inserting a frame after the first frame of image data frame mentioned above. The specific implementation details can be found above and will not be repeated here.

[0330] In addition, it should be understood that the interpolation strategy mentioned in this embodiment is an implementation scenario for generating frame supplement instructions, that is, the scheme of determining the number of interpolation frames after frame loss and then inserting (supplementing) frames as mentioned above. The specific implementation details can be found above and will not be repeated here.

[0331] In other words, the determination of the interpolation strategy mentioned in this embodiment is essentially the processing logic of the interpolation module determining whether first-frame interpolation is required, or determining whether the current rendering timeout has caused frame loss, thereby requiring frame supplementation. In actual applications, when a device implements the data processing method provided by this application, the program instructions may not generate the interpolation strategy step, but instead directly trigger the first-frame interpolation process when the first-frame interpolation conditions are met, and directly determine the number of frames that can be inserted and then interpolate frames when the frame supplementation conditions for lost frames are met.

[0332] For example, when the interpolation strategy includes an interpolation event, it can be determined that the interpolation strategy indicates that interpolation is required. In this case, after the drawing and rendering thread completes drawing and rendering the first image frame (such as frame 1 above) to obtain the first drawing and rendering image, before the next VSync-APP signal, such as VSync-APP signal 2 received at time t2 above, arrives, the drawing and rendering thread uses a frame of image data in the interpolation event (such as the partial data in frame 1 above, frame 1') as the second image frame, inserts the second image frame after frame N, and draws and renders the second image frame.

[0333] For example, when the interpolation strategy includes a frame insertion instruction, it can be determined that the interpolation strategy indicates that interpolation is required. In this case, after the drawing and rendering thread completes the drawing and rendering of the first image frame (such as frame 3 above) to obtain the first drawing and rendering image, before the next VSync-APP signal, such as VSync-APP signal 5 received at time point t5 above, arrives, the drawing and rendering thread determines the timestamp corresponding to the input event of each image frame data to be inserted in the frame number, such as the timestamp of VSync-APP signal 4 corresponding to frame 4, based on the relationship in the interpolation instruction; obtains the image frame data in the input event corresponding to each timestamp as the second image frame, and draws and renders each second image frame.

[0334] It is understandable that when frame 4 is the lost frame, the frame filling operation can be to select only frame 4 as the second image frame according to the interpolation strategy before the VSync-APP signal 6 received at time point t6 arrives, and to draw and render frame 4 first on the premise that it is estimated that it will not affect the drawing of frame 5.

[0335] In addition, it is understandable that when the lost frames are frames 4 and 5, the frame filling operation can be before the arrival of the VSync-APP signal 7 received at the time point t7, and when it is estimated that it will not affect the rendering of frame 6, between the end time of the rendering of frame 3 and the time point t7, if the remaining time allows the rendering of two frames of image data, then both frames 4 and 5 can be used as the second image frames, and frames 4 and 5 can be rendered in turn. After the rendering of frame 5 is completed, frame 6 can be rendered within the cycle.

[0336] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0337] S305, when the interpolation strategy indicates that interpolation is required, after the first image frame is rendered to obtain the first rendered image, before the next vertical synchronization signal that triggers the rendering process arrives, the second image frame is inserted after the first image frame, and the second image frame is rendered.

[0338] For example, when the interpolation strategy includes an interpolation event, it can be determined that the interpolation strategy indicates that interpolation is required. In this case, after the drawing and rendering thread completes drawing and rendering the first image frame (such as frame 1 above) to obtain the first drawing and rendering image, before the next VSync-APP signal, such as VSync-APP signal 2 received at time point t2 above, arrives, the drawing and rendering thread uses a frame of image data in the interpolation event (such as the partial data in frame 1 above, frame 1') as the second image frame, inserts the second image frame after the first image frame, and draws and renders the second image frame.

[0339] For example, when the interpolation strategy includes a frame insertion instruction, it can be determined that the interpolation strategy indicates that interpolation is required. In this case, after the drawing and rendering thread completes the drawing and rendering of the first image frame (such as frame 3 above) to obtain the first drawing and rendering image, before the next VSync-APP signal, such as VSync-APP signal 5 received at time point t5 above, arrives, the drawing and rendering thread determines the timestamp corresponding to the input event of each image frame data to be inserted in the frame number, such as the timestamp of VSync-APP signal 4 corresponding to frame 4, based on the relationship in the interpolation instruction; obtains the image frame data in the input event corresponding to each timestamp as the second image frame, and draws and renders each second image frame.

[0340] It is understandable that when frame 4 is the lost frame, the frame filling operation can be to select only frame 4 as the second image frame according to the interpolation strategy before the VSync-APP signal 6 received at time point t6 arrives, and to draw and render frame 4 first on the premise that it is estimated that it will not affect the drawing of frame 5.

[0341] In addition, it is understandable that when the lost frames are frames 4 and 5, the frame filling operation can be before the arrival of the VSync-APP signal 7 received at the time point t7, and when it is estimated that it will not affect the rendering of frame 6, between the end time of the rendering of frame 3 and the time point t7, if the remaining time allows the rendering of two frames of image data, then both frames 4 and 5 can be used as the second image frames, and frames 4 and 5 can be rendered in turn. After the rendering of frame 5 is completed, frame 6 can be rendered within the cycle.

[0342] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0343] In addition, it can be seen from the above description that in actual applications, you can choose to insert only one frame after the first frame, such as inserting frame 1' after frame 1 in the above text, or you can choose to make up for the lost part or all of the frames after the frame is lost; you can also choose to insert one frame after the first frame, and then make up for the lost part or all of the frames after the frame is lost. The specific implementation details can be found above and will not be repeated here.

[0344] S306 , when receiving the vertical synchronization signal that triggers the synthesis process, the image synthesis system obtains the first drawing and rendering image, synthesizes the first drawing and rendering image to obtain a second picture, where the second picture includes the second content and the third content.

[0345] S307: When a vertical synchronization signal for triggering a display screen refresh process is received, the display driver drives the display screen to display a second picture. Following the sliding operation, the first area displays the second content, and the second area displays the third content.

[0346] For example, when the second screen is as shown in Figure 23, the relevant content of friend B originally displayed in the second area in Figure 22 is displayed in the first area in Figure 23, and the third content (such as the relevant content of friend C in Figures 22 and 23) is displayed in the second area.

[0347] It should be understood that the above description is merely an example listed for a better understanding of the technical solution of this embodiment, and is not intended to be the sole limitation on this embodiment.

[0348] In addition, for specific details about the synthesis thread performing the synthesis operation and the display driver driving the display screen to display the content synthesized by the synthesis thread, please refer to the above and will not be repeated here.

[0349] Therefore, the data processing method provided in this embodiment pre-inserts a frame for drawing and rendering at the initial stage of the sliding operation, and then caches one more frame in the cache queue, reducing the situation of no frame synthesis caused by subsequent drawing and rendering timeout, and reducing display jamming. For example, if only one frame is lost, a smooth transition can be achieved through the inserted frame without jamming, thereby improving the user experience.

[0350] In addition, when the drawing rendering times out, the lost frames are made up by one or more frames through frame filling, reducing the frame loss caused by missing the VSync signal due to the drawing rendering timeout, so that the content displayed on the display can change smoothly, increase the display smoothness, reduce jumps, and further improve the user experience.

[0351] In addition, it is understood that in order to implement the above functions, the terminal device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of this application.

[0352] In addition, it should be noted that the data processing methods provided in the above embodiments implemented by a terminal device in an actual application scenario can also be performed by a chip system included in the terminal device, wherein the chip system may include a processor. The chip system can be coupled to a memory so that when the chip system is running, it calls a computer program stored in the memory to implement the steps performed by the above terminal device. The processor in the chip system can be an application processor or a processor other than an application processor.

[0353] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a terminal device, the terminal device executes the above-mentioned related method steps to implement the data processing method in the above-mentioned embodiment.

[0354] In addition, an embodiment of the present application also provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the above-mentioned related steps to implement the data processing method in the above-mentioned embodiment.

[0355] In addition, an embodiment of the present application also provides a chip (which can also be a component or module), which may include one or more processing circuits and one or more transceiver pins; wherein the transceiver pins and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the above-mentioned related method steps to implement the data processing method in the above-mentioned embodiment to control the receiving pin to receive the signal, so as to control the transmitting pin to send the signal.

[0356] In addition, it can be seen from the above description that the terminal device, computer-readable storage medium, computer program product or chip provided in the embodiments of the present application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0357] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Displaying a first interface of a first application; In response to a sliding operation on the first interface, obtaining an input event corresponding to the sliding operation; Obtaining a first VSync signal, and rendering the Nth frame based on a first MOVE event, where the first MOVE event is extracted from an input event corresponding to the sliding operation based on a timestamp of the first VSync signal; In a case where the rendering time of the Nth frame is longer than one VSync signal cycle, after the rendering of the Nth frame is completed, obtaining the number of lost frames and displaying the Nth frame; Selecting a minimum value from the number of lost frames and the set maximum number of insertable frames as the number of insertable frames M; Before the second VSync signal arrives, M frames are drawn and rendered based on the second MOVE event, and the M frames are displayed; wherein the second VSync signal is the first VSync signal received after the N-th frame is drawn and rendered, and the second MOVE event is extracted from the input event corresponding to the sliding operation based on the timestamps of all or part of the VSync signals received during the N-th frame drawing and rendering process.

2. The method according to claim 1, characterized in that The obtaining of the number of lost frames includes: Determine a first time for starting rendering and a second time for ending rendering of the Nth frame; The number of lost frames is calculated according to the first time, the second time, and the set drawing rendering duration corresponding to the Nth frame, where the set drawing rendering duration is one VSync signal cycle.

3. The method according to claim 2, characterized in that The number of dropped frames is calculated based on the following formula according to the first time, the second time, and the set rendering duration corresponding to the Nth frame: Number of dropped frames = floor[(second time - first time) / VSync signal period] - 1.

4. The method according to claim 2, characterized in that The step of selecting a minimum value from the number of lost frames and the set maximum number of insertable frames as the number of insertable frames M includes: Determining a reception time of a second VSync signal according to the VSync signal period; Determine an average rendering time of each of the N frames that have completed rendering according to the rendering time of the N frames that have completed rendering; Calculate the predicted number of insertable frames according to the receiving time, the second time, and the average drawing rendering time; A minimum value is selected from the predicted number of insertable frames, the number of lost frames, and the set maximum number of insertable frames as the number M of insertable frames.

5. The method according to claim 4, characterized in that The predicted number of insertable frames is calculated according to the receiving time, the second time, and the average rendering time based on the following formula: Predicted number of insertable frames ≤ (receiving time - second time) / average rendering time.

6. The method according to claim 1, characterized in that The method further comprises: When the first application is cold started, obtaining the package name of the first application; determining an application category of the first application according to the package name; When the application category of the first application matches the set application type that supports interpolation, and when the drawing and rendering duration of the Nth frame is greater than one VSync signal cycle, after the drawing and rendering of the Nth frame is completed, the steps of obtaining the number of lost frames, selecting a minimum value from the number of lost frames and the set maximum number of interpolated frames as the number of interpolated frames, and drawing and rendering M frames based on the second MOVE event before the second VSync signal arrives are performed.

7. The method according to claim 6, characterized in that The method further comprises: Determining, based on the reporting point information corresponding to the input event, whether the input event acts on a control in the first interface; When the control acted on is a RecyclerView control or a ListView control, and the drawing and rendering duration of the Nth frame is greater than one VSync signal cycle, after the drawing and rendering of the Nth frame is completed, the steps of obtaining the number of lost frames, selecting a minimum value from the number of lost frames and the set maximum number of insertable frames as the number of insertable frames, and drawing and rendering M frames based on the second MOVE event before the second VSync signal arrives are performed.

8. The method according to claim 7, characterized in that The method further comprises: During the rendering of the Nth frame, determining the number of layers to be rendered; When the number of layers is one and the drawing and rendering time of the Nth frame is longer than one VSync signal cycle, after the drawing and rendering of the Nth frame is completed, the steps of obtaining the number of lost frames, selecting a minimum value from the number of lost frames and the set maximum number of insertable frames as the number of insertable frames, and drawing and rendering M frames based on the second MOVE event before the second VSync signal arrives are performed.

9. The method according to claim 8, characterized in that The method further comprises: When a sliding distance corresponding to a MOVE event obtained by extracting from the input event corresponding to the sliding operation based on the timestamps of two adjacent VSync signals is greater than a minimum sliding distance threshold, and when the drawing and rendering duration of the Nth frame is greater than one VSync signal cycle, after the drawing and rendering of the Nth frame is completed, the steps of obtaining the number of lost frames, selecting a minimum value from the number of lost frames and a set maximum number of insertable frames as the number of insertable frames, and drawing and rendering M frames based on the second MOVE event before the second VSync signal arrives are performed.

10. The method according to claim 1, characterized in that The method further comprises: When the rendering duration of the Nth frame is no longer than one VSync signal cycle and the Nth frame is the first frame of the rendering operation, after the rendering of the Nth frame is completed, the Nth frame is offset by a set offset to obtain the N+1th frame; Before the second VSync signal arrives, the N+1th frame is rendered and displayed.

11. The method according to claim 10, characterized in that The method further comprises: When the event extracted from the input event corresponding to the sliding operation based on the timestamp of the third VSync signal is a DOWN event, determining that the Nth frame is a first frame of the drawing and rendering operation; The third VSync signal is a VSync signal received before the first VSync signal and is adjacent to the first VSync signal.

12. A terminal device, characterized in that: The terminal device includes: a memory and a processor, the memory and the processor are coupled; the memory stores program instructions, and when the program instructions are executed by the processor, the terminal device executes the data processing method according to any one of claims 1 to 11.

13. A computer-readable storage medium, characterized in that The method comprises a computer program, which, when executed on a terminal device, causes the terminal device to execute the data processing method according to any one of claims 1 to 11.

Images