Data processing method and related device

Abstract

The embodiment of the invention provides a data processing method and a related device, and is applied to the technical field of terminals. The method comprises the steps of obtaining an input event corresponding to a sliding operation in response to the sliding operation; drawing and rendering the Nth frame based on a first move event, wherein the first move event is extracted from an input event corresponding to the sliding operation based on the timestamp of the first Vsync signal; displaying the Nth frame; under the condition that the drawing and rendering duration of the Nth frame is larger than the Vsync period, after the Nth frame is drawn and rendered, the (N + 1) th frame is drawn and rendered based on a second move event, the second move event is extracted from an input event corresponding to the sliding operation based on a first timestamp, and the first timestamp is the sum of the timestamp of the first Vsync signal and the Vsync period; and displaying the (N + 1) th frame. In this way, frame loss caused by overtime drawing and rendering is supplemented in a frame insertion mode, and the display fluency is improved.

Description

Data processing method and related device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on October 26, 2022, with application number 202211318725.6 and application name “Data Processing Methods and Related Devices”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a data processing method and related devices. Background Art

[0003] Currently, users can browse various types of content through the display screens of terminal devices. When there is a large amount of content, the display screen cannot display all of it at once. In response to a user's swipe operation on the display screen, the terminal device can control the displayed content to slide with or without the user's swipe, making it easier for the user to browse related content.

[0004] The interface display of a terminal device's display screen typically requires processes such as drawing, rendering, and compositing. For example, the terminal device's interface drawing process may include background drawing, subview drawing, scroll bar drawing, and other processes. The terminal device's interface compositing process may include processes such as vertex processing and pixel processing.

[0005] However, when the terminal device times out during drawing, abnormal phenomena such as display freeze and jump may occur.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide a data processing method and related devices, which are applied to the field of terminal technology. In the scenario of drawing and rendering timeout, the lost frames caused by the drawing and rendering timeout are compensated by interpolation to increase the smoothness of the display.

[0008] In a first aspect, an embodiment of the present application proposes a data processing method. The method includes: a terminal device responding to a sliding operation obtains an input event corresponding to the sliding operation; the terminal device draws and renders the Nth frame based on a first move event, the first move event being extracted from the input event corresponding to the sliding operation based on the timestamp of the first Vsync signal; the terminal device displays the Nth frame; if the drawing and rendering duration of the Nth frame is longer than the Vsync period, the terminal device draws and renders the N+1th frame based on a second move event after the drawing and rendering of the Nth frame is completed, the second move event being extracted from the input event corresponding to the sliding operation based on the first timestamp, the first timestamp being the sum of the timestamp of the first Vsync signal and the Vsync period; the terminal device displays the N+1th frame.

[0009] In this way, the lost frames caused by rendering timeout can be made up by interpolation, thereby increasing the smoothness of the display.

[0010] Optionally, when the duration of the drawing and rendering of the Nth frame is greater than the Vsync cycle, the terminal device draws and renders the N+1th frame based on the second move event after the drawing and rendering of the Nth frame is completed, including: at a first moment, the UI thread of the terminal device sends a Vsync signal request to the SF thread of the terminal device, and the first moment is later than the timestamp of the first Vsync signal; when the first value is greater than the Vsync cycle, the terminal device draws and renders the N+1th frame based on the second move event after the drawing and rendering of the Nth frame is completed, and the first value is the difference between the first moment and the timestamp of the first Vsync signal.

[0011] In this way, the rendering time of the Nth frame image is judged by the time of the Vsync signal request, and the lost frames caused by the late Vsync signal request are compensated to increase the smoothness of the display.

[0012] Optionally, when the first value is greater than the Vsync cycle, the terminal device draws and renders the N+1 frame based on the second move event after the Nth frame is drawn, including: when the first value is greater than the Vsync cycle, the interpolation module of the terminal device sends a first timestamp to the UI thread; the UI thread reads the second move event from the input thread of the terminal device based on the first timestamp; after the Nth frame is drawn and rendered, the UI thread draws and renders the N+1 frame based on the second move event.

[0013] In this way, the first timestamp is determined by the interpolation module, which facilitates the subsequent UI thread to determine the second move event and render the N+1th frame.

[0014] Optionally, when the first value is greater than the Vsync cycle, the terminal device draws and renders the N+1 frame based on the second move event when the Nth frame is drawn, including: when the first value is greater than the Vsync cycle, the interpolation module of the terminal device reads the second move event from the input thread of the terminal device based on the first moment; the interpolation module sends the second move event to the UI thread; the UI thread draws and renders the N+1 frame based on the second move event after the Nth frame is drawn.

[0015] In this way, the second move event is determined by the interpolation module, which facilitates the subsequent UI thread to draw and render the N+1 frame.

[0016] Optionally, the method also includes: the terminal device determines that the input event is a down event based on the timestamp of the second Vsync signal, and the timestamp of the second Vsync signal is earlier than the timestamp of the first Vsync signal; the terminal device draws and renders the Mth frame based on the third move event, and the third move event is extracted from the input event corresponding to the sliding operation based on the timestamp of the third Vsync signal, the timestamp of the third Vsync signal is one Vsync cycle later than the timestamp of the second Vsync signal, and the timestamp of the third Vsync signal is earlier than or equal to the timestamp of the first Vsync signal; the terminal device displays the Mth frame; the terminal device starts to draw and render the interpolated image after the Mth frame is drawn, and the offset between the coordinates corresponding to the interpolated image and the coordinates corresponding to the Mth frame is less than or equal to the second value, and the second value is the offset between the coordinates corresponding to the M frame and the coordinates corresponding to the M-1th frame; the terminal device displays the interpolated image.

[0017] In this way, at the initial stage of the sliding operation, a frame is inserted for drawing and rendering, so that one more frame is cached in the cache queue, which can reduce the situation of no frame synthesis caused by subsequent drawing and rendering timeout, reduce display jamming, and improve user experience.

[0018] Optionally, when the offset between the coordinates corresponding to the Mth frame and the coordinates corresponding to the M-1th frame is less than a first threshold, the offset between the coordinates corresponding to the interpolated image and the coordinates corresponding to the Mth frame is equal to a second value; when the offset between the coordinates corresponding to the Mth frame and the coordinates corresponding to the M-1th frame is greater than or equal to the first threshold, the offset between the coordinates corresponding to the M frame and the coordinates corresponding to the M-1th frame is equal to the first threshold.

[0019] In this way, taking a smaller offset for interpolation can reduce the possibility that the inserted image does not conform to the rules of the sliding operation, improve fluency, and enhance user experience.

[0020] Optionally, the terminal device starts drawing and rendering the interpolated image after the Mth frame is drawn, including: the interpolation module determines the offset between the coordinates corresponding to the interpolated image and the coordinates corresponding to the Mth frame based on the down event and the second move event; the interpolation module sends the offset between the coordinates corresponding to the interpolated image and the coordinates corresponding to the Mth frame to the UI thread; after the Mth frame is drawn, the UI thread draws and renders the interpolated image based on the offset between the coordinates corresponding to the interpolated image and the coordinates corresponding to the Mth frame.

[0021] In this way, when the input events corresponding to two adjacent Vsync signals are down events and move events respectively, a frame of image is inserted, so that one more frame is cached in the cache queue, which can reduce the situation of frameless synthesis caused by subsequent drawing and rendering timeout.

[0022] In a second aspect, embodiments of the present application provide a terminal device, which may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device may be a mobile phone, a smart TV, a wearable device, a tablet computer, a computer with wireless transceiver capabilities, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, etc.

[0023] The terminal device includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the method of the first aspect.

[0024] In a third aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of the first aspect.

[0025] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is run, it enables the computer to execute the method of the first aspect.

[0026] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute the method described in the first aspect.

[0027] It should be understood that the second to fifth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] FIG1 is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application;

[0029] FIG2 is a schematic diagram of a software structure of a terminal device provided in an embodiment of the present application;

[0030] FIG3 is a schematic diagram of an interface of a terminal device provided in an embodiment of the present application;

[0031] FIG4 is a schematic diagram of a flow chart of an interface display process of a terminal device in a possible design;

[0032] FIG5 is a schematic diagram of a flow chart of an interface display process of a terminal device in a possible design;

[0033] FIG6 is a flow chart of a data processing method provided in an embodiment of the present application;

[0034] FIG7 is a schematic diagram of a flow chart of an interface display process of a terminal device provided in an embodiment of the present application;

[0035] FIG8 is a flow chart of a data processing method provided in an embodiment of the present application;

[0036] FIG9 is a schematic diagram of a process flow of an interface display process of a terminal device provided in an embodiment of the present application;

[0037] FIG10 is a schematic diagram of a flow chart of an interface display process of a terminal device provided in an embodiment of the present application;

[0038] FIG11 is a schematic diagram of a flow chart of internal module interaction provided in an embodiment of the present application;

[0039] FIG12 is a schematic diagram of a flow chart of an interface display process of a terminal device provided in an embodiment of the present application;

[0040] FIG13 is a schematic structural diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] To facilitate a clear description of the technical solutions of the embodiments of the present application, some of the terms and technologies involved in the embodiments of the present application are briefly introduced below:

[0042] 1. Frame: This refers to the smallest unit of a single image in an interface display. A frame can be thought of as a still image. Displaying multiple frames in rapid succession can create the illusion of motion. Frame rate refers to the number of frames refreshed per second, or the number of times per second the graphics processor in a terminal device refreshes the image. A higher frame rate results in smoother and more realistic animation. The more frames per second, the smoother the displayed motion.

[0043] It should be noted that before the interface displays a frame, it usually needs to go through processes such as drawing, rendering, and synthesis.

[0044] 2. Frame drawing: refers to the image drawing of the display interface. The display interface can be composed of one or more views. Each view can be drawn by the visual control of the view system. Each view is composed of subviews. A subview corresponds to a small widget in the view. For example, a subview corresponds to a symbol in the image view.

[0045] 3. Frame rendering: This is to shade the drawn view or add 3D effects, etc. For example, 3D effects can be lighting effects, shadow effects, and texture effects.

[0046] 4. Frame synthesis: It is the process of synthesizing multiple or more rendered views into a display interface.

[0047] 5. Vertical synchronization (Vsync) signal: a signal used to control the start of frame rendering, synthesis, and display processes.

[0048] The Vsync signal is a periodic signal, and the Vsync signal period can be set according to the screen refresh rate. For example, when the screen refresh rate is 60Hz, the Vsync signal period can be 16.6ms. In other words, the terminal device generates a control signal every 16.6ms to trigger the Vsync signal period. To ensure smooth display and avoid display freezes and other phenomena, terminal devices generally use the Vsync signal for display synchronization, including image drawing, rendering, compositing, and screen refresh processes.

[0049] Vsync signals include software Vsync (Vsync-APP or Vsync-SF) and hardware Vsync (Vsync-HW). Vsync-APP triggers the rendering process. Vsync-SF triggers the compositing process. The hardware Vsync signal (Vsync-HW) triggers the screen display refresh process. Typically, software Vsync and hardware Vsync maintain cycle synchronization. For example, if Vsync-HW switches from 60Hz to 120Hz, Vsync-APP and Vsync-SF change synchronously, switching from 60Hz to 120Hz.

[0050] 6. Other terms

[0051] In the embodiments of this application, terms such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. For example, the terms "first chip" and "second chip" are used solely to distinguish between different chips and do not define their order. Those skilled in the art will understand that terms such as "first" and "second" do not define the quantity or execution order, and do not necessarily define differences.

[0052] It should be noted that in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0053] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0054] 7. Terminal equipment

[0055] The terminal device of the embodiment of the present application may also be any form of electronic device. For example, the electronic device may include a handheld device with image processing function, a vehicle-mounted device, etc. For example, some electronic devices include: mobile phones, tablet computers, PDAs, laptop computers, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving cars, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to wireless modems, in-vehicle devices, wearable devices, terminal devices in 5G networks or future evolved public land mobile communication networks. The terminal equipment in the network (PLMN), etc., is not limited to this in the embodiments of the present application.

[0056] As an example and not a limitation, in the embodiments of the present application, the electronic device may also be a wearable device. Wearable devices may also be referred to as wearable smart devices, which are a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing, and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not only hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0057] In addition, in the embodiments of the present application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0058] The electronic devices in the embodiments of the present application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, etc.

[0059] In the embodiments of the present application, the electronic device or each network device includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also known as main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as the Linux operating system, the Unix operating system, the Android operating system, the iOS operating system, or the Windows operating system. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0060] For example, FIG1 shows a schematic structural diagram of a terminal device.

[0061] The terminal device may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0062] It should be understood that the structures illustrated in the embodiments of the present invention do not constitute specific limitations on the terminal device. In other embodiments of the present application, the terminal device may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0063] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0064] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0065] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0066] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0067] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the terminal device. In other embodiments of the present application, the terminal device may also adopt a different interface connection method from the above embodiment, or a combination of multiple interface connection methods.

[0068] The terminal device implements display functionality through a GPU, display screen 194, and an application processor. The GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0069] The terminal device can realize the shooting function through the ISP, camera 193, video codec, GPU, display 194 and application processor.

[0070] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0071] The internal memory 121 can be used to store computer executable program code, and the executable program code includes instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the terminal device (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor. For example, the data processing method of an embodiment of the present application can be executed.

[0072] The software system of the terminal device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal device.

[0073] Figure 2 is a block diagram of the software structure of a terminal device in an embodiment of the present application. The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system can be divided into four layers: the application layer (applications), the application framework layer (application framework), the Android runtime (Android runtime) and system libraries, and the kernel layer (kernel).

[0074] The application layer can include a series of application packages. The application layer runs applications by calling the application programming interface (API) provided by the application framework layer. As shown in Figure 2, the application package may include applications such as phone, email, calendar, and camera.

[0075] As shown in FIG2 , the application framework layer may include a window manager, a surface flinger (image composition system, SF thread), a view system, a package manager, an input manager, an activity manager, and a resource manager, etc.

[0076] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0077] The SF thread is used to control image synthesis and generate a vertical synchronization (Vsync) signal.

[0078] Exemplarily, SF threads include: a composition thread, a Vsync thread, and a queue buffer thread. The composition thread is awakened by the Vsync signal to perform composition. The Vsync thread is used to generate the next Vsync signal based on the Vsync signal request. The queue buffer thread is used to store buffers, generate Vsync signal requests, and wake up the composition thread.

[0079] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0080] The package manager is used for program management within the system, such as application installation, uninstallation, and upgrades.

[0081] The input manager is used to manage input devices. For example, the input manager can determine input operations such as mouse clicks, keyboard input operations, and touch swipes.

[0082] The Activity Manager is used to manage the lifecycle of each application and the navigation back function. It is responsible for creating the Android main thread and maintaining the lifecycle of each application.

[0083] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0084] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for scheduling and management of the Android system.

[0085] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0086] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0087] The system library can include multiple functional modules, such as image rendering library, image synthesis library, function library, media library, and input processing library.

[0088] The image rendering library is used to render 2D or 3D images. The image synthesis library is used to synthesize 2D or 3D images.

[0089] In a possible implementation, the application renders the image using the image rendering library, and then sends the rendered image to the SF thread's cache queue. Whenever a Vsync signal arrives, the SF thread sequentially retrieves a frame of image to be synthesized from the cache queue and then uses the image synthesis library to perform image synthesis.

[0090] The function library provides macros, type definitions, string operation functions, mathematical calculation functions, and input and output functions used in the C language.

[0091] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0092] The input processing library is a library used to process input devices, which can implement mouse, keyboard and touch input processing, etc.

[0093] The kernel layer is the layer between hardware and software. The kernel layer includes at least the touch panel (TP) driver, display driver, Bluetooth driver, Wi-Fi driver, keyboard driver, shared memory driver, and camera driver.

[0094] Hardware can be audio devices, Bluetooth devices, camera devices, sensor devices, etc.

[0095] It should be understood that in some embodiments, layers that implement the same function may be referred to by other names, or a layer that can implement the functions of multiple layers may be considered as one layer, or a layer that can implement the functions of multiple layers may be divided into multiple layers. This embodiment of the present application does not limit this.

[0096] The following describes the workflow of the software and hardware of the terminal device 100 by way of example, in conjunction with the scenario of application startup or interface switching in the application.

[0097] When the touch sensor 180K in the touch panel receives a touch operation, the kernel layer processes the touch operation into an original input event (including touch coordinates, touch force, timestamp of the touch operation and other information). The original input event is stored in the kernel layer. The kernel layer reports the original input event to the input manager of the application framework layer through the input processing library. The input manager of the application framework layer parses the information of the original input event (including: operation type and reported point position, etc.) and determines the focus application based on the current focus, and sends the parsed information to the focus application. The focus can be the touch point in a touch operation or the click position in a mouse click operation. The focus application is the application running in the foreground of the terminal device or the application corresponding to the touch position in the touch operation. The focus application determines the control corresponding to the original input event based on the parsed information of the original input event (for example, the reported point position).

[0098] Taking the case where the touch operation is a touch-slide operation and the control corresponding to the touch-slide operation is a list control of the WeChat application, the WeChat application calls the image rendering library in the system library through the view system of the application framework layer to draw and render the image. The WeChat application sends the drawn and rendered image to the cache queue of the SF thread. The image synthesis library in the system library synthesizes the drawn and rendered image in the SF thread into the WeChat interface. The SF thread uses the display driver of the kernel layer to make the screen (display) display the corresponding interface of the WeChat application.

[0099] In some current scenarios, a user may perform a sliding operation on a display screen of a terminal device. In response to the user's sliding operation, the terminal device may control the displayed content to slide with or without the user's hand.

[0100] Exemplarily, Figure 3 is a schematic diagram of the interface of the terminal device provided in an embodiment of the present application. As shown in Figure 3, the terminal device can display the interface of the social application as shown in Figure 3a, the setting-related interface as shown in Figure 3b, the document interface as shown in Figure 3c, and the product browsing interface as shown in Figure 3d. When displaying these interfaces, the terminal device can respond to the user's swipe up or swipe down operation to update the displayed content. The terminal device can also display interfaces such as the interface shown in Figure 3e and the e-book interface as shown in Figure 3f. When displaying these interfaces, the terminal device can respond to the user's left swipe or right swipe operation to update the displayed content. When the terminal device receives a user's swipe operation when displaying the interface shown in Figure 3, the terminal device can perform frame drawing, rendering, synthesis and other processes based on the swipe operation to update the content displayed on the display screen.

[0101] It is understood that the interface display of a terminal device typically requires processes such as drawing, rendering, and compositing. For example, the interface drawing process of the terminal device may include processes such as background drawing, subview drawing, and scroll bar drawing. The interface compositing process of the terminal device may include processes such as vertex processing and pixel processing.

[0102] However, if the terminal device times out when drawing a rendered image (frame) (for example, exceeding one Vsync cycle), the terminal device may experience abnormal phenomena such as display freeze and jump.

[0103] The interface display processing flow of the terminal device is described below with reference to FIG4 and FIG5 .

[0104] For example, Figure 4 illustrates a possible implementation of a terminal device interface display process. As shown in Figure 4, the sequence of images updated by the terminal device based on a sliding operation is shown from time t1 to time t7. The interval between two consecutive times between t1 and t7 is the Vsync period. Whenever a Vsync signal arrives, the UI thread begins drawing and rendering the image, the SF thread begins compositing the image, and the display driver drives the display to refresh the displayed image.

[0105] Specifically, taking the display of frame 1 as an example, if the Vsync signal arrives at time t1, the UI thread of the terminal device draws and renders frame 1 through the view system of the application framework layer. After the drawing and rendering of frame 1 is completed, the UI thread of the terminal device sends the drawn and rendered frame 1 to the SF thread. The SF thread synthesizes the drawn and rendered frame 1. After the synthesis of frame 1 is completed, the terminal device can start the display driver by calling the kernel layer and display the content corresponding to frame 1 on the screen (display). Frames 2 to 7 are synthesized and displayed in a similar process to frame 1, which will not be repeated here.

[0106] It should be noted that when the UI thread is drawing and rendering an image, it sends a Vsync signal request to the SF thread to request the next Vsync signal for drawing and rendering. The SF thread sets a timer for sending the Vsync signal. After the timer expires, the SF thread feeds the Vsync signal back to the UI thread.

[0107] For example, when the UI thread is drawing and rendering frame 1, it will send a Vsync signal request to the SF thread to request the Vsync signal corresponding to time t2 for drawing and rendering. After the timing ends, the SF thread feeds back the Vsync signal corresponding to time t2 to the UI thread.

[0108] In the display processing shown in Figure 4, frames 1 through 7 are rendered within a single Vsync cycle. However, in actual terminal applications, various factors, such as heavy system load (CPU and GPU usage) or poor network connectivity, can increase the time it takes to render frames, leading to frame drops. Furthermore, rendering timeouts can leave the terminal with no frames to synthesize, causing display freezes. This is explained below with reference to Figure 5.

[0109] For example, Figure 5 is a schematic diagram of a terminal device interface display processing flow in a possible implementation. As shown in Figure 5, from time t1 to time t7, the terminal device updates the image based on the sliding operation. From time t1 to time t7, the time interval between two adjacent times is the Vsync period.

[0110] In Figure 5, when the terminal device receives a user sliding operation, the content displayed by the terminal device corresponds to frame 1, frame 2, frame 2, frame 3, and frame 5 in sequence. The terminal device draws, renders, synthesizes, and displays frame 1, frame 2, frame 3, frame 5, frame 6, and frame 7 in Figure 5, respectively. For details, please refer to the corresponding description in Figure 4.

[0111] Unlike Figure 4, the rendering time of frame 3 in Figure 5 exceeds one Vsync cycle due to various reasons, such as the complexity of graphics drawing or fluctuations in system load due to factors such as downloaded data. At t4, the UI thread continues to draw and render frame 3. The SF thread has no images to synthesize, so synthesis is not performed. The display driver drives the display screen to display frame 2. At t5, the UI thread in the terminal device begins drawing and rendering frame 5, the SF thread begins synthesizing frame 3, and the display driver drives the display screen to display frame 2. At t6, the UI thread in the terminal device begins drawing and rendering frame 6, the SF thread begins synthesizing frame 5, and the display driver drives the display screen to display frame 3. At t7, the UI thread in the terminal device begins drawing and rendering frame 7, the SF thread begins synthesizing frame 6, and the display driver drives the display screen to display frame 5.

[0112] It's understandable that in the flow shown in Figure 5, because frame 3's rendering wasn't complete at t4, there were no images to be synthesized in the SF thread at t4, so synthesis didn't occur. Consequently, the display of frame 3 was delayed until t6, and the terminal device displayed frame 2 between t4 and t6. Compared to the flow shown in Figure 4, the display time for frame 2 in Figure 5 is two Vsync cycles, causing the screen to freeze.

[0113] In addition, in the process shown in Figure 5, since the rendering of frame 3 is not completed at time t4, the UI thread continues to render frame 3 and does not render frame 4. Therefore, frame 4 is not displayed subsequently, resulting in frame loss.

[0114] In a possible design, when the UI thread is drawing and rendering frame N, it calls the scheduleVsyncLocked function to send a Vsync signal request to the SF thread to request the next Vsync signal. However, when the drawing of frame N times out, the scheduleVsyncLocked function is called too late, causing the UI thread to not draw frame N+1 due to the lack of the VSync signal from the SF thread.

[0115] In the process shown in Figure 5, the rendering of frame 3 times out. The UI thread does not send a Vsync signal request to the SF thread between t3 and t4, and the SF thread does not send the corresponding Vsync signal to the UI thread at t4. Therefore, the terminal device does not render frame 4, resulting in frame loss.

[0116] As shown in Figures 4 and 5, when a rendering timeout occurs on a terminal device, the SF thread may not be able to read frames, which can cause subsequent display lag. Furthermore, when a rendering timeout occurs on a terminal device, the Vsync signal may be missed, leading to frame drops, resulting in image jitter and a poor user experience.

[0117] It should be noted that during a sliding operation (for example, a scene where the user keeps their hands on the screen), multiple consecutive frames of rendering timeouts usually do not occur. For example, taking the WeChat Moments interface as an example, when an ad or video in Moments is first loaded, it may cause a rendering timeout.

[0118] In view of this, embodiments of the present application provide a data processing method and related apparatus. When sliding starts, a frame of rendering is inserted and cached in a cache queue; when rendering times out, a frame of rendering is inserted for rendering. In this way, when sliding starts, a frame of rendering is inserted and cached in a cache queue, thereby reducing the problem of no frames due to long rendering time and reducing the phenomenon of lag. When rendering times out, a frame of rendering is inserted for rendering, reducing frame loss due to missing Vsync signals and reducing the jump phenomenon during sliding.

[0119] The data processing method of the embodiment of the present application is described in detail below through specific embodiments. The following embodiments can be combined with each other or implemented independently, and the same or similar concepts or processes may not be repeated in some embodiments.

[0120] FIG6 is a flow chart of a data processing method provided in an embodiment of the present application. As shown in FIG6 , the method includes:

[0121] S601: In response to a sliding operation, the terminal device obtains an input event corresponding to the sliding operation.

[0122] It is understood that when a terminal device receives a user's sliding operation, the types of input events corresponding to the sliding operation are press (down), move (move), and lift (up). A sliding operation can include a press event, one or more move events, and a lift event. It can also be understood that the input events corresponding to the sliding operation include: a down event, one or more move events, and an up event.

[0123] Specifically, when the terminal device receives a sliding operation, the display screen of the terminal device may collect an input event corresponding to the sliding operation and report the input event to the input thread for recording.

[0124] S602: The terminal device renders the Nth frame based on a first move event, where the first move event is extracted from an input event corresponding to a sliding operation based on a timestamp of a first Vsync signal.

[0125] In the embodiment of the present application, the first Vsync signal may be any Vsync signal corresponding to the movement of a user's finger or the like during a sliding operation. The first move event corresponds to the first Vsync signal.

[0126] It is understood that when the terminal device receives a sliding operation, the display screen can collect the input event corresponding to the sliding operation and report the input event to the input thread. The UI thread can read the corresponding input event from the input thread based on the timestamp of the Vsync signal to perform drawing and rendering, and update the content displayed on the display screen.

[0127] In an embodiment of the present application, after receiving the Vsync signal, the UI thread reads the input event corresponding to the sliding operation recorded by the input thread based on the timestamp of the Vsync signal to obtain the input event for drawing the image.

[0128] The first move event may be an input event corresponding to a collected sliding operation, or a new input event generated after processing the input event corresponding to a collected sliding operation. The embodiment of the present application does not limit the processing of reported input events by the input thread.

[0129] S603: The terminal device displays the Nth frame.

[0130] S604: When the rendering duration of the Nth frame is longer than the Vsync period, the terminal device renders the N+1th frame based on the second move event after the rendering of the Nth frame is completed.

[0131] In the embodiment of the present application, the second move event is extracted from the input event corresponding to the sliding operation based on the first timestamp. The first timestamp is later than the timestamp of the first Vsync signal, and the first timestamp is earlier than the timestamp of the first Vsync signal received by the UI thread after the rendering of the Nth frame is completed.

[0132] Optionally, the first timestamp is the sum of the timestamp of the first Vsync signal and the Vsync period.

[0133] In this way, the difference between the timestamp corresponding to the inserted image and the timestamp corresponding to the Nth frame image is one Vsync period, which facilitates the terminal device to determine the inserted image information, is simple to calculate, and is easy to implement.

[0134] For example, as shown in Figure 7, if the UI thread sends a Vsync signal request to the SF thread between t4 and t5, and the first value is greater than the Vsync period, the terminal device begins rendering frame 4 after completing the rendering of frame 3. This way, when the rendering timeout occurs, an image frame is inserted, reducing frame drops caused by late Vsync signal requests and improving display smoothness.

[0135] In some embodiments, the duration of a Vsync signal request is used to determine whether the rendering of the Nth frame has timed out. It is understood that when an application renders a frame, it sends a Vsync signal request to the SF thread to render the next frame. Therefore, if the application does not send a Vsync signal request within one Vsync cycle after receiving the Vsync signal, it can be determined that the rendering duration of this frame exceeded one Vsync cycle.

[0136] S605: The terminal device displays the N+1th frame.

[0137] In summary, after the drawing and rendering timeout, the lost frames caused by the drawing and rendering timeout are made up by interpolation to improve the smoothness of the display.

[0138] Optionally, when the duration of the drawing and rendering of the Nth frame is greater than the Vsync cycle, the terminal device draws and renders the N+1th frame based on the second move event after the drawing and rendering of the Nth frame is completed, including: at a first moment, the UI thread of the terminal device sends a Vsync signal request to the SF thread of the terminal device, and the first moment is later than the timestamp of the first Vsync signal; when the first value is greater than the Vsync cycle, the terminal device draws and renders the N+1th frame based on the second move event after the drawing and rendering of the Nth frame is completed, and the first value is the difference between the first moment and the timestamp of the first Vsync signal.

[0139] Exemplarily, as shown in FIG7 , if the Nth frame is frame 3 in FIG7 , the first moment may be any moment between moment t4 and moment t5 .

[0140] Optionally, when the first value is greater than the Vsync cycle, the terminal device draws and renders the N+1 frame based on the second move event after the Nth frame is drawn, including: when the first value is greater than the Vsync cycle, the interpolation module of the terminal device sends a first timestamp to the UI thread; the UI thread reads the second move event from the input thread of the terminal device based on the first timestamp; after the Nth frame is drawn and rendered, the UI thread draws and renders the N+1 frame based on the second move event.

[0141] In this way, the UI thread reads the second move event based on the first timestamp, which facilitates the subsequent recovery of lost frames caused by the late Vsync signal request.

[0142] Optionally, when the first value is greater than the Vsync cycle, the terminal device draws and renders the N+1 frame based on the second move event after the Nth frame is drawn, including: when the first value is greater than the Vsync cycle, the interpolation module of the terminal device reads the second move event from the input thread of the terminal device based on the first moment; the interpolation module sends the second move event to the UI thread; the UI thread draws and renders the N+1 frame based on the second move event after the Nth frame is drawn.

[0143] In this way, the interpolation module reads the second move event based on the first timestamp and sends the second move event to the UI thread to make up for the frame loss caused by the late Vsync signal request.

[0144] The data processing method shown in Figure 6 is described below in conjunction with Figure 7. For example, Figure 7 is a schematic diagram of a terminal device interface display processing flow provided in an embodiment of the present application. Time t1 to time t7 represents the temporal sequence of image updates. The time interval between two adjacent times from t1 to t7 is the Vsync period.

[0145] When the user's finger slides on the display screen, the display screen can report the input event corresponding to the sliding operation to the input thread so that the UI thread in the application can read it for image drawing and then update the image.

[0146] As shown in FIG7 , at time t1 , the UI thread receives the Vsync1 signal, the terminal device reads the move event 1 based on the timestamp of the Vsync1 signal, and starts drawing the rendering frame 1 based on the coordinates corresponding to the move event 1.

[0147] From time t1 to time t2, the UI thread completes the rendering of frame 1 and sends the rendered frame 1 to the cache queue in the SF thread to wait for synthesis.

[0148] At time t2, the UI thread receives the Vsync2 signal. The terminal device reads move event 2 based on the timestamp of the Vsync2 signal and starts drawing and rendering frame 2 based on the coordinates corresponding to move event 2. The SF thread starts to synthesize and draw the rendered frame 1.

[0149] From time t2 to time t3, the UI thread completes the rendering of frame 2 and sends the rendered frame 2 to the cache queue in the SF thread to wait for synthesis.

[0150] At time t3, the UI thread receives the Vsync3 signal. The terminal device reads move event 3 based on the timestamp of the Vsync3 signal and starts drawing and rendering frame 3 based on the coordinates corresponding to move event 3. The SF thread starts to synthesize and draw the rendered frame 2, and the display driver drives the display screen to display the synthesized frame 1.

[0151] At time t4, the UI thread continues to draw and render frame 3. There are no frames to be synthesized in the cache queue, so the SF thread does not perform synthesis. The display driver drives the display screen to display the synthesized frame 2.

[0152] Since frame 3 is not finished rendering before time t4 and no next VSync signal is requested, the Vsync4 signal is missed. The terminal device inserts frame 4 by interpolation, and starts rendering frame 4 when frame 3 is finished rendering.

[0153] The input event is read based on the first timestamp, and the rendering frame 4 is drawn based on the read move event 4. The first timestamp is later than time t3 and earlier than the time when the UI thread sends the Vsync signal request.

[0154] From time t4 to time t5, the UI thread completes the rendering of frames 3 and 4, and sends the rendered frames 3 and 4 to the cache queue in the SF thread to wait for synthesis.

[0155] At time t5, the UI thread receives the Vsync5 signal. The terminal device reads the move event 5 based on the timestamp of the Vsync5 signal and starts drawing and rendering frame 5 based on the coordinates corresponding to the move event 5. The SF thread starts to synthesize and draw the rendered frame 3, and the display driver drives the display screen to display the synthesized frame 2.

[0156] From time t5 to time t6, the UI thread completes the rendering of frame 5 and sends the rendered frame 5 to the cache queue in the SF thread to wait for synthesis.

[0157] At time t6, the UI thread receives the Vsync6 signal. The terminal device reads the move event 6 based on the timestamp of the Vsync6 signal and starts drawing and rendering frame 6 based on the coordinates corresponding to the move event 6. The SF thread starts to synthesize and draw the rendered frame 4, and the display driver drives the display screen to display the synthesized frame 3.

[0158] From time t6 to time t7, the UI thread completes the rendering of frame 6 and sends the rendered frame 6 to the cache queue in the SF thread to wait for synthesis.

[0159] At time t7, the UI thread receives the Vsync7 signal. The terminal device reads the move event 7 based on the timestamp of the Vsync7 signal, and starts drawing and rendering frame 7 based on the coordinates corresponding to the move event 7. The SF thread starts to synthesize and draw the rendered frame 5, and the display driver drives the display screen to display the synthesized frame 4.

[0160] From time t7 to time t8, the UI thread completes the rendering of frame 7 and sends frame 7 at time t3 to the cache queue in the SF thread to wait for synthesis.

[0161] At time t8, the UI thread receives the Vsync7 signal, the SF thread begins synthesizing frame 6, and the display driver drives the display to display frame 5. Compared to the process shown in Figure 5, in the process shown in Figure 7, when the rendering of frame 3 exceeds one Vsync cycle, frame 4 is inserted through a padded frame. The display of frame 4 before frame 5 reduces the jump from frame 3 to frame 5 and improves display smoothness.

[0162] FIG8 is a flow chart of a data processing method provided in an embodiment of the present application. As shown in FIG8 , the method includes:

[0163] S801: The terminal device determines that an input event based on a timestamp of a second Vsync signal is a down event.

[0164] S802: The terminal device renders the Mth frame based on a third move event, where the third move event is obtained based on a timestamp of a third Vsync signal.

[0165] In the embodiment of the present application, the timestamp of the third Vsync signal is one Vsync cycle later than the timestamp of the second Vsync signal, and the timestamp of the third Vsync signal is earlier than or equal to the timestamp of the first Vsync signal. The third move event can also be understood as the first move event in the sliding operation read by the terminal device.

[0166] It is understandable that when the terminal device confirms that the input events corresponding to two adjacent Vsync signals are a down event and a move event respectively, the terminal device performs frame insertion.

[0167] S803: The terminal device displays the Mth frame.

[0168] S804: When the Mth frame is finished drawing, the terminal device starts drawing and rendering the interpolated frame image.

[0169] In the embodiment of the present application, the offset between the coordinates corresponding to the interpolated image and the coordinates corresponding to the Mth frame is less than or equal to the second value, and the second value is the offset between the coordinates corresponding to the Mth frame and the coordinates corresponding to the M-1th frame.

[0170] The offset of the interpolated image relative to the Mth frame is less than or equal to the offset of the Mth frame relative to the M-1th frame, and the offset of the Mth frame relative to the M-1th frame is related to the coordinates corresponding to the third move event and the coordinates corresponding to the down event.

[0171] In some embodiments, the offset may also be understood as a displacement. For example, the displacement of the interpolated image is the difference between the coordinates corresponding to the interpolated image and the coordinates corresponding to the Mth frame.

[0172] In a first possible implementation, a difference between the coordinates corresponding to the interpolated image and the coordinates corresponding to the third move event is equal to a first threshold.

[0173] The first threshold value may be 10 pixels (px) or any other value. This embodiment of the present application is not limited to this. In some embodiments, the first threshold value is related to the user's usage habits, and the terminal device may determine the first threshold value based on the displacement values ​​corresponding to the user's multiple sliding operations.

[0174] In this way, limiting the offset of the inserted image can reduce the possibility that the inserted image does not conform to the rules of the sliding operation, improve fluency, and enhance user experience.

[0175] In a second possible implementation, the difference between the coordinates corresponding to the interpolated image and the coordinates corresponding to the third move event is equal to the first displacement value, and the first displacement value is the difference between the coordinates corresponding to the third move event and the coordinates corresponding to the down event.

[0176] In this way, limiting the offset of the inserted frame image to be the same as the offset of the previous frame image can reduce the inserted image from not conforming to the rules of the sliding operation, improve fluency, and enhance user experience.

[0177] In a third possible implementation, the difference between the coordinates corresponding to the interpolated image and the coordinates corresponding to the third move event is a minimum value between the first threshold and the first displacement value.

[0178] In this way, taking a smaller offset for interpolation can reduce the possibility that the inserted image does not conform to the rules of the sliding operation, improve fluency, and enhance user experience.

[0179] S805: The terminal device displays the interpolated image.

[0180] In summary, inserting a frame in advance for drawing and rendering can cache one more frame in the cache queue, reducing the situation of frameless synthesis caused by subsequent drawing and rendering timeout, reducing display jams, and improving user experience.

[0181] Optionally, when the offset between the coordinates corresponding to the Mth frame and the coordinates corresponding to the M-1th frame is less than a first threshold, the offset between the coordinates corresponding to the interpolated image and the coordinates corresponding to the Mth frame is equal to a second value; when the offset between the coordinates corresponding to the Mth frame and the coordinates corresponding to the M-1th frame is greater than or equal to the first threshold, the offset between the coordinates corresponding to the Mth frame and the coordinates corresponding to the M-1th frame is equal to the first threshold. In this way, using a smaller offset for interpolation can reduce the possibility that the inserted image does not conform to the rules of the sliding operation, improve smoothness, and enhance the user experience.

[0182] Optionally, the terminal device starts drawing and rendering the interpolated image after the Mth frame is drawn, including: the interpolation module determines the offset between the coordinates corresponding to the interpolated image and the coordinates corresponding to the Mth frame based on the down event and the second move event; the interpolation module sends the offset between the coordinates corresponding to the interpolated image and the coordinates corresponding to the Mth frame to the UI thread; after the Mth frame is drawn, the UI thread draws and renders the interpolated image based on the offset between the coordinates corresponding to the interpolated image and the coordinates corresponding to the Mth frame.

[0183] It is understandable that the methods shown in FIG. 6 and FIG. 8 can be used in a terminal device alone, or can be used in combination in a terminal device at the same time.

[0184] The data processing method shown in Figure 8 is described below with reference to Figures 9 and 10. For example, Figure 9 is a schematic diagram of a terminal device interface display processing flow provided in an embodiment of the present application. From time t1 to time t8, the sequence of images updated by the terminal device based on a sliding operation is represented. From time t1 to time t8, the time interval between two adjacent times is the Vsync period.

[0185] As shown in FIG9 , at time t1 , the UI thread receives the Vsync1 signal, the terminal device reads the move event 1 based on the timestamp of the Vsync1 signal, and starts drawing the rendering frame 1 based on the coordinates corresponding to the move event 1.

[0186] Since the input event read by the terminal device at time t0 is a down event, the terminal device inserts move event 1' and draws and renders frame 1' based on the coordinates corresponding to move event 1'. After completing the drawing and rendering of frame 1, the UI thread draws and renders frame 1'.

[0187] The coordinates corresponding to the move event 1' are the sum of the coordinates corresponding to the move event 1 and the first threshold. Alternatively, the coordinates corresponding to the move event 1' are the difference between twice the coordinates corresponding to the move event 1 and the coordinates corresponding to the down event.

[0188] In this way, the displacement corresponding to frame 1' is less than or equal to the displacement corresponding to frame 1, thereby reducing the situation where frame 1' does not conform to the sliding operation, making frame 1' more consistent with the rules of sliding operation and improving the smoothness of display.

[0189] It is understood that the displacement corresponding to frame 1 is the difference between the coordinates corresponding to move event 1 and the coordinates corresponding to the down event. The displacement corresponding to frame 1' is the difference between the coordinates corresponding to move event 1' and the coordinates corresponding to move event 1.

[0190] From time t1 to time t2, the UI thread completes the rendering of frame 1 and frame 1', and sends the rendered frame 1 and frame 1' to the cache queue in the SF thread to wait for synthesis.

[0191] At time t2, the UI thread receives the Vsync2 signal. The terminal device reads move event 2 based on the timestamp of the Vsync2 signal and starts drawing and rendering frame 2 based on the coordinates corresponding to move event 2. The SF thread starts to synthesize and draw the rendered frame 1.

[0192] From time t2 to time t3, the UI thread completes the rendering of frame 2 and sends the rendered frame 2 to the cache queue in the SF thread to wait for synthesis.

[0193] At time t3, the UI thread receives the Vsync3 signal. The terminal device reads move event 3 based on the timestamp of the Vsync3 signal and starts drawing and rendering frame 3 based on the coordinates corresponding to move event 3. The SF thread starts synthesizing and drawing the rendered frame 1', and the display driver drives the display screen to display the synthesized frame 1.

[0194] At time t4, the UI thread continues to draw and render frame 3, the SF thread starts to synthesize and draw the rendered frame 2, and the display driver drives the display screen to display the synthesized frame 1'.

[0195] From time t4 to time t5, the UI thread completes the rendering of frame 3 and sends the rendered frame 3 to the cache queue in the SF thread to wait for synthesis.

[0196] At time t5, the UI thread receives the Vsync5 signal. The terminal device reads the move event 5 based on the timestamp of the Vsync5 signal and starts drawing and rendering frame 5 based on the coordinates corresponding to the move event 5. The SF thread starts to synthesize and draw the rendered frame 3, and the display driver drives the display screen to display the synthesized frame 2.

[0197] From time t5 to time t6, the UI thread completes the rendering of frame 5 and sends the rendered frame 5 to the cache queue in the SF thread to wait for synthesis.

[0198] At time t6, the UI thread receives the Vsync6 signal. The terminal device reads the move event 6 based on the timestamp of the Vsync6 signal and starts drawing and rendering frame 6 based on the coordinates corresponding to the move event 6. The SF thread starts to synthesize and draw the rendered frame 5, and the display driver drives the display screen to display the synthesized frame 3.

[0199] From time t6 to time t7, the UI thread completes the rendering of frame 6 and sends the rendered frame 6 to the cache queue in the SF thread to wait for synthesis.

[0200] At time t7, the UI thread receives the Vsync7 signal. The terminal device reads the move event 7 based on the timestamp of the Vsync7 signal, and starts drawing and rendering frame 7 based on the coordinates corresponding to the move event 7. The SF thread starts to synthesize and draw the rendered frame 6, and the display driver drives the display screen to display the synthesized frame 5.

[0201] From time t7 to time t8, the UI thread completes the rendering of frame 7 and sends the rendered frame 7 to the cache queue in the SF thread to wait for synthesis.

[0202] At time t8, the UI thread receives the Vsync7 signal, the SF thread starts to synthesize and draw the rendered frame 7, and the display driver drives the display screen to display the synthesized frame 6.

[0203] It's understandable that in the process flow shown in Figure 5, when frame 3's rendering times out, frame 2 displays for two Vsync cycles. In contrast, in the process flow shown in Figure 9, when frame 3's rendering times out, frame 2 displays for one Vsync cycle, shortening the display time of frame 2. The insertion of frame 1' in the process flow shown in Figure 9 results in an additional frame being cached in the SF thread. This reduces the number of instances of missing frames when frame 3's rendering times out, thereby reducing display stuttering and improving display smoothness.

[0204] For example, Figure 10 is a schematic diagram of another terminal device interface display processing flow provided by an embodiment of the present application. Time t1 to time t8 represents the order in which images are updated by the terminal device based on a sliding operation. From time t1 to time t8, the time interval between two adjacent times is the Vsync period.

[0205] As shown in FIG10 , at time t1 , the UI thread receives the Vsync1 signal, the terminal device reads the move event 1 based on the timestamp of the Vsync1 signal, and starts drawing the rendering frame 1 based on the coordinates corresponding to the move event 1.

[0206] Since the input event read by the terminal device at time t0 is a press event, the terminal device inserts frame 1'. After the UI thread completes the drawing and rendering of frame 1, it draws and renders frame 1'.

[0207] From time t1 to time t2, the UI thread completes the rendering of frame 1 and sends the rendered frame 1 to the cache queue in the SF thread to wait for synthesis.

[0208] At time t2, the UI thread continues to draw frame 1', and the SF thread starts to synthesize and draw the rendered frame 1.

[0209] Because the UI thread has not completed the rendering of frame 1' at time t2 and misses the Vsync2 signal, the terminal device inserts frame 2 by filling in the frame, and starts rendering frame 2 after the rendering of frame 1' is completed.

[0210] From time t2 to time t3, the UI thread completes the rendering of frame 1' and frame 2, and sends the rendered frame 1' and the rendered frame 2 to the cache queue in the SF thread to wait for synthesis.

[0211] During the subsequent period from time t3 to time t8, the display process of the terminal device can be similar to the process during the period from time t3 to time t8 in Figure 9 above, and will not be repeated here.

[0212] Compared to the process shown in Figure 5, in the process shown in Figure 10, when frame 3's rendering times out, frame 2 displays in 8.3ms, shortening the display time. By inserting frame 1' in the process shown in Figure 10, an additional frame is cached in the SF thread. This reduces the number of instances of missing frames when frame 3's rendering times out, thereby reducing display stuttering and improving display smoothness.

[0213] For ease of understanding, the process of interaction between the various modules involved in the data processing method provided in the embodiment of the present application is explained below in conjunction with Figure 11.

[0214] For example, Figure 11 is a schematic diagram of the process of interaction between various modules in the data processing method provided in an embodiment of the present application. The modules involved include: input thread (input thread), UI thread, interpolation module and SF thread (surface flinger) and display driver. Among them, the input thread, UI thread and interpolation module are all located in the application layer. The UI thread can also be called a logical thread, or the main application thread. The SF thread is located in the framework layer, and the display driver is located in the kernel layer.

[0215] The input thread is responsible for recording the input information uploaded from the input manager and determining the user operation type based on the input information, such as click operation, slide operation, etc. The input thread is also responsible for determining the input information sent to the UI thread based on the timestamp of the Vsync signal.

[0216] The UI thread is responsible for receiving and processing various application update messages and callbacks. For example, it handles input events, animations, and traversal operations. Traversal operations include measurement, layout, and drawing. The UI thread is also responsible for requesting and receiving Vsync signals.

[0217] S1101: During a user's sliding operation, the input thread records input information parsed by the input manager, including but not limited to: input event type, coordinates corresponding to the input event, etc.

[0218] It is understood that during the sliding operation, the terminal device collects input events corresponding to the user operation in real time. The frequency at which the terminal device collects input events can be the same as or different from the refresh rate, and this embodiment of the application does not limit this.

[0219] S1102: The UI thread reads input information from the input thread based on the timestamp of the Vsync0 signal, and transmits the timestamp of the Vsync0 signal and the read first input information to the interpolation module. The first input information includes a down event and a first coordinate.

[0220] In the embodiment of the present application, the Vsync0 signal is the Vsync signal corresponding to when the terminal device reads the down event in this sliding operation.

[0221] Exemplarily, the UI thread sends the timestamp of the Vsync0 signal to the input thread. The input thread determines the first input information based on the timestamp of the Vsync0 signal and sends the first input information to the UI thread.

[0222] In a possible implementation, the input thread processes input events within a Vsync period based on the timestamp of the Vsync0 signal to generate first input information.

[0223] S1103 : After reading the first input information, the UI thread sends a Vsync signal request to the SF thread to request the next Vsync signal.

[0224] Exemplarily, as shown in FIG12 , time t0 is the Vsync0 signal received by the UI thread, and the UI thread reads the first input information from the input thread based on the timestamp of the Vsync0 signal.

[0225] S1104: After the timer expires, the SF thread feeds back the Vsync1 signal requested in S1103 to the UI thread. In a possible implementation, after receiving the Vsync signal request, the SF thread sets a timer for sending the Vsync signal. After the timer expires, the SF thread feeds back the Vsync signal to the UI thread.

[0226] S1105 : The UI thread reads input information based on the timestamp of the Vsync1 signal to obtain second input information.

[0227] In an embodiment of the present application, the second input information includes a move event and a second coordinate. In some embodiments, the second input information also includes an operation type, such as a sliding operation.

[0228] In a first possible implementation, the second input information is calculated based on the timestamp of the Vsync0 signal and the timestamp of the Vsync1 signal and the recorded input information.

[0229] For example, in Figure 12, where time t0 is the Vsync0 signal received by the UI thread, if from time t0 to time t1, the input manager reports input event 1, input event 2, and input event 3 to the UI thread in sequence, when the UI thread receives the Vsync1 signal at time t1, the second input information read is the input information calculated by the input thread based on the input information corresponding to the three input events.

[0230] In a second possible implementation, based on the timestamp of the Vsync1 signal, the input information most recently read is the second input information.

[0231] For example, in Figure 12, where time t0 is the Vsync0 signal received by the UI thread, if from time t0 to time t1, the input manager reports input event 1, input event 2, and input event 3 to the UI thread in sequence, when the UI thread receives the Vsync1 signal at time t1, it reads the input information corresponding to input event 3 as the second input information.

[0232] S1106 : The UI thread draws the rendered image 1 based on the second input information.

[0233] 12 , time t1 is the Vsync1 signal received by the UI thread, and the UI thread reads the second input information from the input thread based on the timestamp of the Vsync1 signal. The UI thread starts drawing the rendering frame 1 based on the first input information.

[0234] In a possible implementation, the UI thread calculates the difference between the second coordinate and the first coordinate to obtain a first displacement value to determine the layout of the image 1 .

[0235] S1107 , the UI thread sends the rendered image 1 to the SF thread.

[0236] S1108 , after receiving the rendered image, the SF thread starts synthesizing image 1 .

[0237] S1109 , the SF thread sends the synthesized image 1 to the display driver.

[0238] S1110 , after receiving the synthesized image, the display driver drives the display screen to display the synthesized image 1 .

[0239] After executing the above S1105, the terminal device further executes S1111.

[0240] S1111 : The UI thread reads the timestamp of sending the Vsync1 signal to the frame insertion module and the read second input information.

[0241] S1112: After receiving the second input information, the interpolation module determines, based on the first input information and the second input information, that a sequence of two adjacent input events is a down event and a move event, and generates third input information.

[0242] It can be understood that the frame insertion module determines that the sequence of two adjacent input events is a down event and a move event, and determines to insert a frame.

[0243] In the embodiment of the present application, the third input information includes a move event and a third coordinate.

[0244] In a first possible implementation, a difference between the third coordinate and the second coordinate is equal to a first threshold.

[0245] In a second possible implementation, the difference between the third coordinate and the second coordinate is equal to the first displacement value, and the first displacement value is the difference between the second coordinate and the first coordinate.

[0246] In a third possible implementation, the difference between the third coordinate and the second coordinate is a minimum value between the first threshold and the first displacement value.

[0247] S1113. The frame insertion module sends the third input information to the UI thread.

[0248] S1114 . After the UI thread finishes drawing the rendered image 1 , it draws a rendered interpolated image based on the third input information.

[0249] It is understood that the UI thread sends the rendered interpolated image to the SF thread. After receiving the rendered interpolated image, the SF thread starts synthesizing the interpolated image. The SF thread sends the synthesized interpolated image to the display driver. The display driver drives the display screen to display the interpolated image.

[0250] For example, as shown in Figure 12, at time t1, the UI thread receives the Vsync1 signal and begins drawing and rendering frame 1. Between time t1 and time t2, the terminal device determines that two adjacent input events are a down event and a move event, generating third input information. After completing the drawing and rendering of frame 1, the UI thread begins drawing and rendering frame 1' based on the third input information.

[0251] S1115 : The UI thread sends a Vsync signal request to the SF thread to request the next Vsync signal.

[0252] In some embodiments, the UI thread sends a Vsync signal request when drawing frame 1; in other embodiments, the UI thread sends a Vsync signal request when drawing frame 1'. The embodiments of the present application do not limit the specific time of sending the Vsync signal request.

[0253] S1116. When sending the Vsync signal request, the UI thread sends the execution time of S1115 to the interpolation module.

[0254] If the difference between the execution time of S1115 and the timestamp of Vsync1 is less than or equal to the Vsync period, the interpolation module does not perform interpolation. If the difference between the execution time of S1115 and the timestamp of Vsync1 is greater than the Vsync period, the interpolation module performs interpolation. The specific process of interpolation can be found in S1127-S1129 below and is not detailed here.

[0255] S1117 . After the timer expires, the SF thread feeds back the Vsync2 signal requested in S1115 to the UI thread.

[0256] Exemplarily, as shown in FIG12 , at time t3 , the UI thread receives the Vsync2 signal and starts drawing the rendering frame 2 .

[0257] S1118 . After receiving the Vsync2 signal, the UI thread reads input information from the input thread based on the timestamp of the Vsync2 signal to obtain fourth input information.

[0258] In the embodiment of the present application, the fourth input information includes a move event and a fourth coordinate.

[0259] In a first possible implementation, the fourth input information is calculated based on the timestamp of the Vsync1 signal and the timestamp of the Vsync2 signal and the recorded input information.

[0260] In a second possible implementation, based on the timestamp of the Vsync2 signal, the input information most recently read is the fourth input information.

[0261] Adaptively, the terminal device performs rendering, synthesis, and display based on the fourth input information. Detailed description is omitted here. The specific process can be referred to above S1106-S1110. Detailed description is omitted here.

[0262] S1119 . The UI thread sends the timestamp of the Vsync2 signal and the fourth input information to the frame insertion module.

[0263] It is understandable that since the two adjacent input information are move events and move events, no interpolation is performed.

[0264] S1120. The UI thread sends a Vsync signal request to the SF thread.

[0265] S1121. The UI thread sends the execution time of S1120 to the frame insertion module.

[0266] If the difference between the execution time of S1120 and the timestamp of Vsync2 is less than or equal to the Vsync period, the interpolation module does not perform interpolation. If the difference between the execution time of S1120 and the timestamp of Vsync2 is greater than the Vsync period, the interpolation module performs interpolation. The specific process of interpolation is similar to that of S1120-S1122 below and is not described in detail here.

[0267] S1122. After the timer expires, the SF thread feeds back the Vsync3 signal requested in S1120 to the UI thread.

[0268] S1123 : After receiving the Vsync3 signal, the UI thread reads input information from the input thread based on the timestamp of the Vsync3 signal to obtain fifth input information.

[0269] In the embodiment of the present application, the fifth input information includes a move event and a fifth coordinate.

[0270] In a first possible implementation, the fourth input information is calculated based on the timestamp of the Vsync2 signal and the timestamp of the Vsync3 signal and the recorded input information.

[0271] In a second possible implementation, based on the timestamp of the Vsync3 signal, the input information most recently read is the fourth input information.

[0272] Adaptively, the terminal device performs rendering, synthesis, and display of the image 3 based on the fifth input information. Detailed description is omitted here. The specific process can be referred to above S1106-S1110. Detailed description is omitted here.

[0273] S1124 . The UI thread sends the timestamp of the Vsync3 signal and the read fifth input information to the frame insertion module and transmits them to the frame insertion module.

[0274] It is understandable that since the two adjacent input information are move events and move events, no interpolation is performed.

[0275] S1125. The UI thread sends a Vsync signal request to the SF thread.

[0276] S1126. The UI thread sends the execution time of S1125 to the frame insertion module.

[0277] When the difference between the execution time of S1125 and the timestamp of Vsync3 is less than or equal to the Vsync period, the interpolation module does not perform interpolation. When the difference between the execution time of S1125 and the timestamp of Vsync3 is greater than the Vsync period, the interpolation module performs interpolation. The specific process of interpolation is similar to that of S1120-S1122 below, and will not be described in detail here.

[0278] S1127 : The frame insertion module determines that the difference between the execution time of S1125 and the timestamp of the Vsync3 signal is greater than one Vsync cycle, and performs frame insertion processing.

[0279] In some embodiments, the frame insertion module sends the first timestamp to the UI thread; and the UI thread reads the sixth input information from the input thread based on the first timestamp.

[0280] The first timestamp is a timestamp between the execution time of S1125 and the timestamp of the Vsync3 signal.

[0281] In a possible implementation manner 1, the first timestamp is the sum of the timestamp of the Vsync3 signal and a Vsync period.

[0282] For example, taking the screen refresh rate as 120 Hz, if the timestamp of the Vsync3 signal is 24.9 ms, then the first timestamp is 24.9 ms + 8.3 ms, that is, 33.2 ms.

[0283] In the second possible implementation, the first timestamp is the sum of the timestamp of the Vsync3 signal and A Vsync cycles, where A is The integer value of .

[0284] For example, taking the screen refresh rate as 120 Hz, if the execution time of S1125 is 33.6 ms and the timestamp of the Vsync3 signal is 24.9 ms, then N is 1, and the first timestamp is 24.9 ms + 8.3 ms, that is, 33.2 ms.

[0285] For example, taking the screen refresh rate as 120 Hz, if the timestamp of the Vsync3 signal is 42 ms and the timestamp of the Vsync3 signal is 24.9 ms, then N is 2, and the first timestamp is 24.9 ms + 8.3 ms + 8.3 ms, that is, 41.5 ms.

[0286] It is understandable that, in addition to the above two methods, the interpolation module may also have other methods for determining the first timestamp, which is not limited in the embodiments of the present application.

[0287] In this embodiment of the present application, the sixth input information includes a move event and a sixth coordinate. The difference between the sixth coordinate and the fifth coordinate is equal to the fourth displacement value; the fourth displacement value is the difference between the fifth coordinate in the fifth input information and the fourth coordinate in the fourth input information.

[0288] In some other embodiments, the frame insertion module reads the sixth input information from the input thread based on the first timestamp; the frame insertion module sends the sixth input information to the UI thread.

[0289] S1128. The frame insertion module sends the first timestamp to the UI thread.

[0290] S1129: The UI thread reads the sixth input information from the input thread based on the first timestamp. Furthermore, when the rendering of the image 3 is completed, the UI thread renders the image 4 based on the sixth input information.

[0291] Adaptively, the terminal device renders, synthesizes, and displays image 3 based on the fifth input information. The terminal device renders, synthesizes, and displays image 4 based on the sixth input information. Detailed description is omitted here. For specific processes, refer to S1106-S1110 above.

[0292] For example, as shown in Figure 12, between time t4 and time t5, since the difference between the time when the UI thread sends the Vsync signal request and time t4 is greater than one Vsync cycle, the terminal device inserts frame 4 by filling in the frame, and starts drawing and rendering frame 4 when the drawing and rendering of frame 3 is completed.

[0293] This allows a frame to be pre-inserted for rendering at the beginning of a slide operation, caching an extra frame in the queue. This reduces the chances of frame loss due to subsequent rendering timeouts, reduces display stuttering, and improves the user experience. If a rendering timeout occurs, an additional frame is added through interpolation, reducing frame drops caused by missing the Vsync signal due to rendering timeouts and improving display smoothness.

[0294] It should be noted that image rendering includes multiple steps such as measurement, layout, and drawing. The UI thread can send a Vsync signal request to the SF thread when the image rendering is completed. The UI thread can also send a Vsync signal request to the SF thread when any of the measurement, layout, and drawing steps are completed. The embodiments of the present application do not specifically limit the triggering conditions for the UI thread to send a Vsync signal request.

[0295] The process shown in FIG. 11 is now described in conjunction with FIG. FIG. 12 is a schematic diagram illustrating a terminal device interface display processing process according to an embodiment of the present application. From time t0 to time t7, the sequence of images updated by the terminal device based on a sliding operation is shown. From time t1 to time t7, the time interval between two adjacent times is the Vsync period.

[0296] As shown in FIG12 , at time t0 , the UI thread receives the Vsync0 signal, and the UI thread reads the first input information based on the timestamp of the Vsync0 signal.

[0297] At time t1, the UI thread receives the Vsync1 signal. Based on the timestamp of the Vsync1 signal, the UI thread reads the second input information and begins rendering frame 1. Furthermore, at time t1, the terminal device identifies two adjacent input events as a down event and a move event, generating a third input information. After completing the rendering of frame 1, the UI thread renders frame 1' based on the third input information.

[0298] In a possible implementation, the displacement corresponding to frame 1' is less than or equal to the displacement corresponding to frame 1. The displacement corresponding to frame 1 is the difference between the second coordinate corresponding to the second input information and the first coordinate corresponding to the first input information. The displacement corresponding to frame 1' is the difference between the third coordinate corresponding to the third input information and the second coordinate.

[0299] From time t1 to time t2, the UI thread completes the rendering of frame 1 and frame 1', and sends the rendered frame 1 and frame 1' to the cache queue in the SF thread to wait for synthesis.

[0300] At time t2, the UI thread receives the Vsync2 signal. The terminal device reads the fourth input information based on the timestamp of the Vsync2 signal and begins drawing and rendering frame 2 based on the fourth coordinates in the fourth input information. The SF thread then begins compositing the rendered frame 1. Furthermore, at time t2, the terminal device determines that the two adjacent input events are move events and move events, and no interpolation is performed. From time t2 to time t3, the UI thread completes the drawing and rendering of frame 2 and sends the rendered frame 2 to the cache queue in the SF thread for compositing.

[0301] At time t3, the UI thread receives the Vsync3 signal. The terminal device reads the fifth input information based on the timestamp of the Vsync3 signal and begins rendering frame 3 based on the fifth coordinate of the fifth input information. The SF thread begins synthesizing and rendering frame 1', and the display driver drives the display to display the synthesized frame 1. In addition, at time t3, the terminal device determines that the two adjacent input events are move events and move events, and does not perform frame interpolation processing.

[0302] At time t4, the UI thread continues drawing and rendering frame 3, the SF thread begins synthesizing and drawing the rendered frame 2, and the display driver drives the display to display the synthesized frame 1'. In addition, at time t4, the terminal device confirms that the two adjacent input events are move events and move events, and no frame insertion processing is performed.

[0303] Since the difference between the time when the UI thread sends the Vsync signal request and time t4 is greater than one Vsync cycle, the terminal device inserts frame 4 by interpolation, and starts rendering frame 4 when the rendering of frame 3 is completed.

[0304] From time t4 to time t5, the rendering of frame 3 and frame 4 is completed, and the rendered frame 3 and the rendered frame 4 are sent to the cache queue in the SF thread to wait for synthesis.

[0305] At t5, the UI thread receives the Vsync5 signal. The terminal device reads the fifth input information based on the timestamp of the Vsync5 signal and begins drawing and rendering frame 5 based on the coordinates in the fifth input information. The SF thread begins synthesizing and drawing the rendered frame 3, and the display driver drives the display to display the synthesized frame 2. Furthermore, at t5, the terminal device determines that the two adjacent input events are move events and move events, and no frame interpolation is performed. From t5 to t6, the UI thread completes the drawing and rendering of frame 5 and sends frame 5 to the cache queue in the SF thread to await synthesis.

[0306] At time t6, the UI thread receives the Vsync6 signal. The terminal device reads the sixth input information based on the timestamp of the Vsync6 signal and begins drawing and rendering frame 6 based on the coordinates in the sixth input information. The SF thread begins synthesizing and drawing the rendered frame 5, and the display driver drives the display to display the synthesized frame 3. Furthermore, at time t6, the terminal device determines that the two adjacent input events are move events and move events, and no frame interpolation is performed. From time t6 to time t7, the UI thread completes the drawing and rendering of frame 6 and sends the rendered frame 6 to the cache queue in the SF thread to await synthesis.

[0307] At time t7, the UI thread receives the Vsync7 signal, the terminal device reads the seventh input information based on the timestamp of the Vsync7 signal, and starts drawing and rendering frame 7 based on the coordinates in the seventh input information. The SF thread starts to synthesize and draw the rendered frame 6, and the display driver drives the display screen to display the synthesized frame 4.

[0308] From t7 to t8, the UI thread completes the rendering of frame 7 and sends it to the cache queue in the SF thread for compositing. At t8, the UI thread receives the Vsync7 signal, and the SF thread begins compositing frame 7. The display driver then drives the display to display the composited frame 4.

[0309] Compared to the process flow shown in Figure 5, the process flow shown in Figure 12 inserts frame 1', caching an additional frame in the SF thread. This reduces the number of instances of missing frames when frame 3 rendering times out, thereby reducing display stuttering and improving display smoothness. Furthermore, in the process flow shown in Figure 12, if frame 3 rendering exceeds one Vsync cycle, frame 4 is inserted via a filler frame. Frame 4 is displayed before frame 5, minimizing the jump from frame 3 to frame 5 and improving display smoothness.

[0310] It should be noted that the "at..." in the embodiments of the present application can be the instant when a certain situation occurs, or it can be a period of time after the situation occurs, and the embodiments of the present application do not specifically limit this. In addition, the display interface provided in the embodiments of the present application is only an example, and the display interface can also include more or less content.

[0311] The data processing method of the embodiment of the present application has been described above. The following describes the apparatus for executing the above data processing method provided in the embodiment of the present application. Those skilled in the art will understand that the method and apparatus can be combined and referenced with each other, and the relevant apparatus provided in the embodiment of the present application can execute the steps in the above data processing method.

[0312] As shown in Figure 13, Figure 13 is a structural diagram of a data processing device provided in an embodiment of the present application. The data processing device can be a terminal device in an embodiment of the present application, or it can be a chip or chip system in the terminal device.

[0313] As shown in FIG13 , a data processing device 2100 can be used in a communication device, circuit, hardware component, or chip, and includes a display unit 2101 and a processing unit 2102. The display unit 2101 is used to support the display steps performed by the data processing device 2100, and the processing unit 2102 is used to support the information processing steps performed by the data processing device 2100.

[0314] In a possible implementation, the data processing device 2100 may also include a communication unit 2103. Specifically, the communication unit is used to support the data processing device 2100 in executing the steps of sending and receiving data. The communication unit 2103 may be an input or output interface, pin, or circuit.

[0315] In a possible embodiment, the data processing device may further include a storage unit 2104. The processing unit 2102 and the storage unit 2104 are connected via a circuit. The storage unit 2104 may include one or more memories, which may be devices in one or more devices or circuits for storing programs or data. The storage unit 2104 may exist independently and be connected to the processing unit 2102 of the data processing device via a communication circuit. The storage unit 2104 may also be integrated with the processing unit 2102.

[0316] The storage unit 2104 can store computer-executable instructions for the method in the terminal device, so that the processing unit 2102 executes the method in the above embodiment. The storage unit 2104 can be a register, a cache, or a RAM, etc. The storage unit 2104 can be integrated with the processing unit 2102. The storage unit 2104 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions. The storage unit 2104 can be independent of the processing unit 2102.

[0317] The data processing method provided in the embodiment of the present application can be applied to electronic devices with display functions. The electronic devices include terminal devices. The specific device form of the terminal device can refer to the above related descriptions and will not be repeated here.

[0318] An embodiment of the present application provides a terminal device, which includes: a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the above method.

[0319] The present embodiment provides a chip. The chip includes a processor configured to invoke a computer program stored in a memory to execute the technical solution of the above embodiment. The implementation principles and technical effects are similar to those of the above-mentioned related embodiments and will not be further described here.

[0320] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the above-mentioned method is implemented. The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0321] In one possible implementation, computer-readable media may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium designed to carry or store the desired program code in the form of instructions or data structures and accessible by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL) or wireless technologies such as infrared, radio and microwave are used to transmit software from a website, server or other remote source, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of medium. Disk and optical disk as used herein include optical disk, laser disk, optical disk, digital versatile disk (DVD), floppy disk and Blu-ray disk, where disks generally reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0322] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above method.

[0323] The present application embodiment is described with reference to the flow chart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application.It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions.These computer program instructions can be provided to the processing unit of general-purpose computer, special-purpose computer, embedded processing machine or other programmable device to produce a machine, so that the instruction executed by the processing unit of computer or other programmable data processing device produces the device for realizing the function specified in one flow chart flow or multiple flows and / or one block or multiple blocks of block diagram.

[0324] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: The method comprises: The terminal device obtains an input event corresponding to the sliding operation in response to the sliding operation; The terminal device renders the Nth frame based on a first move event, where the first move event is extracted from an input event corresponding to the sliding operation based on a timestamp of a first Vsync signal; The terminal device displays the Nth frame; When a duration of rendering of the Nth frame is longer than a Vsync period, the terminal device renders the N+1th frame based on a second move event after the rendering of the Nth frame is completed, where the second move event is extracted from the input event corresponding to the sliding operation based on a first timestamp, where the first timestamp is a sum of a timestamp of the first Vsync signal and the Vsync period; The terminal device displays the N+1th frame.

2. The method according to claim 1, characterized in that When the rendering duration of the Nth frame is longer than the Vsync period, the terminal device renders the N+1th frame based on the second move event after the rendering of the Nth frame is completed, including: At a first moment, the UI thread of the terminal device sends a Vsync signal request to the SF thread of the terminal device, where the first moment is later than the timestamp of the first Vsync signal; When the first value is greater than the Vsync period, the terminal device renders the N+1th frame based on the second move event after the Nth frame is rendered, and the first value is the difference between the first moment and the timestamp of the first Vsync signal.

3. The method according to claim 2, characterized in that When the first value is greater than the Vsync period, the terminal device renders the N+1 frame based on the second move event after the N frame is completed, including: When the first value is greater than the Vsync period, the interpolation module of the terminal device sends the first timestamp to the UI thread; The UI thread reads the second move event from the input thread of the terminal device based on the first timestamp; After the rendering of the Nth frame is completed, the UI thread renders the N+1th frame based on the second move event.

4. The method according to claim 2, characterized in that When the first value is greater than the Vsync period, the terminal device renders the N+1 frame based on the second move event when the N frame is completed, including: When the first value is greater than the Vsync period, the interpolation module of the terminal device reads the second move event from the input thread of the terminal device based on the first timestamp; The interpolation module sends the second move event to the UI thread; After the Nth frame is drawn, the UI thread renders the N+1th frame based on the second move event.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: The input event determined by the terminal device based on the timestamp of the second Vsync signal is a down event, and the timestamp of the second Vsync signal is earlier than the timestamp of the first Vsync signal; The terminal device renders the Mth frame based on a third move event, where the third move event is extracted from the input event corresponding to the sliding operation based on a timestamp of a third Vsync signal, the timestamp of the third Vsync signal is later than the timestamp of the second Vsync signal by one Vsync cycle, and the timestamp of the third Vsync signal is earlier than or equal to the timestamp of the first Vsync signal; The terminal device displays the Mth frame; The terminal device draws and renders an interpolated frame image after the M-th frame is drawn, wherein an offset between coordinates corresponding to the interpolated frame image and coordinates corresponding to the M-th frame is less than or equal to a second value, and the second value is an offset between coordinates corresponding to the M-th frame and coordinates corresponding to the M-1-th frame; The terminal device displays the interpolated image.

6. The method according to claim 5, characterized in that When the offset between the coordinates corresponding to the M-th frame and the coordinates corresponding to the M-1-th frame is less than a first threshold, the offset between the coordinates corresponding to the interpolated image and the coordinates corresponding to the M-th frame is equal to the second value; when the offset between the coordinates corresponding to the M-th frame and the coordinates corresponding to the M-1-th frame is greater than or equal to the first threshold, the offset between the coordinates corresponding to the M-th frame and the coordinates corresponding to the M-1-th frame is equal to the first threshold.

7. The method according to claim 5 or 6, characterized in that The terminal device starts drawing and rendering an interpolated frame image after the Mth frame is drawn, including: The interpolation module determines, based on the down event and the second move event, an offset between coordinates corresponding to the interpolation image and coordinates corresponding to the Mth frame; The interpolation module sends the offset between the coordinates corresponding to the interpolation image and the coordinates corresponding to the Mth frame to the UI thread; After the M-th frame is drawn, the UI thread draws and renders the interpolated frame image based on an offset between coordinates corresponding to the interpolated frame image and coordinates corresponding to the M-th frame.

8. A terminal device, characterized in that: include: processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the terminal device executes the method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that The method comprises a computer program, which, when being executed, causes a computer to execute the method according to any one of claims 1 to 7.

Images