Data processing method and related device

Through the frame rate control system, the synthesis thread and the display driver operate at the second frame rate in the terminal device, the lag problem caused by screen refresh rate switching is solved, the consistency of image drawing, rendering and display rhythm is achieved, and the improvement of the User experience.

CN116521115BActive Publication Date: 2024-11-12HONOR DEVICE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211502249.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-30
Publication Date
2024-11-12
Estimated Expiration
2042-01-30

AI Technical Summary

Technical Problem

During the change of the screen interface, terminal devices may experience lag caused by screen refresh rate switching, affecting the user experience.

Method used

Using a data processing method, the frame rate control system receives messages carrying the second frame rate sent by the cache thread, controls the application, synthesis thread and display driver to draw, synthesize and display images at the second frame rate, ensuring image drawing, rendering and The display rhythm is consistent and reduces the sliding speed jump.

Benefits of technology

It effectively reduces the lag phenomenon during screen refresh rate switching, improves user experience, and ensures the smoothness and consistency of image display.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116521115B_ABST
    Figure CN116521115B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a data processing method and related devices, which are applied in the field of terminal technology. The method includes: when the application draws a rendered image at a first frame rate, the frame rate control system receives a message carrying a second frame rate sent by a cache thread, and the ratio of the second frame rate to the first frame rate is an integer greater than 1; in response to the message, the frame rate control system controls the application to draw the rendered image at the second frame rate; at a first moment, the frame rate control system controls the synthesis thread to synthesize the rendered image at the second frame rate, and controls the display driver to drive the screen to display the synthesized image at the second frame rate. In this way, the drawing and rendering are switched first, and then the synthesis and display are switched, so that the display interval of each frame image between the previous frame image and the drawing and rendering interval is consistent, thereby reducing the jump in sliding speed caused by the inconsistency between the display interval and the drawing and rendering interval, reducing jams, and increasing user experience.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application. The application number of the original application is 202210114699.9, and the original application date is January 30, 2022. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a data processing method and related devices. Background Art

[0003] Currently, users can view various types of content through the display screen of terminal devices. When there is a lot of content, the display screen cannot display all the content at once, and users can slide on the display screen to browse related content.

[0004] The interface display of the display screen of the terminal device usually needs to go through the processes of drawing, rendering and synthesis. Exemplarily, the interface drawing process of the terminal device may include the processes of background drawing, subview drawing, scroll bar drawing, etc. The interface synthesis process of the terminal device may include the processes of vertex processing and pixel processing.

[0005] However, when the screen refresh rate of a terminal device switches while the screen interface is changing, the terminal device may experience a freeze. Summary of the invention

[0006] The embodiment of the present application provides a data processing method and a related device, which are applied to a terminal device. The method is used to solve the problem of freezing caused by switching of the screen refresh rate when the screen interface of the terminal device changes.

[0007] In a first aspect, an embodiment of the present application proposes a data processing method, which is applied to a terminal device, wherein the terminal device includes an application, a frame rate control system, a synthesis thread, a cache thread, and a display driver.

[0008] The method includes: when an application draws a rendered image at a first frame rate, a frame rate control system receives a message carrying a second frame rate sent by a cache thread, and a ratio of the second frame rate to the first frame rate is an integer greater than 1; in response to receiving the message carrying the second frame rate, the frame rate control system controls the application to draw the rendered image at the second frame rate; at a first moment, the frame rate control system controls the synthesis thread to synthesize the image drawn by the application at the second frame rate, and controls the display driver to drive the screen to display the image synthesized by the synthesis thread at the second frame rate; wherein the first moment is between the 1+Ath Vsync-HW signal and the 2+Ath Vsync-HW signal after the frame rate control system receives the message carrying the second frame rate, A is the number of caches cached in the cache queue corresponding to the application in the cache thread when the frame rate control system receives the message carrying the second frame rate, and the Vsync-HW signal is used to trigger the screen to display the synthesized image.

[0009] In this way, when the screen refresh rate of the terminal device switches, the image drawing and rendering process is switched first, and then the image synthesis process and the image display process are switched, so that the display interval of each frame of the image is consistent with the drawing and rendering interval with the previous frame of the image, and then the display rhythm of each frame of the image is consistent with the drawing and rendering rhythm, reducing the sliding speed jump caused by the inconsistency between the display interval and the drawing and rendering interval, reducing the jamming and improving the user experience.

[0010] Optionally, the terminal device also includes a Vsync thread; in response to receiving a message carrying a second frame rate, the frame rate control system controls the application to draw a rendered image at the second frame rate, including: after the frame rate control system receives the message carrying the second frame rate, the frame rate control system sends a first message to the Vsync thread, and the first message is used to indicate that the application frame rate is switched to the second frame rate; the Vsync thread generates a Vsync-APP signal at the second frame rate based on the first message and sends the Vsync-APP signal to the application; the application draws a rendered image based on the Vsync-APP signal.

[0011] In the embodiment of the present application, generating the Vsync-APP signal at the second frame rate can be understood as generating the Vsync-APP signal according to the Vsync signal period of the second frame rate.

[0012] In this way, by modifying the time interval between the Vsync-APP signals, the image rendering process is switched from the first frame rate to the second frame rate, which is simple and saves computing resources.

[0013] Optionally, the terminal device also includes: a Vsync thread; at a first moment, a frame rate control system controls the synthesis thread to synthesize the rendered image drawn by the application at a second frame rate, and controls the display driver to drive the screen to display the image synthesized by the synthesis thread at the second frame rate, including: at a first moment, the frame rate control system sends a second message to the Vsync thread, and the second message is used to indicate that the synthesis frame rate and the screen refresh rate are both switched to the second frame rate; the Vsync thread generates a Vsync-SF signal at the second frame rate based on the second message and sends the Vsync-SF signal to the synthesis thread; the synthesis thread synthesizes and draws the rendered image based on the Vsync-SF signal; the Vsync thread sends a third message to the display driver based on the second message, and the third message is used to indicate that the screen refresh rate is switched to the second frame rate; the display driver controls the screen to generate a Vsync-HW signal at the second frame rate based on the third message; and the display driver controls the screen to display the synthesized image after receiving the Vsync-HW signal.

[0014] In the embodiment of the present application, generating a Vsync-SF signal at the second frame rate can be understood as generating a Vsync-SF signal according to the Vsync signal period of the second frame rate. Generating a Vsync-HW signal at the second frame rate can be understood as generating a Vsync-HW signal according to the Vsync signal period of the second frame rate.

[0015] In this way, by modifying the time interval between Vsync-SF signals and the time interval between Vsync-HW signals, the synthesis process and the display process of the image are switched from the first frame rate to the second frame rate, respectively. The method is simple and saves computing resources.

[0016] Optionally, the difference between the first moment and the moment when the frame rate control system receives the message carrying the second frame rate satisfies: (buffer quantity+1)*Vsync cycle length corresponding to the first frame rate.

[0017] In this way, the calculation at the first moment can be simplified and computing resources can be saved.

[0018] Optionally, the number of caches is the sum of the number of caches that have been rendered in the cache queue corresponding to the application and the number of caches that are being rendered in the cache queue corresponding to the application.

[0019] Optionally, when the application draws a rendered image at a first frame rate, after the frame rate control system receives a message carrying the second frame rate sent by the synthesis thread, it also includes: the frame rate control system calculates the ratio of the second frame rate to the first frame rate; when the ratio is an integer greater than 1, the frame rate control system obtains the cache quantity from the cache queue corresponding to the application; the frame rate control system determines the first moment based on the cache quantity.

[0020] Optionally, when the ratio is an integer greater than 1, the frame rate control system obtains the cache quantity from the cache queue corresponding to the application, including: the frame rate control system obtains the focus window from the window manager, the focus window corresponds to the application; the frame rate control system obtains the cache quantity from the cache queue corresponding to the application based on the focus window.

[0021] Optionally, when the application draws the rendered image at the first frame rate, before the frame rate control system receives the message carrying the second frame rate sent by the synthesis thread, the method also includes: when the application draws the rendered image at the first frame rate, the cache thread receives other images after drawing and rendering, and the other images correspond to other windows corresponding to the focus window; after the cache thread receives the other images after drawing and rendering, it sends a message carrying the second frame rate to the frame rate control system.

[0022] In this way, when other windows are displayed on the display interface, the screen refresh rate is increased, making the display smooth and delicate, improving the user experience.

[0023] Optionally, other images correspond to pop-up reminders of the terminal device, or other images correspond to screenshot animations of the terminal device.

[0024] In this way, when the terminal device displays a pop-up reminder or a screenshot animation, the screen refresh rate is increased, making the display smooth and delicate, improving the user experience.

[0025] In a second aspect, an embodiment of the present application provides a terminal device, which may also be referred to as a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device may be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc.

[0026] The terminal device comprises a processor, and the processor is used to call a computer program in a memory to execute the method according to the first aspect.

[0027] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a terminal device, the terminal device executes the method of the first aspect.

[0028] In a fourth aspect, an embodiment of the present application provides a computer program product, which, when the computer program is executed, enables a terminal device to execute the method of the first aspect.

[0029] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute a method such as the first aspect.

[0030] It should be understood that the second to fifth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of the hardware system structure of a terminal device provided in an embodiment of the present application;

[0032] Figure 2 A schematic diagram of the terminal device software system structure provided in the embodiment of the present application;

[0033] Figure 3 A schematic diagram of a terminal device interface display processing flow in a possible implementation;

[0034] Figure 4 A schematic diagram of an interface display processing flow corresponding to a frame rate switching in a possible implementation;

[0035] Figure 5 A schematic diagram showing a processing flow for one interface in a possible implementation;

[0036] Figure 6 A schematic diagram showing an interface in a possible implementation;

[0037] Figure 7 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0038] Figure 8 A schematic diagram of the process of interaction between various modules in the data processing method provided in an embodiment of the present application;

[0039] Fig. 9 A schematic diagram of a terminal device interface display processing flow provided in an embodiment of the present application;

[0040] Fig.10 A schematic diagram of a data processing method provided in an embodiment of the present application;

[0041] Fig.11 A schematic diagram of a terminal device interface display processing flow provided in an embodiment of the present application;

[0042] Fig.12 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0043] Fig.13 A schematic diagram of the hardware structure of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first" and "second" are used to distinguish the same items or similar items with basically the same functions and effects. For example, the first chip and the second chip are only used to distinguish different chips, and their order is not limited. Those skilled in the art can understand that the words "first" and "second" do not limit the quantity and execution order, and the words "first" and "second" do not necessarily limit them to be different.

[0045] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0046] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0047] The embodiments of the present application provide a data processing method, which can be applied to electronic devices with display functions.

[0048] Electronic devices include terminal devices, which may also be referred to as terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. Terminal devices may be mobile phones, smart TVs, wearable devices, tablet computers (Pad), computers with wireless transceiver functions, virtual reality (VR) terminal devices, augmented reality (AR) terminal devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc. The embodiments of the present application do not limit the specific technology and specific device form used by the terminal devices.

[0049] In order to better understand the embodiment of the present application, the structure of the terminal device in the embodiment of the present application is introduced below:

[0050] Figure 1 The structure diagram of the terminal device 100 is shown. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0051] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0052] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0053] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0054] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0055] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuits sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0056] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the terminal device 100.

[0057] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0058] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to implement the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0059] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0060] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal device 100.

[0061] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0062] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transmit data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0063] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0064] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal device 100. While the charging management module 140 is charging the battery 142, it may also power the terminal device through the power management module 141.

[0065] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0066] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0067] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of the antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0068] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0069] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0070] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0071] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. GNSS may include the global positioning system (GPS), the global navigation satellite system (GLONASS), the Beidou navigation satellite system (BDS), the quasi-zenith satellite system (QZSS) and / or the satellite based augmentation system (SBAS).

[0072] The terminal device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0073] The display screen 194 is used to display images, display videos, and receive sliding operations, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light emitting diode or an active-matrix organic light emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0074] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0075] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0076] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0077] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0078] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0079] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0080] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0081] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0082] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0083] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0084] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0085] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0086] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording function, etc.

[0087] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0088] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the terminal device 100 detects the intensity of the touch operation according to the pressure sensor 180A. The terminal device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0089] The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the terminal device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0090] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0091] The magnetic sensor 180D includes a Hall sensor. The terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the terminal device 100 is a flip phone, the terminal device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover automatic unlocking and other features are set.

[0092] The acceleration sensor 180E can detect the magnitude of the acceleration of the terminal device 100 in all directions (generally three axes). When the terminal device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the terminal device and applied to applications such as horizontal and vertical screen switching and pedometers.

[0093] The distance sensor 180F is used to measure the distance. The terminal device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0094] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light outward through the light emitting diode. The terminal device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 can use the proximity light sensor 180G to detect that the user holds the terminal device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, and pocket mode automatically unlocks and locks the screen.

[0095] The ambient light sensor 180L is used to sense the ambient light brightness. The terminal device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal device 100 is in a pocket to prevent accidental touch.

[0096] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0097] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 heats the battery 142 to avoid abnormal shutdown of the terminal device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0098] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal device 100, which is different from the position of the display screen 194.

[0099] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180M can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180M can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the bone conduction sensor 180M to realize a voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the bone conduction sensor 180M to realize a heart rate detection function.

[0100] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100.

[0101] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0102] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0103] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the terminal device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The terminal device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0104] The software system of the terminal device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture, etc. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal device 100.

[0105] Figure 2 : is a software structure block diagram of the terminal device of the embodiment of the present application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Androidruntime) and the system library, the hardware abstraction layer, and the kernel layer.

[0106] The application layer can include a series of application packages. Figure 2 As shown, the application package may include applications such as phone, email, calendar, camera, etc.

[0107] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0108] like Figure 2 As shown, the application framework layer may include a window manager, a frame rate control system, an image synthesis system, a view system, a package manager, an input manager, an activity manager, and a resource manager, etc.

[0109] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0110] The frame rate control system is used to adjust the screen refresh rate.

[0111] The image synthesis system is used to control image synthesis and generate a vertical synchronization (Vsync) signal.

[0112] The image synthesis system includes: synthesis thread, Vsync thread, cache (quene buffer) thread. The synthesis thread is used to be awakened by the Vsync signal for synthesis. The Vsync thread is used to generate the next Vsync signal according to the Vsync signal request. The cache thread is used to store the cache, generate the Vsync signal request, and wake up the synthesis thread. There are one or more cache queues in the cache thread, which are used to store the cache corresponding to different applications.

[0113] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0114] The package manager is used for program management within the system, such as application installation, uninstallation, and upgrades.

[0115] The input manager is used to manage input device programs. For example, the input system can determine input operations such as mouse click operations, keyboard input operations, and touch slide operations.

[0116] The activity manager is used to manage the life cycle of each application and the navigation back function. It is responsible for creating the Android main thread and maintaining the life cycle of each application.

[0117] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0118] Android runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0119] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0120] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0121] The system library can include multiple functional modules, such as image rendering library, image synthesis library, function library, media library, and input processing library.

[0122] The image rendering library is used to render two-dimensional or three-dimensional images. The image synthesis library is used to synthesize two-dimensional or three-dimensional images.

[0123] In a possible implementation, the application renders the image through the image rendering library, and then the application sends the rendered image to the cache queue of the image synthesis system. Whenever the Vsync signal arrives, the image synthesis system (e.g., surface flinger) sequentially obtains a frame of image to be synthesized from the cache queue, and then synthesizes the image through the image synthesis library.

[0124] The function library provides macros, type definitions, string operation functions, mathematical calculation functions, and input and output functions used in the C language.

[0125] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG and PNG, etc.

[0126] The input processing library is a library for processing input devices, which can implement mouse, keyboard, and touch input processing, etc.

[0127] The hardware abstraction layer can contain multiple library modules, such as hardware synthesizer (hwcomposer, HWC), camera library module, etc. The Android system can load the corresponding library module for the device hardware, thereby enabling the application framework layer to access the device hardware. Device hardware can include LCD displays, cameras, etc. in electronic devices.

[0128] The kernel layer is the layer between hardware and software. The kernel layer includes at least touch panel (TP) driver, display driver, Bluetooth driver, WIFI driver, keyboard driver, shared memory driver and camera driver.

[0129] Hardware can be audio devices, Bluetooth devices, camera devices, sensor devices, etc.

[0130] The following is an illustrative description of the software and hardware workflow of the terminal device 100 in conjunction with a scenario where an application is started or an interface switch occurs in an application.

[0131] When the touch sensor 180K in the touch panel receives a touch operation, the kernel layer processes the touch operation into an original input event (including touch coordinates, touch strength, timestamp of the touch operation, and other information). The original input event is stored in the kernel layer. The kernel layer reports the original input event to the input manager of the application framework layer through the input processing library. The input manager of the application framework layer parses the information of the original input event (including: operation type and reported point position, etc.) and determines the focus application according to the current focus, and sends the parsed information to the focus application. The focus can be the touch point in a touch operation or the click position in a mouse click operation. The focus application is the application running in the foreground of the terminal device or the application corresponding to the touch position in the touch operation. The focus application determines the control corresponding to the original input event based on the parsed information of the original input event (for example, the reported point position).

[0132] Taking the touch operation as a touch sliding operation, and the control corresponding to the touch sliding operation as a list control of the WeChat application as an example, the WeChat application calls the image rendering library in the system library to draw and render the image through the view system of the application framework layer. The WeChat application sends the drawn and rendered image to the cache queue of the image synthesis system. The image synthesis library in the system library synthesizes the drawn and rendered image in the image synthesis system into the WeChat interface. The image synthesis system uses the display driver of the kernel layer to make the screen (display) display the corresponding interface of the WeChat application.

[0133] To facilitate understanding, some of the examples given are provided for reference to the description of concepts related to the embodiments of the present application.

[0134] 1. Frame: refers to the smallest unit of a single picture in the interface display. A frame can be understood as a still picture. Displaying multiple connected frames quickly and continuously can create the illusion of object movement. Frame rate refers to the number of frames that refresh the picture in 1 second, and can also be understood as the number of times the graphics processor in the terminal device refreshes the picture per second. A high frame rate can produce smoother and more realistic animations. The more frames per second, the smoother the displayed action will be.

[0135] It should be noted that before the interface displays a frame, it usually needs to go through processes such as drawing, rendering, and synthesis.

[0136] 2. Frame drawing: refers to the picture drawing of the display interface. The display interface can be composed of one or more views, each view can be drawn by the visual control of the view system, each view is composed of subviews, and a subview corresponds to a small widget in the view, for example, one of the subviews corresponds to a symbol in the picture view.

[0137] 3. Frame rendering: It is to color the drawn view or add 3D effects, etc. For example, 3D effects can be lighting effects, shadow effects, and texture effects.

[0138] 4. Frame synthesis: It is the process of synthesizing multiple or more rendered views into a display interface.

[0139] 5. Hand-off sliding: refers to the terminal device interface continuing to slide and display at the initial speed after the touch sliding ends.

[0140] For example, in applications such as Settings and Headlines, the interface displays a list layout. After the finger sliding the interface is lifted, the interface can continue to slide for a period of time at the initial speed after the finger is released.

[0141] It is understandable that when the interface includes a list control, the interface displays a list layout, which is a list interface. The list control can be a listview or a recyleview, and the embodiment of the present application does not limit the list control.

[0142] The following describes the display process of the interface of the terminal device 100 in combination with software and hardware.

[0143] It should be noted that in order to improve the smoothness of display and reduce the occurrence of display freezes, terminal devices generally display based on Vsync signals to synchronize the processes of image drawing, rendering, synthesis and screen refresh display.

[0144] It can be understood that the Vsync signal is a periodic signal, and the Vsync signal period can be set according to the screen refresh rate. For example, when the screen refresh rate is 60Hz, the Vsync signal period can be 16.6ms, that is, the terminal device generates a control signal every 16.6ms to trigger the Vsync signal period.

[0145] It should be noted that the Vsync signal can be divided into software Vsync signal and hardware Vsync signal. Software Vsync signals include Vsync-APP and Vsync-SF. Vsync-APP is used to trigger the drawing and rendering process. Vsync-SF is used to trigger the synthesis process. The hardware Vsync signal (Vsync-HW) is used to trigger the screen display refresh process.

[0146] Normally, the software Vsync signal and the hardware Vsync signal maintain periodic synchronization. Taking the change between 60Hz and 120Hz as an example, if Vsync-HW switches from 60Hz to 120Hz, Vsync-APP and Vsync-SF change synchronously, both switching from 60Hz to 120Hz.

[0147] For example, Figure 3The following is a schematic diagram of a terminal device interface display processing flow in a possible implementation. In chronological order, the content displayed by the terminal device corresponds to frame 1, frame 2, and frame 3 in sequence.

[0148] Specifically, taking the display of frame 1 as an example, the application of the terminal device draws and renders frame 1 through the view system of the application framework layer. After the drawing and rendering of frame 1 is completed, the application of the terminal device sends the drawn and rendered image data of frame 1 to the image synthesis system (for example, surface flinger). The image synthesis system synthesizes the drawn and rendered frame 1. After the synthesis of frame 1 is completed, the terminal device can start the display driver by calling the kernel layer to display the content corresponding to frame 1 on the screen (display). It can be understood that the application of the terminal device sends the drawn and rendered image data to the image synthesis system. The above-mentioned drawn and rendered frame 1 may refer to the drawn and rendered image data corresponding to frame 1.

[0149] Frames 2 and 3 are synthesized and displayed in a similar process to frame 1, which will not be described again here. Figure 3 Each frame lags by 2 Vsync signal cycles from drawing to display, so the display of the terminal device has a lag.

[0150] It should be noted that the terminal device may reduce the screen refresh rate to reduce lag when the system load is heavy, or increase the screen refresh rate to increase display smoothness when the system load is light.

[0151] For example, Figure 4 The following is a schematic diagram of an interface display processing flow corresponding to a frame rate switching in a possible implementation. In chronological order, the content displayed by the terminal device corresponds to frame 0, frame 1, frame 2, frame 3, frame 4, frame 5 and frame 6 in sequence.

[0152] Specifically, taking the display of frame 2 as an example, the application of the terminal device draws and renders frame 2 through the view system of the application framework layer. After the drawing and rendering of frame 2 is completed, the application of the terminal device sends the drawn and rendered frame 2 to the image synthesis system (for example, surface flinger). The image synthesis system synthesizes the drawn and rendered frame 2. After the synthesis of frame 2 is completed, the terminal device can start the display driver by calling the kernel layer to display the content corresponding to frame 2. Frames 3, 4, 5 and 6 are also synthesized and displayed in a similar process to frame 2, which will not be repeated here.

[0153] When frame 3 is drawn and rendered, the frame rate control system of the terminal device decides to switch the frame rate (for example, from 60Hz to 120Hz). When frame 4 is drawn and rendered, the frame rate is switched. The Vsync signal cycle length corresponding to frame 4 is changed to complete the frame rate switching.

[0154] It should be noted that the terminal device determines the layout of the image through the displacement. In some sliding scenarios (such as hand-off sliding), the displacement of the image is related to the frame interval (the length of the previous Vsync cycle) corresponding to the drawing and rendering of the previous frame. Specifically, taking the uniform sliding scenario as an example, the displacement of the current image (frame) during drawing and rendering is obtained by multiplying the frame interval of the previous frame (current frame Vsync-App timestamp-previous frame Vsync-App timestamp) by the sliding speed of the current frame. For example, Figure 4 Taking frame 3 in FIG. 1 as an example, the displacement of frame 3 is obtained by multiplying the frame interval of frame 2 (time stamp of Vsync2-time stamp of Vsync1) by the sliding speed of frame 3.

[0155] The sliding speed of the terminal device image display is obtained by dividing the displacement difference between the current frame displacement and the previous frame displacement (the displacement amount of the current frame) by the corresponding frame interval when the previous frame was displayed (the display duration of the previous frame). Figure 4 Taking frame 3 in the video as an example, the sliding speed of frame 3 is the displacement of frame 3 divided by the frame interval corresponding to the display of frame 2 (timestamp of Vsync4-timestamp of Vsync3).

[0156] Therefore, when the frame interval corresponding to the image drawing and rendering is consistent with the frame interval corresponding to the display, the image is displayed at the preset sliding speed. If the frame interval corresponding to the image drawing and rendering is inconsistent with the frame interval corresponding to the display, the displayed sliding speed may jump, which may cause the display screen to freeze and become unsmooth, resulting in a poor user experience.

[0157] from Figure 4 It can be seen that Figure 4 Each frame lags behind 2 Vsync signal cycles from drawing to display. When the screen refresh rate switches, the frame interval corresponding to the drawing and rendering of frame 2 is inconsistent with the frame interval corresponding to the display of frame 2. Similarly, the frame interval corresponding to the drawing and rendering of frame 3 is inconsistent with the frame interval corresponding to the display of frame 3. This may cause the sliding speed of frame 3 and frame 4 to be inconsistent with the preset sliding speed, and then the sliding speed of frame 3 and frame 4 will jump.

[0158] Combine the following Figure 5 and Figure 6 right Figure 4 The displacement and sliding speed involved in the process are explained.

[0159] For example, a list is slid at a uniform speed, the screen refresh rate is switched from 60 Hz to 120 Hz, and the sliding speed is 2 pixels every 16.6 milliseconds (ms). Figure 5 A schematic diagram showing the processing flow of one interface in a possible implementation.

[0160] Figure 5 In the chronological order, the contents displayed by the terminal device correspond to frame 0, frame 1, frame 2, frame 3, frame 4, frame 5 and frame 6.

[0161] It can be understood that the displacement is the product of the frame interval of the previous frame (current frame Vsync-App timestamp-previous frame Vsync-App timestamp) and the sliding speed of the current frame. Figure 5 The displacement of frame 3 is (16.6ms-0ms)×2pixel / 16.6ms, that is, 2pixel; similarly, the displacement of frame 4 is (33.2ms-16.6ms)×2pixel / 16.6ms, that is, 2pixel.

[0162] like Figure 5 As shown, the terminal device decides to switch the frame rate when drawing and rendering frame 3. When frame 4 starts drawing and rendering at 33.2ms, the frame rate switch has not been completed. Therefore, the displacement of frame 2, the displacement of frame 3, and the displacement of frame 4 are all related to the screen refresh rate before switching (or the length of the Vsync cycle before the frame rate switch), which is 2 pixels. At 41.5ms, the frame rate switch is completed. The displacement of frame 5 and the displacement of frame 6 are both related to the screen refresh rate after switching (or the length of the Vsync cycle after the frame rate switch), which is 1 pixel.

[0163] from Figure 5 It can be seen that the frame interval corresponding to frame 2 when drawing and rendering is 16.6ms-0ms, that is, 16.6ms, and the frame interval corresponding to frame 2 when displayed is 41.5ms-33.2ms, that is, 8.3ms. The display rhythm of the terminal device is accelerated, and the sliding speed increases when switching to frame 3 display. Similarly, the sliding speed increases when switching to frame 4 display. The frame interval corresponding to frame 4 when drawing and rendering is 41.5ms-33.2ms, that is, 8.3ms, and the frame interval corresponding to frame 4 when displayed is 58.1ms-49.8ms, that is, 8.3ms. The display rhythm of frame 4 is the same as the drawing and rendering rhythm, and the sliding speed remains unchanged when switching to frame 5. When switching from 60Hz to 120Hz, the sliding speed displayed by the terminal device first increases and then decreases, causing the user to perceive the screen as stuck.

[0164] For ease of understanding, the following Figure 6 right Figure 5 The display speed is described below.

[0165] It is understandable that the user perceives a speed change when the screen switches. Therefore, the sliding speed can be expressed by the difference between the displacement of the current frame and the displacement of the previous frame (the displacement of the current frame) divided by the display time of the previous frame.

[0166] For example, Figure 6 for Figure 5 The interface display diagram corresponding to frame 0, frame 1, frame 2, frame 3, frame 4, frame 5 and frame 6. Figure 6 As shown, there is a triangle in the list interface. Take the display screen (screen) with an absolute position of 0-18 pixels as an example. If the triangle position in frame 0 is at 0, and the displacement in frame 1 is 2 pixels, then in frame 1, the triangle position is at 2 pixels. The displacements in frame 2, frame 3, and frame 4 are all 2 pixels. The displacements in frame 5 and frame 6 are all 1 pixel. In frames 2, 3, 4, 5, and 6, the triangle positions are at 4 pixels, 6 pixels, 8 pixels, 9 pixels, and 10 pixels, respectively.

[0167] Combination Figure 5 At 16.6ms, the Vsync signal arrives, the display interface of the terminal device changes from frame 0 to frame 1, the position of the triangle moves from 0 to 2pixel, the moving speed is 2pixel / 16.6ms, and the sliding speed perceived by the user is 2pixel / (16.6ms-0ms), that is, 2pixel / 16.6ms. At 33.2ms, the Vsync signal arrives, the display interface of the terminal device changes from frame 1 to frame 2, the moving speed of the triangle is 2pixel / 16.6ms, and the sliding speed perceived by the user is 2pixel / (33.2ms-16.6ms), that is, 2pixel / 16.6ms.

[0168] At 41.5ms, the Vsync signal arrives, the display interface of the terminal device changes from frame 2 to frame 3, the movement speed of the triangle is 2pixel / 8.3ms, and the sliding speed perceived by the user is 2pixel / (41.5ms-33.2ms), that is, 2pixel / 8.3ms. At 49.8ms, the Vsync signal arrives, the display interface of the terminal device changes from frame 3 to frame 4, the movement speed of the triangle is 2pixel / 8.3ms, and the sliding speed perceived by the user is 2pixel / (49.8ms-41.5ms), that is, 2pixel / 8.3ms. At 58.1ms, the Vsync signal arrives, the display interface of the terminal device changes from frame 4 to frame 5, the movement speed of the triangle is 1pixel / 8.3ms, and the sliding speed perceived by the user is 1pixel / (58.1ms-49.8ms), that is, 1pixel / 8.3ms. At 66.4ms, the Vsync signal arrives, and the display interface of the terminal device changes from frame 5 to frame 6. The movement speed of the triangle is 1pixel / 8.3ms, and the sliding speed perceived by the user is 1pixel / (66.4ms-58.1ms), that is, 1pixel / 8.3ms.

[0169] Figure 5 In the example, the sliding speed changes from 2pixel / 16.6ms to 2pixel / 8.3ms and then to 1pixel / 8.3ms. The change in sliding speed makes users feel stuck and the user experience is poor.

[0170] In summary, the screen refresh rate of the terminal device changes during the sliding process, and the frame interval corresponding to image drawing and rendering may be larger or smaller than the frame interval corresponding to image display, resulting in a jump in the sliding speed (increase or decrease) during display, causing the screen to freeze.

[0171] In view of this, the embodiment of the present application provides a data processing method. When the frame rate of the terminal device switches, the application frame rate is switched first, and the synthesis frame rate and the screen refresh rate are switched later. In this way, the frame interval corresponding to the image display can be consistent with the frame interval corresponding to the image drawing and rendering, reducing the jump of the sliding speed during display, thereby reducing the jamming phenomenon, making the picture display uniform and smooth, and improving the user experience.

[0172] The application scenarios provided by the embodiments of the present application are described below with reference to the accompanying drawings. Figure 7 A schematic diagram of an application scenario provided for an embodiment of the present application.

[0173] When the terminal device is Figure 7 When the list interface shown in a receives an upward sliding operation, it enters Figure 7 Compared with the list interface shown in b in Figure 7 The list interface shown in a, Figure 7 The corresponding content displayed in the list in the list interface shown in b changes.

[0174] When the terminal device is sliding the list, other content may be displayed, and the terminal device may increase the screen refresh rate to improve the smoothness of the display. Figure 7 The pop-up window reminder 701 in the interface shown in c in FIG. Figure 7 The screenshot animation 702 in the interface shown in d, etc.

[0175] In one possible scenario, the terminal device Figure 7 After an up event in the sliding operation is detected in the list interface of a, the sliding list display continues based on the initial velocity at the time of the up event. The terminal device may reduce the screen refresh rate in the display after the up event to reduce power consumption; when the terminal device displays other content in the list interface, it may switch the screen refresh rate based on the method of the embodiment of the present application to improve the smoothness of the list interface. Other content includes but is not limited to: pop-up reminders, screenshot animations, or charging reminders. Pop-up reminders include: application message reminders, such as SMS message reminders and WeChat message reminders.

[0176] It should be noted that the types of input events corresponding to the sliding operation can be divided into press (down), move (move) and lift (up).

[0177] It is understandable that the list interface of the embodiment of the present application can be an interface of a social application, a setting related interface, a document interface, or a product browsing interface. The terminal device can receive a sliding operation of the user on the list interface. The sliding operation can be an upward sliding operation, a downward sliding operation, a left sliding operation, or a right sliding operation.

[0178] It is understandable that the data processing method provided in the embodiment of the present application can also be applied to other speed-related scenarios, which are not limited here. In addition, the method provided in the embodiment of the present application can be applied to a uniform speed scenario, a speed reduction scenario, or a speed increase scenario. The embodiment of the present application does not limit the change in speed.

[0179] For ease of understanding, the following Figure 8 and Fig. 9 The process of interaction between various modules involved in the data processing method provided in the embodiment of the present application is explained.

[0180] For example, Figure 8 A schematic diagram of the process of interaction between various modules in the data processing method provided in an embodiment of the present application.

[0181] like Figure 8 As shown, the system may include: an application, a frame rate control system, an image synthesis system (surfaceflinger), a window manager, a hardware synthesizer, and a display driver. Among them, the application includes an application main thread and an application rendering thread. The image synthesis system includes a Vsync thread, a cache thread, and a synthesis thread. The application main thread can also be called a logical thread, or a UI thread.

[0182] Take the pop-up window reminder of the terminal device during the hand-off sliding process as an example.

[0183] When the touch panel receives a touch operation, the kernel layer processes the touch operation into a raw input event (including touch coordinates, touch force, timestamp of the touch operation, etc.). The raw input event is stored in the kernel layer. The kernel layer reports the raw input event to the input manager of the application framework layer through the input processing library. The input manager of the application framework layer parses the information of the raw input event (including: operation type and reported point location, etc.) and determines the focus application according to the current focus, and sends the parsed raw input event information (for example, down event, move event, up event, etc.) to the synthesis thread.

[0184] When the terminal device receives a sliding operation while displaying at the first frame rate, the synthesis thread receives the down event, move event, and up event sent by the input manager. When the synthesis thread receives the down event sent by the input manager, it sends a message to the frame rate control system for instructing to switch the screen refresh rate, and the message carries the second frame rate. Exemplarily, the first frame rate is 60 Hz and the second frame rate is 120 Hz.

[0185] After receiving the preset duration of the up event, the synthesis thread sends a message for instructing the frame rate control system to switch the screen refresh rate, where the message carries the first frame rate.

[0186] S801. When the terminal device issues a pop-up window reminder, the cache thread receives cache queues of other layers except the focus window layer.

[0187] It should be noted that the focus window layer can be understood as the layer corresponding to the list in the display interface, corresponding to the list in the display interface. Other layers except the focus window layer can be understood as the layers corresponding to the pop-up window reminder, corresponding to the pop-up window in the display interface.

[0188] In the embodiment of the present application, the focus window layer and other layers may correspond to the same application or may correspond to different applications respectively, which is not limited here.

[0189] For example, Fig. 9 A display flow diagram provided in an embodiment of the present application. Fig. 9 As shown, if the first frame rate is 60 Hz, the second frame rate is 120 Hz. And, Fig. 9 In the chronological order, the contents displayed by the terminal device correspond to frame 0, frame 1, frame 2, frame 3, frame 4, frame 5 and frame 6.

[0190] Fig. 9 At 16.6ms, the Vsync-APP signal arrives, and the main thread of the application starts drawing and rendering frame 3. During 16.6ms-33.2ms, the cache thread of the terminal device receives cache queues of other layers except the focus window layer.

[0191] S802: The cache thread sends a message for instructing the frame rate control system to switch the screen refresh rate, where the message carries the second frame rate.

[0192] For example, Fig. 9 In the process shown, during the period of 16.6ms-33.2ms, the cache thread receives cache queues of other layers except the focus window layer (such as Fig. 9 The synthesis thread sends a message to the frame rate control system for instructing to switch the screen refresh rate. The message carries a second frame rate, and the second frame rate is 120 Hz.

[0193] S803: The frame rate control system confirms that the ratio of the second frame rate to the first frame rate is an integer.

[0194] For example, Fig. 9 In the process shown, the frame rate control system determines that the ratio of the second frame rate to the first frame rate is 2, which is an integer.

[0195] S804, the frame rate control system queries the window manager for the focus window. Adaptively, the window manager feeds back the application package name and focus window layer corresponding to the focus window to the frame rate control system. The focus window is the window corresponding to the application running in the foreground of the terminal device.

[0196] For example, Fig. 9 In the process shown, the focus window is the window corresponding to the input event.

[0197] It is understandable that the above S803 and S804 may be executed simultaneously or at different times, and the embodiment of the present application does not limit the sequence of S803 and S804.

[0198] In a possible implementation, the frame rate control system can determine the focus application according to the application package name corresponding to the focus window, and then confirm whether the focus application is in the pre-stored delayed switching application list, and if it is in the delayed switching application list, execute subsequent S805-S830. If it is not in the delayed switching application list, the application frame rate and screen refresh rate are switched synchronously.

[0199] S805. The frame rate control system queries the cache thread for the number of caches in the cache queue corresponding to the focus window according to the focus window; the cache thread adaptively feeds back the number of caches to the frame rate control system.

[0200] It should be noted that the buffer in the buffer queue includes one or more of the following: a buffer that has been queued (queued buffer), a buffer that is being rendered (dequeued buffer), a buffer that is being synthesized (acquired buffer), and an unused buffer (free buffer).

[0201] The cached cache can be understood as a cache that stores the rendered image. The rendering cache can be understood as a cache used to store the image being rendered by the application. The synthesizing cache can be understood as a cache that is synthesized in the synthesis thread. The unused cache can be understood as a cache that does not store an image.

[0202] For example, if the total number of caches in the cache queue is 20, Fig. 9At 17ms, the application is drawing and rendering frame 3, the cache being rendered is frame 3, and the number is 1; there is no cached cache in the cache queue, and the number is 0; the synthesis thread is synthesizing frame 2, and the cache being synthesized is frame 2, and the number is 1; the number of unused caches is 20-1-0-1, that is, 18.

[0203] exist Fig. 9 At 30ms, the application is not rendering, and the number of buffers being rendered is 0; the cached buffer is frame 3, and the number is 1; the synthesis thread is not synthesizing, and the number of buffers being synthesized is 0; the number of unused buffers is 20-0-1-0, that is, 19.

[0204] In the embodiment of the present application, the number of buffers refers to the sum of the number of queued buffers and the number of dequeued buffers being rendered.

[0205] For example, Fig. 9 In the process shown, frame 3 drawing rendering is completed, the number of cached buffers is 1 and the number of buffers being rendered is 0, the number of buffers is 1, and the cache thread feeds back the number of buffers as 1 to the frame rate control system.

[0206] S806: The frame rate control system determines, based on the buffer quantity, a delay time M between the screen refresh rate switching to the second frame rate and the application frame rate switching to the second frame rate. M satisfies the following formula: M=(buffer quantity+1)*Vsync period corresponding to the first frame rate.

[0207] When the ratio of the second frame rate to the first frame rate is an integer, the delay time M = (buffer number + 1) * Vsync period corresponding to the first frame rate. When the buffer number is 1, the delay time M is 2 * Vsync period corresponding to the first frame rate.

[0208] For example, Fig. 9 In the process shown, the frame rate control system determines that the delay time between the screen refresh rate switching to the second frame rate and the application frame rate switching to the second frame rate is 2*16.6ms, that is, 33.2ms.

[0209] S807. The frame rate control system sends a first message to the Vsync thread; the first message is used to instruct the application frame rate to switch to the second frame rate.

[0210] The application frame rate refers to the frame rate corresponding to the application's drawing and rendering.

[0211] In a possible implementation, the first message carries a second frame rate. Exemplarily, the second frame rate is 120 Hz.

[0212] S809. After receiving the first message, the Vsync thread stores the Vsync signal period corresponding to the second frame rate.

[0213] It is understandable that after receiving the first message, the Vsync thread stores the Vsync signal period corresponding to the application frame rate as the Vsync period corresponding to the second frame rate. Subsequent Vsync-APP signals are generated according to the rhythm corresponding to the second frame rate.

[0214] S810: The frame rate control system sends a first message to the Vsync thread and then goes into sleep mode.

[0215] When the terminal device executes the above S802-S809 or before executing the above S802, the terminal device may also execute S811.

[0216] S811. The application main thread sends a Vsync-APP request to the Vsync thread.

[0217] S812. The Vsync thread generates a Vsync-APP according to a Vsync signal period corresponding to the first frame rate.

[0218] For example, Fig. 9 At 16.6ms-33.2ms, the decision to switch the application frame rate is made. The Vsync thread generates Vsync-APP according to the Vsync signal period (16.6ms) corresponding to 60Hz. At 33.2ms, the Vsync-APP signal is sent to the main thread of the application.

[0219] S813. When receiving Vsync-APP, the main application thread calculates the frame interval according to the timestamp of Vsync-APP.

[0220] Specifically, the main application thread calculates the difference between the timestamp of the Vsync-APP signal received this time and the timestamp of the Vsync-APP signal received last time, and the difference is the frame interval corresponding to the drawing and rendering of the previous frame.

[0221] For example, Fig. 9 In the example, the timestamp of the Vsync-APP signal received this time is 33.2ms, and the timestamp of the Vsync-APP signal received last time is 16.6ms. The frame interval corresponding to the calculation and rendering of frame 3 by the main thread of the application is 33.2ms-16.6ms, that is, 16.6ms.

[0222] S814. Apply the main thread to calculate the displacement.

[0223] S815. The application main thread sends the displacement of the current frame to the application rendering thread to wake up the application rendering thread.

[0224] In a possible implementation, the displacement is the product of the frame interval and the speed. It should be noted that the application main thread can determine the speed based on a pre-stored speed curve. Fig. 9 For example, the Vsync-APP signal of 33.2ms in the image has a speed of 2pixel / 16.6ms. The current frame is frame 4, and the displacement of frame 4 is the product of the frame interval (16.6ms) corresponding to the rendering of frame 3 and 2pixel / 16.6ms, that is, 2pixel. The main thread of the application sends the displacement of the current frame (2pixel) to the application rendering thread.

[0225] In a possible implementation, the application main thread does not execute S813-S815. The application main thread sends the speed of the current frame, the current Vsync-APP timestamp, and the timestamp of the previous frame Vsync-APP to the application rendering thread to wake up the application rendering thread. Exemplarily, the application main thread sends the speed (2pixel / 16.6ms), the Vsync-APP timestamp of the current frame (33.2ms), and the timestamp of the previous frame Vsync-APP (16.6ms) to the application rendering thread.

[0226] S816: After receiving the displacement or the timestamp of the Vsync-APP signal, the application rendering thread is awakened and starts drawing a rendered image.

[0227] In a possible implementation, after receiving the displacement, the main thread of the application is awakened and starts drawing the rendered image.

[0228] S817: After being awakened, the application rendering thread requests a cache from the cache process to store the rendered image.

[0229] Adaptively, after receiving the cache request command sent by the application rendering thread, the cache thread reserves space for storing the rendered image and sends an instruction for instructing the cache to be dequeued to the application rendering thread.

[0230] S818: After receiving the instruction for instructing cache dequeue, the application rendering thread draws a rendering image according to the displacement.

[0231] S819, the application rendering thread sends the rendered image to the cache thread (cache queue).

[0232] S820. After receiving the rendered image sent by the application rendering thread, the cache thread requests a Vsync-SF signal from the Vsync thread.

[0233] After the above S814, when the terminal device executes S814-S820, the main thread of the application can also execute S821.

[0234] It is understandable that the terminal device can simultaneously execute processes such as rendering, synthesis, and display. Fig. 9 As shown, when frame 4 is rendered, frame 3 and the pop-up window reminder are synthesized, and frame 2 is displayed.

[0235] S821. The application main thread sends a Vsync-APP request to the Vsync thread.

[0236] S822. The Vsync thread generates a Vsync-APP according to the Vsync signal period corresponding to the second frame rate, and sends the Vsync-APP to the application main thread.

[0237] For example, Fig. 9 In the process shown, the application main thread sends a Vsync-APP request to the Vsync thread during 33.2ms-41.5ms; the Vsync thread generates Vsync-APP according to the Vsync signal period (8.3ms) corresponding to the second frame rate, and sends Vsync-APP to the application main thread at 33.2ms+8.3ms, i.e. 41.5ms.

[0238] It is understandable that, usually, the execution time of S822 is after the execution time of S819.

[0239] It is understandable that after receiving the Vsync-APP in S822, the application main thread calculates the displacement and draws the rendered image through the application rendering thread. After the image is drawn and rendered, it is sent to the cache queue of the cache thread to wait for synthesis. After receiving the drawn and rendered image sent by the application rendering thread, the cache thread requests the Vsync-SF signal from the Vsync thread.

[0240] After the above S822, after the main thread of the application main line calculates the displacement, the main thread of the application can also execute S823 to continue requesting Vsync-APP.

[0241] S823. The application main thread sends a Vsync-APP request to the Vsync thread.

[0242] S824, the Vsync thread generates a Vsync-APP according to the Vsync signal period corresponding to the second frame rate, and sends the Vsync-APP to the application main thread. In addition, the Vsync thread generates a Vsync-SF according to the Vsync signal period corresponding to the first frame rate, and sends the Vsync-SF to the synthesis thread.

[0243] It can be understood that the Vsync thread generates a Vsync-APP signal in S824 in response to the Vsync-APP requested in S823. The Vsync thread generates a Vsync-SF signal in S824 in response to the Vsync-SF requested in S820.

[0244] For example, Fig. 9 As shown, the Vsync thread sends a Vsync-APP signal to the application main thread at 49.8 ms, and the Vsync thread sends a Vsync-SF signal to the synthesis thread at 49.8 ms. S825: After receiving the Vsync-SF signal, the synthesis thread synthesizes the image.

[0245] In a possible implementation, after receiving the Vsync-SF signal, the composition thread may query the window manager for the focus application. The composition thread queries the cache queue corresponding to the focus application in the cache thread according to the focus application to confirm the image to be composited.

[0246] It can be understood that in the above S824, the generation time of the Vsync-APP signal is consistent with the generation time of the Vsync-SF signal. Therefore, when the synthesis thread synthesizes the image, the application is drawing the rendered image. Specifically, after receiving the Vsync-APP in S824, the application main thread calculates the displacement and draws the rendered image through the application rendering thread. After the image is drawn and rendered, it is sent to the cache queue to wait for synthesis. After receiving the drawn and rendered image sent by the application rendering thread, the cache thread requests the Vsync-SF signal from the Vsync thread.

[0247] For example, Fig. 9 As shown, at 49.8ms, the synthesis thread is synthesizing frame 4, and the application main thread is drawing rendering frame 6.

[0248] S826: The synthesis thread sends the synthesized image to the hardware synthesizer. The hardware synthesizer sends the synthesized image to the display driver for display.

[0249] S827: After the Vsync-HW signal arrives, the display driver displays the synthesized image.

[0250] S828: After the frame rate control system reaches M time, the sleep state ends.

[0251] For example, Fig. 9 In the process shown, the frame rate control system sleeps for 33.2ms, and the 33.2ms after S810 is the period from 49.8ms to 66.4ms. Therefore, when the terminal device executes S825 (49.8ms-66.4ms), that is, when synthesizing frame 4, the sleep ends.

[0252] S829. The frame rate control system sends a second message to the Vsync thread, where the second message is used to indicate that both the synthetic frame rate and the screen refresh rate are switched to the second frame rate.

[0253] S830. After receiving the second message, the Vsync thread stores the Vsync signal period corresponding to the second frame rate.

[0254] It is understandable that after receiving the second message, the Vsync thread stores the Vsync signal period corresponding to the synthesized frame rate as the Vsync period corresponding to the second frame rate. Subsequent Vsync-SF signals are generated according to the rhythm corresponding to the second frame rate.

[0255] For example, Fig. 9 In the process shown, a Vsync-SF signal is generated at 66.4ms, and the synthesis thread synthesizes frame 5 (i.e., draws the rendered image according to the Vsync signal in S822). A Vsync-SF signal is generated at 66.4ms+8.3ms, i.e., 74.7ms, and the synthesis thread synthesizes frame 6.

[0256] S831. After receiving the second message, the Vsync thread sends the second frame rate to the hardware synthesizer.

[0257] S832: After receiving the second frame rate, the hardware synthesizer sends the second frame rate to the display driver.

[0258] S833: The display driver drives the screen to switch the screen refresh rate to the second frame rate.

[0259] It can be understood that the screen generates a Vsync-HW signal according to the Vsync period corresponding to the second frame rate.

[0260] For example, Fig. 9 In the process shown, the Vsync-HW signal is generated at 66.4ms, and the screen displays frame 4. The Vsync-HW signal is generated at 66.4ms+8.3ms, that is, 74.7ms, and the screen displays frame 5.

[0261] S834. The Vsync thread generates a Vsync-SF according to the Vsync signal period corresponding to the first frame rate, and sends the Vsync-SF to the synthesis thread.

[0262] It is understandable that the terminal device responds to the Vsync-SF signal requested after S822 and before S824, and sends the Vsync-SF to the synthesis thread. After receiving the Vsync-SF signal, the synthesis thread synthesizes the image and sends the synthesized image to the display driver for display.

[0263] For example, Fig. 9 As shown, the Vsync thread sends a Vsync-SF signal to the synthesis thread at 66.4 ms, and the synthesis thread synthesizes frame 5.

[0264] S835. The Vsync thread generates a Vsync-SF according to the Vsync signal period corresponding to the second frame rate, and sends the Vsync-SF to the synthesis thread.

[0265] It is understandable that the terminal device responds to the Vsync-SF signal requested after S824 and sends the Vsync-SF to the synthesis thread. After receiving the Vsync-SF signal, the synthesis thread synthesizes the image and sends the synthesized image to the display driver for display.

[0266] For example, Fig. 9 As shown, the Vsync thread sends a Vsync-SF signal to the synthesis thread at 74.7 ms, and the synthesis thread synthesizes frame 6.

[0267] It should be noted that when the terminal device displays a screenshot animation or a charging reminder during the hand-off sliding process, the implementation process of the terminal device is the same as the above Figure 8 The process shown is similar and will not be described in detail here.

[0268] The data processing method of the embodiment of the present application is described in detail below through specific embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0269] Fig.10 A flowchart of a data processing method is provided for an embodiment of the present application. Fig.10 As shown, the method may include:

[0270] S1001. Determine that the second frame rate is an integer multiple of the first frame rate.

[0271] Specifically, the frame rate control system determines that the second frame rate is an integer multiple of the first frame rate.

[0272] In a possible implementation, after receiving a message carrying the second frame rate sent by the synthesis thread, the frame rate control system determines that the second frame rate is an integer multiple of the first frame rate.

[0273] In the embodiment of the present application, the first frame rate is the frame rate before the screen refresh rate is switched; the second frame rate is the frame rate after the screen refresh rate is switched.

[0274] It is understandable that the ratio of the second frame rate to the first frame rate is an integer. For example, the first frame rate may be 120 Hz and the second frame rate may be 60 Hz; the first frame rate may be 120 Hz and the second frame rate may be 40 Hz. This is not limited here.

[0275] S1002: Query the cache quantity in the cache queue.

[0276] Specifically, the frame rate control system queries the number of buffers in the buffer queue. The number of buffers refers to the sum of the number of buffers that have been cached (queued buffer) and the number of buffers being rendered (dequeued buffer).

[0277] In a possible implementation, the focus window is queried, and the number of caches in the cache queue corresponding to the focus window is queried based on the application package name corresponding to the focus window. Exemplarily, the frame rate control system obtains the focus application from the window manager. The cache queue corresponding to the focus application in the image synthesis system is queried according to the focus application, and the number of caches in the corresponding cache queue is confirmed.

[0278] In this way, it is possible to determine how many Vsync cycles are between drawing and rendering and displaying each frame of the image, making it easier to determine the delay time for switching the screen refresh rate later.

[0279] S1003: Determine, based on the cache quantity, a delay time between switching the screen refresh rate to the second frame rate and switching the application frame rate to the second frame rate.

[0280] Specifically, the frame rate control system determines, based on the cache quantity, a delay time between switching the screen refresh rate to the second frame rate and switching the application frame rate to the second frame rate.

[0281] In the embodiment of the present application, the delay duration may be the sum of the first duration and the second duration. The first duration is the difference between the first Vsync-HW time after the frame rate control system receives the message carrying the second frame rate sent by the synthesis thread and the time when the frame rate control system receives the message carrying the second frame rate sent by the synthesis thread.

[0282] The second duration is between (buffer quantity*Vsync cycle duration corresponding to the first frame rate) and (buffer quantity+1)*Vsync cycle duration corresponding to the first frame rate.

[0283] In a possible implementation, the delay duration satisfies: (buffer quantity+1)*Vsync cycle duration corresponding to the first frame rate.

[0284] It is understandable that the difference between the rendering and display of each frame image in the terminal device is 2 Vsync cycles. When a buffer pile is added to the buffer queue, the difference between the rendering and display of each frame image is increased by 1.

[0285] Delaying the screen refresh rate by (buffer number + 1) * switching the Vsync cycle duration corresponding to the first frame rate can make the rhythm of the image when drawing and rendering consistent with the rhythm of the frame image when displaying, making the image display speed consistent with the expected speed, thereby reducing jumps and jams.

[0286] S1004: Switch the application frame rate to a second frame rate.

[0287] Specifically, the frame rate control system controls the image synthesis system to switch the application frame rate to the second frame rate.

[0288] It is understandable that the application frame rate is controlled by the Vsync-APP signal. By controlling the time interval between adjacent Vsync-APP signals, the application frame rate is switched to the second frame rate. The adjacent time interval becomes the Vsync cycle duration corresponding to the second frame rate.

[0289] S1005. After a delay period, switch the screen refresh rate to a second frame rate.

[0290] Specifically, after the delay time, the frame rate control system controls the display driver to switch the screen refresh rate to the second frame rate via the image synthesis system.

[0291] It is understandable that the screen refresh rate is controlled by the Vsync-HW signal. By controlling the time interval between adjacent Vsync-HW signals, the screen refresh rate is switched to the second frame rate. The adjacent time interval becomes the Vsync cycle duration corresponding to the second frame rate.

[0292] In a possible implementation, the switching time of the synthesis frame rate is consistent with the switching time of the screen refresh rate.

[0293] It is understandable that the composite frame rate is controlled by the Vsync-SF signal. By controlling the time interval between adjacent Vsync-SF signals, the composite frame rate is switched to the second frame rate. The adjacent time interval becomes the Vsync cycle duration corresponding to the second frame rate.

[0294] In summary, when switching the frame rate, first switch the application frame rate, and then switch the screen refresh rate, so that the display interval between the Mth frame and the M-1th frame is consistent with the drawing rendering interval, and then the display rhythm of the image is consistent with the drawing rendering rhythm, thereby reducing the sliding speed jump caused by the inconsistency between the display interval and the drawing rendering interval, reducing the jamming, and improving the user experience.

[0295] The following describes the situations where the second frame rate is greater than the first frame rate. Fig. 9 and Fig.11 This corresponds to the display process when the second frame rate is greater than the first frame rate.

[0296] For example, Fig. 9 A schematic diagram of an interface display processing flow provided by an embodiment of the present application is shown in FIG. Fig. 9 In the chronological order, the contents displayed by the terminal device correspond to frame 0, frame 1, frame 2, frame 3, frame 4, frame 5 and frame 6.

[0297] like Fig. 9 As shown, when the terminal device is drawing and rendering in frame 3, it receives a pop-up reminder and decides to switch the application frame rate. After switching, the interval of the Vsync-APP signal becomes the Vsync period (8.3ms) corresponding to the second frame rate. During the period of 16.6ms-33.2ms, when the decision to switch the application frame rate is made, the number of caches is 1. After (1+1)*16.6, that is, 33.2ms, the decision to switch the screen refresh rate is made. Therefore, during the period of 49.8ms-66.4ms, the decision to switch the screen refresh rate is made. After switching, the interval of the Vsync-SF signal and the interval of the Vsync-HW signal both become the Vsync period (8.3ms) corresponding to the second frame rate.

[0298] The generation time of the Vsync-APP signal after the decision application frame rate is switched (the timestamp of Vsync-APP) is 33.2ms, 41.5ms, and 49.8ms, respectively. When frame 4 starts drawing and rendering at 33.2ms, the application frame rate has not completed the frame rate switch. Therefore, the displacement of frame 2, the displacement of frame 3, and the displacement of frame 4 are all related to the application frame rate before switching (the first frame rate), which is 2pixel. At 41.5ms, the application frame rate completes the frame rate switch. The displacement of frame 5 and the displacement of frame 6 are both related to the application frame rate after switching (the second frame rate), which is 1pixel.

[0299] The generation time of the Vsync-HW signal after the decision screen refresh rate is switched (the timestamp of Vsync-HW) is 66.4ms, 74.7ms and 83ms respectively; the screen refresh rate does not complete the frame rate switching at 66.4ms, and the sliding speed calculation at 66.4ms is related to the screen refresh rate before switching (the first frame rate). The screen refresh rate completes the frame rate switching at 74.7ms, and the sliding speed calculation at 74.7ms is related to the screen refresh rate after switching (the second frame rate).

[0300] Fig. 9At 16.6ms, the display interface of the terminal device changes from frame 0 to frame 1, and the sliding speed is 2pixel / 16.6ms. At 33.2ms, the display interface of the terminal device changes from frame 1 to frame 2, and the sliding speed is 2pixel / 16.6ms. At 49.8ms, the display interface of the terminal device changes from frame 2 to frame 3, and the sliding speed is 2pixel / 16.6ms. At 66.4ms, the display interface of the terminal device changes from frame 3 to frame 4, and the sliding speed is 2pixel / 16.6ms. At 74.7ms, the display interface of the terminal device changes from frame 4 to frame 5, and the sliding speed is 1pixel / 8.3ms. At 83ms, the display interface of the terminal device changes from frame 5 to frame 6, and the sliding speed is 1pixel / 8.3ms. When the screen switches, the speed is consistent, and the screen displays smoothly without lag.

[0301] For example, Fig.11 A display flow diagram provided in an embodiment of the present application. Fig.11 As shown, in the scenario where the list slides at a uniform speed, the screen refresh rate is switched from 60 Hz to 120 Hz, and the sliding speed is 2 pixels / 16.6 ms. Fig.11 In the chronological order, the contents displayed by the terminal device correspond to frame-1, frame 0, frame 1, frame 2, frame 3, frame 4, frame 5 and frame 6.

[0302] like Fig.11 As shown, when the terminal device is drawing and rendering in frame 3, it decides to switch the application frame rate. After switching, the interval of the Vsync-APP signal becomes the Vsync period (8.3ms) corresponding to the second frame rate. During the period of 16.6ms-33.2ms, when the decision to switch the application frame rate is made, the number of caches is 2. After (2+1)*16.6, that is, 49.8ms, the decision to switch the screen refresh rate is made. Therefore, during the period of 66.4ms-83ms, the decision to switch the screen refresh rate is made. After switching, the interval of the Vsync-SF signal and the interval of the Vsync-HW signal both become the Vsync period (8.3ms) corresponding to the second frame rate.

[0303] The generation time of Vsync-APP after the decision application frame rate switching is 33.2ms, 41.5ms and 49.8ms respectively; when frame 4 starts drawing and rendering at 33.2ms, the application frame rate has not completed the frame rate switching. Therefore, the displacement of frame 2, the displacement of frame 3 and the displacement of frame 4 are all related to the application frame rate before switching (first frame rate), which is 2pixel. At 41.5ms, the application frame rate completes the frame rate switching. The displacement of frame 5 and the displacement of frame 6 are both related to the application frame rate after switching (second frame rate), which is 1pixel.

[0304] The generation time of Vsync-HW after the decision screen refresh rate is switched is 83ms, 91.3ms and 99.6ms; the screen refresh rate does not complete the frame rate switch at 83ms, and the sliding speed calculation at 83ms is related to the screen refresh rate before the switch (first frame rate). The screen refresh rate completes the frame rate switch at 91.3ms, and the sliding speed calculation at 91.3ms is related to the screen refresh rate after the switch (second frame rate).

[0305] Fig.11 At 16.6ms, the display interface of the terminal device changes from frame -1 to frame 0, and the sliding speed is 2pixel / 16.6ms. At 33.2ms, the display interface of the terminal device changes from frame 0 to frame 1, and the sliding speed is 2pixel / 16.6ms. At 49.8ms, the display interface of the terminal device changes from frame 1 to frame 2, and the sliding speed is 2pixel / 16.6ms. At 66.4ms, the display interface of the terminal device changes from frame 2 to frame 3, and the sliding speed is 2pixel / 16.6ms. At 83ms, the display interface of the terminal device changes from frame 3 to frame 4, and the sliding speed is 2pixel / 16.6ms. At 91.3ms, the display interface of the terminal device changes from frame 4 to frame 5, and the sliding speed is 1pixel / 8.3ms. At 99.6ms, the display interface of the terminal device changes from frame 5 to frame 6, and the sliding speed is 1pixel / 8.3ms. When the screen switches, the speed is consistent, and the screen displays smoothly without lag.

[0306] The data processing method of the embodiment of the present application has been described above, and the terminal device for executing the above data processing method provided by the embodiment of the present application is described below. Those skilled in the art can understand that the method and the apparatus can be combined and referenced with each other, and the terminal device provided by the embodiment of the present application can execute the steps in the above data processing method.

[0307] like Fig.12 As shown, Fig.12 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application is shown. The data processing device may be a terminal device in an embodiment of the present application. The data processing device includes: a display screen 1801, the display screen is used to display images; one or more processors 1802; a memory 1803; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory 1803, and the one or more computer programs include instructions, and when the instructions are executed by the data processing device, the data processing device executes the steps in the above-mentioned data processing method.

[0308] Fig.13 A hardware structure diagram of a data processing device provided in an embodiment of the present application. Fig.13, the device includes: a memory 1901, a processor 1902 and an interface circuit 1903. The device may also include a display screen 1904, wherein the memory 1901, the processor 1902, the interface circuit 1903 and the display screen 1904 can communicate; illustratively, the memory 1901, the processor 1902, the interface circuit 1903 and the display screen 1904 can communicate through a communication bus, the memory 1901 is used to store computer execution instructions, the execution is controlled by the processor 1902, and the communication is performed by the interface circuit 1903, so as to realize the data processing method provided in the embodiment of the present application.

[0309] Optionally, the interface circuit 1903 may also include a transmitter and / or a receiver. Optionally, the processor 1902 may include one or more CPUs, or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0310] In possible implementations, the computer-executable instructions in the embodiments of the present application may also be referred to as application code, which is not specifically limited in the embodiments of the present application.

[0311] The data processing device provided in the embodiment of the present application is used to execute the data processing method of the above embodiment. The technical principles and technical effects are similar and will not be repeated here.

[0312] The present application embodiment provides a terminal device, the structure of which is shown in FIG. Figure 1 The memory of the terminal device can be used to store at least one program instruction, and the processor is used to execute at least one program instruction to implement the technical solution of the above method embodiment. Its implementation principle and technical effect are similar to those of the above method related embodiments, and will not be repeated here.

[0313] The embodiment of the present application provides a chip. The chip includes a processor, and the processor is used to call a computer program in a memory to execute the technical solution in the above embodiment. Its implementation principle and technical effect are similar to those of the above related embodiments, and will not be repeated here.

[0314] The embodiment of the present application provides a computer program product, when the computer program product is run in an electronic device, the terminal device executes the technical solution in the above embodiment. Its implementation principle and technical effect are similar to those of the above related embodiments, and will not be repeated here.

[0315] The embodiment of the present application provides a computer-readable storage medium on which program instructions are stored. When the program instructions are executed by a terminal device, the terminal device executes the technical solution of the above embodiment. Its implementation principle and technical effect are similar to those of the above related embodiments, and will not be repeated here.

[0316] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0317] The above specific implementation methods further illustrate the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that: Applied to a terminal device, the terminal device includes an application, a frame rate control system, a synthesis thread, and a cache thread, and the method includes: When the application draws a rendered image at a first frame rate, the frame rate control system receives a message carrying a second frame rate sent by the cache thread, and the second frame rate is greater than the first frame rate; In response to receiving the message carrying the second frame rate, At a first moment, the frame rate control system controls the application to draw a rendered image at the second frame rate; At a second moment, the frame rate control system controls the synthesis thread to synthesize the image drawn and rendered by the application at the second frame rate, and the second moment is later than the first moment; the difference between the second moment and the moment when the frame rate control system receives the message carrying the second frame rate satisfies: (the number of caches cached in the cache queue corresponding to the application + 1) * the length of the Vsync cycle corresponding to the first frame rate.

2. The method according to claim 1, characterized in that The second moment is between the Mth Vsync-HW signal and the M+1th Vsync-HW signal after the frame rate control system receives the message carrying the second frame rate, where M is an integer greater than 1, and the Vsync-HW signal is used to trigger the screen of the terminal device to display the synthesized image.

3. The method according to claim 2, characterized in that The M is 1+A, and A is the number of caches cached in the cache queue corresponding to the application in the cache thread when the frame rate control system receives the message carrying the second frame rate.

4. The method according to claim 1, characterized in that: The number of buffers is the sum of the number of buffers that have been rendered in the buffer queue corresponding to the application and the number of buffers that are being rendered in the buffer queue corresponding to the application.

5. The method according to any one of claims 1 to 4, characterized in that: The terminal device also includes a Vsync thread; The frame rate control system controls the application to draw a rendered image at the second frame rate, including: The frame rate control system sends a first message to the Vsync thread; In response to the first message, the Vsync thread controls the application to draw a rendered image at the second frame rate.

6. The method according to claim 5, characterized in that The Vsync thread controls the application to draw a rendered image at the second frame rate, including: The Vsync thread generates a Vsync-APP signal at the second frame rate; The Vsync thread sends the Vsync-APP signal to the application; In response to receiving the Vsync-APP signal, the application draws a rendered image.

7. The method according to claim 6, characterized in that The application includes a UI thread and a rendering thread, and the application draws a rendered image including: The UI thread draws the image; The rendering thread starts rendering the image when the image drawing is completed.

8. The method according to any one of claims 1 to 4, characterized in that: The terminal device also includes: a Vsync thread; The frame rate control system controls the synthesis thread to synthesize the image rendered by the application at the second frame rate, including: The frame rate control system sends a second message to the Vsync thread; In response to the second message, the Vsync thread synthesizes and draws the rendered image at the second frame rate.

9. The method according to claim 8, characterized in that The Vsync thread synthesizes and draws the rendered image at the second frame rate, including: The Vsync thread generates a Vsync-SF signal at the second frame rate; The Vsync thread sends the Vsync-SF signal to the synthesis thread; In response to the Vsync-SF signal, the composition thread composites and draws the rendered image.

10. The method according to any one of claims 1 to 4, characterized in that: The terminal device further includes a display driver, and the method further includes: At the second moment, the frame rate control system controls the display driver to drive the screen to display the image synthesized by the synthesis thread at the second frame rate.

11. The method according to claim 10, characterized in that The frame rate control system controls the display driver to drive the screen to display the image synthesized by the synthesis thread at the second frame rate, including: The frame rate control system sends a second message to the Vsync thread in the terminal device; In response to the second message, the Vsync thread sends a third message to the display driver; In response to the third message, the display driver controls the screen to display the synthesized image at the second frame rate.

12. The method according to claim 11, characterized in that The display driver controls the screen to display the synthesized image at the second frame rate, comprising: The display driver controls the screen to generate a Vsync-HW signal at the second frame rate; In response to the Vsync-HW signal, the display driver controls the screen to display the synthesized image.

13. The method according to any one of claims 1 to 4, characterized in that: When the application renders the image at the first frame rate, after the frame rate control system receives a message carrying the second frame rate sent by the cache thread, the method further includes: When the second frame rate is greater than the first frame rate, the frame rate control system obtains a cache quantity from a cache queue corresponding to the application, and the cache quantity is used to determine the second moment.

14. The method according to claim 13, characterized in that The frame rate control system obtains the cache quantity of the cache from the cache queue corresponding to the application, including: The frame rate control system obtains a focus window from a window manager, where the focus window corresponds to the application; The frame rate control system obtains the cache quantity from the cache queue corresponding to the application based on the focus window.

15. The method according to any one of claims 1 to 4, characterized in that: A ratio of the second frame rate to the first frame rate is an integer greater than 1.

16. The method according to claim 15, characterized in that When the application renders the image at the first frame rate, after the frame rate control system receives a message carrying the second frame rate sent by the cache thread, the method further includes: The frame rate control system calculates a ratio of the second frame rate to the first frame rate; When the ratio is an integer greater than 1, the frame rate control system obtains the cache quantity of the cache from the cache queue corresponding to the application.

17. The method according to claim 14, characterized in that When the application draws the rendered image at the first frame rate, before the frame rate control system receives the message carrying the second frame rate sent by the cache thread, the method further includes: When the application renders the image at the first frame rate, the cache thread receives a second image after rendering, the second image corresponds to a second window, and the second window is different from the focus window; After receiving the rendered second image, the cache thread sends a message carrying the second frame rate to the frame rate control system.

18. The method according to claim 17, characterized in that The second image corresponds to a pop-up reminder of the terminal device, or the second image corresponds to a screenshot animation of the terminal device.

19. A terminal device, characterized in that: The terminal device comprises a processor, and the processor is used to call a computer program in a memory to execute the method according to any one of claims 1 to 18.

20. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 18.

21. A computer program product, characterized in that The method comprises a computer program, which, when executed, causes a terminal device to execute a method according to any one of claims 1 to 18.

22. A chip, characterized in that: The chip comprises a processor, and the processor is used to call a computer program in a memory to execute the method according to any one of claims 1-18.