Frame rendering time anchoring method, electronic device and storage medium

Abstract

A frame display time anchoring method, an electronic device and a storage medium relate to the technical field of terminals. The method comprises: when a VSYNC signal is detected to have a jump, the terminal device first calculates an expected display time of a target frame through SF, then judges whether the target frame is a first frame and whether the current time is less than the earliest display time, if so, the expected display time is sent to the HWC of the HAL layer, then, according to the expected display time and the period of the VSYNC signal, a time period in which the HWC needs to pause display is calculated, and then the HWC is controlled to perform a sleep operation according to the time period in which the HWC needs to pause display, until the wrong display time is missed, and when the expected display time is reached, the target frame is displayed to the screen. In this way, by only packaging the expected display time at the first frame and delivering it to the HWC to anchor the display time, the anchoring cost can be reduced, the parameter transmission path can be shortened, and the maintainability of the anchoring method can be improved.

Description

Frame display time anchoring method, electronic device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on January 10, 2024, with application number 202410038650.9 and invention name “A frame display time anchoring method, electronic device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a frame transmission and display time anchoring method, electronic equipment and storage medium. Background Art

[0003] With the continuous development of technology, terminal devices represented by mobile phones and tablet computers are increasingly used in people's lives and work, bringing great convenience to people. For example, people can use terminal devices to take photos, watch videos, or play music.

[0004] Currently, when a terminal device's display system (SurfaceFlinger, SF) synthesizes each frame signal, it packages the expected display time of the current frame and transmits it to the display driver for processing. This ensures that the frame display time is anchored, preventing delays and frame drops that affect the user experience. However, this method of anchoring frame display time is costly, has a long transmission path, and is difficult to maintain, resulting in poor frame display time anchoring effectiveness. Summary of the Invention

[0005] In order to solve the above problems, the present application provides a frame display time anchoring method, electronic device and storage medium, the purpose of which is to reduce the anchoring cost of the frame display time, shorten the parameter transmission path, improve the maintainability of the anchoring method, and thereby improve the anchoring effect of the frame display time.

[0006] In the first aspect, the present application provides a frame display time anchoring method, which includes: when a jump in the virtual level VSYNC signal is detected, the terminal device first calculates the expected display time of the target frame through SF, and then determines whether the target frame is the first frame, and whether the current time is less than the earliest display time. If so, the expected display time is sent to the HWC of the HAL layer, and then, based on the expected display time and the period of the VSYNC signal, the time period for which the HWC needs to suspend display is calculated, and then the HWC is controlled to perform a sleep operation according to the time period for which the HWC needs to suspend display until the erroneous display time is missed, and when the expected display time is reached, the target frame is displayed on the screen.

[0007] It can be seen that in the above-mentioned frame display time anchoring method, the expected display time is no longer packaged and transmitted for each frame. Instead, after identifying that the target frame to be displayed is the first frame, the expected display time is packaged only at the first frame, the layer parameters are calculated, and then passed to the HWC, which effectively reduces the anchoring cost of the frame display time and improves the anchoring efficiency. In addition, the expected display time and other parameters are no longer passed to the lower-level display driver for processing. Instead, the parameters are blocked from being sent down in the HWC, the relevant frame display logic in the display driver is removed, and deployed in the HWC, so that the display time can be anchored through the HWC, and the display operation can be further performed to the screen, thereby achieving the optimization of the anchoring expected display time, that is, shortening the parameter transmission path, improving the maintainability of the anchoring method, and thus improving the anchoring effect.

[0008] In one possible implementation, the VSYNC signal is a virtual signal fitted according to the refresh rate of the terminal device display screen; the VSYNC signal is used to control the timing of the display system SF obtaining the layer for preprocessing.

[0009] In a possible implementation, calculating the expected display time of the target frame by the display system SF may include: calculating the expected display time of the target frame by the display system SF according to the VSYNC signal and its continuous working time VysncWorkDuration.

[0010] In one possible implementation, before determining whether the target frame is the first frame and whether the current time is less than the earliest display time, the expected display time can be packaged and assigned to the refresh rate parameter via the display system SF synthesis process interface and then transmitted to the output file. The expected display time is then transmitted to the SF display file and the first synthesized file via the output file selection synthesis strategy interface.

[0011] In one possible implementation, determining whether the target frame is the first frame and determining whether the current time is less than the earliest display time may include: determining whether the target frame is the first frame through the first synthetic file and determining whether the current time is less than the earliest display time.

[0012] In one possible implementation, sending the expected display sending time to the hardware abstraction layer module HWC for window synthesis and display may include: first passing the expected display sending time to the second synthesis file through the first synthesis file; and then passing the expected display sending time to the hardware abstraction layer module HWC for window synthesis and display through the expected display sending time setting interface of the second synthesis file.

[0013] In one possible implementation, the expected display sending time is passed to the hardware abstraction layer module HWC for window synthesis and display through the expected display sending time setting interface of the second synthesis file, which may include: passing the expected display sending time in a preset format to the hardware abstraction layer module HWC for window synthesis and display through the expected display sending time setting interface of the second synthesis file.

[0014] In one possible implementation, the expected display sending time is 64-bit data; the expected display sending time is passed to the hardware abstraction layer module HWC for window synthesis and display through the expected display sending time setting interface of the second synthesis file, which may include: splitting the 64-bit expected display sending time into two 32-bit expected display sending times through the expected display sending time setting interface of the second synthesis file, and passing the two 32-bit expected display sending times to the hardware abstraction layer module HWC for window synthesis and display to provide the accuracy of the transmission of the expected display sending time.

[0015] In one possible implementation, the time period during which the HWC needs to suspend display transmission is calculated based on the expected display transmission time and the period of the virtual level VSYNC signal, which may include: first receiving two 32-bit expected display transmission times through the HWC display file, and recombining the two 32-bit expected display transmission times to obtain the recombined expected display transmission time; then calculating the difference between the recombined expected display transmission time and the period of the virtual level VSYNC signal, and using the obtained time difference as the time period during which the HWC needs to suspend display transmission.

[0016] In a second aspect, the present application provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor is used to call and execute the computer program to implement the frame display time anchoring method described in any one of the first aspects above.

[0017] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor of an electronic device, is used to implement the frame display time anchoring method described in any one of the first aspects above.

[0018] In a fourth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the frame display time anchoring method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG1 is a schematic diagram of a scenario provided by an embodiment of the present application;

[0020] FIG2 is a schematic diagram of frame loss without anchoring the frame transmission and display time according to an embodiment of the present application;

[0021] FIG3 is a schematic diagram showing the frame transmission and display time according to an embodiment of the present application;

[0022] FIG4 is a schematic diagram of the time anchoring of the current frame display transmission according to an embodiment of the present application;

[0023] FIG5 is a schematic diagram of an electronic device provided in an embodiment of the present application;

[0024] FIG6 is a block diagram of the software structure of the electronic device provided in an embodiment of the present application;

[0025] FIG7 is a flow chart of a frame display time anchoring method provided in an embodiment of the present application;

[0026] FIG8 is a signaling diagram of a frame transmission and display time anchoring method provided in an embodiment of the present application;

[0027] FIG9 is a schematic diagram of a frame transmission and display time anchoring principle provided by an embodiment of the present application;

[0028] FIG10 is a schematic diagram of the frame display time anchoring effect provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the specification and claims of this application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless the context clearly indicates otherwise.

[0030] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0031] The "multiple" involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first" and "second" are only used for the purpose of distinguishing the description and cannot be understood as indicating or implying relative importance or order.

[0032] In order to enable people skilled in the art to more clearly understand the solution of the present application, the application scenario of the technical solution of the present application will be first explained.

[0033] Refer to Figure 1, which shows a scenario schematic diagram provided by an embodiment of the present application.

[0034] In this example scenario, when an image is currently displayed on the display screen of any terminal device, the frame display time is usually controlled based on the TE (Tearing Effect Signal) sequence, where TE is a signal sequence determined by the display driver of the terminal device according to the refresh rate of the display screen. Taking the refresh rate of the display screen as 120Hz as an example, when the screen refresh rate is 120Hz, 20 frames need to be refreshed per second on the display screen, that is, the interval between each frame is 8.3 milliseconds, and the TE sequence can be determined. Further, a virtual level vertical synchronization (Vertical Synchronization, VSYNC) signal can be fitted based on the TE sequence to trigger the display system (Surface Flinger, SF) to obtain the layer, and after processing and synthesis by the HWC (hwcomposer) module for window synthesis and display in the Hardware Abstraction Layer (HAL), the screen interface of the terminal device is rendered and displayed through the display driver.

[0035] Among them, Surface Flinger is usually set in the system library of the software framework layer of the terminal device, and is used to obtain the layer rendered by the APP (which can be regarded as the producer) for preprocessing when the VSYNC signal jumps. In some embodiments, Surface Flinger generates a software VSYNC model based on the hardware VSYNC signal sent by the display driver (or the virtual level VSYNC signal fitted according to TE). The software VSYNC model is used to generate a virtual level VSYNC signal according to the VSYNC cycle simulated by the software VSYNC model based on TE when receiving an image display request sent by the application. The VSYNC signal is used to control the rhythm of drawing and rendering of the display screen and layer synthesis, that is, to control the timing of SF obtaining the layer for preprocessing (consumption), that is, to wake up SF to obtain the parameters of each layer for preprocessing, and pass the obtained processing results to the HWC of the HAL layer for further processing and synthesis.

[0036] As shown in Figure 1, it shows the top-down display process of screen synthesis from SF to HWC. When the virtual level VSYNC signal used by Surface Flinger (i.e. Vsync-sf in Figure 1) jumps, SF is triggered to obtain the layer for preprocessing (consumption), and the preprocessed layer parameters and other information are packaged and passed to the two main threads of HWC, and then submitted to the display driver's thread function crtc_commit to complete the display.

[0037] Specifically, as shown in Figure 1, in actual applications, the layers rendered by the APP (producer) are further processed and synthesized by the SF (the corresponding consumer of the producer) before being sent to the screen for display. The timing of the SF obtaining the layer for pre-processing (consumption) is controlled by the virtual level VSYNC (i.e., Vsync-sf in Figure 1). When the VSYNC-sf signal jumps (as shown in Figure 1, a rising or falling edge appears), the SF is awakened. The work content of the SF can be summarized as pre-processing various layer parameters, such as clipping each layer according to parameters such as Z-axis order (Z_order), transparency, size, and position, calculating the visible area of ​​each layer, filtering the layers that will eventually participate in the synthesis, querying the layer synthesis method, etc., and passing the obtained pre-processing results (i.e., pre-processed parameters) to the HWC of the HAL layer for further processing and synthesis. In other words, the actual layer overlay and screen synthesis are completed in the HWC.

[0038] HWC (hwcomposer) is a module of the HAL layer in the Android system that performs layer synthesis and display. It completes the overlay of each layer. It mainly includes but is not limited to two key threads: binder and vendor.qti.hardware.display.composer-service. As shown in Figure 1, the two threads are used to receive the pre-processed parameters sent by SF and submit the overlaid data to the display driver of the driver layer to send it to the screen for display.

[0039] The display driver, a driver layer within the Android system, is responsible for the final compositing and display of layered images. This is accomplished primarily by calling the complete_commit function from the crtc_commit thread. The completion of the complete_commit function is aligned with the TE signal. As shown in Figure 1, this indicates that the currently composed frame has been displayed on the screen, demonstrating that the TE signal determines the timing of the final frame display.

[0040] It's important to note that when a frame is synthesized by the SF, HWC, and the crtc_commit thread in the display driver and ultimately displayed on the screen at the TE signal, the TE signal at which it will be displayed is actually anchored when the frame is synthesized from the SF source. This is because if the display timing is not anchored, the phenomenon shown in Figure 2 will occur: after the Vsync model is adjusted, the synthesized frame will be synthesized immediately after encountering a TE signal. This phenomenon superficially shortens the interval between the display of the previous and next frames, making the display more continuous and compensating for frame drop, but in reality, it simply delays the timing of frame drop. This delay often brings frame drop outside of the animation (such as when a user clicks an app icon on the screen to launch it) into the animation, making it more noticeable to the user and reducing the user experience. Frame drop within the animation is more noticeable, and can cause lag and other issues, such as when the user interacts with app launch / exit animations, swipes up to enter multitasking, and transition animations.

[0041] Therefore, when the display driver sends a frame to the screen, the specific TE signal at which it is sent for display can be calculated based on the VSYNC-sf signal and its continuous working time VysncWorkDuration-sf. As shown in Figure 3, taking the value of VysncWorkDuration-sf as 32 milliseconds as an example, when the VSYNC-sf signal jump triggers SF frame synthesis, it can be calculated based on the value of VysncWorkDuration-sf (i.e., 32 milliseconds) that the frame needs to be sent for display at point ② as shown in Figure 3, rather than point ①.

[0042] Refer to FIG4 , which shows a schematic diagram of the current frame display time anchoring provided by an embodiment of the present application.

[0043] In the current frame display time anchoring process, it can be summarized as follows: every time SF synthesizes a frame, it will package the expected display time of the current frame, and then pass it to HWC and then to the display driver following the synthesis process. Finally, the received expected display time is processed in the display driver to achieve the anchoring of the frame display time. However, the current frame display time anchoring method has two problems: First, the expected display time is packaged and transmitted for each frame, which has a large power consumption overhead for the terminal device. The other is that the current frame display time anchoring method needs to pass the expected display time parameter to the lower-level display driver of the Android system for processing, which not only lengthens the parameter transmission and calculation path, but also leads to poor maintainability of the method due to more business collected at the bottom level, which reduces the anchoring effect of the frame display time.

[0044] To overcome the above technical problems, this application provides a frame display time anchoring method, electronic device, and storage medium. These methods can reduce the cost of anchoring the frame display time, shorten the parameter transmission path, improve the maintainability of the anchoring method, and thus enhance the anchoring effect of the frame display time.

[0045] The frame display time anchoring method provided in the embodiments of the present application can be applied to electronic devices (i.e., terminal devices) such as mobile phones, tablet computers, personal digital assistants (PDAs), desktop computers, laptop computers, notebook computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, and wearable devices.

[0046] In order to enable people skilled in the art to more clearly understand the frame display time anchoring method provided in this application, the hardware architecture and software system architecture of the electronic device are first introduced in detail below.

[0047] Referring to FIG5 , it shows a schematic diagram of an electronic device provided in an embodiment of the present application.

[0048] As shown in Figure 5, the electronic device 500 may include a processor 510, a mobile communication module 520, a wireless communication module 530, a display screen 540, an internal memory 541, a camera 542, an audio module 543, a speaker 543A, a receiver 543B, a microphone 543C, an earphone interface 543D, an antenna group 1 and an antenna group 2.

[0049] It should be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 500. In other embodiments of the present application, the electronic device 500 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0050] The processor 510 may include one or more processing units, for example: the processor 510 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units can be independent devices or integrated into one or more processors. The controller can generate an operation control signal based on the instruction opcode and timing signal to complete the control of instruction fetching and execution, such as the TE signal sequence determined by the refresh rate of the display screen, etc., to control the frame display time.

[0051] Processor 510 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 510 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 510. If processor 510 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of processor 510, and thus improves system efficiency.

[0052] The internal memory 541 can be used to store computer executable program code, which includes instructions. The internal memory 541 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound acquisition function, an image shooting function, etc.), etc. The data storage area can store data created during the use of the electronic device 500 (such as audio data, image data, etc.), etc. In addition, the internal memory 541 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 510 executes various functional applications and data processing of the electronic device 500 by running instructions stored in the internal memory 541 and / or instructions stored in a memory provided in the processor.

[0053] In some embodiments, the internal memory 541 stores instructions for executing a frame display time anchoring method. The processor 510 can execute the instructions stored in the internal memory 541 to implement, when detecting a jump in the virtual level VSYNC signal, the calculation of the expected display time of the target frame through the display system SF, and then determine whether the target frame is the first frame and whether the current time is less than the earliest display time; if so, the expected display time is sent to the HWC. The HWC can then calculate the time period for which the HWC needs to suspend display based on the expected display time and the period of the virtual level VSYNC signal, and control the HWC to perform a sleep operation based on the time period for which the HWC needs to suspend display until the incorrect display time is missed, and then display the target frame on the screen when the correct expected display time is reached.

[0054] The display screen 540 is used to display images, videos, etc., such as displaying a layer image sent to the display via the HWC or display driver. The display screen 540 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 500 may include one or N display screens 540, where N is a positive integer greater than one.

[0055] The camera 542 is used to capture static images or videos. In some embodiments, the electronic device 500 may include 1 or N cameras 542, where N is a positive integer greater than 1.

[0056] Electronic device 500 implements display functionality through a GPU, display screen 540, and an application processor. The GPU is a microprocessor for image processing that connects display screen 540 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 510 may include one or more GPUs that execute program instructions to generate or modify display information.

[0057] The electronic device 500 can implement audio functions such as music playback, recording, and voice input and output through the audio module 543, speaker 543A, receiver 543B, microphone 543C, headphone jack 543D, and application processor.

[0058] The audio module 543 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 543 can also be used to encode and decode audio signals. In some embodiments, the audio module 543 can be provided in the processor 510, or some functional modules of the audio module 543 can be provided in the processor 510.

[0059] The speaker 543A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 500 can listen to music or listen to hands-free calls through the speaker 543A.

[0060] The receiver 543B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 500 receives a call or a voice message, the user can place the receiver 543B close to the ear to hear the voice.

[0061] Microphone 543C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 543C to input the sound signal into the microphone 543C. The electronic device 500 can be provided with at least one microphone 543C. In other embodiments, the electronic device 500 can be provided with two microphones 543C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 500 can also be provided with three, four or more microphones 543C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0062] The headphone jack 543D is used to connect a wired headphone, and does not limit the standard attributes of the interface.

[0063] It should be understood that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 500.

[0064] The wireless communication function of the electronic device 500 can be implemented through the antenna 1, the antenna 2, the mobile communication module 520, the wireless communication module 530, the modem processor and the baseband processor.

[0065] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 500 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0066] The mobile communication module 520 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 500. The mobile communication module 520 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 520 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 520 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 520 can be set in the processor 510. In some embodiments, at least some of the functional modules of the mobile communication module 520 can be set in the same device as at least some of the modules of the processor 510.

[0067] The wireless communication module 530 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., applied to the electronic device 500. The wireless communication module 530 can be one or more devices integrating at least one communication processing module. The wireless communication module 530 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 510. The wireless communication module 530 can also receive the signal to be sent from the processor 510, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0068] In addition, on top of the above components, the electronic device 500 runs an operating system, such as an iOS operating system, an Android operating system, a Windows operating system, etc. Applications can be installed and run on the operating system.

[0069] See FIG6 , which shows a schematic diagram of the software structure of the electronic device provided in an embodiment of the present application.

[0070] The software system of the electronic device 500 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 500.

[0071] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: the application program layer (APK), the application framework layer (Framework), the hardware abstraction layer (HAL), and the driver layer.

[0072] The application layer can include a series of application packages (APPs). As shown in Figure 6, the application package may include applications such as camera, call, navigation, WLAN, Bluetooth, and map. When a user takes the electronic device 500 to take photos, make video calls, record videos, or live stream videos, the camera application can be used as a producer to communicate with the framework layer (specifically, Surface Flinger) to request camera functions and obtain layer data for subsequent processing.

[0073] The application framework layer (framework layer) provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions. As shown in Figure 6, the application framework layer may include a window manager, a notification manager, a resource manager, a Surface Flinger, and a VSYNC signal (i.e., a VSYNC-sf signal) used to trigger Surface Flinger to obtain layer parameters for preprocessing. The application framework layer can be used to implement interaction between camera services and camera APIs. That is, it provides a unified interface that enables different camera hardware to interact with different camera applications. The framework layer is also used to handle many common aspects of camera functions, such as autofocus, exposure control, etc.

[0074] Among them, when a jump in the VSYNC-sf signal is detected, Surface Flinger recognizes that the target frame to be displayed is the first frame, and then only packages the expected display time, transmits and calculates the layer parameters at the first frame, replacing the existing solution of anchoring the corresponding TE for each frame and transmitting and calculating the layer parameters, so as to reduce the anchoring cost of the frame display time.

[0075] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0076] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager can be used to notify users of download completions, message reminders, and so on.

[0077] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0078] In this embodiment, the hardware abstraction layer (HAL) provides a set of standard interfaces so that the framework layer can communicate with camera hardware from various manufacturers without having to understand the underlying hardware details. The hardware abstraction layer (HAL) stores a module HWC (hwcomposer) for window synthesis and display. It should be noted that in order to shorten the transmission path of layer parameters and improve the maintainability of the anchoring method, the present application no longer requires the receiving and processing logic of the relevant layer parameters in the existing display driver. Instead, it blocks the distribution of layer parameters in the HWC, removes the relevant frame display logic in the display driver, and deploys it in the HWC, so that the display delivery time can be anchored through the HWC, thereby replacing the existing solution of anchoring the frame display delivery time in the underlying display driver, and realizing the optimization of anchoring the expected display delivery time, that is, shortening the parameter transmission path and improving the maintainability of the anchoring method.

[0079] In this way, the display of different layers of images on the Android platform can be achieved through the APP of the application layer, the VSYNC-sf signal of the framework layer, and the HWC of the SF and HAL layers, and the anchoring cost of the frame display time is reduced, the parameter transmission path is shortened, the maintainability of the anchoring method is improved, and the anchoring effect of the frame display time is improved.

[0080] The technical solutions involved in the following embodiments can all be implemented in electronic devices having the above-mentioned hardware architecture and software architecture.

[0081] Next, the implementation process of the frame display time anchoring method provided by this application will be introduced in detail:

[0082] As shown in FIG7 , the specific implementation process of the frame display time anchoring method may include the following steps S701-S705:

[0083] S701: When the terminal device detects that the virtual level VSYNC signal jumps, the display system SF calculates the expected display time of the target frame.

[0084] In this embodiment, when an image is displayed on the display screen of any terminal device, the TE sequence is first determined according to the refresh rate of the terminal device display screen, and then a virtual level VSYNC signal (i.e., VSYNC-sf signal) is fitted according to the TE sequence to control the timing of Surface Flinger obtaining the layer for preprocessing (consumption).

[0085] Next, when a jump in the VSYNC signal is detected, that is, when a rising edge or a falling edge of the VSYNC signal is detected, the expected display time (expressed by expectedPresentTime) of the target frame (referring to the frame that currently needs to be synthesized and displayed) is calculated through Surface Flinger. Specifically, when the VSYNC-sf signal jumps, the expected display time of the target frame can be calculated based on the VSYNC signal and its continuous working time VysncWorkDuration (such as 32 milliseconds) when synthesizing the SF frame. For example, as shown in Figure 3, when the VSYNC-sf signal jump triggers SF frame synthesis, the expected display time of the target frame can be calculated at ② based on the value of VysncWorkDuration-sf (that is, 32 milliseconds), that is, it needs to be displayed at ② instead of ①.

[0086] S702: The terminal device determines whether the target frame is the first frame, and determines whether the current time is less than the earliest display time.

[0087] It should be noted that after analyzing the frame display time anchoring process, the applicant discovered that when SFs are continuously synthesized, after the first frame is anchored to the corresponding TE and the display time is determined, subsequent frames will automatically be displayed at the corresponding TE, eliminating the situation where the previous frame's TE is incorrectly anchored. Therefore, anchoring the TE on the first frame can ensure the accuracy of subsequent display times. The existing method of packaging each frame, transmitting the expected display time, and repeatedly calculating the display time for each frame is actually redundant.

[0088] Therefore, in order to reduce the anchoring cost of the frame display time and improve the anchoring efficiency, after the terminal device in this application calculates the expected display time of the target frame through step S701, it can further package the expected display time expectedPresentTime and assign it to the refresh rate parameter (refreshArgs) through the synthesis process interface of the display system SF (expressed by composite()), and pass it to the output file (Output.cpp). Then, through the selection synthesis strategy interface of the output file (Output.cpp) (expressed by chooseCompositionStrategy()), the expected display time expectedPresentTime is passed to the SF display file (expressed by Display.cpp) and the first synthesis file (expressed by HWComposer.cpp). Then, the first synthesis file is used to determine whether the target frame is the first frame, and whether the current time is less than the earliest display time. If so, continue to execute the subsequent step S703. That is, when the synthesis time point of VSYNC-sf is later than the display time of the previous frame, the currently synthesized frame (ie, the target frame) is determined to be the first frame, and the subsequent step S703 is continued. Otherwise, it is determined to be a non-first frame and anchor TE processing is not performed.

[0089] S703: If yes, the expected display sending time is sent to the hardware abstraction layer module HWC for window synthesis and display.

[0090] If the terminal device determines in step S702 that the target frame is the first frame and the current time is less than the earliest presentation time, the expected presentation time can be further transmitted to the second presentation file (represented by Hwc2.cpp) through the first composition file (HWComposer.cpp); and the expected presentation time expectedPresentTime can be transmitted to the HWC of the HAL layer through the expected presentation time setting interface (represented by setExpectedPresentTime()) of the second composition file. Specifically, the expected presentation time expectedPresentTime can be transmitted to the HWC of the HAL layer according to a preset format.

[0091] The specific content of the preset format is not limited and can be determined based on the hardware composition of the HWC. One optional implementation method is that when the expected display time is 64-bit data, the 64-bit expected display time can be split into two 32-bit expected display times through the set expected display time interface of the second composite file, and the two 32-bit expected display times are transmitted to the HWC.

[0092] S704: Calculate the time period during which the HWC needs to suspend display according to the expected display time and the period of the virtual level VSYNC signal.

[0093] It should be noted that in order to shorten the transmission path of layer parameters and improve the maintainability of the anchoring method, this application no longer receives and processes relevant layer parameters in the display driver, but blocks the issuance of layer parameters in the HWC above the display driver, removes the relevant frame display sending logic in the display driver, and deploys it in the HWC to anchor the display sending time through the HWC, thereby replacing the existing solution of anchoring the frame display sending time in the underlying display driver, and then realizing the optimization of anchoring the expected display sending time, which not only shortens the parameter transmission path, but also improves the maintainability of the anchoring method.

[0094] Specifically, after the second synthesized file in the terminal device SF splits the 64-bit expected display time into two 32-bit expected display times and sends them to the HWC, the two 32-bit expected display times can be further received through the HWC display file (represented by hwc_display.cpp) and reassembled to obtain the reassembled expected display time. The difference between the reassembled expected display time and the period of the VSYNC signal is then calculated, and the resulting time difference is used as the time period during which the HWC needs to suspend display, that is, the time period during which it needs to sleep to avoid the incorrect display time.

[0095] S705: According to the time period when the HWC needs to suspend display transmission, the HWC is controlled to perform a sleep operation until the wrong display transmission time is missed, and when the expected display transmission time is reached, the target frame is transmitted to the screen.

[0096] After the terminal device calculates the time period during which the HWC needs to suspend display through step S704, it can further use the time period during which the HWC needs to suspend display to control the HWC to perform a sleep operation until the erroneous display time is missed, and when the expected display time is reached, the target frame is displayed on the screen.

[0097] Specifically, as shown in FIG8 , the specific implementation process of the frame display time anchoring method may include the following steps S801-S810:

[0098] S801: SurfaceFlinger.cpp uses the synthesis process Composite() to pass expectedPresentTime to Output.cpp.

[0099] In this embodiment, when a jump in the VSYNC-sf signal is detected, the SF acquisition layer can be triggered for preprocessing, and the expected presentation time expectedPresentTime of the target frame can be calculated. Then, the expected presentation time expectedPresentTime can be packaged and assigned to the refresh rate parameter (refreshArgs) along with the synthesis process composite() and passed to the output file (Output.cpp).

[0100] S802: Output.cpp calls and executes the function writeCompositionState() to determine the composition state.

[0101] S803: Output.cpp calls and executes the function prepareFrame().

[0102] S804: Output.cpp transmits the expected presentation time expectedPresentTime to Display.cpp via the vendorchooseCompositionStrategy() synthesis strategy interface.

[0103] S805: Display.cpp calls and executes the function getDeviceCompositionChanges() to obtain the terminal device composition strategy and its changes, and transmits the expected presentation time expectedPresentTime to HWComposer.cpp.

[0104] S806: HWComposer.cpp calls the execution function presentOrValidate() to query the currently valid composition strategy from the bottom layer, and transmits the expected presentation time expectedPresentTime to Hwc2.cpp.

[0105] Before transmitting the expected Presentation Time (expectedPresentTime) to Hwc2.cpp, HWComposer.cpp can determine whether the current target frame is the first frame and whether the current time is less than the earliest Presentation Time. If so, and previousPresentFence is equal to SIGNAL_TIME_PENDING, it continues transmitting the expected Presentation Time (expectedPresentTime) to Hwc2.cpp. Otherwise, the current target frame is determined to be non-first frame and anchor TE processing is not performed.

[0106] S807: Hwc2.cpp calls and executes the function setExpectedPresentTime() to set the expected presentation time expectedPresentTime to the lower-layer HWC.

[0107] S808: Hwc2.cpp calls and executes the function setCursorPosition(), and transmits the expected display time expectedPresentTime to hwc_display.cpp.

[0108] S809: The display file hwc_display.cpp in the HWC receives the expected presentation time (expectedPresentTime) passed by Hwc2.cpp through the setCursorPosition() interface and calculates the time required to sleep to miss it: mEarliestPresentTime = expectedPresentTime – vsyncPeriod. vsyncPeriod represents the vsync period. For example, when the display screen refresh rate is 120Hz, the vsync period can be determined to be 8.3 milliseconds, i.e., vsyncPeriod = 8.3 milliseconds.

[0109] It should be noted that, for ease of understanding, the pre-marked erroneous display time may be T0, and the correct expected display time (ie, the expected display time) may be T1, then T0 = T1 - vsyncPeriod.

[0110] S810: The file drm_atomic_req.cpp in the HWC calls the execution interface function Commit() to sleep, and after the time period required to suspend the display during sleep, when the correct expected display time is reached, the subsequent layer switching synthesis display process is continued.

[0111] In summary, as shown in Figure 9, when the synthesis time point of VSYNC-sf is later than the display time of the previous frame, the currently synthesized frame (target frame) is determined to be the first frame (i.e., frame a in Figure 9); otherwise, it is determined to be a non-first frame (i.e., frame b in Figure 9). Furthermore, after identifying the first target frame (i.e., frame a in Figure 9) and anchoring the expected display TE, frame a will be displayed at position ① in Figure 9, frame b will be automatically displayed at position ② in Figure 9, frame c will be automatically displayed at position ③ in Figure 9, and so on. This achieves anchoring of the display time of each frame, thereby reducing the anchoring cost of the frame display time and improving anchoring efficiency.

[0112] As shown in Figure 10, by calculating the display in the HWC and avoiding the first erroneous TE instead of waiting in the display driver, the anchoring expected display time is optimized, which shortens the parameter transmission path, improves the maintainability of the anchoring method, and thus improves the anchoring effect.

[0113] In addition, an embodiment of the present application further provides an electronic device (i.e., a terminal device). For the hardware structure and software framework of the electronic device, please refer to the corresponding descriptions of Figures 5 and 6. The electronic device includes a memory and a processor. The memory stores a computer program, and the processor is used to call and execute the computer program to implement the frame display time anchoring method provided in the above description.

[0114] The present application also provides a computer-readable storage medium in an embodiment, on which a computer program is stored. When the computer program is executed by a processor of a terminal device, it is used to implement the frame display time anchoring method provided in the above description.

[0115] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A frame display time anchoring method, characterized in that, Applied to a terminal device, the method includes: When it is detected that the virtual level VSYNC signal jumps, calculate the expected display time of the target frame through the display system SF; Judge whether the target frame is the first frame, and judge whether the current time is less than the earliest display time; If so, send the expected display time to the hardware abstraction layer module HWC for window composition and display; Calculate the time period during which the HWC needs to pause the display according to the expected display time and the period of the virtual level VSYNC signal; Control the HWC to perform a sleep operation according to the time period during which the HWC needs to pause the display until the wrong display time is missed, and perform an operation of displaying the target frame on the screen when the expected display time is reached.

2. The method according to claim 1, wherein The VSYNC signal is a virtual signal fitted according to the refresh rate of the display screen of the terminal device; the VSYNC signal is used to control the timing for the display system SF to obtain a layer for preprocessing.

3. The method according to claim 1, wherein The calculating the expected display time of the target frame through the display system SF includes: Calculate the expected display time of the target frame through the display system SF according to the VSYNC signal and its continuous working time VysncWorkDuration.

4. The method according to claim 1, wherein Before judging whether the target frame is the first frame and judging whether the current time is less than the earliest display time, the method further includes: Package and assign the expected display time to the refresh rate parameter through the composition process interface of the display system SF and transfer it to the output file; Transfer the expected display time to the SF display file and the first composition file through the select composition strategy interface of the output file.

5. The method according to claim 4, characterized in that, Judging whether the target frame is the first frame and judging whether the current time is less than the earliest display time includes: Judge whether the target frame is the first frame and judge whether the current time is less than the earliest display time through the first composition file.

6. The method according to claim 5, characterized in that, The sending the expected display time to the hardware abstraction layer module HWC for window composition and display includes: Transfer the expected display time to the second composition file through the first composition file; Transfer the expected display time to the hardware abstraction layer module HWC for window composition and display through the set expected display time interface of the second composition file.

7. The method according to claim 6, characterized in that, The transferring the expected display time to the hardware abstraction layer module HWC for window composition and display through the set expected display time interface of the second composition file includes: Transfer the expected display time to the hardware abstraction layer module HWC for window composition and display in a preset format through the set expected display time interface of the second composition file.

8. The method according to claim 7, characterized in that, The expected display time is 64-bit data; the transferring the expected display time to the hardware abstraction layer module HWC for window composition and display through the set expected display time interface of the second composition file includes: The expected display time interface is set through the second synthesis file, the 64-bit expected display time is split into two 32-bit expected display times, and the two 32-bit expected display times are passed to the hardware abstraction layer module HWC for window synthesis and display.

9. The method according to claim 8, characterized in that, Calculating the time period during which the HWC needs to pause display according to the expected display time and the period of the virtual level VSYNC signal includes: Receiving the two 32-bit expected display times through the HWC display file, and reorganizing the two 32-bit expected display times to obtain the reorganized expected display time; Calculating the difference between the reorganized expected display time and the period of the virtual level VSYNC signal, and using the obtained time difference as the time period during which the HWC needs to pause display.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory stores a computer program, and the processor is configured to call and execute the computer program to implement the method according to any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by the electronic device, the method according to any one of claims 1-9 is implemented.

Images