Frame sending and displaying time anchoring method, electronic equipment and storage medium

Abstract

The invention provides a frame sending and displaying time anchoring method, electronic equipment and a storage medium, and relates to the technical field of terminals. The method comprises the following steps: when detecting that a VSYNC signal jumps, terminal equipment firstly calculates the expected sending and displaying time of a target frame through SF, then judges whether the target frame is the first frame and judges whether the current time is shorter than the earliest sending and displaying time, if yes, sends the expected sending and displaying time to an HWC of an HAL layer, and then sends the expected sending and displaying time to the HWC of the HAL layer; the method comprises the following steps of: calculating a time period in which the HWC needs to pause transmission display according to expected transmission display time and a period of a VSYNC signal, further controlling the HWC to carry out sleep operation according to the time period in which the HWC needs to pause transmission display until wrong transmission display time is missed, and carrying out operation of transmitting and displaying a target frame to a screen when the expected transmission display time is reached. In the embodiment of the invention, the expected transmit-display time is only packaged at the first frame and transmitted to the HWC to anchor the transmit-display time, so that the anchoring cost can be reduced, the parameter transmission path can be shortened, and the maintainability of the anchoring method can be improved.

Description

Technical Field

[0001] This application relates to the technical field of terminals, and in particular, to a frame display time anchoring method, an electronic device, and a storage medium. Background Art

[0002] With the continuous development of technology, terminal devices represented by mobile phones and tablet computers are increasingly used in people's lives and work, bringing great convenience to people. For example, people can take pictures, watch videos, or play music through terminal devices.

[0003] Currently, for each frame signal synthesized by the display system (SurfaceFlinger, SF) of a terminal device, the expected display time of the current frame is packaged and transmitted to the display driver for processing to achieve the anchoring of the frame display time, preventing the occurrence of delayed frame loss and affecting the user experience. However, this method of anchoring the frame display time has high costs, a long transmission path, and poor maintainability, resulting in a poor anchoring effect of the frame display time. Summary of the Invention

[0004] To solve the above problems, this application provides a frame display time anchoring method, an electronic device, and a storage medium, aiming to reduce the anchoring cost of the frame display time, shorten the parameter transmission path, improve the maintainability of the anchoring method, and thereby improve the anchoring effect of the frame display time.

[0005] In a first aspect, this application provides a frame display time anchoring method, which includes: when it is detected that the virtual level VSYNC signal changes, the terminal device first calculates the expected display time of the target frame through SF, and then determines whether the target frame is the first frame and whether the current time is less than the earliest display time. If so, the expected display time is sent to the HWC of the HAL layer. Then, according to the expected display time and the period of the VSYNC signal, the time period during which the HWC needs to pause the display is calculated. Furthermore, the HWC can be controlled to perform a sleep operation according to the time period during which the HWC needs to pause the display until the wrong display time is missed, and when the expected display time is reached, the target frame is displayed on the screen.

[0006] It can be seen that in the above frame display time anchoring method, instead of packing and transmitting the expected display time for each frame, after identifying the target frame that needs to be displayed as the first frame, the expected display time is only packed at the first frame, the layer parameters are calculated, and they are transmitted to the HWC, effectively reducing the anchoring cost of the frame display time and improving the anchoring efficiency. Moreover, parameters such as the expected display time are no longer continuously transmitted to the lower-level display driver for processing. Instead, the transmission of parameters is blocked in the HWC, the relevant frame display logic in the display driver is removed, and it is deployed in the HWC, so that the display time can be anchored through the HWC, and then the operation of displaying to the screen is further carried out, thus realizing the optimization of anchoring the expected display time, that is, shortening the parameter transmission path, improving the maintainability of the anchoring method, and further improving the anchoring effect.

[0007] In a possible implementation manner, the VSYNC signal is a virtual signal fitted according to the refresh rate of the display screen of the terminal device; the VSYNC signal is used to control the timing for the display system SF to obtain the layer for preprocessing.

[0008] In a possible implementation manner, calculating the expected display time of the target frame by the display system SF may include: calculating the expected display time of the target frame by the display system SF according to the VSYNC signal and its continuous working time VysncWorkDuration.

[0009] In a possible implementation manner, before determining whether the target frame is the first frame and determining whether the current time is less than the earliest display time, the expected display time may be first packed and assigned to the refresh rate parameter through the composition process interface of the display system SF and transmitted to the output file. Then, through the selected composition strategy interface of the output file, the expected display time is transmitted to the SF display file and the first composition file.

[0010] In a possible implementation manner, determining whether the target frame is the first frame and determining whether the current time is less than the earliest display time may include: determining whether the target frame is the first frame and determining whether the current time is less than the earliest display time through the first composition file.

[0011] In a possible implementation manner, sending the expected display time to the hardware abstraction layer module HWC for window composition and display may include: first transmitting the expected display time to the second composition file through the first composition file; then transmitting the expected display time to the hardware abstraction layer module HWC for window composition and display through the set expected display time interface of the second composition file.

[0012] In a possible implementation, by setting the expected display time interface in the second composition file, the expected display time is passed to the Hardware Abstraction Layer module HWC for window composition and display, which may include: by setting the expected display time interface in the second composition file, the expected display time is passed to the Hardware Abstraction Layer module HWC in a preset format.

[0013] In a possible implementation, the expected display time is 64-bit data; by setting the expected display time interface in the second composition file, the expected display time is passed to the Hardware Abstraction Layer module HWC for window composition and display, which may include: by setting the expected display time interface in the second composition file, the 64-bit expected display time is split into two 32-bit expected display times, and the two 32-bit expected display times are passed to the Hardware Abstraction Layer module HWC for window composition and display to provide the transmission accuracy of the expected display time.

[0014] In a possible implementation, according to the expected display time and the period of the virtual level VSYNC signal, calculating the period during which HWC needs to pause display may include: first, receiving two 32-bit expected display times through the HWC display file, and reorganizing the two 32-bit expected display times to obtain the reorganized expected display time; then calculating the difference between the reorganized expected display time and the period of the virtual level VSYNC signal, and using the obtained time difference as the period during which HWC needs to pause display.

[0015] In a second aspect, the present application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor is used to call and execute the computer program to implement the frame display time anchoring method described in any one of the above first aspects.

[0016] In a third aspect, the present application provides a computer-readable storage medium, on which a computer program is stored. When the program is run by the processor of the electronic device, it is used to implement the frame display time anchoring method described in any one of the above first aspects.

[0017] In a fourth aspect, the present application provides a computer program product. When the computer program product runs on a computer, it causes the computer to execute the frame display time anchoring method described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a schematic diagram of the scenario provided by the embodiment of the present application;

[0019] Figure 2 It is a schematic diagram of dropped frames without anchoring the frame display time provided by the embodiment of the present application;

[0020] Figure 3 Schematic diagram of the frame display time provided by the embodiment of the present application;

[0021] Figure 4 Schematic diagram of the current frame display time anchoring provided by the embodiment of the present application;

[0022] Figure 5 Schematic diagram of the electronic device provided by the embodiment of the present application;

[0023] Figure 6 Software structure block diagram of the electronic device provided by the embodiment of the present application;

[0024] Figure 7 Flowchart of the frame display time anchoring method provided by the embodiment of the present application;

[0025] Figure 8 Signaling diagram of the frame display time anchoring method provided by the embodiment of the present application;

[0026] Figure 9 Schematic diagram of the frame display time anchoring principle provided by the embodiment of the present application;

[0027] Figure 10 Schematic diagram of the frame display time anchoring effect provided by the embodiment of the present application. Detailed implementation manners

[0028] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to limit the present application. As used in the specification and claims of the present application, the singular forms "a", "an", "the", "above", "said", "this" are also intended to include, for example, the expression "one or more", unless there is a clear opposite indication in the context.

[0029] Referring to "one embodiment" or "some embodiments" described in this specification means that specific features, structures, or characteristics described in conjunction with the embodiment are included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in other some embodiments", "in still other embodiments" and the like that appear in different places in this specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variants all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0030] The "multiple" involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, terms such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying an order.

[0031] To enable those skilled in the art to understand the solution of the present application more clearly, the application scenario of the technical solution of the present application will be described first.

[0032] See Figure 1 , which shows a schematic diagram of a scenario provided by the embodiments of the present application.

[0033] In this example scenario, when an image is currently displayed on the display screen of any terminal device, the frame sending and display time is usually controlled based on the TE (Tearing Effect Signal) sequence. Among them, TE is a signal sequence determined by the display driver of the terminal device according to the refresh rate of the display screen. Taking the refresh rate of the display screen as 120Hz as an example, when the screen refresh rate is 120Hz, 20 frames need to be refreshed per second on the display screen, that is, the interval between each frame is 8.3 milliseconds, and then the TE sequence can be determined. Further, a virtual level vertical synchronization (VSYNC) signal can be fitted according to the TE sequence, which is used to trigger the display system (Surface Flinger, SF) to obtain layers, and after being processed and synthesized by the module HWC (hwcomposer) in the Hardware Abstraction Layer (HAL) for window composition and display, the screen interface of the terminal device is rendered and displayed through the display driver.

[0034] Among them, Surface Flinger is usually set in the system library of the software framework layer of the terminal device, and is used to obtain the layers rendered by the APP (which can be regarded as the producer) for preprocessing when the VSYNC signal jumps. In some embodiments, Surface Flinger generates a software VSYNC model based on receiving the hardware VSYNC signal sent by the display driver (or the virtual level VSYNC signal fitted according to TE). This software VSYNC model is used to generate a virtual level VSYNC signal according to the VSYNC cycle simulated by the software VSYNC model based on TE when receiving an image display request sent by the application program. This VSYNC signal is used to control the rhythm of drawing and rendering the display screen and layer composition, that is, to control the timing of SF obtaining layers for preprocessing (consumption), that is, to wake up SF to obtain the parameters of each layer for preprocessing, and pass the obtained processing results to HWC in the HAL layer for further processing and synthesis.

[0035] As shown Figure 1 in the figure, it shows the display sending process of top-down implementation screen composition from SF to HWC. When the virtual level VSYNC signal (i.e., Figure 1 Vsync-sf in [reference]) used by Surface Flinger jumps, it triggers SF to obtain layers for preprocessing (consumption), packs information such as parameters of each preprocessed layer and passes it to two main threads of HWC, and then submits it to the thread function crtc_commit of the display driver to complete the display sending.

[0036] Specifically, as Figure 1 shown in the figure, in practical applications, the layers rendered by the APP (producer) are further processed and synthesized by SF (the consumer corresponding to the producer) before being sent to the screen for display. The timing for SF to obtain layers for preprocessing (consumption) is controlled by the virtual level VSYNC (i.e., Figure 1 Vsync-sf in [reference]) signal. When the VSYNC-sf signal jumps (such as Figure 1 the appearance of a rising edge or a falling edge shown in the figure), it wakes up SF to work. The work content of SF can be summarized as preprocessing various layer parameters. For example, it clips each layer according to parameters such as the order of the Z-axis (Z_order), transparency, size, position, calculates the visible area of each layer, filters the layers finally participating in the composition, queries the layer composition method, etc., and passes the obtained preprocessing results (i.e., the preprocessed parameters) to HWC in the HAL layer for further processing and synthesis. That is, the actual layer superposition and screen composition are completed in HWC.

[0037] And HWC (hwcomposer) is a module in the HAL layer of the Android system for layer screen composition and display. The superposition work of each layer is completed by it; it mainly can include but is not limited to two key threads, binder and vendor.qti.hardware.display.composer-service, as Figure 1 shown in the figure. The two are used to receive the preprocessed parameters sent by SF and submit the superimposed data to the display driver in the driver layer to send to the screen for screen display respectively.

[0038] The display driver is in the driver layer of the Android system, and is used for the final composition and display sending operations of the layer screen. It mainly realizes the display sending by the crtc_commit thread calling the complete_commit function, and the completion timing of the complete_commit function is aligned with the TE signal, as Figure 1As shown, it indicates that a currently synthesized frame is sent to the screen for display, and it also reflects that the TE signal determines the timing of the final frame display.

[0039] It should be noted that during the process of a frame synthesized by SF, HWC, and the thread crtc_commit in the display driver being finally sent to the screen at the TE signal, in fact, the display time of the frame is anchored at the time of frame synthesis at the SF source. This is because if the display time is not anchored, the following phenomenon will occur as Figure 2 shown: When the Vsync model is adjusted, a synthesized frame will be immediately synthesized after encountering a TE signal; this phenomenon seemingly shortens the interval between the display of two consecutive frames, making the display more continuous and compensating for the impact of dropped frames. In fact, it just delays the timing of dropped frames. And this delay usually brings the dropped frames outside the animation effects (such as when the user clicks on any application icon on the screen to start the application) into the animation effects, which is more obviously perceived by the user, reducing the user experience. Moreover, dropped frames within the animation effects are more easily perceived. For example, in scenarios such as the application startup / exit animation effects, swiping up to enter the multitasking mode, and transition animation effects, there may be phenomena such as stuttering.

[0040] Therefore, when the display driver sends a frame to the screen, the specific TE signal at which the frame is sent for display can be calculated based on the VSYNC-sf signal and its continuous working time VysncWorkDuration-sf. As Figure 3 shown, taking the value of VysncWorkDuration-sf as 32 milliseconds as an example, when the VSYNC-sf signal jumps to trigger the SF frame synthesis, the display position of the frame can be calculated according to the value of VysncWorkDuration-sf (i.e., 32 milliseconds) to be at Figure 3 the position ② shown, rather than at the position ①.

[0041] See Figure 4 , which shows a schematic diagram of the current frame display time anchoring provided by the embodiment of the present application.

[0042] In the current frame display time anchoring process, it can be generally summarized that when the SF synthesizes each frame, it will package the expected display time of the current frame, and then follow the synthesis process to be transmitted to the HWC, and then to the display driver. Finally, the received expected display time is processed in the display driver to achieve the anchoring of the frame display time. However, there are two problems with the current frame display time anchoring method: one is that packaging and transmitting the expected display time for each frame incurs a relatively large power consumption overhead on the terminal device. The other is that the current frame display time anchoring method needs to transmit the parameter of the expected display time to the lower-level display driver of the Android system for processing, which not only lengthens the parameter transmission and calculation path, but also results in poor maintainability of the method due to more underlying aggregated services, reducing the anchoring effect of the frame display time.

[0043] To overcome the above technical problems, this application provides a frame display time anchoring method, an electronic device, and a storage medium. It can reduce the anchoring cost of the frame display time, shorten the parameter transmission path, improve the maintainability of the anchoring method, and thus improve the anchoring effect of the frame display time.

[0044] The frame display time anchoring method provided by the embodiments of this application can be applicable to electronic devices (i.e., terminal devices) such as mobile phones, tablet computers, personal digital assistants (PDAs), desktop, laptop, notebook computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, and wearable devices.

[0045] To enable those skilled in the art to more clearly understand the frame display time anchoring method provided by this application, the following first introduces the hardware architecture and software system architecture of the electronic device in detail.

[0046] See Figure 5 , which shows a schematic diagram of the electronic device provided by the embodiments of this application.

[0047] As Figure 5 shown, the electronic device 500 may include a processor 510, a mobile communication module 520, a wireless communication module 530, a display screen 540, an internal memory 541, a camera 542, an audio module 543, a speaker 543A, a receiver 543B, a microphone 543C, a headphone jack 543D, an antenna group 1, and an antenna group 2.

[0048] It can be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 500. In some other embodiments of the present application, the electronic device 500 may include more or fewer components than those illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0049] The processor 510 may include one or more processing units. For example, the processor 510 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors. The controller can generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions. For example, according to the TE signal sequence determined by the refresh rate of the display screen, etc., to control the frame display time.

[0050] A memory may also be provided in the processor 510 for storing instructions and data. In some embodiments, the memory in the processor 510 is a cache memory. This memory can store the instructions or data that the processor 510 has just used or recycled. If the processor 510 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 510, and thus improves the efficiency of the system.

[0051] The internal memory 541 can be used to store computer-executable program code, and the executable program code includes instructions. The internal memory 541 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound collection function, an image capture function, etc.). The data storage area can store data created during the use of the electronic device 500 (such as audio data, image data, etc.). In addition, the internal memory 541 can include high-speed random access memory and can also include non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 510 executes various functional applications and data processing of the electronic device 500 by running the instructions stored in the internal memory 541 and / or the instructions stored in the memory provided in the processor.

[0052] In some embodiments, the instructions stored in the internal memory 541 are for executing the frame display time anchoring method. The processor 510 can, by executing the instructions stored in the internal memory 541, calculate the expected display time of the target frame through the display system SF when detecting a jump in the virtual level VSYNC signal, and then determine whether the target frame is the first frame and whether the current time is less than the earliest display time; if so, send the expected display time to the HWC. So that the HWC calculates the period during which the HWC needs to pause the display according to the expected display time and the period of the virtual level VSYNC signal, and controls the HWC to perform a sleep operation according to the period during which the HWC needs to pause the display until the wrong display time is missed, and perform an operation of displaying the target frame on the screen when the correct expected display time is reached.

[0053] The display screen 540 is used to display images, videos, etc., such as displaying the layer images sent by HWC or the display driver. The display screen 540 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 500 may include one or N display screens 540, where N is a positive integer greater than 1.

[0054] The camera 542 is used to capture still images or videos. In some embodiments, the electronic device 500 may include one or N cameras 542, where N is a positive integer greater than 1.

[0055] The electronic device 500 realizes the display function through the GPU, the display screen 540, and the application processor, etc. The GPU is a microprocessor for image processing, connecting the display screen 540 and the application processor. The GPU is used to execute mathematical and geometric calculations for graphics rendering. The processor 510 may include one or more GPUs, which execute program instructions to generate or change display information.

[0056] The electronic device 500 can realize the audio function through the audio module 543, the speaker 543A, the receiver 543B, the microphone 543C, the headphone jack 543D, and the application processor, etc. For example, the input and output of voices such as music playback and recording.

[0057] The audio module 543 is used to convert digital audio information into an analog audio signal for output, and is also used to convert analog audio input into digital audio signals. The audio module 543 can also be used to encode and decode audio signals. In some embodiments, the audio module 543 may be set in the processor 510, or some functional modules of the audio module 543 may be set in the processor 510.

[0058] The speaker 543A, also known as the "horn", is used to convert an audio electrical signal into a sound signal. The electronic device 500 can listen to music or hands-free calls through the speaker 543A.

[0059] The receiver 543B, also known as the "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 500 answers a call or a voice message, the receiver 543B can be brought close to the human ear to receive the voice.

[0060] The microphone 543C, also known as the "microphone" or "transmitter", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak close to the microphone 543C to input the sound signal into the microphone 543C. The electronic device 500 can be provided with at least one microphone 543C. In some other embodiments, the electronic device 500 can be provided with two microphones 543C, which can not only collect sound signals but also implement a noise reduction function. In some other embodiments, the electronic device 500 can also be provided with three, four or more microphones 543C, which can collect sound signals, reduce noise, identify the sound source, and implement functions such as directional recording.

[0061] The headphone jack 543D is used to connect a wired headphone, and the standard attributes of the interface are not limited.

[0062] It can be understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is only for illustrative purposes and does not constitute a structural limitation on the electronic device 500.

[0063] The wireless communication function of the electronic device 500 can be implemented through antenna 1, antenna 2, the mobile communication module 520, the wireless communication module 530, the modulation and demodulation processor, and the baseband processor, etc.

[0064] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the electronic device 500 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example: Antenna 1 can be multiplexed as the diversity antenna of the wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0065] The mobile communication module 520 can provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the electronic device 500. The mobile communication module 520 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 520 can receive electromagnetic waves through antenna 1, filter, amplify, and perform other processing on the received electromagnetic waves, and then transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 520 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves through antenna 1 and radiate it out. In some embodiments, at least some functional modules of the mobile communication module 520 may be provided in the processor 510. In some embodiments, at least some functional modules of the mobile communication module 520 and at least some modules of the processor 510 may be provided in the same device.

[0066] The wireless communication module 530 can provide solutions for wireless communications applied to the electronic device 500, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. The wireless communication module 530 can be one or more devices integrating at least one communication processing module. The wireless communication module 530 receives electromagnetic waves through antenna 2, performs frequency modulation and filtering processing on the electromagnetic wave signals, and sends the processed signals to the processor 510. The wireless communication module 530 can also receive the signals to be sent from the processor 510, perform frequency modulation and amplification on them, and convert them into electromagnetic waves through antenna 2 and radiate them out.

[0067] In addition, on top of the above components, the electronic device 500 runs an operating system. For example, iOS operating system, Android operating system, Windows operating system, etc. Application programs can be installed and run on the operating system.

[0068] See Figure 6 , which shows a schematic diagram of the software structure of the electronic device provided by the embodiments of the present application.

[0069] The software system of the electronic device 500 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservices architecture, or a cloud architecture. In the embodiments of this application, taking the Android system with a layered architecture as an example, the software structure of the electronic device 500 is exemplarily described.

[0070] The layered architecture divides the software into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom are the application layer (APK), the application framework layer (Framework), the hardware abstraction layer (HAL), and the driver layer.

[0071] The application layer may include a series of application packages (APP). As Figure 6 shown, the application packages may include applications such as cameras, calls, navigation, WLAN, Bluetooth, maps, etc. When the user gets the electronic device 500 to take pictures, make video calls, record videos or conduct video live broadcasts, the camera application can be used as a producer to communicate with the framework layer (specifically, Surface Flinger) to request camera functions and obtain layer data for subsequent processing.

[0072] The application framework layer (framework layer) provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some predefined functions. As Figure 6 shown, the application framework layer may include a window manager, a notification manager, a resource manager, Surface Flinger, and a VSYNC signal (i.e., VSYNC-sf signal) used to trigger Surface Flinger to obtain layer parameters for preprocessing, etc. The application framework layer can be used to implement the interaction between the camera service and the camera API. That is, it provides a unified interface, enabling different camera hardware to interact with different camera applications. The framework layer is also used to handle many general aspects of camera functions, such as autofocus, exposure control, etc.

[0073] Among them, when it is detected that the VSYNC-sf signal jumps, after Surface Flinger identifies that the target frame to be displayed is the first frame, it only packages the expected display time, transfers and calculates the layer parameters at this first frame, instead of the existing scheme of anchoring the corresponding TE for each frame and transferring and calculating the layer parameters, so as to reduce the anchoring cost of the frame display time.

[0074] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0075] The notification manager enables an application to display notification information in the status bar. It can be used to convey messages of the notification type, and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to inform that the download is complete, message reminders, etc.

[0076] The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.

[0077] In this embodiment, the hardware abstraction layer (HAL) provides a set of standard interfaces, enabling the framework layer to communicate with camera hardware of various different manufacturers without the need to understand the underlying hardware details. Modules such as HWC (hwcomposer) for window composition and display are stored in the hardware abstraction layer (HAL). It should be noted that in order to shorten the transmission path of layer parameters and improve the maintainability of the anchoring method in this application, the logic for receiving and processing relevant layer parameters in the existing display driver is no longer required. Instead, the distribution of layer parameters is blocked in HWC, the relevant frame display logic in the display driver is removed, and it is deployed to HWC. Thus, the display time can be anchored through HWC, replacing the existing solution of anchoring the frame display time in the underlying display driver, achieving the optimization of the expected display time for anchoring, that is, shortening the parameter transmission path and improving the maintainability of the anchoring method.

[0078] In this way, through the APP in the application layer, the VSYNC-sf signal and SF in the framework layer, and the HWC in the HAL layer, the display of different layer images on the Android platform can be achieved, reducing the anchoring cost of the frame display time, shortening the parameter transmission path, improving the maintainability of the anchoring method, and thus improving the anchoring effect of the frame display time.

[0079] The technical solutions involved in the following embodiments can all be implemented in an electronic device with the above-mentioned hardware architecture and software architecture.

[0080] Next, the implementation process of the frame display time anchoring method provided in this application will be introduced in detail:

[0081] As Figure 7 shown, the specific implementation process of this frame display time anchoring method may include the following steps S701 - S705:

[0082] S701: When the terminal device detects a jump in the virtual level VSYNC signal, calculate the expected display time of the target frame through the display system SF.

[0083] In this embodiment, when an image is displayed on the display screen of any terminal device, first, a TE sequence is determined according to the refresh rate of the display screen of the terminal device, and then a virtual level VSYNC signal (i.e., the VSYNC-sf signal) is fitted according to the TE sequence, which is used to control the timing for Surface Flinger to obtain a layer for preprocessing (consumption).

[0084] Next, when it is detected that the VSYNC signal jumps, that is, when it is detected that the VSYNC signal has a rising edge or a falling edge, Surface Flinger calculates the expected display time (represented by expectedPresentTime) of the target frame (referring to the frame that needs to be synthesized and sent for display currently). Specifically, when the VSYNC-sf signal jumps, during SF frame synthesis, according to the VSYNC signal and its continuous working time VysncWorkDuration (such as 32 milliseconds), the expected display time of the target frame can be calculated. For example, as Figure 3 shown, when the VSYNC-sf signal jump triggers SF frame synthesis, the expected display time of the target frame at ② can be calculated according to the value of VysncWorkDuration-sf (i.e., 32 milliseconds), that is, it needs to be displayed at ② instead of at ①.

[0085] S702: The terminal device determines whether the target frame is the first frame and whether the current time is less than the earliest display time.

[0086] It should be noted that after the applicant analyzed the process of anchoring the frame display time, it was found that when SF continuously synthesizes frames, after the first frame anchors the corresponding TE and determines the display time, subsequent frames will automatically be displayed at the corresponding TE, and there will no longer be a situation of erroneously anchoring the TE of the previous frame. Therefore, anchoring the TE of the first frame can ensure the accuracy of the subsequent display time, and the existing method of packing and transmitting the expected display time for each frame and repeatedly calculating the display time for each frame is actually redundant.

[0087] Therefore, in order to reduce the anchoring cost of the frame display time and improve the anchoring efficiency, after calculating the expected display time of the target frame through step S701, the terminal device in this application can further first package and assign the expected display time expectedPresentTime to the refresh rate parameter (refreshArgs) through the synthesis process interface of the display system SF (represented by composite()), and transfer it to the output file (Output.cpp). Then, through the select synthesis strategy interface of the output file (Output.cpp) (represented by chooseCompositionStrategy()), the expected display time expectedPresentTime is transferred to the SF display file (represented by Display.cpp) and the first synthesis file (represented by HWComposer.cpp). Next, the first synthesis file is used to determine whether the target frame is the first frame and whether the current time is less than the earliest display time. If so, the subsequent step S703 is continued. That is, when the synthesis time point of VSYNC-sf is later than the display time of the previous frame, the currently synthesized frame (i.e., the target frame) is determined to be the first frame, and the subsequent step S703 is continued. Otherwise, it is determined to be a non-first frame and no anchor TE processing is performed.

[0088] S703: If so, the expected display time is sent to the hardware abstraction layer module HWC for window composition and display.

[0089] If the terminal device determines through step S702 that the target frame is the first frame and the current time is less than the earliest display time, the expected display time can be further transferred to the second synthesis file (represented by Hwc2.cpp) through the first synthesis file (HWComposer.cpp); and through the set expected display time interface of the second synthesis file (represented by setExpectedPresentTime()), the expected display time expectedPresentTime is transferred to the HWC of the HAL layer. Specifically, the expected display time expectedPresentTime can be transferred to the HWC of the HAL layer in a preset format.

[0090] Among them, the specific content of the preset format is not limited and can be determined according to the hardware composition result of the HWC. An optional implementation method is that when the expected display time is 64-bit data, the 64-bit expected display time can be split into two 32-bit expected display times through the set expected display time interface of the second synthesis file, and these two 32-bit expected display times are transferred to the HWC.

[0091] S704: Calculate the time period during which HWC needs to pause the display transmission according to the expected display transmission time and the period of the virtual level VSYNC signal.

[0092] It should be noted that in order to shorten the transmission path of layer parameters and improve the maintainability of the anchoring method, this application no longer receives and processes relevant layer parameters in the display driver. Instead, it blocks the distribution of layer parameters in HWC, which is above the display driver, removes the relevant frame display transmission logic in the display driver, and deploys it to HWC to anchor the display transmission time through HWC, thus replacing the existing solution of anchoring the frame display transmission time in the underlying display driver. Furthermore, it realizes the optimization of the anchored expected display transmission time, not only shortening the parameter transmission path but also improving the maintainability of the anchoring method.

[0093] Specifically, after the second composite file in the terminal device SF splits the 64-bit expected display transmission time into two 32-bit expected display transmission times and sends them to HWC, HWC can further receive these two 32-bit expected display transmission times through the HWC display file (represented by hwc_display.cpp) and recombine them to obtain the recombined expected display transmission time. Then, calculate the difference between the recombined expected display transmission time and the period of the VSYNC signal, and use the obtained time difference as the time period during which HWC needs to pause the display transmission, that is, the time period during which it needs to sleep to avoid incorrect display transmission times.

[0094] S705: Control HWC to perform a sleep operation according to the time period during which HWC needs to pause the display transmission until it misses the incorrect display transmission time, and when the expected display transmission time is reached, perform the operation of sending the target frame to the screen for display.

[0095] After the terminal device calculates the time period during which HWC needs to pause the display transmission through step S704, it can further use this time period during which HWC needs to pause the display transmission to control HWC to perform a sleep operation until it misses the incorrect display transmission time, and when the expected display transmission time is reached, perform the operation of sending the target frame to the screen for display.

[0096] Specifically, as Figure 8 shown, the specific implementation process of the above frame display transmission time anchoring method may include the following steps S801 - S810:

[0097] S801: SurfaceFlinger.cpp uses the composite process Composite() to transfer expectedPresentTime to Output.cpp.

[0098] In this embodiment, when it is detected that the VSYNC-sf signal jumps, the SF can be triggered to obtain the layer for preprocessing, and the expected presentation time expectedPresentTime of the target frame can be calculated. Then, along with the composite process composite(), the expected presentation time expectedPresentTime is packaged and assigned to the refresh rate parameter (refreshArgs) and passed to the output file (Output.cpp).

[0099] S802: Output.cpp calls the execution function writeCompositionState() to determine the composition state.

[0100] S803: Output.cpp calls the execution function prepareFrame().

[0101] S804: Output.cpp transmits the expected presentation time expectedPresentTime to Display.cpp through the selection of the composite strategy interface vendorchooseCompositionStrategy().

[0102] S805: Display.cpp calls the execution function getDeviceCompositionChanges() to obtain the terminal device composite strategy and its changes, and transmits the expected presentation time expectedPresentTime to HWComposer.cpp.

[0103] S806: HWComposer.cpp calls the execution function presentOrValidate() to query the currently valid composite strategy from the bottom layer, and transmits the expected presentation time expectedPresentTime to Hwc2.cpp.

[0104] Among them, before transmitting the expected presentation time expectedPresentTime to Hwc2.cpp, HWComposer.cpp can determine whether the current target frame is the first frame, and determine whether the current time is less than the earliest presentation time. If so, and when previousPresentFence is equal to SIGNAL_TIME_PENDING, continue to transmit the expected presentation time expectedPresentTime to Hwc2.cpp. Otherwise, it is determined that the current target frame is not the first frame, and no anchor TE processing is performed.

[0105] S807: Hwc2.cpp calls the execution function setExpectedPresentTime() to set the expected presentation time expectedPresentTime into the underlying HWC.

[0106] S808: Hwc2.cpp calls the execution function setCursorPosition() to transfer the expected presentation time expectedPresentTime to hwc_display.cpp.

[0107] S809: The display file hwc_display.cpp in HWC receives the expected presentation time expectedPresentTime passed by Hwc2.cpp through the interface setCursorPosition() and calculates the time to sleep to miss: mEarliestPresentTime = expectedPresentTime - vsyncPeriod. Here, vsyncPeriod represents the vsync period. For example, when the refresh rate of the display screen is 120Hz, the value of the vsync period can be determined to be 8.3 milliseconds, that is, vsyncPeriod = 8.3 milliseconds.

[0108] It should be noted that for easy understanding, the pre-marked incorrect presentation time can be set as T0, and the marked correct expected presentation time (i.e., the expected presentation time) as T1, then T0 = T1 - vsyncPeriod.

[0109] S810: The file drm_atomic_req.cpp in HWC calls the execution interface function Commit() to perform sleep, and after the period when sleep needs to pause the presentation, when it reaches the correct expected presentation time, it continues to execute the subsequent composite presentation process of layer switching.

[0110] In summary, as Figure 9 shown, when the composite time point of VSYNC - sf is later than the presentation time of the previous frame, the currently composite frame (target frame) is determined as the first frame (i.e., Figure 9 frame a in Figure 9 ), otherwise, it is determined as a non - first frame (i.e., Figure 9 frame b in Figure 9 ). And after identifying the target frame as the first frame (i.e., Figure 9 frame a in Figure 9 ) and anchoring the expected presentation TE, frame a will be presented at ① in Figure 9 , frame b will be automatically presented at ② in Figure 9 , frame c will be automatically presented at ③ in Figure 9 , and so on, to achieve the anchoring of the presentation time of each frame, thereby reducing the anchoring cost of the frame presentation time and improving the anchoring efficiency.

[0111] In addition, as Figure 10 shown, by calculating the display sending in the HWC and avoiding the first incorrect TE instead of waiting in the display driver, the optimization of the anchored expected display sending time is also achieved, that is, the parameter transmission path is shortened, the maintainability of the anchoring method is improved, and thus the anchoring effect is improved.

[0112] In addition, the embodiment of the present application also provides an electronic device (i.e., a terminal device). For the hardware structure and software framework of the electronic device, reference can be made to Figure 5 and Figure 6 the corresponding descriptions. The electronic device includes a memory and a processor. The memory stores a computer program, and the processor is configured to call and execute the computer program to implement the frame display sending time anchoring method provided in the above description.

[0113] The embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is run by the processor of the terminal device, it is used to implement the frame display sending time anchoring method provided in the above description.

[0114] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A frame display time anchoring method, characterized in that, Applied to a terminal device, the method includes: When it is detected that the virtual level VSYNC signal changes, calculate the expected display time of the target frame through the display system SF; Judge whether the target frame is the first frame, and judge whether the current time is less than the earliest display time; If so, send the expected display time to the hardware abstraction layer module HWC for window composition and display; Calculate the time period during which the HWC needs to pause the display according to the expected display time and the period of the virtual level VSYNC signal; Control the HWC to perform a sleep operation according to the time period during which the HWC needs to pause the display until the wrong display time is missed, and perform an operation of displaying the target frame on the screen when the expected display time is reached.

2. The method according to claim 1, wherein The VSYNC signal is a virtual signal fitted according to the refresh rate of the display screen of the terminal device; the VSYNC signal is used to control the display system SF to obtain the timing for preprocessing the layer.

3. The method according to claim 1, characterized in that, The calculating the expected display time of the target frame through the display system SF includes: Calculate the expected display time of the target frame through the display system SF according to the VSYNC signal and its continuous working time VysncWorkDuration.

4. The method according to claim 1, wherein Before judging whether the target frame is the first frame and judging whether the current time is less than the earliest display time, the method further includes: Package and assign the expected display time to the refresh rate parameter through the composition process interface of the display system SF and transfer it to the output file; Transfer the expected display time to the SF display file and the first composition file through the selection composition strategy interface of the output file.

5. The method according to claim 4, characterized in that Judging whether the target frame is the first frame and judging whether the current time is less than the earliest display time includes: Judge whether the target frame is the first frame and judge whether the current time is less than the earliest display time through the first composition file.

6. The method according to claim 5, wherein The sending the expected display time to the hardware abstraction layer module HWC for window composition and display includes: Transfer the expected display time to the second composition file through the first composition file; Transfer the expected display time to the hardware abstraction layer module HWC for window composition and display through the set expected display time interface of the second composition file.

7. The method according to claim 6, characterized in that, The transferring the expected display time to the hardware abstraction layer module HWC for window composition and display through the set expected display time interface of the second composition file includes: Transfer the expected display time to the hardware abstraction layer module HWC for window composition and display in a preset format through the set expected display time interface of the second composition file.

8. The method according to claim 7, characterized in that, The expected display time is 64-bit data; the transferring the expected display time to the hardware abstraction layer module HWC for window composition and display through the set expected display time interface of the second composition file includes: The expected display time interface is set through the second composite file, and the 64-bit expected display time is split into two 32-bit expected display times, and the two 32-bit expected display times are passed to the Hardware Abstraction Layer module HWC for window composition and display.

9. The method according to claim 8, characterized in that, Calculating the period during which the HWC needs to pause display according to the expected display time and the period of the virtual level VSYNC signal includes: Receiving the two 32-bit expected display times through the HWC display file, and reorganizing the two 32-bit expected display times to obtain the reorganized expected display time; Calculating the difference between the reorganized expected display time and the period of the virtual level VSYNC signal, and using the obtained time difference as the period during which the HWC needs to pause display.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory stores a computer program, and the processor is configured to call and execute the computer program to implement the method according to any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by the electronic device, the method according to any one of claims 1-9 is implemented.

Images