Display method and related device

Abstract

The application provides a display method and related equipment, and relates to the technical field of device display. The display method comprises the following steps: performing drawing rendering on image content of a first image frame to be displayed; determining an offset parameter for enabling image frame synthesis, and determining a first target image frame based on the offset parameter and the first image frame; in the case where it is detected that a display fence of the first target image frame is not released, updating the offset parameter, and determining a second target image frame based on the updated offset parameter and the first image frame; in the case where it is detected that the display fence of the second target image frame is released, performing synthesis on the image content of the first image frame to obtain the first image frame; and performing display of the first image frame. By using the above method, the probability of frame loss of the electronic equipment can be reduced, and the user's equipment use experience can be improved.

Description

Technical Field

[0001] This application relates to the field of device display technology, and in particular to a display method and related equipment. Background Technology

[0002] The fence mechanism is a resource locking mechanism in the operating system, and it also exists in the graphics display process.

[0003] For example, assuming a screen refresh rate of 60Hz, when SurfaceFlinger is compositing image data for the current frame, it checks whether the display fence of the previous frame has been released. If the display fence of the previous frame has been released, SurfaceFlinger proceeds with compositing the image data for the current frame; if the display fence of the previous frame has not been released, SurfaceFlinger abandons compositing the image data for the current frame and waits for the next vertical sync signal to wake it up. In cases of insufficient scheduling capabilities or thread anomalies in the operating system, the display fence of image frames may be released abnormally, potentially leading to frequent frame drops on electronic devices. Summary of the Invention

[0004] In view of the above, it is necessary to provide a display method and related device that can solve the problem of frequent frame dropping in electronic devices when the display fence of an image frame experiences an abnormal release.

[0005] In a first aspect, this application provides a display method, comprising: rendering the image content of a first image frame to be displayed; determining an offset parameter for enabling image frame composition, and determining a first target image frame based on the offset parameter and the first image frame; updating the offset parameter when it is detected that the display fence of the first target image frame has not been released, and determining a second target image frame based on the updated offset parameter and the first image frame; compositing the image content of the first image frame to obtain the first image frame when it is detected that the display fence of the second target image frame has been released; and displaying the first image frame.

[0006] Using the above technical solution, when it is detected that the fence used to enable the composition of the first image frame (the display fence of the first target image frame) has not been released, the offset parameters can be updated. For example, the offset parameters can be updated through the display composition service (SurfaceFlinger), so that the fence used to enable the composition of the first image frame is changed from the display fence of the first target image frame to the display fence of the second target image frame. Since the second target image frame is displayed before the first target image frame, the display fence of the second target image frame will be released earlier than the display fence of the first target image frame, thereby realizing the display fence of the first target image frame. When the fence for displaying the first image frame is not released, SurfaceFlinger synthesizes the image content of the first image frame. Compared with related technologies, when the fence for displaying the first target image frame is not released, SurfaceFlinger needs to wait for the next Vsync-sf signal before attempting to synthesize the image content of the first image frame. This can extend the working time of SurfaceFlinger, improve the fault tolerance of image frame synthesis and display, and make it less prone to frame dropping. Furthermore, since the working time of SurfaceFlinger is extended, the system scheduling frequency can be slowed down. For example, the working frequency of processing chips (such as CPU and GPU) can be appropriately reduced, thereby reducing the power consumption of electronic devices.

[0007] In one possible implementation, the offset parameters for enabling image frame composition are determined, including: determining the offset parameters for enabling image frame composition based on the operating duration of SurfaceFlinger and the period of the vertical synchronization signal.

[0008] Using the above technical solution, the offset parameter used to enable image frame synthesis can be determined based on the working duration of SurfaceFlinger and the period of the vertical synchronization signal. For example, the value of the offset parameter is the integer part of the value obtained by dividing the working duration of SurfaceFlinger by the period of the vertical synchronization signal.

[0009] In one possible implementation, when it is detected that the display fence of the first target image frame has not been released, the offset parameter is updated, including: when SurfaceFlinger detects that the display fence of the first target image frame has not been released within a preset time, the offset parameter is updated, and the preset time is less than the working time of SurfaceFlinger.

[0010] Using the above technical solution, before SurfaceFlinger composites the image content of the first image frame, it can first determine whether the display fence of the first target image frame has been released within a preset time. If the display fence of the first target image frame is not released, it indicates that compositing the image content of the first image frame is not currently allowed. SurfaceFlinger can update the offset parameter to change the fence used to enable the compositing of the first image frame from the display fence of the first target image frame to the display fence of the second target image frame, thereby achieving the compositing of the image content of the first image frame based on the display fence of the second target image frame. The preset time can be configured in advance. The preset time is shorter than the working time of SurfaceFlinger, and to ensure that SurfaceFlinger has enough time to composite the image content, the preset time can be set to a small value relative to the working time of SurfaceFlinger, for example, a preset time of 1ms.

[0011] In one possible implementation, when the display fence of the second target image frame is detected to be released, the image content of the first image frame is composited, including: when SurfaceFlinger detects that the display fence of the second target image frame has been released, it calls the Hardware Hybrid Renderer (HWC) or the Graphics Processing Unit (GPU) to composite the image content of the first image frame.

[0012] Using the above technical solution, when frame drops occur, the method of modifying the offset parameters can be applied to HWC or GPU synthesis, or hybrid synthesis of HWC and GPU.

[0013] In one possible implementation, the display method further includes: abandoning the synthesis of image content for the first image frame if the display fence for the second target image frame is not released.

[0014] Using the above technical solution, if the fence for the display of the second target image frame is not released, it indicates that the image content of the first image frame is not allowed to be synthesized at present. SurfaceFlinger can give up synthesizing the image content of the first image frame and enter sleep mode, for example, waiting for the next Vsync-sf signal to wake it up before trying to synthesize the image content of the first image frame.

[0015] In one possible implementation, the first image frame to be displayed is the interface of an application. If it is detected that the display fence of the first target image frame has not been released, the offset parameter is updated, including: if the application is a preset first application and it is detected that the display fence of the first target image frame has not been released, the offset parameter is updated.

[0016] By adopting the above technical solution, the phenomenon of frame dropping in electronic devices can be reduced by modifying the offset parameter, but it may not be possible to completely avoid frame dropping. For some applications with high real-time requirements (such as game applications), frame dropping will affect the user experience. Therefore, the offset parameter can be configured to be modified only for a preset first application, such as an application with low real-time requirements, to improve the user experience.

[0017] In one possible implementation, the display method further includes: abandoning the synthesis of the image content of the first image frame when the application is a preset second application and the fence for displaying the first target image frame is detected to be not released.

[0018] By adopting the above technical solution, the phenomenon of frame dropping in electronic devices can be reduced by modifying the offset parameter, but it may not be able to completely avoid frame dropping. For some applications with high real-time requirements (such as game applications), frame dropping may still occur after modifying the offset parameter, thus limiting the effect of modifying the offset parameter. Therefore, the offset parameter can be configured not to be applied to the preset second application, such as applications with high real-time requirements. In this case, if SurfaceFlinger detects that the display fence of the first target image frame has not been released, it will directly abandon the synthesis of the image content of the first image frame and wait for the next Vsync-sf signal to wake up before trying to synthesize the image content of the first image frame.

[0019] In one possible implementation, the first preset application is an application in the application whitelist, and the second preset application is an application not in the application whitelist.

[0020] Using the above technical solution, an application whitelist can be built to specify which applications or types of applications use the scheme of modifying offset parameters. The application whitelist can be maintained by developers or users.

[0021] In one possible implementation, when it is detected that the display fence of the first target image frame has not been released, the offset parameter is updated, including: when it is detected that the display fence of the first target image frame has not been released, updating the offset parameter used to enable the synthesis of the first image frame to be displayed and the N image frames after the first image frame to be displayed, wherein the first image frame and the N image frames after the first image frame all belong to the same continuous frame refresh event.

[0022] Using the above technical solution, for a continuous frame refresh event, when compositing an image frame and encountering a situation where the fence is released before the target image frame is displayed, the offset parameter is modified. This ensures that the display fence release detection for that image frame and every subsequent image frame is based on the updated offset parameter. This lengthens the work-duration of the SurfaceFlinger for that image frame and every subsequent image frame, increasing fault tolerance and reducing the likelihood of frame drops. Furthermore, the extended SurfaceFlinger work-duration can slow down the system scheduling frequency; for example, the operating frequency of the processing chip can be appropriately reduced, thereby lowering the power consumption of the electronic device. The offset parameter is determined again in the original manner for the next continuous frame refresh event until another frame drop occurs, at which point the offset parameter is modified again.

[0023] In one possible implementation, updating the offset parameter involves incrementing the offset parameter by one.

[0024] By adopting the above technical solution, by incrementing the offset parameter by one, the fence used to enable the synthesis of the first image frame is changed from the display fence of the first target image frame to the display fence of the previous image frame of the first target image frame. This enables the synthesis of the image content of the first image frame when the display fence of the first target image frame is not released, but the display fence of the previous image frame of the first target image frame is released, which can extend the work-duration of SurfaceFlinger.

[0025] Secondly, this application provides an electronic device, which includes a touch screen, a memory, and a processor; the touch screen, the memory, and the processor are coupled; the memory is used to store program instructions; and the processor is used to read the program instructions stored in the memory to implement the display method of the first aspect described above.

[0026] Thirdly, this application provides a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the display method of the first aspect described above.

[0027] Fourthly, this application provides a chip coupled to a memory in an electronic device, the chip being used to control the electronic device to implement the display method described in the first aspect above.

[0028] Furthermore, the technical effects brought about by the second to fourth aspects can be found in the descriptions of the methods in the above-mentioned method section, and will not be repeated here. Attached Figure Description

[0029] Figure 1 A schematic diagram illustrating the process from drawing and rendering to display of an image frame provided in an embodiment of this application;

[0030] Figure 2 A schematic diagram illustrating the tracking of a fence release when the SF has not waited for the display of the previous image frame, according to an embodiment of this application;

[0031] Figure 3 This is a schematic diagram illustrating the tracking of the SF waiting for the display of the previous image frame to be released according to an embodiment of this application;

[0032] Figure 4A A schematic diagram illustrating the tracking of the work-duration mechanism of an application provided in an embodiment of this application;

[0033] Figure 4B A schematic diagram illustrating the tracking of the SF work-duration mechanism provided in an embodiment of this application;

[0034] Figure 5A A schematic diagram illustrating frame dropping tracking according to an embodiment of this application;

[0035] Figure 5B A schematic diagram illustrating the tracking of frame loss issues according to another embodiment of this application;

[0036] Figure 6 A schematic diagram illustrating the optimization of frame dropping tracking according to an embodiment of this application;

[0037] Figure 7 A schematic diagram illustrating the optimization of frame dropping tracking according to another embodiment of this application;

[0038] Figures 8A-8D A schematic diagram illustrating the process from drawing and rendering to display of an image frame, provided in another embodiment of this application;

[0039] Figure 9 A flowchart illustrating the interaction between APP, SF, HWC and display driver in an electronic device provided in an embodiment of this application;

[0040] Figure 10This is a flowchart of a display method provided in an embodiment of this application;

[0041] Figure 11 This is a hardware architecture diagram of an electronic device provided in one embodiment of this application;

[0042] Figure 12 This is a software architecture diagram of an electronic device provided in one embodiment of this application. Detailed Implementation

[0043] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0044] In the following description, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, words such as "exemplary," "or," and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of words such as "exemplary," "or," and "for example" is intended to present the relevant concepts in a concrete manner.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. It should be understood that, unless otherwise stated, " / " in this application means "or". For example, A / B can mean A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. "At least one" means one or more. "More than one" means two or more. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, and a, b, and c. It should be understood that the order of steps shown in the flowcharts herein can be changed, and some can be omitted.

[0046] To facilitate understanding of the embodiments of this application, the technical terms involved in this application will first be introduced:

[0047] A user interface (UI) is the medium through which an application or operating system interacts and exchanges information with the user. It converts the internal form of information into a form that the user can understand. The user interface is written in specific computer languages ​​such as Java or Extensible Markup Language (XML). This source code is parsed and rendered on the electronic device, ultimately presenting content that the user can recognize. A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operation displayed graphically. It can be visible interface elements displayed on the screen of an electronic device, such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0048] Application (APP): A software program that can perform one or more specific functions. Examples include instant messaging applications, video applications, audio applications, image capture applications, cloud desktop applications, etc.

[0049] Vertical synchronization (Vsync) signals are divided into hardware Vsync signals and software Vsync signals. Hardware Vsync signals are abbreviated as Vsync-hw signals, which can be understood as tearing effect (TE) signals. Software Vsync signals include applied vertical synchronization (Vsync-app) signals and synthesized vertical synchronization (Vsync-sf) signals. For ease of description, Vsync-hw, Vsync-app, and Vsync-sf signals are simplified to Vsync-hw, Vsync-app, and Vsync-sf, respectively.

[0050] Vsync-app and Vsync-sf are generated based on Vsync-hw. Specifically, DispSyncSource.cpp in the SurfaceFlinger (SF) virtualizes Vsync-hw into Vsync-app and Vsync-sf. DispSyncSource.cpp samples Vsync-hw, and the sampling results are input into the software Vsync model. DispSyncSource.cpp runs the software Vsync model to output Vsync-app and Vsync-sf. Vsync-hw can be generated by the display driver based on the refresh rate.

[0051] The process by which electronic devices use Vsync-app, Vsync-sf, and Vsync-hw to draw, render, and display image frames is as follows: Figure 1 As shown. Under the action of Vsync-app, the APP generates image data (this process is the APP's rendering process). Figure 1 In Vsync-sf, `render` refers to image data. After rendering, the image data can be stored in a buffer. Under Vsync-sf, the SF reads the image data from the buffer and sends it to the hardware hybrid renderer (HWComposer, HWC). The HWC composites the image data to generate a single image frame (this process is called SF compositing). The HWC sends the image frame to the display driver, and under Vsync-hw, the display driver sends the image frame to the display screen for display (this process is called HWC display sending). After the display screen completes the display of the image frame, the display driver calls the kernel thread `crtc_commit`. `crtc_commit` releases the fence, which indicates the completion of displaying one image frame.

[0052] like Figure 1 As shown, an image frame goes through three cycles from drawing, rendering to display. The first cycle completes the APP drawing and rendering, the second cycle completes SF compositing, and the third cycle completes HWC display. The periods of the three signals Vsync-app, Vsync-sf, and Vsync-hw can be determined according to the refresh rate of the APP. For example, if the refresh rate of the APP is 120Hz, the period of these three signals is 8.3 milliseconds (ms); if the refresh rate is 90Hz, the period is 11.1 milliseconds (ms); and if the refresh rate is 60Hz, the period is 16.6 ms.

[0053] In some embodiments, two fence mechanisms may exist during the drawing, rendering, and display of an image frame. For example, when an app performs a drawing and rendering task for an image frame, it can register a fence for drawing and rendering in the buffer. If the fence is released, it indicates that the drawing and rendering of the image frame is complete, and only in this case can the SF compositing of the image frame be allowed. Another example is that when the SF is compositing the image data of the current image frame, it also checks whether the display fence of the previous image frame has been released. If the display fence of the previous image frame is released, it indicates that the display of the previous image frame is complete, and only in this case is the SF allowed to composit the image data of the current image frame. If the display fence of the previous image frame is not released, it indicates that the display of the previous image frame is not complete, and the SF abandons the compositing of the image data of the current image frame, waiting for the next Vsync-sf wake-up call before attempting to composit the image data of the current image frame again.

[0054] In some embodiments, the tracking of the fence used for displaying image frames is as follows: Figure 2 and Figure 3 As shown, Figure 2 This illustrates a scenario where Vsync-sf wakes up SF to synthesize image data for the current image frame. If the SF detects that the fence for the previous image frame has not been released, it abandons the synthesis of the current image frame's image data and goes to sleep, waiting for the next Vsync-sf wake-up call. This results in frame dropping. Figure 3 This illustrates the scenario where Vsync-sf wakes up SF to synthesize the image data of the current image frame. It is detected that the fence of the previous image frame has been released, and SF performs the synthesis of the image data of the current image frame.

[0055] In some embodiments, during the process of drawing, rendering, and displaying an image frame, two work-duration mechanisms may exist: the application's work-duration mechanism and the SF's work-duration mechanism. The application's work-duration mechanism can refer to the minimum time interval from when the application starts drawing and rendering the image data of the image frame to when the image data of that image frame is consumed (used by the SF for compositing). The SF's work-duration mechanism can refer to the minimum time interval from when the SF starts compositing the image data of the image frame to when the image frame is displayed.

[0056] Figure 4A This illustrates the application's work-duration mechanism.

[0057] Assuming the first application's work-duration is 16.6ms, for the image data drawn and rendered by the first application (image data stored in a buffer), for example, the image data drawn and rendered at point ①, it will not be consumed until at least 16.6ms later. Figure 4A As shown, after 16.6ms, the SF at ② checks all buffers in the buffer queue (BufferQueue) to determine if the buffer produced at ① has reached its consumption time. The SF can then use that buffer for compositing. If the SF finds that a buffer has not reached its work-duration, the SF will directly abandon its compositing; that is, the SF will not use that buffer for compositing.

[0058] Figure 4B This illustrates SF's work-duration mechanism.

[0059] Assuming SF's work-duration is 15.6ms, meaning the image frame currently synthesized by SF will not be displayed until at least 15.6ms later. Figure 4B As shown, assuming the SF work-duration is 15.6ms, the image frame synthesized at point ① will not be displayed on the screen until at least 15.6ms later, meaning it can be displayed on the screen at point ②.

[0060] In some embodiments, the application's work-duration and the SF's work-duration are generally pre-configured in the operating system and can be configured differently according to the refresh rate of the APP. For example, if the refresh rate of the APP is 120Hz, the application's work-duration is 11.6ms and the SF's work-duration is 10.3ms; if the refresh rate of the APP is 60Hz, the application's work-duration is 16.6ms and the SF's work-duration is 15.6ms.

[0061] In some embodiments, before synthesizing the image data of the current image frame, SF needs to detect whether the display fence of the target image frame has been released. SF will only synthesize the image data of the current image frame if the display driver calls the kernel thread crtc_commit to release the display fence of the target image frame.

[0062] For example, when the SF is woken up by Vsync-sf to perform image data composition for the current image frame, the SF checks whether the display fence of the target image frame has been released. If the fence is not released, the SF waits for a preset time (usually 1ms). If the fence is still not released, the SF abandons the composition of the current image frame's image data (i.e., the SF will not perform image data composition for the current image frame until the display fence of the target image frame is released), and waits for the next Vsync-sf wake-up call before attempting to compose the image data for the current image frame again. In other words, when the SF is woken up by Vsync-sf to perform image data composition for the current image frame, the SF will only perform image data composition for the current image frame after detecting / waiting for the display fence of the target image frame to be released. The target image frame can refer to the i-th frame above the current image frame, where i is a positive integer.

[0063] In some embodiments, the function code for determining the target image frame that SF should wait for is as follows:

[0064] shift = static_cast<size_t> ((mExpectedPresentTime-now) / vsyncPeriod);

[0065] Here, `shift` is the offset parameter. The value of `shift` indicates which frame before the current image frame the target image frame is. For example, if `shift = 0`, it means the target image frame is the previous image frame; if `shift = 1`, it means the target image frame is the second-to-last image frame. For example, if the current image frame is frame n, the previous image frame is frame (n-1), and the second-to-last image frame is frame (n-2), where n is a positive integer. `vsyncPeriod` is the period of the current Vsync signal, `now` is the synthesis time of the image data for the current image frame, and `mExpectedPresentTime` is the expected fastest display time for the current image frame. The value of `(mExpectedPresentTime - now)` is the work-duration of SF. `static_cast`<size_t> () is the floor function.

[0066] That is, the shift value is the integer part of (mExpectedPresentTime-now) divided by the Vsync signal period, which is also the integer part of the work-duration of SF divided by the Vsync signal period.

[0067] For example, if the refresh rate of the app currently displayed on the screen is 60Hz, the Vsync signal period is 16.6ms, and the SF work-duration is 15.6ms, then shift = 0 (rounded down from 15.6 / 16.6). This means that the prerequisite for SF to synthesize the image data of the current image frame is that the fence of the previous image frame is released. If the refresh rate of the app currently displayed on the screen is 120Hz, the Vsync signal period is 8.3ms, and the SF work-duration is 10.3ms, then shift = 1 (rounded down from 10.3 / 8.3). This means that the prerequisite for SF to synthesize the image data of the current image frame is that the fence of the two image frames preceding the current image frame is released.

[0068] In some embodiments, during the display of an APP interface, if SF fails to wait for the release of the display fence of the target image frame and abandons the synthesis of image data for the current image frame, it may be due to insufficient system resources (software / hardware resources) causing system scheduling abnormalities, or abnormalities in the APP's threads.

[0069] like Figure 5A As shown, when the refresh rate of the APP is 60Hz, SF abandons the synthesis of the image data of the current image frame because it cannot wait for the fence of the previous image frame to be released, resulting in frame dropping.

[0070] exist Figure 5A In the process, at point ①, Vsync-sf wakes up SF. Since the display fence of the previous image frame has been released, SF can perform image data synthesis for the first image frame at point ①. At point ②, Vsync-sf wakes up SF. SF can detect that the display fence of the first image frame has not been released, and after waiting for 1ms (starting from wake-up), the display fence of the first image frame is still not released. SF will not synthesize the image data for the second image frame. In this case, SF will abandon the synthesis of the image data for the second image frame and enter sleep mode, waiting for the next wake-up of Vsync-sf to attempt to synthesize the image data for the second image frame, resulting in frame dropping (the synthesis of the image data for the second image frame should have occurred at point ②). At point ③, Vsync-sf wakes up SF. Since the display fence of the first image frame has been released, SF can perform image data synthesis for the second image frame at point ③.

[0071] like Figure 5BAs shown, when the refresh rate of the APP is 120Hz, SF abandons the synthesis of image data for the current image frame because it cannot wait for the fence of the two previous image frames to be released, resulting in frame dropping.

[0072] exist Figure 5B In the process, at point ①, Vsync-sf wakes up SF, and SF successfully performs the synthesis of image data for the first image frame. At point ②, Vsync-sf wakes up SF, and SF successfully performs the synthesis of image data for the second image frame. At point ③, Vsync-sf wakes up SF. SF can detect that the display fence for the first image frame has not been released, and after waiting for 1ms (starting from wake-up), the display fence for the first image frame is still not released. SF will not synthesize image data for the third image frame. In this case, SF will abandon the synthesis of image data for the third image frame and enter sleep mode, waiting for the next wake-up of Vsync-sf to attempt to synthesize image data for the third image frame, thus causing frame dropping (the synthesis of image data for the third image frame should have been performed at point ③).

[0073] like Figure 6 As shown, assuming the refresh rate of the app currently displayed on the screen is 60Hz, the Vsync signal period is 16.6ms, and the SF's work-duration is 15.6ms, we can calculate that shift is 0. This means that the prerequisite for SF to synthesize an image frame is waiting for the display fence of the previous image frame to be released. At point ①, Vsync-sf wakes up SF. Before SF synthesizes the image data of the first image frame, it can detect that the display fence of the previous image frame has not been released. After waiting for 1ms, if the display fence of the previous image frame is still not released, SF can abandon the synthesis of the image data of the first image frame and enter sleep mode, waiting for the next wake-up of Vsync-sf. At point ② (16.6ms after point ①), Vsync-sf wakes up SF. SF can detect that the display fence of the previous image frame has been released. SF can synthesize the image data of the first image frame at point ②, and the synthesized first image frame is displayed on the screen at point ③.

[0074] from Figure 6As can be seen, since the first image frame is displayed on the screen at point ③, even if the SF is woken up at point ① to force the synthesis of the image data of the first image frame, due to the serial execution characteristics of the display driver, the display of the first image frame will still wait on the display driver side. That is, the first image frame will still be displayed at point ③, which is the same as the display effect of the SF synthesizing the image data of the first image frame at point ②. Therefore, waking up the SF at point ① to force the synthesis of the image data of the first image frame achieves the same display effect as waking up the SF at point ② to perform the synthesis of the image data of the first image frame. That is, using the ratio of the SF's work-duration to the current Vsync signal period to determine the parameter shift, although frame dropping may occur, can meet the display requirements of the image frame.

[0075] like Figure 7 As shown, assuming a refresh rate of 60Hz, if the SF (Surfing Frame) fails to wait for the release of the fence for the target image frame, the shift value can be modified. For example, shift' = shift + m, where m is a positive integer. Optionally, m = 1. This means the SF can re-determine the target image frame based on the value of shift', reducing the probability of frame drops. For example, shift = 0, m = 1, the target image frame is the previous frame. Normally, the SF synthesizes the image data of the current image frame while waiting for the release of the fence for the previous image frame. When the SF fails to wait for the release of the fence for the previous image frame, modifying the shift value (shift' = 0 + 1 = 1) changes the target image frame to the two frames above the current image frame. The SF then synthesizes the image data of the current image frame while waiting for the release of the fence for the two frames above the current image frame.

[0076] exist Figure 7 In the middle, by modifying the shift value, it is possible to enable the synthesis of image data for the first image frame when SF is awakened at point ① (the display fence of the two image frames preceding the first image frame has been released), compared to Figure 6 At point ②, the SF is awakened to synthesize the image data of the first image frame. The work-duration of the SF can be extended. In other words, if the SF synthesizes the image data of the first image frame at point ①, compared to synthesizing the image data of the first image frame at point ②, the SF can have more time to perform the synthesis of the image data of the first image frame.

[0077] For example, in Figure 6 In the middle, at point ②, SF is awakened to synthesize the image data of the first image frame. The work-duration of SF is 15.6ms. Figure 7 In the process, SF is woken up at ① to synthesize the image data of the first image frame. The work-duration of SF is 15.6ms + 16.6ms = 32.2ms.

[0078] like Figure 7 As shown, by modifying the shift value, you can make... Figure 7 The work-duration of the image frame at point ① and the corresponding SF for each subsequent image frame is lengthened (from 15.6ms to 32.2ms), allowing more time for system scheduling exception recovery, thread exception recovery, etc., resulting in higher fault tolerance and making it less likely for frame drops to occur. Furthermore, because the work-duration of SF is lengthened, the system scheduling frequency can be slowed down. For example, the operating frequency of processing chips (such as CPU and GPU) can be appropriately reduced, thereby reducing the power consumption of electronic devices.

[0079] like Figure 8A As shown, taking an APP with a refresh rate of 60Hz and a SF work-duration of 15.6ms as an example, the shift value of SF can be calculated to be 0. That is, the premise for SF to synthesize the image data of the current image frame is to wait for the release of the fence of the previous image frame.

[0080] Under the control of Vsync-app, the APP draws and renders the image data of the nth image frame. Figure 8A In this context, "render" refers to image data, and the image data of the nth frame after rendering is stored in the buffer.

[0081] Under the action of Vsync-sf, SF is awakened, and SF waits until the fence of the previous image frame of the nth image frame is released. SF reads the image data of the nth image frame from the buffer to synthesize the nth image frame.

[0082] Under the influence of Vsync-hw, the display driver sends the nth image frame to the display screen for display.

[0083] After the display screen completes the display of the nth image frame, the display driver calls the kernel thread crtc_commit to release the display fence of the nth image frame.

[0084] like Figure 8BAs shown, under the action of Vsync-sf, SF is awakened, but SF does not wait for the fence of the previous image frame of the nth image frame to be released. SF abandons reading the image data of the nth image frame from the buffer and enters sleep mode, waiting for the next Vsync-sf. Under the action of the next Vsync-sf, SF is awakened, and SF waits for the fence of the previous image frame of the nth image frame to be released. SF reads the image data of the nth image frame from the buffer to synthesize the nth image frame. The nth image frame, which should have been synthesized at ①, is actually synthesized at ② and displayed at ③, resulting in a dropped frame.

[0085] like Figure 8C As shown, under the action of Vsync-sf, SF is awakened, and the fence that SF has been waiting for to display the previous image frame of the nth image frame is released. SF modifies the shift value, for example, by incrementing the shift value by one, so that the precondition for compositing the nth image frame is updated from: the fence waiting for the display of the previous image frame of the nth image frame is released to the fence waiting for the display of the two image frames preceding the nth image frame is released. Since the fence for the display of the two image frames preceding the nth image frame has been released, SF reads the image data of the nth image frame from the buffer to compose the nth image frame. That is, the nth image frame can still be composed at ①, although due to the serial execution characteristics of the display driver, on the display driver side, the display of the nth image frame composed at ① will still wait, that is, the nth image frame will still be displayed at ③.

[0086] However, compared to Figure 8B At point ②, SF is activated to synthesize the image data of the nth image frame. Figure 8C The work-duration of the SF shown can be extended. Figure 8B In the middle, at point ②, SF is awakened to synthesize the image data of the nth image frame. The work-duration of SF is 15.6ms. Figure 8C In the process, SF is awakened at ① to synthesize the image data of the first image frame. SF's work-duration can reach 32.2ms (15.6ms + 16.6ms), which has higher fault tolerance and is less likely to cause frequent frame drops.

[0087] like Figure 8DAs shown, under the action of Vsync-sf, SF is awakened, and the fence that SF has been waiting for to display the previous image frame of the nth image frame is released. SF modifies the shift value, for example, by incrementing the shift value by two, so that the prerequisite for compositing the nth image frame is updated from: the fence waiting for the display of the previous image frame of the nth image frame is released to the fence waiting for the display of the three image frames preceding the nth image frame is released. Since the fence for the display of the three image frames preceding the nth image frame has been released, SF reads the image data of the nth image frame from the buffer to compose the nth image frame. That is, the nth image frame can still be composed at ①, although due to the serial execution characteristics of the display driver, on the display driver side, the display of the nth image frame composed at ① will still wait, that is, the nth image frame will still be displayed at ③.

[0088] Moreover, in Figure 8D In the middle, at ①, SF is awakened to synthesize the image data of the first image frame. The work-duration of SF is still 32.2ms, that is, adding two to the shift value has the same effect as adding one to the shift value.

[0089] Similarly, if SF modifies the shift value by adding three, four, etc., it has the same effect as adding one to the shift value.

[0090] In some embodiments, if the SF (Synchronous Frame Controller) fails to wait for the display of the target image frame and abandons the synthesis of image data for the current image frame, resulting in a dropped frame, the electronic device can use a frame interpolation scheme to reduce the probability of frame drops again. The frame interpolation scheme requires an additional vertical synchronization signal (i.e., a vertical synchronization signal not generated according to the screen refresh rate). This additional vertical synchronization signal is used to wake up the SF to synthesize the image data of the current image frame. Adding an additional vertical synchronization signal not only requires providing a false timestamp to wake up the Vsync timer when it is asleep to generate the additional vertical synchronization signal, but the additional vertical synchronization signal may also affect the rhythm at which the subsequently generated vertical synchronization signal (the vertical synchronization signal generated based on the screen refresh rate) wakes up the SF to work. However, the solution in this application reduces the probability of frame drops again by modifying the shift value without involving the addition of an additional vertical synchronization signal.

[0091] In some embodiments, in a scenario of continuous frame refresh, if the SF (Synthetic Frame) abandons the synthesis of image data for the current image frame before waiting for the release of the fence for the display of the target image frame, the shift value can be modified so that the target image frame is re-determined based on the modified shift value for the current image frame and each subsequent image frame, thereby lengthening the work-duration of the SF corresponding to the current image frame and each subsequent image frame to reduce frame drop.

[0092] In some embodiments, continuous frame refresh can refer to the continuous drawing, rendering, compositing, and displaying of multiple consecutive image frames, which together form an animation. For example, an app displayed on a screen can initiate continuous frame refresh in response to user actions. The modified shift value can be configured to be effective only for a segment of continuous frame refresh. That is, during a segment of continuous frame refresh, if the SF (Synchronous Rendering) fails to wait for the target image frame's display fence to be released and abandons compositing the image data of the current image frame, the shift value is modified so that the target image frame is re-determined based on the modified shift value, thereby lengthening the work-duration of the SF corresponding to the current image frame and each subsequent image frame. After the end of this segment of continuous frame refresh, the next segment of continuous frame refresh is still based on "shift = static_cast".<size_t> The shift value is determined by "((mExpectedPresentTime-now) / vsyncPeriod)". If the continuous frame refresh occurs again and the SF fails to wait for the target image frame to be displayed and abandons the synthesis of the image data of the current image frame, the shift value is modified.

[0093] In some embodiments, the electronic device can also construct an application whitelist. The operation of modifying the shift value in SF only applies to applications in the application whitelist. The applications or application types in the application whitelist can be set by the developer or the user, and this application embodiment does not limit this. For example, since game applications have high real-time requirements, modifying the shift value in SF cannot completely prevent frame drops in applications. That is, the applications in the application whitelist may not include game applications. On the other hand, applications such as video playback and instant messaging applications have relatively low real-time requirements. Modifying the shift value in SF can greatly reduce frame drops in applications and improve the user's application experience. Therefore, the applications in the application whitelist may include video playback, instant messaging, and other similar applications.

[0094] To further understand the reduction of frame dropping by modifying the shift value involved in this application, the following will be combined with... Figure 9The software processing flow of the first APP responding to and displaying information when a user interacts with the application interface of the first APP through a swipe operation is described in detail. Figure 9 Taking the display screen of an electronic device showing the application interface of the first APP as an example, the first APP can be an APP that supports swiping operation to update the application interface. The application type of the first APP is not limited in this application example.

[0095] S90: In response to the user's swipe gesture on the application interface of the first APP, the first APP, under the action of Vsync-app, draws and renders the image data of the nth image frame. The rendered image data of the nth image frame can be stored in the buffer.

[0096] In some embodiments, after a user performs a swipe operation on the application interface of the first APP, the first APP can respond to the swipe operation by performing a series of continuous frame refreshes under the action of Vsync-app, so that the first APP can respond to the swipe operation by displaying a series of application interface refresh animations. It is assumed that the series of continuous frame refreshes may include p image frames, where p is a positive integer, and the nth image frame may refer to a certain image frame among the p image frames.

[0097] In some embodiments, the first APP can call the CPU and / or GPU to draw and render the image data of the nth image frame under the action of Vsync-app, where n is a positive integer.

[0098] S91: Under the action of Vsync-sf, SF determines the shift value corresponding to the nth image frame.

[0099] S92: Based on this shift value, detect whether the display fence of the m-th image frame preceding the n-th image frame has been released. m is a positive integer.

[0100] In some embodiments, SF can be implemented using the aforementioned "shift=static_cast".<size_t> The `((mExpectedPresentTime-now) / vsyncPeriod)` parameter determines the shift value corresponding to the nth image frame. After determining the shift value, it can be used to determine which frame before the nth image frame the mth image frame is. For example, if the shift value is 0, the mth image frame is the previous image frame of the nth image frame; if the shift value is 1, the mth image frame is the second image frame before the nth image frame.

[0101] S93: If the SF detects that the display fence of the m-th image frame has been released or released within a preset time, the SF reads the image data of the n-th image frame from the buffer and sends the image data of the n-th image frame to the HWC.

[0102] In some embodiments, the preset time can be set in advance, and this application embodiment does not limit this. For example, the preset time is 1ms.

[0103] In some embodiments, the SF can also send the image data of the nth image frame to the GPU for compositing. This application embodiment illustrates the example of the SF sending the image data of the nth image frame to the HWC for compositing, but is not limited thereto.

[0104] S94: If SF does not wait for the display fence of the m-th image frame to be released within the preset time, SF modifies the calculated shift value.

[0105] S95: Detect whether the display fence of the j-th image frame preceding the n-th image frame has been released based on the modified shift value. j is a positive integer.

[0106] In some embodiments, modifying the calculated shift value by SF can mean incrementing the calculated shift value by one, two, or similar operations. Optionally, modifying the calculated shift value by SF can mean incrementing the calculated shift value by one. For example, if the shift value calculated in step S802 is equal to 0, and the m-th image frame is the previous image frame of the n-th image frame, then the modified shift value is 1, meaning the j-th image frame is the second image frame above the n-th image frame.

[0107] In some embodiments, for the synthesis of each image frame after the nth image frame (from the (n+1)th image frame to the pth image frame), SF can detect whether the fence of the two preceding image frames of that image frame has been released based on the modified shift value.

[0108] S96: If the SF detects that the display fence of the j-th image frame has been released or released within a preset time, the SF reads the image data of the n-th image frame from the buffer and sends the image data of the n-th image frame to the HWC.

[0109] In some embodiments, taking the shift value calculated in step S92 as 0 as an example, for the synthesis of each image frame after the nth image frame (from the (n+1)th image frame to the pth image frame), if the SF detects that the display fence of the two image frames preceding the image frame has been released or released within a preset time, the SF reads the image data of the image frame from the buffer and sends the image data of the image frame to the HWC for synthesis.

[0110] In some embodiments, if the SF does not wait for the display fence of the j-th image frame to be released within a preset time, the SF may abandon the synthesis of the image data of the n-th image frame. That is, the SF will not read the image data of the n-th image frame from the buffer. In this case, the SF may enter sleep mode and wait for the next Vsync-sf to wake it up.

[0111] S97: HWC synthesizes the image data of the nth image frame to generate the nth image frame.

[0112] S98: HWC sends the nth image frame to the display driver.

[0113] S99: Under the action of Vsync-hw, the display driver sends the nth image frame to the display screen for display.

[0114] In some embodiments, after the display screen has finished displaying the nth image frame, the display driver may call the kernel thread crtc_commit to release the display fence of the nth image frame.

[0115] See Figure 10 The image shows a display method provided in an embodiment of this application. The display method can be applied to an electronic device, which includes a display screen. The display method may include:

[0116] S101: In response to the Vsync-app signal, render the image content of the first image frame to be displayed.

[0117] In some embodiments, taking an electronic device with a certain application installed as an example, when using the application, the user may need to refresh the currently displayed application interface. The user can perform an operation on the application interface, such as a swipe or click operation on the application interface, and the application can respond to the operation and refresh the application interface.

[0118] For example, refreshing the application interface involves displaying multiple image frames. The application's thread can respond to the Vsync-app signal to draw and render the image content of one image frame. The following explanation uses the image content of the first image frame as an example of drawing and rendering the image content in response to the Vsync-app signal.

[0119] S102: In response to the Vsync-sf signal, determine the offset parameters for enabling image frame synthesis, and determine the first target image frame based on the offset parameters and the first image frame.

[0120] In some embodiments, the SF can composite the image content of a first image frame in response to a Vsync-sf signal. Before compositing the image content of the first image frame, the SF determines an offset parameter (i.e., the shift value mentioned above) to enable image frame compositing. For example, the SF can determine the offset parameter to enable image frame compositing based on the current work-duration and the period of the Vsync-sf signal. After obtaining the offset parameter to enable image frame compositing, the SF can further determine a first target image frame based on the offset parameter and the first image frame.

[0121] For example, the value of the offset parameter is the integer part of the value obtained by dividing the operating duration of the SF by the period of the Vsync-sf signal.

[0122] For example, if the offset parameter is 0, the first target image frame is the image frame preceding the first image frame; if the offset parameter is 1, the first target image frame is the image frame two images preceding the first image frame.

[0123] S103: Detect whether the fence displayed in the first target image frame has been released.

[0124] In some embodiments, detecting whether the display fence of the first target image frame has been released may refer to whether the SF detects whether the display fence of the first target image frame has been released, or whether it has been released within a preset time.

[0125] S104: When the fence for displaying the first target image frame is released, the image content of the first image frame is synthesized to obtain the first image frame to be displayed.

[0126] In some embodiments, if the fence for displaying the first target image frame is released, it indicates that the image content of the first image frame is currently allowed to be composited. SF can call HWC or GPU to composite the image content of the first image frame to obtain the first image frame to be displayed.

[0127] S105: If it is detected that the fence displayed on the first target image frame has not been released, update the offset parameter, and determine the second target image frame based on the updated offset parameter and the first image frame.

[0128] In some embodiments, if the fence for displaying the first target image frame is not released, indicating that image content composition of the first image frame is currently not permitted, SF can update the offset parameter to change the fence used to enable composition of the first image frame from the fence for displaying the first target image frame to the fence for displaying the second target image frame, thereby enabling image content composition of the first image frame based on the fence for displaying the second target image frame. The preset time can be configured in advance. The preset time is less than the working duration of SurfaceFlinger, and to ensure that SurfaceFlinger has sufficient time for image content composition, the preset time can be set to a small value relative to the working duration of SurfaceFlinger, for example, a preset time of 1ms.

[0129] In some embodiments, updating the offset parameter can be done by incrementing the value of the offset parameter by one.

[0130] In some embodiments, the application can be set as a preset first application, and the offset parameter is updated when the SF detects that the display fence of the first target image frame has not been released. When the application is a preset second application, and the SF detects that the display fence of the first target image frame has not been released, the SF can directly abandon the compositing of the image content of the first image frame, and wait for the next Vsync-sf signal to wake it up before attempting to compose the image content of the first image frame again. The preset first application can be an application in the application whitelist, and the preset second application can be an application not in the application whitelist. The application whitelist can be maintained by the developer or the user of the electronic device.

[0131] In some embodiments, updating the offset parameter may include updating the offset parameter used to enable the composite of the first image frame to be displayed and the N image frames following the first image frame to be displayed, wherein the first image frame and the N image frames following the first image frame all belong to the same consecutive frame refresh event, and N is a positive integer.

[0132] S106: Detect whether the fence displayed on the second target image frame has been released.

[0133] In some embodiments, detecting whether the display fence of the second target image frame has been released may refer to whether the SF detects whether the display fence of the second target image frame has been released, or whether it has been released within a preset time.

[0134] S107: When the fence for displaying the second target image frame is released, the image content of the first image frame is synthesized to obtain the first image frame to be displayed.

[0135] In some embodiments, if the fence for displaying the second target image frame is released, it indicates that the image content of the first image frame is currently allowed to be composited. SF can call HWC or GPU to composite the image content of the first image frame to obtain the first image frame to be displayed.

[0136] S108: In response to the Vsync-hw signal, the first image frame to be displayed is sent for display.

[0137] In some embodiments, the display driver may send the first image frame to be displayed in response to the Vsync-hw signal.

[0138] S109: If the fence for displaying the second target image frame is not released, abandon the synthesis of the image content of the first image frame.

[0139] In some embodiments, if the fence for displaying the second target image frame is not released, it indicates that the image content of the first image frame is not currently allowed to be synthesized. The SF can abandon the synthesis of the image content of the first image frame and enter a sleep state, waiting for the next Vsync-sf signal to wake it up before attempting to synthesize the image content of the first image frame again.

[0140] See Figure 11 As shown, the electronic device 100 involved in the embodiments of this application will be described below. The electronic device in the embodiments of this application may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, and other devices including touch screens. The embodiments of this application do not impose special limitations on the specific form of the electronic device. Please refer to... Figure 11 , Figure 11 This is a schematic diagram of the structure of the electronic device 100 provided in the embodiments of this application.

[0141] Electronic device 100 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0142] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0143] In addition, an operating system runs on top of the aforementioned components. Examples include Apple's iOS operating system, Google's Android open-source operating system, and Microsoft's Windows operating system.

[0144] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors.

[0145] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0146] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instructions or data again, it can directly retrieve them from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves system efficiency.

[0147] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0148] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0149] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0150] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0151] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0152] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0153] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0154] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a minimized display, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1. In this embodiment, the display screen 194 can be a touch screen, that is, the display screen 194 integrates a touch sensor 180K.

[0155] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0156] Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, fifth generation DDR SDRAM is generally called DDR5 SDRAM), etc.

[0157] Non-volatile memory can include disk storage devices and flash memory.

[0158] Flash memory can be classified according to its operating principle, including NOR FLASH, NAND FLASH, 3D NAND FLASH, etc.; according to the level of the storage cell, including single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc.; and according to the storage specification, including universal flash storage (UFS) and embedded multimedia card (eMMC), etc.

[0159] The random access memory can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0160] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110.

[0161] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions.

[0162] The display methods described in the above embodiments can all be implemented in the electronic device 100 having the above hardware structure.

[0163] The operating system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture Android system as an example to illustrate the software structure of an electronic device. Figure 12 This is a software architecture diagram of an electronic device. A layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. Taking the Android system as an example, in some implementations, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer (Framework), the Hardware Abstraction Layer (HAL), and the kernel layer.

[0164] The application layer can include a series of application packages. For example, application packages can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS.

[0165] The application framework layer provides an Application Programming Interface (API) and programming framework for applications within the application layer. The application framework layer includes predefined functions. For example, it may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc. The application framework layer may also include a Service Provider (SF), which includes DispSyncSource.cpp.

[0166] The hardware abstraction layer may include HWC, which can be used to synthesize image data.

[0167] The kernel layer is the layer between hardware and software. The kernel layer may contain display drivers, camera drivers, audio drivers, sensor drivers, etc.

[0168] The kernel layer is the core of the operating system for electronic devices. It is the first layer of software extension based on the hardware, providing the most basic functions of the operating system. It is the foundation for the operating system's operation, responsible for managing system processes, memory, device drivers, files, and network systems, and determining the system's performance and stability. For example, the kernel can determine the timing of an application's operations on certain parts of the hardware.

[0169] The kernel layer includes hardware-dependent programs such as interrupt handlers and device drivers, as well as basic, common, and frequently running modules such as clock management and process scheduling modules, and critical data structures. The kernel layer can be located in the processor or embedded in internal memory.

[0170] This embodiment also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on the electronic device 100, the electronic device 100 performs the aforementioned related method steps to implement the display method in the above embodiment.

[0171] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement the display method described in the above embodiment.

[0172] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the display methods in the above-described method embodiments.

[0173] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0175] The unit described as a separate component may or may not be physically separate. The component shown as a unit can be one physical unit or multiple physical units, that is, it can be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0176] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0177] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0178] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A display method, characterized in that, The method includes: The image content of the first image frame to be displayed is drawn and rendered. Based on the working duration of the SurfaceFlinger display compositing service and the period of the vertical synchronization signal, an offset parameter for enabling image frame compositing is determined, and a first target image frame is determined based on the offset parameter and the first image frame. If it is detected that the display fence of the first target image frame has not been released, the offset parameter is updated based on the fence release of other image frames, and a second target image frame is determined based on the updated offset parameter and the first image frame. The second target image frame is displayed before the first target image frame, and the other image frames are image frames that are displayed before the first target image frame. If the display fence of the second target image frame is released, the image content of the first image frame is synthesized to obtain the first image frame. The first image frame is then displayed.

2. The display method as described in claim 1, characterized in that, The step of updating the offset parameter based on the release of the fence in other image frames when it is detected that the fence displayed in the first target image frame has not been released includes: If the SurfaceFlinger detects that the display fence of the first target image frame has not been released within a preset time, it updates the offset parameter based on the fence release of the other image frames. The preset time is less than the working time of the SurfaceFlinger.

3. The display method as described in claim 1, characterized in that, The step of synthesizing the image content of the first image frame when the display fence of the second target image frame is detected to be released includes: When SurfaceFlinger detects that the display fence of the second target image frame has been released, it invokes a hardware hybrid renderer or graphics processor to composite the image content of the first image frame.

4. The display method as described in claim 1, characterized in that, The method further includes: If the display fence of the second target image frame is not released, the synthesis of the image content of the first image frame is abandoned.

5. The display method as described in claim 1, characterized in that, The first image frame to be displayed is the application's interface. The step of updating the offset parameter based on the release of the fences of other image frames when it is detected that the display fence of the first target image frame has not been released includes: If the application is a preset first application and the fence of the first target image frame is not released, the offset parameter is updated based on the fence release of the other image frames.

6. The display method as described in claim 5, characterized in that, The method further includes: If the application is a preset second application and the fence displaying the first target image frame is not released, the synthesis of the image content of the first image frame is abandoned.

7. The display method as described in claim 6, characterized in that, The preset first application is an application in the application whitelist, and the preset second application is an application that is not in the application whitelist.

8. The display method according to any one of claims 1 to 7, characterized in that, The step of updating the offset parameter based on the release of the fence in other image frames when it is detected that the fence displayed in the first target image frame has not been released includes: If the display fence of the first target image frame is not released, the offset parameter used to enable the synthesis of the first image frame to be displayed and the N image frames after the first image frame to be displayed is updated based on the fence release of the other image frames, where the first image frame and the N image frames after the first image frame all belong to the same continuous frame refresh event.

9. The display method according to any one of claims 1 to 7, characterized in that, Updating the offset parameter includes incrementing the offset parameter by one.

10. An electronic device, characterized in that, The electronic device includes a display screen, a memory, and a processor; The display screen, the memory, and the processor are coupled together; The memory is used to store program instructions; The processor is configured to read the program instructions stored in the memory to implement the display method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the display method as described in any one of claims 1 to 9.

Images