Display method and apparatus

Abstract

The application provides a display method and device. The display method comprises: determining that the refresh rate of an electronic device is switched from a first refresh rate to a second refresh rate, the first refresh rate being greater than the second refresh rate; in response to the refresh rate being switched from the first refresh rate to the second refresh rate, generating a second image frame after the electronic device completes display of a first image frame; and displaying the second image frame, so as to solve the frame loss problem that occurs when switching from a high refresh rate to a low refresh rate, reduce the sense of lag of the electronic device, and improve user experience.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a display method and apparatus. Background Technology

[0002] Electronic devices can install various applications (APPs). During the operation of an APP, the display of image frames is controlled by the refresh rate corresponding to that APP. Some APPs have different refresh rates, and the refresh rate changes synchronously when switching between APPs. However, when switching from a high refresh rate to a low refresh rate, frame drops occur, causing noticeable lag and degrading the user experience. Summary of the Invention

[0003] This application provides a display method and apparatus, aiming to solve the frame dropping problem that occurs when switching from a high refresh rate to a low refresh rate, thereby reducing the stuttering sensation of electronic devices and improving the user experience. To achieve the above objective, this application provides the following technical solution:

[0004] In a first aspect, this application provides a display method applied in an electronic device, the method comprising: determining that the refresh rate of the electronic device switches from a first refresh rate to a second refresh rate, wherein the first refresh rate is greater than the second refresh rate; responding to the switch from the first refresh rate to the second refresh rate, generating a second image frame after the electronic device has completed the display of a first image frame; and displaying the second image frame.

[0005] Normally, for an electronic device to generate an image frame, two conditions must be met: one is that the level change of the first signal (Vsync-sf) meets a preset condition, which is either a change from low to high (corresponding to a rising edge) or a change from high to low (corresponding to a falling edge); the other condition is that the electronic device has completed the display of the previous image frame. However, when the electronic device switches from a first refresh rate to a second refresh rate, the latency of the second signal (Vsync-hw) increases (i.e., the period increases). The increased latency of the second signal can overwrite at least one of the first signals. The end of the second signal indicates that the electronic device has completed the display of an image frame. Under the overwritten first signal, the electronic device cannot determine whether the display of an image frame has been completed, making it unable to process other image frames, resulting in image frame loss. After determining that the display of an image frame has been completed, it must wait for the rising or falling edge of the first signal, and the waiting process may also cause the electronic device to lose image frames. The display method provided in this application, when determining that the refresh rate is switching from the first refresh rate to the second refresh rate, generates and displays the second image frame after the electronic device has completed the display of the first image frame. Thus, the electronic device does not need to wait for the rising or falling edge of the first signal, and generates the image frame immediately after determining that the display of an image frame has been completed, reducing the loss of image frames, thereby reducing the lag of the electronic device and improving the user experience.

[0006] Optionally, the method further includes: after the refresh rate is switched to the second refresh rate, aligning the first signal with the second signal, wherein the first signal is used to indicate the generation of an image frame and the second signal is used to indicate the display of an image frame; generating a third image frame after the electronic device completes the display of the second image frame and the level change of the first signal meets a preset condition, wherein the preset condition is that the level of the first signal changes from a low level to a high level, or the level of the first signal changes from a high level to a low level; and displaying the third image frame under the action of the second signal.

[0007] Alignment refers to the fact that the periods of the first and second signals are the same, and that their rising edges and falling edges are aligned, meaning the waveforms of the first and second signals are identical, although their amplitudes can be the same or different. During the refresh rate switch to the second refresh rate, the period of the second signal generated by the electronic device increases, causing the first and second signals, which were aligned before the switch, to lose alignment after the refresh rate switch. In this case, image frames can be generated once every first signal interval, resulting in the electronic device merging one frame and losing one frame. Merging one frame can mean generating one image frame and then displaying it. The display method provided in this application can control the alignment of the first and second signals after the refresh rate switch, allowing the electronic device to generate one image frame under each first signal, preventing the merging and loss of frames, reducing image frame loss, thereby reducing the stuttering sensation of the electronic device and improving the user experience.

[0008] Optionally, before determining that the refresh rate of the electronic device switches from a first refresh rate to a second refresh rate, the method further includes: responding to the first refresh rate, generating a (i+n)th first signal based on the i-th second signal, wherein the interval between the (i+n)-th first signal and the i-th second signal is the sum of the periods of n first signals, where n is a natural number greater than 1; after the refresh rate switches to the second refresh rate, controlling the alignment of the first and second signals includes: sampling the first second signal after the refresh rate switch, and generating a first signal using the sampling result of the first second signal, wherein the interval between the first signal and the first second signal is the sum of the periods of (n-1) second signals. The reason why the first and second signals are not aligned after the refresh rate switch is because a periodically increasing second signal was used in the process of generating the first signal. Therefore, in this application, after determining the refresh rate switch, the electronic device adjusts the generation mechanism of the first signal after the switch, using the first first signal after the refresh rate switch to generate the second signal, so that the first and second signals can be aligned after the refresh rate switch. The sampling results of the second signal can be stored in the result sequence. When the first signal is generated, the sampling results of the second signal are read from the result sequence. The sampling results of the second signal can be multiple timestamps of the second signal, and the result sequence can be a timestamp sequence storing the multiple timestamps of the second signal.

[0009] Optionally, the method also includes: discarding a special signal, which is a second signal generated during the process of switching the refresh rate from the first refresh rate to the second refresh rate, and the special signal is the signal preceding the first second signal, thereby reducing the amount of data processed by the electronic device.

[0010] Optionally, discarding special signals may include: prohibiting the sampling of special signals; or, sampling special signals but not storing the sampling results of the special signals in the result sequence.

[0011] Optionally, the second image frame is an image frame lost during the process of switching the refresh rate from the first refresh rate to the second refresh rate. In some examples, after determining that the first image frame display is complete, the first image frame of the lost multi-frame image can be generated immediately to ensure image continuity; in other examples, after determining that the first image frame display is complete, the last image frame of the lost multi-frame image can be synthesized immediately to shorten the delay time of the last image frame. The next image frame displayed by the electronic device is the image frame after the lost last image frame. Immediately generating and displaying the lost last image frame ensures that the next displayed image frame and the lost last image frame are continuous, thus guaranteeing continuity.

[0012] Optionally, the electronic device completes the display of the first image frame by: after the electronic device calls the kernel thread crtc_commit to release the fence resources, it determines that the display of the first image frame is complete.

[0013] Optionally, determining whether the refresh rate of the electronic device switches from the first refresh rate to the second refresh rate includes: determining the end time and the start time of the first signal when the refresh rate of the electronic device is the first refresh rate; determining the difference between the end time and the start time; and if the difference meets a preset switching condition, determining that the refresh rate of the electronic device switches from the first refresh rate to the second refresh rate. The preset switching condition is used to indicate an increase in the period of the first signal. For example, the preset switching condition can be a preset threshold or a preset value range. The values ​​of the preset threshold and the preset value range are determined according to the refresh rate. For example, when switching from 90Hz to 60Hz, the preset threshold can be a value less than 19.4ms, and the preset value range can be (11.1, 19.4).

[0014] Optionally, the application framework layer of the electronic device includes: a refresh rate processing unit and a data reading unit; the hardware abstraction layer of the electronic device includes a hardware hybrid renderer; the kernel layer of the electronic device includes a display driver; the refresh rate processing unit is used to determine when the refresh rate of the electronic device switches from a first refresh rate to a second refresh rate; the data reading unit is used to read the first image data after the electronic device completes the display of the first image frame in response to the refresh rate switching from the first refresh rate to the second refresh rate; the hardware hybrid renderer is used to synthesize the first image data to generate a second image frame; and the display driver is used to display the second image frame.

[0015] Optionally, the application architecture layer further includes: a signal generation unit, used to control the alignment of a first signal and a second signal after the refresh rate is switched to a second refresh rate, wherein the first signal is used to indicate the generation of an image frame and the second signal is used to indicate the display of an image frame; a data reading unit, further used to read the second image data after the electronic device completes the display of the second image frame and the level change of the first signal meets a preset condition, wherein the preset condition is that the level of the first signal changes from a low level to a high level, or the level of the first signal changes from a high level to a low level; a hardware hybrid renderer, further used to synthesize the second image data to generate a third image frame; and a display driver, further used to display the third image frame under the action of the second signal.

[0016] Secondly, this application provides an electronic device, which includes: one or more processors; one or more memories; the memories store one or more programs, and when one or more programs are executed by the processor, the electronic device performs the above-described display method.

[0017] Thirdly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the above-described display method. Attached Figure Description

[0018] Figure 1 This is a schematic diagram illustrating the process of rendering and displaying an image frame using Vsync-app, Vsync-sf, and Vsync-hw, as provided in this application.

[0019] Figure 2 A schematic diagram illustrating the generation of Vsync-hw for the display driver provided in this application;

[0020] Figure 3 A schematic diagram illustrating the tracking of frame loss issues provided in this application;

[0021] Figure 4 An enlarged schematic diagram of the frame dropping problem provided in this application;

[0022] Figure 5 A schematic diagram of software / hardware Vsync after a refresh rate switch provided in this application;

[0023] Figure 6 This is a schematic diagram illustrating the process of rendering an image frame to its display using Vsync-app, Vsync-sf, and Vsync-hw after a refresh rate switch, as provided in this application.

[0024] Figure 7 A schematic diagram illustrating the optimized tracking for frame dropping issues provided in this application;

[0025] Figure 8 An enlarged schematic diagram illustrating the optimization of the frame dropping problem provided in this application;

[0026] Figure 9 A schematic diagram illustrating the optimization of software / hardware Vsync after a refresh rate switch provided in this application;

[0027] Figure 10 This is a schematic diagram illustrating the process of rendering and displaying an image frame using the optimized Vsync-app, Vsync-sf, and Vsync-hw provided in this application.

[0028] Figure 11 Hardware architecture diagram of the electronic device provided in this application;

[0029] Figure 12 A software architecture diagram of the electronic device provided in this application;

[0030] Figure 13 Signaling diagram of the display method provided in this application;

[0031] Figure 14 and Figure 15 A schematic diagram of the timestamp sequence and predicted Vsync-sf provided for this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions "a," "an," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the embodiments of this application, "one or more" refers to one, two, or more; "and / or" describes the relationship between related objects, indicating that three relationships may exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0033] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0034] The "multiple" mentioned in the embodiments of this application refers to two or more. It should be noted that in the description of the embodiments of this application, terms such as "first" and "second" are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance, nor should they be construed as indicating or implying order.

[0035] First, let's introduce the technical terms used in this application:

[0036] Vertical synchronization (Vsync) signals are divided into hardware Vsync signals and software Vsync signals. Hardware Vsync signals are abbreviated as Vsync-hw signals, which can be understood as tearing effect (TE) signals. Software Vsync signals include Vsync-app signals and Vsync-sf signals. For ease of description, Vsync-hw, Vsync-app, and Vsync-sf signals are simplified to Vsync-hw, Vsync-app, and Vsync-sf, respectively.

[0037] Vsync-app and Vsync-sf are generated based on Vsync-hw. Specifically, DispSyncSource.cpp in SF (SurfaceFlinger) virtualizes Vsync-hw into Vsync-app and Vsync-sf. DispSyncSource.cpp samples Vsync-hw, and the sampling results are input into the software Vsync model. DispSyncSource.cpp runs the software Vsync model to output Vsync-app and Vsync-sf. Vsync-hw can be generated by the display driver based on the refresh rate.

[0038] The process by which electronic devices render and display image frames using Vsync-app, Vsync-sf, and Vsync-hw is as follows: Figure 1 As shown. Under the action of Vsync-app, the APP generates image data (this process is the APP rendering process). Figure 1 In Vsync-sf, `render` refers to image data, which is stored in a buffer. Under Vsync-sf, the SF reads the image data from the buffer and sends it to the hardware blending renderer (HWC). The HWC then composites the image data to generate a single image frame (this process is called SF compositing). The HWC sends the image frame to the display driver, which, under Vsync-hw, displays the image frame on the screen (this process is called HWC display delivery). After the image frame is displayed on the screen, the display driver calls the kernel thread `crtc_commit`. `crtc_commit` releases the fence resources, indicating the completion of displaying one image frame.

[0039] However, the display of one image frame is not entirely independent of the previous image frame. After the previous image frame has finished displaying and crtc_commit has released the fence resources, SF can read image data under the action of Vsync-sf. Figure 1 In this process, a single image frame goes through three cycles from rendering to display. The first cycle completes the APP rendering, the second cycle completes the SF compositing, and the third cycle completes the HWC display.

[0040] The periods of the three signals Vsync-app, Vsync-sf, and Vsync-hw can be determined based on the refresh rate of the app. For example, if the refresh rate of the app is 90Hz, the period of these three signals is 11.1 milliseconds (ms); if the refresh rate of the app is 60Hz, the period of these three signals is 16.6ms. When an electronic device switches apps, the refresh rate also switches synchronously. However, when the electronic device switches from a high refresh rate to a low refresh rate, an excessively long period of Vsync-hw can cause frame drops during app switching. Furthermore, after the switch is completed, Vsync-app and Vsync-sf are not aligned with Vsync-hw, resulting in alternating occurrences of synthesized and dropped frames after the switch. This causes noticeable lag in electronic devices and degrades the user experience. Synthesizing a frame refers to synthesizing a single image frame using SF, while dropping a frame refers to delaying SF synthesis, such as performing SF synthesis on the rising or falling edge of the next Vsync-sf to synthesize the image frame data in the next Vsync-sf. Alternatively, dropping a frame may mean discarding an image frame.

[0041] Taking the refresh rate of an electronic device switching from 90Hz to 60Hz as an example, the display driver in the electronic device can generate Vsync-hw. When the refresh rate changes, the display driver can simulate the refresh rate switch and adjust Vsync-hw according to the new refresh rate. (See diagram below.) Figure 2 As shown, at a refresh rate of 90Hz, the display driver generates a Vsync-hw with a duration of 11.1ms. This Vsync-hw includes a low level with a duration of 8.3ms and a high level with a duration of 2.8ms. The display driver determines that the refresh rate switch occurs on the rising edge of Vsync-hw, and the refresh rate switches from 90Hz to 60Hz. The display driver uses the rising edge as the start of a Vsync-hw, which includes a high level lasting 2.8ms and a low level lasting 16.6ms. The duration of this Vsync-hw is 19.4ms. Therefore, the display driver's mechanism of simulating Vsync-hw at 90Hz results in a Vsync-hw lasting 19.4ms when the refresh chip switches from 90Hz to 60Hz. This phenomenon not only causes the SF to lose 1 to 2 image frames because it cannot wait for fence resources, but also causes the rising and falling edges of Vsync-sf and Vsync-hw to be continuously misaligned, resulting in alternating situations of combining one frame and losing one frame, bringing obvious stuttering.

[0042] Tracing the frame dropping problem, such as Figure 3 As shown, when switching from 90Hz to 60Hz, there is a Vsync-hw that takes as long as 19.4ms. Figure 3 In this context, TE = 19.4ms indicates that within 19.4ms, SF did not wait for crtc_commit to release the fence resources, and the SF compositing process could not execute, resulting in the loss of two image frames within 19.4ms. Figure 3 As shown in 1 and 2. Where Figure 3 Enlarged diagrams of 1 and 2 in the diagram are as follows: Figure 4 As shown, the process from `onMessageReceived` to `HwcPresentOrValidateDisplay` indicates normal execution of SF composition; if SF continues to wait, it means that SF has not waited for `crtc_commit` to release the fence resources, and there is a possibility of frame dropping. Figure 4 In the middle, crtc_commit only releases the fence resources after SF loses two frames.

[0043] On the rising and falling edges of Vsync-sf, if SF waits until crtc_commit releases the fence resource, SF can read image data. If SF does not wait for crtc_commit to release the fence resource, SF cannot read image data. After 19.4ms, SF waits for the fence resource once every Vsync-sf interval, indicating that SF can read image data once every Vsync-sf interval, resulting in an alternating situation of synthesizing one frame and dropping one frame. Figure 3 As shown in Figure 3, the large box pointed to by 3 indicates alternating periods of frame synthesis and frame loss. The small squares within the large box indicate frame loss. For the first small square pointing to the large box, because the SF did not wait for crtc_commit to release the fence resource on the rising edge of Vsync-sf, the SF did not read image data, resulting in the loss of one image frame. Then, on the falling edge of Vsync-sf, the SF waits for crtc_commit to release the fence resource, allowing the SF to read image data and synthesize it by HWC. Therefore, on the falling edge of Vsync-sf, one image frame is synthesized. After the SF loses 5 image frames, the electronic device begins normal synthesis at 60Hz.

[0044] For example, when an electronic device swipes to exit an app, the refresh rate switches from 90Hz to 60Hz. During this switch, the device loses two frames. After the switch is complete, it refreshes one frame and loses another, resulting in a stuttering experience and a poor user experience.

[0045] After the refresh rate changes, the software / hardware Vsync is illustrated as follows: Figure 5 As shown, it illustrates a schematic diagram of software / hardware Vsync switching from 90Hz to 60Hz. Figure 5 The vertical line can serve as a reference for software / hardware Vsync, pointing to the start time of software / hardware Vsync.

[0046] At a refresh rate of 90Hz, the three signals Vsync-app, Vsync-sf, and Vsync-hw have the same period, and at least in each period, the reference values ​​for Vsync-sf and Vsync-hw are also the same. That is, Vsync-sf and Vsync-hw are aligned in each period. Alignment means that the periods are the same and the rising and falling edges of Vsync-sf and Vsync-hw are aligned. For Vsync-app, the period is the same as Vsync-sf and Vsync-hw, but the rising and falling edges can differ slightly from Vsync-sf and Vsync-hw. Figure 5Vsync-app is aligned with Vsync-sf and Vsync-hw. However, when switching the refresh rate from 90Hz to 60Hz, a longer Vsync-hw process occurs, such as... Figure 5 In the fourth cycle, Vsync-hw takes 19.4ms, meaning that a longer Vsync-hw occurs in the fourth cycle, with its cycle increasing from 11.1ms to 19.4ms. However, the cycles of Vsync-app and Vsync-sf remain at 11.1ms in the fourth cycle. Therefore, the cycle of Vsync-hw in the fourth cycle is different from that of Vsync-app and Vsync-sf, resulting in a different baseline for Vsync-hw in the fifth cycle compared to that of Vsync-app and Vsync-sf in the fourth cycle. Starting from the fifth cycle, Vsync-app, Vsync-sf, and Vsync-hw are not aligned.

[0047] like Figure 5 As shown, although the period of Vsync-app, Vsync-sf and Vsync-hw is 16.6ms when the refresh rate is switched to 60Hz, the longer time of Vsync-hw in the fourth period causes Vsync-hw, which originally ended at (1), to end at (2). This results in the period of Vsync-hw in the fourth period being different from the period of Vsync-app and Vsync-sf in the fourth period. The end of Vsync-hw in the fourth period is also different from the end of Vsync-app and Vsync-sf in the fourth period. Consequently, starting from the fifth period, the actual Vsync-app, Vsync-sf and Vsync-hw are continuously misaligned.

[0048] Furthermore, because Vsync-hw ends at (2) in the fourth cycle, the release of fence resources, originally scheduled for (1) by crtc_commit, is delayed until (2). Since (2) is neither the rising nor falling edge of Vsync-sf, SF synthesis is delayed by one Vsync-sf, i.e. Figure 5 As shown, at (3), a frame is lost because the SF misses the fence resource. The SF synthesis that was originally performed at (3) is actually performed at (4).

[0049] exist Figure 5 The process of rendering and displaying an image frame using Vsync-app, Vsync-sf, and Vsync-hw, as shown in the software / hardware Vsync documentation, is as follows: Figure 6 As shown, from Figure 6It can be seen that before the refresh rate switches from 90Hz to 60Hz, the app completes rendering in one cycle, and during this cycle, `crtc_commit` can release fence resources. This means that SF can wait for fence resources in the next cycle and perform SF compositing in the next cycle. After SF compositing, the display driver sends the image to the screen in the next cycle, allowing the screen to display the image frame in the next cycle, thus completing the rendering and display of the image frame within three cycles. However, when switching the refresh rate from 90Hz to 60Hz, the time taken for `crtc_commit` to release fence resources increases, such as... Figure 6 As shown in (1), the fence resources that should have been released at the end of the fourth cycle were released after the start of the fifth cycle, causing the SF synthesis of the fifth cycle to be delayed to the sixth cycle. That is, the fence resources were missed at (1), and the SF synthesis that should have been performed at (1) was actually performed at (2). The delay in SF synthesis resulted in the loss of one frame. Similarly, the fence resources were missed at (3), and the SF synthesis that should have been performed at (3) was actually performed at (4), resulting in the loss of one frame. That is, instead of generating one image frame under each Vsync-app, after the refresh rate switch, it becomes generating one image frame every Vsync-app interval.

[0050] From the above Figures 3 to 6 It is known that when an electronic device switches from a high refresh rate app to a low refresh rate app, the excessively long Vsync-hw cycle causes frame drops during the app switching process. After the switch, Vsync-app and Vsync-sf are not aligned with Vsync-hw, resulting in alternating instances of combining one frame and dropping another after the switch, causing noticeable stuttering on the electronic device and degrading the user experience.

[0051] To address this issue, this application provides a display method that, in response to a switch from a high refresh rate to a low refresh rate, synthesizes a single frame of image data upon receiving a `crtc_commit` instruction to release fence resources, but without receiving a `Vsync-sf` instruction. In other words, when switching from a high refresh rate to a low refresh rate, although the SF instruction is not at the rising or falling edge of `Vsync-sf`, the SF instruction determines that `crtc_commit` has released the fence resources. Therefore, the SF instruction can immediately read the image data after `crtc_commit` releases the fence resources and immediately perform SF synthesis. Compared to waiting to receive `Vsync-sf` to read the image data, this method synthesizes a single frame of image data in advance, thereby enabling rapid synthesis of a single frame of image data when switching from a high refresh rate to a low refresh rate, reducing the number of dropped frames.

[0052] In some examples, in response to switching to a low refresh rate, Vsync-app, Vsync-sf, and Vsync-hw are synchronized. That is, after switching to a low refresh rate, Vsync-app, Vsync-sf, and Vsync-hw are aligned to resolve the situation where one frame is synthesized and another is dropped due to the misalignment of Vsync-app, Vsync-sf, and Vsync-hw after the switch.

[0053] like Figure 7 The diagram illustrates the trace of frame dropping optimization after an electronic device implements the display method of this application. Figure 7 When switching the refresh rate from 90Hz to 60Hz, the Vsync-hw latency per frame increased from 11.1ms to 19.4ms (e.g., ...). Figure 7 (As shown in TE = 19.4ms). A 19.4ms Vsync-hw overwrites a Vsync-sf. Normally, crtc_commit releases the fence resources at the end of the Vsync-hw. Therefore, under the overwritten Vsync-sf, SF cannot wait for crtc_commit to release the fence resources, resulting in the loss of two frames during the 19.4ms Vsync-hw. After the 19.4ms Vsync-hw ends, SF determines that crtc_commit has released the fence resources and immediately performs SF compositing to force the synthesis of one frame of image data. In other words, although SF is not at the rising or falling edge of Vsync-sf, SF determines that crtc_commit has released the fence resources and immediately performs SF compositing to force the synthesis of one frame of image data.

[0054] In general, SF synthesis requires two conditions: one is that it is at the rising or falling edge of Vsync-sf, and the other is that crtc_commit has released the fence resources. However, when switching from a high refresh rate to a low refresh rate, a long Vsync-hw occurs. SF synthesis only needs to meet one condition: that crtc_commit has released the fence resources. In this way, SF does not need to wait for the rising or falling edge of the next Vsync-sf and will be synthesized in advance in the next Vsync-sf, thus allowing the synthesis of one frame of image data to be done in advance.

[0055] In this method, the pre-synthesized image data can be any one of the two lost image frames. Although two frames are lost during the 19.4ms Vsync-hw process, the image data of any one of the two lost image frames can be synthesized immediately after the 19.4ms Vsync-hw process ends, effectively reducing the number of lost frames. In some examples, the image data of the first frame of the two lost image frames can be synthesized immediately after the 19.4ms Vsync-hw process ends, ensuring image continuity; in other examples, the image data of the second frame of the two lost image frames can be synthesized immediately after the 19.4ms Vsync-hw process ends, shortening the delay in synthesizing the second image frame.

[0056] Because the 19.4ms Vsync-hw process can cause misalignment of subsequent Vsync-app, Vsync-sf, and Vsync-hw, this misalignment can lead to situations where one frame is synthesized while another is lost. To address this issue, after switching to a lower refresh rate, Vsync-app, Vsync-sf, and Vsync-hw are aligned. After alignment, SF can read image data from the buffer on the rising and falling edges of Vsync-sf, respectively, and HWC synthesizes the image data. This allows SF synthesis to be performed on the rising and falling edges of Vsync-sf, enabling the electronic device to perform SF synthesis on the rising and falling edges of Vsync-sf at 60Hz after switching to 60Hz, thus avoiding frame loss.

[0057] The following diagram, with reference to the accompanying illustration, illustrates the optimization of software / hardware Vsync after a refresh rate switch, as shown in Figure 8. Figure 9 In the example, at a refresh rate of 90Hz, the periods of the three signals Vsync-app, Vsync-sf, and Vsync-hw are the same, and the baseline for each signal is also the same. However, when switching the refresh rate from 90Hz to 60Hz, a longer Vsync-hw signal appears, such as... Figure 9In the fourth cycle, Vsync-hw takes 19.4ms, meaning that a long Vsync-hw occurs in the fourth cycle. Originally, crtc_commit would release the fence resource before the end of (1), but because the refresh rate switched from 90Hz to 60Hz, crtc_commit released the fence resource at (2) and completed the release at (5) (the end of the fourth cycle). As a result, SF compositing was not performed at (3) because the fence resource was missed, resulting in a lost frame. (4) marks the end of Vsync-hw in the fourth cycle. At (4), crtc_commit releases the fence resource, and SF compositing is immediately performed at (4) to synthesize the image data of one frame, thereby forcibly refreshing one frame, that is, forcibly displaying one frame in the fifth cycle.

[0058] In the fifth cycle, the end times of the three signals Vsync-app, Vsync-sf, and Vsync-hw are the same. Therefore, in the sixth cycle, the start times of these three signals are the same. Since the periods of these three signals are the same, in the sixth cycle, Vsync-app, Vsync-sf, and Vsync-hw are vertically aligned. Figure 9 As shown, starting from the sixth cycle, Vsync-app, Vsync-sf, and Vsync-hw are vertically aligned. Taking Vsync-hw as an example, the sixth cycle can be... Figure 9 The sixth vertical line in the middle indicates the start time, and the seventh vertical line indicates the end time, so that subsequent software / hardware Vsync can be aligned to stably refresh (display) the image at a refresh rate of 60Hz.

[0059] After software / hardware Vsync optimization, the process of rendering and displaying image frames using Vsync-app, Vsync-sf, and Vsync-hw is as follows: Figure 10 As shown, at (1), missing the fence resource should result in a frame being dropped, but at (2), when the fence resource is available, the dark gray frame is refreshed immediately instead of waiting for the next Vsync_sf (specifically, waiting for the rising or falling edge of Vsync_sf) to arrive, thus avoiding the dropping of the light gray frame; at (3), the fence resource is released normally after the light gray frame is released, and SF compositing and HWC display are performed normally; at (4), the fence resource is released normally after the dark gray frame is released, and SF compositing and HWC display are performed normally, and so on. Subsequently, there is no phenomenon of alternating between compositing one frame and dropping one frame, which reduces the stuttering feeling and improves the user experience.

[0060] The above-described display method is applied to electronic devices. In some embodiments, the electronic device may be a mobile phone, tablet computer, desktop computer, laptop computer, notebook computer, ultra-mobile personal computer (UMPC), handheld computer, netbook, personal digital assistant (PDA), wearable electronic device, smartwatch, etc. This application does not impose any special limitations on the specific form of the electronic device.

[0061] like Figure 11 As shown, the electronic device may include: a processor, an external memory interface, internal memory, a Universal Serial Bus (USB) interface, a charging management module, a power management module, a battery, antenna 1, antenna 2, a mobile communication module, a wireless communication module, a sensor module, buttons, a motor, an indicator, a camera, a display screen, and a Subscriber Identification Module (SIM) card interface, etc. The audio module may include a speaker, receiver, microphone, headphone jack, etc., and the sensor module may include a pressure sensor, gyroscope sensor, barometric pressure sensor, magnetic sensor, accelerometer, distance sensor, proximity sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor, bone conduction sensor, etc.

[0062] A processor may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units can be independent devices or integrated into one or more processors. The processor is the nerve center and command center of an electronic device. The controller generates operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0063] Display screens are used to display images, videos, and a range of graphical user interfaces (GUIs).

[0064] An external memory interface can be used to connect external memory cards, such as Micro SD cards, to expand the storage capacity of electronic devices. The external memory card communicates with the processor through the external memory interface to perform data storage functions. For example, music, video, and other files can be saved on the external memory card. Internal memory can be used to store computer executable program code, which includes instructions. The processor executes the instructions stored in the internal memory to perform various functional applications and data processing of the electronic device. For example, in this application, the processor executes the instructions stored in the internal memory to cause the electronic device to perform the display method provided in this application.

[0065] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device. In other embodiments, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0066] In addition, an operating system runs on top of the aforementioned components. Examples include Apple's iOS operating system, Google's Android open-source operating system, and Microsoft's Windows operating system.

[0067] The operating system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture Android system as an example to illustrate the software structure of an electronic device. Figure 12 This is a software architecture diagram of an electronic device. A layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. Taking the Android system as an example, in some implementations, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer (Framework), the Hardware Abstraction Layer (HAL), and the kernel layer.

[0068] The application layer can include a series of application packages. These packages can include apps such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and SMS. The application framework layer provides application programming interfaces (APIs) and a programming framework for the applications in the application layer. The application framework layer includes predefined functions. For example, it can include a window manager, content provider, view system, phone manager, resource manager, and notification manager. The application framework layer can also include SF, which includes DispSyncSource.cpp, SF.cpp, and VsyncReactor.cpp. These can be considered three units within SF: DispSyncSource.cpp is the signal generation unit, SF.cpp is the data reading unit, and VsyncReactor.cpp is the refresh rate processing unit. The HAL can include HWC. The kernel layer is the layer between hardware and software. The kernel layer includes at least a display driver, camera driver, audio driver, and sensor driver. HWC can be used to synthesize image data. During the image data synthesis process, functions such as setCallback, registerCallback, and onComposerhalVsync can be called.

[0069] VsyncReactor.cpp samples Vsync-hw, and DispSyncSource.cpp generates Vsync-app and Vsync-sf based on the sampling results. VsyncReactor.cpp can call addResyncSample to sample Vsync-hw, obtaining its timestamp, which serves as an example of the sampling result. The Vsync-hw timestamp is stored in the timestamp sequence using addHwVsyncTimestamp. VsyncReactor.cpp can also call periodConfirmed for scene recognition. When VsyncReactor.cpp detects a high refresh rate to low refresh rate transition and a long-lasting Vsync-hw, it sends a command to SF.cpp, instructing SF.cpp to perform SF synthesis with HWC after confirming that crtc_commit has released fence resources. VsyncReactor.cpp then adjusts the timestamp sequence to ensure that Vsync-app, Vsync-sf, and Vsync-hw are aligned after the refresh rate switch.

[0070] Taking an electronic device with a refresh rate of 90Hz and a refresh rate switch from 90Hz to 60Hz as an example, the interaction process between DispSyncSource.cpp, SF.cpp, VsyncReactor.app, and HWC is explained, and its signaling diagram is as follows. Figure 13 As shown, the following steps may be included:

[0071] S101. VsyncReactor.cpp calls addResyncSample to sample Vsync-hw(i) and obtain the timestamp of Vsync-hw(i). The timestamp of Vsync-hw(i) is then stored in a timestamp sequence. For example, VsyncReactor.cpp calls addHwVsyncTimestamp to store the timestamp of Vsync-hw(i) in a timestamp sequence.

[0072] S102, DispSyncSource.cpp generates Vsync-sf(i+2) based on the timestamp of Vsync-hw(i). Vsync-sf(i+2) is the second Vsync-sf after Vsync-sf(i). Vsync-hw(i) and Vsync-sf(i) correspond to the same period. In this embodiment, Vsync-hw(i) and Vsync-sf(i) are Vsync-hw and Vsync-sf in the i-th period. SF can also generate Vsync-app(i+2) when generating Vsync-sf(i+2). Under the action of Vsync-app(i+2), the APP generates image data, which will not be described in detail here. The generation of Vsync-sf(i+2) based on the timestamp of Vsync-hw(i) is only an example and is not limited in this embodiment.

[0073] In S103 and SF.cpp, after confirming that crtc_commit has released the fence resource and that it is at the rising or falling edge of Vsync-sf(i+2), image data is read. A rising edge of Vsync-sf(i+2) indicates that the level of Vsync-sf(i+2) has changed from low to high, and a falling edge of Vsync-sf(i+2) indicates either a rising or falling edge.

[0074] S104 and HWC synthesize image data to complete SF synthesis through SF.cpp and HWC, generating an image frame.

[0075] S105 and HWC can send the synthesized image data to the display driver, which then drives the screen to display the image frames.

[0076] In S106, VsyncReactor.cpp calls periodConfirmed and recognizes that the time taken for Vsync-hw(i+1) is 19.4ms, thus determining that a refresh rate switch has occurred. It then sends a refresh rate switch instruction to SF.cpp. This instruction, in addition to indicating a refresh rate switch, can also carry the time taken for Vsync-hw(i+1). Vsync-hw(i+1) is the next Vsync-hw after Vsync-hw(i).

[0077] Ideally, when the next Vsync-hw signal ends at the current time, the difference between the current time and the start time of that Vsync-hw is equal to the period of that Vsync-hw. Therefore, periodConfirmed can identify whether a preset scenario has occurred by checking if this difference exceeds a preset threshold. The preset scenario is a scenario where Vsync-hw takes a long time during refresh rate switching. When switching from 90Hz to 60Hz, the preset threshold can be, but is not limited to, a value less than 19.4ms. For example, it can be a value between 11.1 and 19.4ms. Alternatively, periodConfirmed can identify whether a preset scenario has occurred by checking if the difference is within a preset value range. The preset value range can be between 11.1 and 19.4ms, but may not include 11.1, although it may include 19.4ms. In other refresh rate switching scenarios, the preset threshold is not limited to 19.4ms. This embodiment does not impose any limitations on the preset threshold.

[0078] The S107 and SF.cpp response instructions immediately read the image data after confirming that crtc_commit has released the fence resources.

[0079] S108 and HWC synthesize image data to generate an image frame.

[0080] S109 and HWC can send the synthesized image data to the display driver, which then drives the screen to display the image frames.

[0081] In other words, after receiving the instruction from VsyncReactor.cpp indicating a refresh rate switch and that Vsync-hw(i) takes 19.4ms, SF.cpp does not need to wait for the rising and falling edges of Vsync-sf. After determining that crtc_commit has released the fence resources, SF can read the image data, allowing SF to read the image data in advance. Similarly, the HWC synthesized image data is also read in advance.

[0082] S110. In VsyncReactor.cpp, the call to addResyncSample to sample Vsync-hw(i+1) is prohibited, so that the timestamp of Vsync-hw(i+1) is not stored in the timestamp sequence.

[0083] S110' and VsyncReactor.cpp call addResyncSample to sample Vsync-hw(i+1), but prohibit the call to addHwVsyncTimestamp to store the timestamp of Vsync-hw(i+1) in the timestamp sequence, thus achieving the purpose of not storing the timestamp of Vsync-hw(i+1) in the timestamp sequence.

[0084] S111. VsyncReactor.cpp calls addResyncSample to sample Vsync-hw(i+2) to obtain the timestamp of Vsync-hw(i+2), and calls addHwVsyncTimestamp to store the timestamp of Vsync-hw(i+2) into the timestamp sequence.

[0085] S112. DispSyncSource.cpp generates Vsync-sf(i+3) based on the timestamp of Vsync-hw(i+2). The generated Vsync-sf(i+3) is aligned with Vsync-hw(i+3).

[0086] S113 and SF.cpp read image data after determining that crtc_commit has released the fence resources and is at the rising or falling edge of Vsync-sf(i+3).

[0087] S114 and HWC synthesize image data to complete SF synthesis through SF.cpp and HWC, generating an image frame.

[0088] S115 and HWC can send the synthesized image data to the display driver, which then drives the screen to display the image frames.

[0089] Vsync-sf(i+3) and Vsync-hw(i+3) are aligned to ensure that Vsync-app, Vsync-sf, and Vsync-hw are aligned after a refresh rate switch. On the subsequent rising or falling edge of Vsync-sf, SF reads the image data and HWC synthesizes it. This process, combining SF and HWC, generates a single image frame. Thus, starting from Vsync-sf(i+3), the electronic device can generate image frames normally, avoiding the phenomenon of synthesizing one frame and then losing another.

[0090] Figure 14 An example of a timestamp sequence and a predicted Vsync-sf is shown. When sampling Vsync-hw, multiple time points of Vsync-hw can be sampled to obtain multiple timestamps. The Vsync-sf is then generated using these multiple timestamps. Figure 14 Taking the use of multiple timestamps of the i-th Vsync-hw to generate the (i+2)-th Vsync-sf as an example, this embodiment does not limit which Vsync-hw is used to generate which Vsync-sf. The multiple timestamps of the Vsync-hw can be stored as a timestamp set in a timestamp queue. Figure 14 The example below illustrates how one of the multiple timestamps in Vsync-hw (such as the first timestamp) is stored in a timestamp queue.

[0091] As in Figure 14 In the second cycle, the first timestamp TS1 of Vsync-hw (abbreviated as Vsync-hw2) is stored in the timestamp queue, and the timestamp of Vsync-hw2 is used to generate Vsync-sf4; the first timestamp TS2 of Vsync-hw3 is stored in the timestamp queue, and Vsync-sf5 is generated based on the timestamp of Vsync-hw3; the first timestamp TS3 of Vsync-hw4 is stored in the timestamp queue, and Vsync-sf6 is generated based on the timestamp of Vsync-hw4. Because Vsync-hw3 takes a long time, its sampling time is increased, causing a delay in the generation of Vsync-sf5. Furthermore, before the refresh rate switch, the period of Vsync-hw should be the same as that of Vsync-sf. However, the increased time of Vsync-hw3 makes their periods different from Vsync-sf3, resulting in a difference in the end times of Vsync-sf3 and Vsync-hw3. Vsync-hw3 ends later than Vsync-sf3. Vsync-sf4 starts after Vsync-sf3 finishes, and vice versa. Therefore, with the end times of Vsync-sf3 and Vsync-hw3 different, the start times of Vsync-sf4 and Vsync-hw4 also differ. From Vsync-hw4 onwards, the software / hardware Vsync is not aligned.

[0092] Figure 15The diagram illustrates the optimization of the timestamp sequence and predicted Vsync-sf. When a refresh rate switch is detected, and the time taken for Vsync-hw3 before the switch is 19.4ms, addHwVsyncTimestamp skips the timestamp of Vsync-hw3 and uses the timestamp of Vsync-hw4 to generate Vsync-sf5. Vsync-sf5 and Vsync-hw3 are separated by one Vsync-hw cycle, and Vsync-hw5 is also separated by one Vsync cycle from Vsync-hw3. The -hw period indicates that the start time of both Vsync-hw5 and Vsync-sf5 is one Vsync-hw period (i.e., one Vsync-hw4 period) from the end time of Vsync-hw3. Vsync-hw5 and Vsync-sf5 correspond to the same refresh rate, indicating that the period of Vsync-hw5 is the same as that of Vsync-sf5. Therefore, Vsync-hw5 and Vsync-sf5 not only have the same period, but also the same reference, that is, Vsync-hw5 and Vsync-sf5 are aligned.

[0093] Because `addHwVsyncTimestamp` skips the timestamp of `Vsync-hw3` and uses the timestamp of `Vsync-hw4` to generate `Vsync-sf5`, and `Vsync-hw4` is the next `Vsync-hw` after `Vsync-hw3`, using the timestamp of `Vsync-hw4` to generate `Vsync-sf5` means that the generation of `Vsync-sf5` is delayed, and the end time of `Vsync-sf4` is also delayed, causing the period of `Vsync-sf4` to increase. For example... Figure 15 The period of Vsync-sf4 is 19.4ms. The dashed line indicates that Vsync-sf5 is generated using Vsync-hw3. When generating Vsync-sf5 using the timestamp of Vsync-hw4, the period of Vsync-sf4 is increased from 16.6ms to 19.4ms.

[0094] Furthermore, this application provides an electronic device, which includes: one or more processors; one or more memories; the memories store one or more programs, which, when executed by the processor, cause the electronic device to perform the above-described display method.

[0095] This application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the above-described display method.

Claims

1. A display method, characterized in that, When applied to electronic devices, the method includes: The refresh rate of the electronic device is determined to switch from a first refresh rate to a second refresh rate, wherein the first refresh rate is greater than the second refresh rate; In response to the refresh rate switching from the first refresh rate to the second refresh rate, the condition for the electronic device to generate the next image frame of the first image frame changes from the first condition to the second condition. The first condition is that the level of the first signal changes and the first image frame is completed and displayed. The first signal is used to indicate the generation of the image frame. The second condition is that the first image frame is completed and displayed. After the electronic device completes the display of the first image frame, a second image frame is generated, and the second image frame is the image frame following the first image frame. Display the second image frame; After the refresh rate is switched to the second refresh rate, the first signal is aligned with the second signal, and the second signal is used to indicate the display of image frames; When the electronic device completes the display of the second image frame and the level change of the first signal meets the preset conditions, a third image frame is generated, wherein the preset conditions are that the level of the first signal changes from a low level to a high level, or the level of the first signal changes from the high level to the low level. Under the action of the second signal, the third image frame is displayed.

2. The method according to claim 1, characterized in that, Before determining that the refresh rate of the electronic device switches from a first refresh rate to a second refresh rate, the method further includes: in response to the refresh rate being the first refresh rate, generating a (i+n)th first signal based on the i-th second signal, wherein the duration of the interval between the (i+n)th first signal and the i-th second signal is the sum of the periods of the n first signals, where n is a natural number greater than 1; The step of aligning the first signal with the second signal after the refresh rate is switched to the second refresh rate includes: sampling the first second signal after the refresh rate is switched to the second refresh rate, and generating the first signal using the sampling result of the first second signal, wherein the time interval between the first signal and the first second signal is the sum of the periods of (n-1) second signals.

3. The method according to claim 2, characterized in that, The method further includes: discarding a special signal, wherein the special signal is a second signal generated during the process of switching the refresh rate from the first refresh rate to the second refresh rate, and the special signal is the signal preceding the first second signal.

4. The method according to claim 3, characterized in that, The discarding of special signals includes: prohibiting the sampling of the special signals; or, The special signal is sampled, but the sampling result of the special signal is not stored in the result sequence.

5. The method according to claim 1, characterized in that, The second image frame is an image frame that is lost during the process of switching the refresh rate from the first refresh rate to the second refresh rate.

6. The method according to claim 1, characterized in that, The electronic device completes the display of the first image frame by: after the electronic device calls the kernel thread crtc_commit to release the fence resources, it determines that the display of the first image frame is complete.

7. The method according to claim 1, characterized in that, The step of determining the refresh rate of the electronic device to switch from the first refresh rate to the second refresh rate includes: determining the end time of the first signal and the start time of the first signal when the refresh rate of the electronic device is the first refresh rate; Determine the difference between the end time and the start time; If the difference satisfies the preset switching conditions, the refresh rate of the electronic device is determined to switch from the first refresh rate to the second refresh rate.

8. The method according to any one of claims 1 to 7, characterized in that, The application framework layer of the electronic device includes a refresh rate processing unit and a data reading unit; the hardware abstraction layer of the electronic device includes a hardware hybrid renderer; the kernel layer of the electronic device includes a display driver. The refresh rate processing unit is used to determine whether the refresh rate of the electronic device switches from a first refresh rate to a second refresh rate; The data reading unit is used to respond to the refresh rate switching from the first refresh rate to the second refresh rate, and the condition for the electronic device to generate the next image frame of the first image frame changes from the first condition to the second condition. The first condition is that the level of the first signal changes and the first image frame is completed. The first signal is used to indicate the generation of the image frame. The second condition is that the first image frame is completed. After the electronic device completes the display of the first image frame, the first image data is read. The hardware hybrid renderer is used to synthesize the first image data to generate a second image frame, the second image frame being the image frame following the first image frame. The display driver is used to display the second image frame.

9. The method according to claim 8, characterized in that, The application architecture layer further includes: a signal generation unit, which is used to control the alignment of a first signal and a second signal after the refresh rate is switched to the second refresh rate, wherein the second signal is used to indicate the display of an image frame; The data reading unit is further configured to read the second image data when the electronic device completes the display of the second image frame and the level change of the first signal meets a preset condition, wherein the preset condition is that the level of the first signal changes from a low level to a high level, or the level of the first signal changes from the high level to the low level. The hardware hybrid renderer is also used to synthesize the second image data to generate a third image frame; The display driver is also used to display the third image frame under the action of the second signal.

10. An electronic device, characterized in that, The electronic device includes: One or more processors; One or more memory units; The memory stores one or more programs that, when executed by the processor, cause the electronic device to perform the display method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the display method as described in any one of claims 1 to 9.

Images