Image sending and displaying method and electronic device

Abstract

An image display method and an electronic device, and relate to the technical field of terminals. The method comprises: a SoC sends image data to a display screen at a first frame rate. After a continuous time length during which the SoC does not send new image data to the display screen exceeds a first time length, the SoC sends image data to the display screen at a second frame rate. After the SoC processes a first image, the SoC switches to a third frame rate, and after the end of a frame at the third frame rate, switches to the first frame rate, and sends the first image to the display screen, the first image being a frame of new image data. The second frame rate is less than the first frame rate, the third frame rate is greater than the second frame rate, and the first frame rate, the second frame rate and the third frame rate are all frame rates supported by the display screen. In this way, the third frame rate supported by the display screen can be switched to, and new image data can be displayed in a timely manner.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image display method and electronic device. Background Technology

[0002] In electronic devices such as mobile phones and tablets, an idle mode is typically provided for image processing scenarios. When no new image data is generated for an extended period of time, the underlying system (including drivers and hardware) can enter the idle mode, which includes disabling some interrupts and reducing the frame rate, thereby reducing device power consumption.

[0003] After entering IDLE mode, when new image data is generated, the underlying layer can enable corresponding interrupts and increase the frame rate, thereby exiting IDLE mode and resuming image processing and display at a higher frequency. Furthermore, after exiting IDLE mode, the underlying layer can display the new image data.

[0004] However, in existing technologies, the transmission and display of new image data may experience high latency, affecting display performance. Summary of the Invention

[0005] This application provides an image display method and electronic device, which can exit IDLE mode more promptly when new image data is generated, thereby reducing the delay in displaying new image data and improving display performance.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] Firstly, an image display method is provided, applied to an electronic device including a system-on-a-chip (SoC) and a display screen. Specifically, the method includes: the SoC sending image data to the display screen at a first frame rate; after the SoC has not sent new image data to the display screen for a continuous period exceeding a first duration, the SoC sends image data to the display screen at a second frame rate. That is, after the SoC has not generated new image data for a period of time, it can reduce the frame rate, thereby saving device power consumption. Furthermore, after the SoC processes and obtains the first image, i.e., after obtaining new image data, the SoC switches to a third frame rate; after one frame of the third frame rate ends, it switches back to the first frame rate and sends the first image, which is one frame of new image data, to the display screen. The second frame rate is lower than the first frame rate, the third frame rate is higher than the second frame rate, and all three frame rates are frame rates supported by the display screen.

[0008] In summary, by adopting this application, after reducing to the second frame rate, when new image data is generated, the SoC can first switch to the third frame rate, which is greater than or equal to the second frame rate. This shortens the current frame length and allows the current frame to end earlier, enabling the new image data to be quickly displayed after the end of a frame at the third frame rate, reducing display latency. Furthermore, the third frame rate is a frame rate supported by the display, ensuring display performance during frame rate switching.

[0009] Furthermore, the aforementioned switch from the first frame rate to the second frame rate is triggered by the SoC based on the absence of new image data for an extended period, and the switch from the second frame rate to the third frame rate is triggered by the SoC based on the generation of new image data. These are not frame-switching requirements of the upper-layer application. Therefore, after one frame of the third frame rate ends, the SoC reverts to the first frame rate, thus maintaining consistency with the frame rate requirements of the upper layer.

[0010] In one possible implementation of the first aspect, after sending the first image to the display, the method further includes: the SoC sending image data to the display screen at a first frame rate.

[0011] Using this implementation, after restoring to the first frame rate, the SoC will process the image and send it to the display according to the first frame rate, thereby maintaining consistency with the frame rate requirements of the upper layer.

[0012] In one possible implementation of the first aspect, the frame length corresponding to the third frame rate is greater than or equal to the second duration, the second duration is the time interval between the first moment and the second moment, the second moment is the moment after the first image is obtained, and the first moment is the start moment of the current frame in which the first moment is located.

[0013] In other words, the frame length corresponding to the third frame rate is greater than the second duration, indicating that when the third frame rate is used, the end time of the current frame is located at or after the second time. This ensures that the current frame ends and is displayed only after new image data is obtained.

[0014] In one possible implementation of the first aspect, if the second duration is less than or equal to the frame length corresponding to the first frame rate, the third frame rate is greater than or equal to the first frame rate.

[0015] In other words, the SoC obtains new image data very early, making the second duration very short, which is less than or equal to the frame length corresponding to the first frame rate. In this case, the SoC can switch to a third frame rate that is greater than or equal to the first frame rate. In this way, compared to the second frame rate, the current frame can end quickly, and the new image data can be sent for display, significantly shortening the display latency.

[0016] Additionally, if the third frame rate equals the first frame rate, it indicates that the frame rate has been restored to the first frame rate, and subsequent frames will not need to be restored again at the end of the current frame.

[0017] In one possible implementation of the first aspect, if the second duration is greater than the frame length corresponding to the first frame rate and less than or equal to the frame length corresponding to the fourth frame rate, the third frame rate is greater than the second frame rate and less than the first frame rate. The fourth frame rate is the minimum frame rate supported by the display screen that is greater than the second frame rate.

[0018] In other words, the time it takes for the SoC to process new image data is moderate, so that the second duration is greater than the frame length corresponding to the first frame rate and less than or equal to the frame length corresponding to the fourth frame rate. In this case, the SoC can switch to a third frame rate that is less than the first frame rate and greater than the second frame rate. In this way, compared with the second frame rate, the frame length of the current frame can be shortened to a certain extent, so as to end the current frame earlier, send the new image for display, and reduce the display latency.

[0019] In one possible implementation of the first aspect, after the SoC processes and obtains the first image, the method further includes: if the second duration is greater than the frame length corresponding to the fourth frame rate but less than or equal to the frame length corresponding to the second frame rate, the SoC switches to the first frame rate after one frame of the second frame rate ends, and sends the first image to the display screen. Here, the fourth frame rate is the minimum frame rate supported by the display screen that is greater than the second frame rate.

[0020] In other words, if the SoC receives new image data late, close to the second frame rate, the SoC can maintain the second frame rate and switch to the first frame rate after the current frame of the second frame rate ends. This avoids switching to a frame rate that the display does not support if the condition of sending the data to the display in advance is not met.

[0021] In one possible implementation of the first aspect, the SoC switches to a third frame rate, including: upon obtaining a first identifier, the SoC switches to a third frame rate. The first identifier indicates that the frame rate should be switched after the current frame has ended.

[0022] By adopting this implementation method, obtaining the first identifier indicates that in order to avoid the occurrence of frame rates that the display does not support, the display can be sent slightly later. The SoC first switches the frame rate to the third frame rate, and then sends the display after one frame of the third frame rate is completed. This not only avoids the occurrence of frame rates that the display does not support, but also allows the display to be sent earlier.

[0023] In one possible implementation of the first aspect, the method further includes: upon obtaining the second identifier, the SoC switches to the first frame rate and sends the first image to the display screen.

[0024] Using this implementation, obtaining the second identifier indicates extremely high latency requirements for display delivery, necessitating immediate display delivery. The SoC immediately reverts to the first frame rate and executes the display delivery. Thus, for scenarios with high latency requirements, the immediate frame rate switching method can still be used to reduce display delivery latency.

[0025] In one possible implementation of the first aspect, before the SoC switches to the third frame rate, the method further includes: the SoC calculating the time interval between the first moment and the second moment to obtain a second duration. The SoC determines a third frame rate with a frame length greater than or equal to the second duration from the frame rates supported by the display. Here, the second moment is the moment after the first image is obtained, and the first moment is the start time of the frame in which the first moment occurs.

[0026] Using this implementation, the SoC can determine the third frame rate that the display supports and whose frame length is greater than the second duration. This can avoid the display not supporting the frame rate and ensure that the current frame ends after the second time period, thus enabling the display to be sent.

[0027] In one possible implementation of the first aspect, the SoC determines a third frame rate with a frame length greater than the second duration from the frame rates supported by the display, including: if there are multiple frame rates with a frame length greater than the second duration, the SoC determines the third frame rate as the largest frame rate among the multiple frame rates.

[0028] By adopting this implementation method, the frame length of the current frame can be shortened to the greatest extent, and new image data can be sent for display earlier.

[0029] In one possible implementation of the first aspect, after the continuous duration during which the SoC has not sent new image data to the display exceeds a first duration, the method further includes: the SoC switching to a second frame rate and entering an idle (IDLE) mode. The SoC sending image data to the display at the second frame rate includes: in IDLE mode, the SoC sending image data to the display at the second frame rate.

[0030] In other words, after a long period of no new image data is generated, the SoC will perform a series of operations such as reducing the frame rate and enter idle mode, thus operating in a low-power state.

[0031] In one possible implementation of the first aspect, after switching to the first frame rate, the method further includes: the SoC exiting IDLE mode. Sending the first image to the display includes: the SoC sending the first image to the display after exiting IDLE mode. That is, the SoC sends new image data to the display only after exiting IDLE mode.

[0032] In one possible implementation of the first aspect, the first image includes: a first frame of image data processed by the SoC after the SoC has not sent new image data to the display screen for a continuous period exceeding a first duration.

[0033] In other words, after a long period of no new image data and a reduction in the frame rate, when the SoC receives the first new frame of image data, it will increase the frame rate to the third frame rate and then return to the first frame rate, thus continuing to process the image and send it for display at a higher speed.

[0034] Secondly, this application also provides an electronic device, which includes a System-on-a-Chip (SoC), a display screen, a memory, and one or more processors. The SoC, display screen, memory, and processors are coupled. The memory stores computer program code, which includes computer instructions. When the computer instructions are executed by the processor, the electronic device performs the following steps:

[0035] The SoC sends image data to the display at a first frame rate. If the SoC does not send new image data to the display for a continuous period exceeding a first duration, the SoC sends image data to the display at a second frame rate. After the SoC processes and obtains the first image, it switches to a third frame rate. After one frame at the third frame rate ends, it switches back to the first frame rate and sends the first image to the display. The first image is one new frame of image data. The second frame rate is lower than the first frame rate, and the third frame rate is higher than the second frame rate. All three frame rates are supported by the display.

[0036] In one possible implementation of the second aspect, when computer instructions are executed by the processor, the electronic device performs the following steps: the SoC sends image data to the display screen at a first frame rate.

[0037] In one possible implementation of the second aspect, the frame length corresponding to the third frame rate is greater than or equal to the second duration, the second duration is the time interval between the first moment and the second moment, the second moment is the moment after the first image is obtained, and the first moment is the start moment of the current frame in which the first moment is located.

[0038] In one possible implementation of the second aspect, if the second duration is less than or equal to the frame length corresponding to the first frame rate, the third frame rate is greater than or equal to the first frame rate.

[0039] In one possible implementation of the second aspect, if the second duration is greater than the frame length corresponding to the first frame rate and less than or equal to the frame length corresponding to the fourth frame rate, the third frame rate is greater than the second frame rate and less than the first frame rate. The fourth frame rate is the minimum frame rate supported by the display screen that is greater than the second frame rate.

[0040] In one possible implementation of the second aspect, when computer instructions are executed by the processor, the electronic device performs the following steps: if the second duration is greater than the frame length corresponding to the fourth frame rate but less than or equal to the frame length corresponding to the second frame rate, the SoC switches to the first frame rate after one frame of the second frame rate ends and sends a first image to the display screen. The fourth frame rate is the minimum frame rate supported by the display screen that is greater than the second frame rate.

[0041] In one possible implementation of the second aspect, when computer instructions are executed by the processor, the electronic device performs the following steps: upon acquiring a first identifier, the SoC switches to a third frame rate. The first identifier indicates that the frame rate should be switched after the current frame has ended.

[0042] In one possible implementation of the second aspect, when computer instructions are executed by the processor, the electronic device performs the following steps: upon obtaining a second identifier, the SoC switches to a first frame rate and sends a first image to the display screen.

[0043] In one possible implementation of the second aspect, when computer instructions are executed by the processor, the electronic device performs the following steps: the SoC calculates the time interval between a first moment and a second moment to obtain a second duration. The SoC determines a third frame rate from the frame rates supported by the display, with a frame length greater than or equal to the second duration. Here, the second moment is the moment after the first image is obtained, and the first moment is the start time of the frame in which the first moment occurs.

[0044] In one possible implementation of the second aspect, when computer instructions are executed by the processor, the electronic device performs the following steps: when there are multiple frame rates with a frame length greater than the second duration, the SoC determines the third frame rate as the largest of the multiple frame rates.

[0045] In one possible implementation of the second aspect, when computer instructions are executed by the processor, the electronic device performs the following steps: the SoC switches to a second frame rate and enters an idle mode. In idle mode, the SoC displays data to the screen at the second frame rate.

[0046] In one possible implementation of the second aspect, when computer instructions are executed by the processor, the electronic device performs the following steps: the SoC exits IDLE mode; after exiting IDLE mode, the SoC sends a first image to the display screen.

[0047] In one possible implementation of the second aspect, the first image includes: a first frame of image data processed by the SoC after the SoC has not sent new image data to the display screen for a continuous period exceeding a first duration.

[0048] Thirdly, this application provides a chip system applied to an electronic device including a display panel and a memory; the chip system includes one or more interface circuits and one or more processors; the interface circuits and processors are interconnected via lines; the interface circuits are used to receive signals from the memory of the electronic device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the electronic device performs the method as described in the first aspect and any of its possible design embodiments.

[0049] Fourthly, this application provides a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform a method as described in the first aspect and any of its possible design embodiments.

[0050] Fifthly, this application provides a computer program product that, when run on an electronic device, causes the electronic device to perform the method as described in the first aspect and any possible design thereof.

[0051] Understandably, the beneficial effects achieved by the electronic device of the second aspect, the chip system of the third aspect, the computer-readable storage medium of the fourth aspect, and the computer program product of the fifth aspect provided above can be referred to the beneficial effects of the first aspect and any of its possible design embodiments, which will not be repeated here. Attached Figure Description

[0052] Figure 1 This is a schematic diagram illustrating the structure of a signal for display.

[0053] Figure 2 This is a schematic diagram illustrating the principle of entering and exiting IDLE mode.

[0054] Figure 3 This is a schematic diagram illustrating another principle of entering and exiting IDLE mode.

[0055] Figure 4 A schematic diagram illustrating the principles of entering and exiting IDLE mode as provided in the embodiments of this application;

[0056] Figure 5 A hardware configuration diagram of a mobile phone provided in an embodiment of this application;

[0057] Figure 6 A hardware and software architecture diagram of a mobile phone provided for an embodiment of this application;

[0058] Figure 7 A flowchart illustrating an image display method provided in this application embodiment;

[0059] Figure 8One of the schematic diagrams for setting the third frame rate provided in the embodiments of this application;

[0060] Figure 9 The second schematic diagram for setting the third frame rate provided in the embodiments of this application;

[0061] Figure 10 The third schematic diagram for setting the third frame rate is provided for the embodiments of this application. Detailed Implementation

[0062] The technical solutions of the embodiments of this application are described below with reference to the accompanying drawings. In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to limit the application. As used in the specification and appended claims of this application, the singular expressions "a," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" refer to one or more (including two). The term "and / or" is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0063] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0064] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0065] Before introducing the embodiments of this application, the terminology involved in the embodiments of this application will be briefly explained.

[0066] 1. Image submission mode.

[0067] Image sending mode refers to the mode in which the host sends image data to the display screen. The host can be a system-on-chip (SoC) or an application processor (AP) of an electronic device; the following explanation uses an SoC as an example.

[0068] A System-on-Chips (SoC) can send image data to a display screen via the Mobile Industry Processor Interface (MIPI) protocol of the Display Serial Interface (DSI). The MIPI protocol defines the following image transmission modes: video mode and command mode. The following section primarily describes the video mode involved in this application.

[0069] Video Mode: Similar to traditional RGB interfaces, the SoC needs to continuously refresh the display. Since no dedicated data signal is used to transmit synchronization information, control signals and RGB data are transmitted via the MIPI bus in the form of messages. Furthermore, the SoC refreshes the display periodically, eliminating the need for a frame buffer.

[0070] Command Mode: The MIPI bus controller sends pixel data streams to the display screen using display command messages. The display screen has a frame buffer for storing image data. After the image data is placed in the display screen's frame buffer, the timing controller can retrieve the image data from the frame buffer and display the image on the screen.

[0071] The image display method provided in this application embodiment is mainly applicable to the above-mentioned video mode. For specific reasons, please refer to the explanation after "3. Method of switching frame rate in video mode" below.

[0072] 2. Transmission signals for image data in video mode.

[0073] An image frame consists of multiple rows of pixels, with each pixel's color represented using several bits of data. For example, on a monochrome display, each pixel uses 1 bit. On a 256-color display, each pixel uses 8 bits.

[0074] The SoC transmits image data to the display screen via the MIPI protocol. The following section combines... Figure 1 This section introduces the signals included in the MIPI signals used to transmit image data in video mode and their meanings.

[0075] Vertical Synchronization (VSync) signal: VSync represents the width of the vertical synchronization pulse. The VSync signal can also be called the frame synchronization signal, used to indicate the start of a frame of data.

[0076] Horizontal Synchronization (HSync) signal: HSync represents the width of the horizontal synchronization pulse, calculated in lines. The HSync signal can also be called the line synchronization signal, used to indicate the start of a line of data.

[0077] Pixel clock (VCLK) signal, Figure 1 Not shown in the image: Each VCLK signal represents the data being transmitted for one pixel.

[0078] Vertical Back Porch (VBP) signal: The VBP signal indicates the number of lines that are invalid after the VSync signal at the beginning of a frame.

[0079] Vertical Front Porch (VFP) signal: The VFP signal indicates the number of invalid lines between the end of the valid line count in a frame and the next VSync.

[0080] Horizontal Back Porch (HBP) signal: The HBP signal represents the number of VCLK signals between the start of the HSync signal and the start of valid data in a line.

[0081] Horizontal Front Porch (HFP) signal: The HFP signal indicates the number of VCLK signals between the end of a line of valid data and the start of the next HSync signal.

[0082] See Figure 2 Here is an example of a MIPI waveform. Figure 2 The diagram shows the MIPI waveforms corresponding to the transmission of four image frames. Taking the transmission of the second frame as an example, a low level transmits the image information of the second frame. For example, a low level transmits... Figure 1 The image information within the vertical effective area includes the HSync, HBP, and HFP signals for the corresponding rows (effective rows) within that area. The high level consists of two parts: one part is the high level before the low level, transmitting the VSync and VBP signals; the other part is the high level after the low level, transmitting the VFP signal. Therefore, the high level does not transmit valid image data; these are invalid rows.

[0083] Furthermore, in the MIPI waveform, a Start of Frame (SOF) signal can be included before the valid data. The SOF signal indicates the start of valid data. Clearly, the SOF signal follows the VSync signal.

[0084] 3. How to switch frame rates in video mode.

[0085] When the image frame rate changes, the time interval between the SoC processing the image and sending the image data to the display screen must change accordingly to meet the frame rate requirements after the switch. For example, if the frame rate before the switch is 120Hz, the SoC can process one frame of image and send the processed image to the display screen every 8.3ms; if the frame rate after the switch is 60Hz, the SoC can process one frame of image and send the processed image to the display screen every 16.6ms.

[0086] In some implementations, the SoC can change the time interval for transmitting image data by altering the number of rows transmitted. For example, the SoC can modify the number of rows for the VBP and VFP signals in the MIPI waveform, changing the number of invalid rows transmitted to meet the frame rate requirements after switching. For instance, reducing invalid rows can increase the frame rate, while increasing invalid rows can decrease the frame rate.

[0087] See also Figure 1 VBP and VFP can be collectively referred to as the Vertical Blank (VBLK) region. Modifying the number of rows of the VBP and VFP signals is equivalent to modifying the number of rows in the VBLK region.

[0088] In other implementations, the SoC can change the time interval for transmitting image data by altering the linetime of each line. For example, the SoC can modify the number of VCLK signals in the HBP and HFP signals of each line in the MIPI waveform, changing the linetime to meet the frame rate requirements after switching. For instance, changing the linetime can alter the time interval for transmitting image data. Based on this, using real switching, the SoC can modify the number of VCLK signals in the HBP and HFP signals of each line in the MIPI waveform (also known as porch modification), changing the linetime to meet the frame rate requirements after switching. For example, reducing the number of VCLK signals in the HFP signal can increase the frame rate, while increasing the number of VCLK signals in the HFP signal can decrease the frame rate.

[0089] The image display method provided in this application embodiment can switch the frame rate through any of the above implementation methods.

[0090] As can be seen, in video mode, the SoC can switch frame rates by modifying the MIPI signal. This allows for negotiation with the display without sending commands, resulting in high flexibility in frame rate switching, such as switching at any time or to any frame rate. However, this characteristic also makes it easier for unexpected frame rates to occur when switching frame rates in video mode. Therefore, the embodiments of this application can avoid unexpected frame rates in video mode. Of course, the image processing method provided in these embodiments can also be applied to command mode, and this application does not specifically limit its application to this.

[0091] The image display method provided in this application can be applied to image processing scenarios. For example, in a scenario where an electronic device processes and displays multiple consecutive video frames during video playback. As another example, in a reading scenario where an electronic device processes and displays image frames containing text to be read.

[0092] Typically, image processing includes steps such as rendering, compositing, and display. For example, modules such as the Central Processing Unit (CPU) and Graphics Processing Unit (GPU) in a System-on-a-Chip (SoC) can perform rendering and compositing processes. Then, the display driver in the SoC can send the composited image data to the display screen for display (i.e., display).

[0093] In some situations, such as pausing video playback or displaying static images, there may be a prolonged period (e.g., 50ms, 60ms, 3 frames, 5 frames) without a need to process new image data. The CPU, GPU, and other modules will not receive new image data for an extended period, and consequently, the display driver will not receive new image data for a correspondingly long time. In such cases, the underlying system (including drivers and hardware, the same applies below) can enter IDLE mode. Entering IDLE mode disables some interrupts and reduces the frame rate, thereby reducing device power consumption.

[0094] See Figure 2 and Figure 3 Before entering IDLE mode, the frame rate is 60 FPS, so a VSync signal is generated approximately every 16.67 ms, triggering the display. Specifically, the display shows the 0th frame; after VSync1 (VSync x in the diagram can represent the xth VSync signal, and the same applies below) arrives, the display driver can send the 1st image (represented by the number "1" in the diagram, and the same applies below), and then the display can show the 1st image; after VSync2 arrives, the display driver can send the 2nd image, and then the display can show the 2nd image. However, after VSync3, VSync4, and VSync5 arrive, no new image data is generated, and correspondingly, the display driver has no new image data to send, causing the display to continuously show multiple frames of the 2nd image. When VSync5 arrives, three consecutive frames have not received new image data, and correspondingly, the display driver will not receive any new image data. In this situation, the underlying layer can enter IDLE mode. The underlying layer can reduce the frame rate from 60FPS to 30FPS. Then, a VSync signal is generated approximately 33.33ms after VSync5. As shown in the figure, the time interval between adjacent VSync signals between VSync5 and VSync8 is 33.33ms. In this way, the CPU, GPU and other modules can process images at a slower speed, and the display driver can trigger the display at a lower frequency.

[0095] Subsequently, after the CPU, GPU, and other modules process and obtain new image data, the display driver can receive the new image data. In this case, the underlying layer can exit IDLE mode. Exiting IDLE mode enables some interrupts and can also increase the frame rate, thereby restoring faster image processing and display, meeting the needs of efficient image processing.

[0096] In principle, upon entering IDLE mode and receiving new image data, the underlying layer can immediately restore the frame rate before entering IDLE mode and exit IDLE mode. After exiting IDLE mode, the display driver can send the new image data to the screen, thus enabling rapid display of new image data. For ease of explanation, this method will be referred to as Method 1.

[0097] See also Figure 2 After VSync8, the display driver receives the third image. At this point, the underlying layer can recover from 30FPS to 60FPS. However, since the current frame's duration of 19.23ms exceeds the 16.67ms frame length corresponding to 60FPS when the third image is received, the current frame is immediately interrupted, ending at the 19.23ms position with a frame rate of 52FPS. The underlying layer then exits IDLE mode and generates VSync9, starting a new frame. After VSync9 arrives, the display driver can send the third image to the screen quickly.

[0098] See also Figure 2 Once the frame rate recovers to 60 FPS, a VSync signal is generated approximately every 16.67 ms, as shown in the diagram where the time interval between adjacent VSync signals (VSync9-VSync11) is 16.67 ms. This allows the CPU, GPU, and other modules to process images more quickly, and the display driver to trigger display at a higher frequency. Specifically, after VSync10 arrives, the display driver can send the fourth image to the display, and then the screen can display the fourth image; after VSync11 arrives, the display driver can send the fifth image to the display, and then the screen can display the fifth image.

[0099] Method one described above will result in a random frame rate for the frame before exiting IDLE mode (i.e., the frame in which new image data is received). See also... Figure 2 The frame before exiting IDLE mode is the frame between VSync8 and VSync9, and that frame has a frame rate of 52 FPS. This can be understood as... Figure 2 In the example, if the display driver receives the third image earlier, it can exit IDLE mode earlier, and the frame rate of that frame can be higher. Conversely, if the display driver receives the third image later, it can exit IDLE mode later, and the frame rate of that frame can be lower.

[0100] However, some displays do not support all frame rates. When an unsupported frame rate is encountered, display performance will be affected, such as screen freezing. In a specific scenario, for some Touch and Display Driver Integration (TDDI) displays, the VBLK area corresponds to the Touch Panel (TP) slot. When exiting IDLE mode, the current frame can be immediately terminated by modifying the row number of the VBLK area (see the previous section on "3. Methods of Switching Frame Rates in Video Mode"), and then the frame rate can be restored to the frame rate before entering IDLE mode. If the VBLK area is modified to an unexpected row number, i.e., the row number of the VBLK area corresponding to an unsupported frame rate, it is very likely that the TP slot will be interrupted, affecting the normal use of the TP and thus affecting display performance.

[0101] For example, the above Figure 2 The display screen used for display supports frame rates including 120FPS, 90FPS, 60FPS, 55FPS, 45FPS, 40FPS, and 30FPS. However, after receiving the third image, the current frame ends at 52FPS by modifying the row number of the VBLK area, which interrupts the TP slot and affects the display performance.

[0102] As can be seen, although the first method can quickly send new image data to the display, it may result in a frame rate that the display does not support, thus affecting display performance.

[0103] Based on the issues with Method 1, some other implementations (hereinafter referred to as Method 2) allow the underlying layer to wait until the current frame ends after receiving new image data after entering IDLE mode, then restore the frame rate to the level before entering IDLE mode, and exit IDLE mode. This allows the system to run the current frame at the frame rate established after entering IDLE mode after receiving new image data, and then increase the frame rate, thus avoiding the occurrence of unsupported frame rates.

[0104] See Figure 3 After VSync8, upon receiving the third image, the underlying layer can wait for the current 30FPS frame to finish before reverting to 60FPS, exiting IDLE mode, and then generating VSync9', starting a new frame. After VSync9' arrives, the display driver can send the third image to the screen. This prevents frame rates that the display does not support.

[0105] See also Figure 3 Subsequently, when VSync10' and VSync11' arrive in sequence, the display driver can send the 4th and 5th images for display. Please refer to the previous explanation of sending the 4th and 5th images for display, which will not be repeated here.

[0106] In method two above, the frame rate of the current frame receiving new image data remains at the reduced frame rate after entering IDLE mode, rather than being random, thus avoiding frame rates that the display does not support. See also... Figure 3 If new image data is received after VSync8, the current frame is the frame between VSync8 and VSync9', and the frame rate of this frame is still 30 FPS.

[0107] However, if the frame rate is switched only after the current frame ends, it may take a long time to send new image data for display, resulting in high display latency. See also... Figure 3 After the third image is synthesized, it needs to wait for a duration of t1 until the current 30FPS frame ends, and then the display driver sends the third image to the screen. Of course, if the third image is synthesized earlier, the display latency may be higher, and vice versa.

[0108] As can be seen, although the second method can avoid frame rates that the display does not support, there is a certain possibility that the transmission and display of new image data will have a high latency.

[0109] In summary, to address the issues of the two implementation methods, this application provides an image display method, which includes: after entering IDLE mode, running at a second frame rate (e.g., 30 FPS), and after generating new image data, switching the frame rate from the second frame rate to a third frame rate, exiting IDLE mode after one frame of the third frame rate has ended, and displaying the new image data.

[0110] In this method, the third frame rate is higher than the second frame rate. Because the third frame rate is higher than the second frame rate, compared to method two, the frame length corresponding to the current frame can be shortened, allowing new image data to be displayed earlier and reducing display latency. For example, with a second frame rate of 30 FPS and a third frame rate of 45 FPS, see [link to example]. Figure 4 After VSync8, upon receiving the third image, the underlying layer switches to 45FPS. After the current frame at 45FPS ends, it exits IDLE mode and generates VSync9. The display driver then sends the third image to the screen with lower latency. Figure 4 The display delay t2 in the middle is less than Figure 3 The display delay t1 in the middle.

[0111] Furthermore, the third frame rate is a frame rate supported by the display. Therefore, compared to method one described above, it avoids switching to a frame rate not supported by the display, thus ensuring display performance. For example, Figure 4The 45FPS in the figure refers to the frame rate supported by the display. Therefore, modifying the VBLK area to meet 45FPS will not interrupt the TP slot, thus ensuring display performance.

[0112] Furthermore, after a frame at the third frame rate ends, the underlying layer can restore the frame rate to the first frame rate, thus resuming operation at the frame rate before entering IDLE mode. See also... Figure 4 After a frame at 45 FPS ends, the underlying layer can restore the frame rate to 60 FPS, generating a VSync signal approximately every 16.67 ms. As shown in the diagram, the time interval between adjacent VSync signals between VSync9” and VSync11” is 16.67 ms. This allows the CPU, GPU, and other modules to process images faster, and the display driver to trigger display at a higher frequency. Specifically, after VSync10” arrives, the display driver can send the fourth image to the display, and then the screen can display the fourth image; after VSync11” arrives, the display driver can send the fifth image to the display, and then the screen can display the fifth image.

[0113] For example, the electronic devices in this application embodiment may be mobile phones, tablets, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices, and / or smart city devices, etc., that have image processing and display requirements. This application embodiment does not impose any special limitations on the specific form of the electronic device.

[0114] See Figure 5 This is a hardware structure diagram of an electronic device. For example... Figure 5As shown, taking a mobile phone as an example, the electronic device may include a processor 210, an external memory interface 220, an internal memory (RAM) 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, a sensor module 280, buttons 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, etc.

[0115] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the mobile phone. In other embodiments, the mobile phone may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0116] The processor 210 may include one or more processing units, such as an application processor (AP), a modem, a central processing unit (CPU), a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural network processing unit (NPU).

[0117] Different processing units can be independent devices or integrated into one or more processors. For example, an AP can integrate a CPU, GPU, etc.

[0118] In some embodiments, the processor 210 can execute an image processing method by running instructions stored in the internal memory 221.

[0119] In addition, one or more processing units in the aforementioned processor 210, as well as other components of the mobile phone (such as memory, input / output interfaces, etc.), can be integrated into the SoC.

[0120] The charging management module 240 receives charging input from the charger. The power management module 241 connects the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240, providing power to the processor 210, internal memory 221, display screen 294, camera 293, and wireless communication module 260, etc. The power management module 241 can also monitor parameters such as battery capacity, battery cycle count, and battery health status.

[0121] The wireless communication function of a mobile phone can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor, and baseband processor.

[0122] Mobile phones can achieve display functions through GPU, display screen 294, and application processor. The GPU is a microprocessor for image processing, connecting the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering.

[0123] Mobile phones can achieve camera functions through the camera 293, ISP, video codec, GPU, display 294, application processor AP, neural network processor NPU, etc.

[0124] Mobile phones can perform audio functions, such as music playback and recording, through an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headphone jack 270D, and an application processor.

[0125] Display screen 294 is used to display images, videos, etc. Display screen 294 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, a mobile phone may include one or more displays screens 294.

[0126] The software system of the aforementioned electronic device (such as the software system of a SoC) can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses a layered architecture of Android. TM Taking a system as an example, this section illustrates the software structure of an electronic device. A layered architecture divides the software system of an electronic device into several layers, each with a clear role and function, and the layers communicate with each other through software interfaces.

[0127] See Figure 6 Taking a mobile phone as an example, the hardware and software architecture of an electronic device can include the Application Layer, Application Framework Layer, Native Layer, Hardware Abstraction Layer (HAL), Kernel Layer, and Hardware Layer. Among them, the Application Layer, Application Framework Layer, Native Layer, and Hardware Abstraction Layer belong to user space, while the Kernel Layer and Hardware Layer belong to kernel space.

[0128] The application layer can accommodate various applications such as email, video players, memo pads, alarm clocks, settings, and games. When these applications run in the foreground, they can all process images and display them through the electronic device.

[0129] The application framework layer provides the application programming interface (API) and programming framework for the application layer. The application framework layer includes some predefined functions.

[0130] The application framework layer can include system services such as the window manager, activity manager, and resource manager.

[0131] The local layer includes an image compositer (SurfaceFlinger). SurfaceFlinger can be used for layer compositing, backlighting, screen brightness control, and more.

[0132] For example, SurfaceFlinger can assign compositing tasks to the rendered layers, either to the Hardware Composer (HWC) in the Hardware Abstraction Layer or to the GPU. For instance, if the compositing task includes tasks such as rounded corner clipping, it can be assigned to the GPU for image compositing; if the compositing task only includes simple layer overlay tasks, it can be assigned to the HWC. This allows for control over image compositing.

[0133] The Hardware Abstraction Layer (HWC) can encapsulate kernel-level drivers, providing calling interfaces to higher layers. The HWC can include HWC. HWC can be used for image compositing and for submitting the composited image data to the display driver. It should be noted that the image data obtained by the GPU during image compositing is also submitted to the display driver via HWC.

[0134] The kernel layer (also known as the driver layer) includes drivers that enable the hardware to function. Specifically, the kernel layer includes display control commit modules (such as CRTC commit) and display drivers (such as the LCD Driver).

[0135] The display control submission module can be used for information transfer between the upper layer and the driver layer to achieve display control. For example, the display control submission module can transmit an image from HWC to the display driver.

[0136] A display driver can be used to send images to a display screen for display, i.e., to send them for display. For example, a display driver can send images for display in video mode.

[0137] In this embodiment of the application, the display driver can also be used to calculate the third frame rate, thereby balancing display latency and display performance.

[0138] The hardware layer can include hardware such as CPU, GPU, and display screen.

[0139] It is understandable that the above Figure 5 and Figure 6 In descriptions of electronic devices, the division of hardware and software modules is exemplary and not limited to this in practice.

[0140] The image display method provided in this application can be executed in an electronic device having the above-described hardware and software structure. See also Figure 7 Before entering IDLE mode, the system runs at a first frame rate (e.g., 60 FPS as mentioned above). Correspondingly, the SoC can trigger image display at the first frame rate. Image display methods include:

[0141] S701: If the duration of unprocessed new image data exceeds the first duration, the SoC reduces the frame rate to the second frame rate and enters IDLE mode.

[0142] The first duration can be measured in time, such as 50ms, 80ms, etc. Alternatively, the first duration can be measured in frames, such as 3 frames, 5 frames, etc. This application does not impose specific limitations on this.

[0143] Reducing the frame rate includes, but is not limited to, the following operations: modifying the number of rows in the VBLK region or modifying the transmission time of each row of data to match the second frame rate; modifying the clock frequency of the graphics card (including the GPU) to adjust the speed at which the graphics card processes images.

[0144] The new image data can also be understood as new commit data, such as the data submitted by the display control commit module to the display driver. It can be understood that the display control commit module submits data to the display driver by storing the data in a cache, and then the display driver retrieves it from the cache.

[0145] If no new image data is processed within the first time period, the SoC can reduce the frame rate and enter IDLE mode to run at a lower speed. For example, the GPU can process images at a lower speed, and the display driver can send images to the display at a lower second frame rate, thereby reducing device power consumption.

[0146] In one specific implementation, the second frame rate is the minimum frame rate supported by the display, which minimizes device power consumption.

[0147] Taking a minimum supported frame rate of 30 FPS as an example, the second frame rate can be 30 FPS. Before entering IDLE mode, the SoC runs at 60 FPS and can generate a VSync signal at 16.67ms intervals to trigger the display. Figure 4 VSync1 to VSync5 are shown in the diagram. After entering IDLE mode, the SoC runs at 30 FPS and can generate VSync signals at 33.33ms intervals to trigger the display, as shown below. Figure 4 The VSync6 to VSync8 diagrams are shown in the table.

[0148] It should be noted that in this embodiment, the SoC frame rate switching, including S701 and the frame rate switching in S706-S707 below, mainly refers to the switching of the frame rate by the underlying modules used for image processing and display, thereby adjusting the speed of image processing and display. The underlying modules used for image processing and display (hereinafter referred to as the underlying layer) include the display driver and display control submission module at the kernel layer, and the CPU and GPU at the hardware layer. That is to say, the upper layers (including the application layer, application framework layer, and local layer) are generally unaware of the frame rate reduction and do not enter IDLE mode. For example, the underlying layer can call the interface for entering IDLE mode to perform operations such as reducing the frame rate to enter IDLE mode.

[0149] It should be noted that the operations to be performed when entering IDLE mode may also include other operations, such as disabling some interrupts, which will not be elaborated on in this embodiment.

[0150] S702. The SoC detects whether new image data exists. If yes, execute S703; otherwise, repeat S702.

[0151] For example, when new image data is generated, the new image data is submitted to the display driver, which can detect whether new image data exists based on whether it has been received.

[0152] If no new image data is detected, the SoC can remain in IDLE mode. It's understandable that after entering IDLE mode, there may be no new image data for a period of time. Figure 4 During the transition from VSync5 to VSync8, the SoC can remain in IDLE mode to reduce device power consumption. Meanwhile, the SoC will repeatedly execute S702 until new image data is detected, at which point it will terminate.

[0153] When new image data is detected, the SoC can exit IDLE mode. Similarly, exiting IDLE mode mainly refers to the lower-level system exiting IDLE mode; the upper layers are unaware of this. For example, the lower-level system can call the IDLE mode exit interface to perform the exit operation. The exit operation includes switching the frame rate, as shown in S703-S707 below. It should be noted that other operations related to exiting IDLE mode, such as disabling interrupts, will not be discussed in detail here.

[0154] S703: The SoC checks whether to wait until the current frame ends to switch the frame rate. If yes, then execute S704; otherwise, immediately revert to the first frame rate.

[0155] In short, the S703 switches the frame rate when it detects whether to use method two mentioned above.

[0156] In one specific implementation, the SoC records an identifier indicating when the frame rate will switch. If this identifier is the first identifier (e.g., 1), it indicates that the frame rate should be switched after the current frame ends, i.e., the timing for switching the frame rate in method two described above is used. If this identifier is the second identifier (e.g., 0), it indicates that the frame rate can be switched immediately, i.e., the timing for switching the frame rate in method one described above is used.

[0157] In some embodiments, if it is detected that the frame rate needs to be switched after the current frame ends, it indicates that in order to avoid the display from displaying a frame rate that is not supported, the display can be sent later. The following steps S704 and subsequent steps can be performed to switch the frame rate to the third frame rate first, and then send the display after one frame of the third frame rate ends. Compared with the second method, the new image data can be sent earlier.

[0158] If it is detected that the frame rate needs to be switched immediately without waiting for the current frame to end, then the frame rate can be immediately restored to the first frame rate and the display can be executed. In this way, for some scenarios with high requirements for display latency, the method of switching the frame rate immediately can still be used to reduce the display latency.

[0159] In other embodiments, the underlying layer can directly adopt the following S704 and subsequent steps for all scenarios, first switching the frame rate to the third frame rate. This avoids the occurrence of frame rates that the display does not support compared to method one, and allows for more timely display delivery compared to method two. In other words, the above-mentioned S703 can be omitted.

[0160] The S704 SoC retrieves a list of frame rates supported by the display.

[0161] In one specific implementation, the SoC can record a list of frame rates supported by the display. Before switching frame rates, the SoC can obtain this list to determine the third frame rate. For example, if the display supports frame rates including 120FPS, 90FPS, 60FPS, 55FPS, 45FPS, 40FPS, and 30FPS, then the frame rate list could be {120FPS, 90FPS, 60FPS, 55FPS, 45FPS, 40FPS, 30FPS}.

[0162] The S705 and SoC calculate the second duration between the current moment and the first moment, where the first moment is the start time of the current frame.

[0163] The current moment (also known as the second moment) refers to the moment when the S705 process executes after the SoC generates new image data (or the display driver receives new image data). See also Figure 8 , Figure 8 For the aforementioned Figure 4 The magnified view between "VSync8" and "VSync9" shows that the current time is T0 after the third image is synthesized.

[0164] Here, the current frame refers to the frame that generates new image data, and the first moment is the start moment of that frame.

[0165] If the start of a frame is measured by the VSync signal, then the first moment is the moment corresponding to the most recent VSync signal preceding the current moment. See also... Figure 8 If the most recent VSync signal before time T0 is VSync8, then the first time is time T3, which corresponds to VSync8.

[0166] The SoC subtracts the first moment from the current moment, such as... Figure 8From T0-T3, we can obtain the second duration. This second duration can be understood as the duration that the current frame has been running.

[0167] The S706 and SoC switch the frame rate to a third frame rate based on the second duration. The third frame rate is greater than or equal to the second frame rate and is listed in the frame rate list.

[0168] Taking a second frame rate of 30 FPS as an example, the third frame rate can be 30 FPS or higher, and the frame rate supported by the display screen can also be used.

[0169] Furthermore, the frame length corresponding to the third frame rate is greater than or equal to the second duration mentioned above, and correspondingly, the end time of the current frame corresponding to the third frame rate is at or after the current time. In this way, it can be ensured that the current frame corresponding to the third frame rate ends only after the current time is received, and then the subsequent sending and display of new image data is performed.

[0170] See also Figure 8 The second duration between T0 and T3 is 19.23ms. The third frame rate can be 45FPS. The frame length corresponding to 45FPS is 22.22ms. 22.22ms > 19.23ms, that is, the end time T4 of the current frame corresponding to 45FPS is after time T0. The display driver can send the third image to the display at time T4 after time T0.

[0171] Furthermore, among the frame rates greater than or equal to the second frame rate and supported by the display, there may be multiple frame rates with frame lengths greater than or equal to the second duration. In this case, the third frame rate can be any of the multiple frame rates. In one specific implementation, the third frame rate can be the maximum frame rate among the multiple frame rates. For example, the multiple frame rates include 45 FPS and 40 FPS, and the underlying layer can set the frame rate to 45 FPS. In this way, the current frame corresponding to the third frame rate has the shortest frame length, which can end the current frame earlier, thereby allowing new image data to be displayed earlier.

[0172] Generally speaking, the earlier the new image data is received after the first moment, the earlier the current moment in S705 mentioned above, and the higher the third frame rate that the SoC can determine. Conversely, the later the new image data is received, the later the current moment in S705 mentioned above, and the lower the third frame rate that the SoC can determine. Specifically, the third frame rate can have the following possibilities:

[0173] The first scenario is that the third frame rate is greater than or equal to the first frame rate.

[0174] If new image data is received very early, and the second duration is less than or equal to the frame length corresponding to the first frame rate, the third frame rate can be the first frame rate. For example, if the first frame rate is 60 FPS, and the corresponding frame length is 16.67 ms, see [link to relevant documentation]. Figure 9 If the second duration between the current time T5 and time T3 is 10ms, and 10ms < 16.67ms, then the end time T6 of the current frame corresponding to 60FPS is after the current time T5, and the SoC can switch the frame rate to at least 60FPS.

[0175] The second type is where the third frame rate equals the second frame rate.

[0176] If the new image data is received very late, and the second duration is greater than the frame length corresponding to the fourth frame rate, but less than or equal to the frame length corresponding to the second frame rate, then the third frame rate can be the second frame rate.

[0177] The fourth frame rate is the smallest frame rate among those supported by the display and greater than the second frame rate. For example, if the second frame rate is the smallest frame rate supported by the display, then the fourth frame rate is the second largest frame rate among those supported by the display, ordered from smallest to largest. For instance, if the display supports a list of frame rates {120FPS, 90FPS, 60FPS, 55FPS, 45FPS, 40FPS, 30FPS}, and the second frame rate is 30FPS, and the display supports frame rates greater than the second frame rate (40FPS, 45FPS, 55FPS, 60FPS, 90FPS, 120FPS), with the smallest being 40FPS, then the fourth frame rate is 40FPS.

[0178] For example, the second frame rate is 30 FPS, corresponding to a frame length of 33.33 ms, and the fourth frame rate is 40 FPS, corresponding to a frame length of 25 ms. See [link to documentation]. Figure 10 The second duration between the current time T7 and time T3 is 28ms. Since 25ms < 28ms < 33.33ms, the end time T8 of the current frame corresponding to 40FPS is before the current time T7, and the end time T9 of the current frame corresponding to 30FPS is after the current time T9. The underlying layer can maintain a frame rate of 30FPS.

[0179] It should be noted that in the second case, since the new image data is received too late, the frame rate can be maintained at the second frame rate. Correspondingly, the data can only be displayed after the current frame at the second frame rate has finished running, and the effect is the same as the second method mentioned above.

[0180] The third type is where the third frame rate is greater than the second frame rate but less than the first frame rate.

[0181] When the timing of receiving new image data is appropriate, and the second duration is greater than the frame length corresponding to the first frame rate and less than or equal to the frame length corresponding to the fourth frame rate, the third frame rate is greater than the second frame rate and less than the first frame rate.

[0182] That is, the above Figure 8 and Figure 4 The situation shown will not be elaborated further here.

[0183] Furthermore, the timing of the execution of S704 mentioned above is not based on... Figure 7 As shown, S704 can be executed at any time after S701 and before S706.

[0184] S707: After a frame at the third frame rate ends, the SoC resumes the first frame rate and exits IDLE mode.

[0185] The end time of a frame can be the time when the last line of VFP signal transmission ends, which can be approximately understood as the time when the next VSync signal arrives.

[0186] After switching the frame rate to the third frame rate, when the time interval from the first moment reaches the frame length corresponding to the third frame rate, the end time of the current frame is reached, that is, the end of one frame of the third frame rate. The SoC can generate a VSync signal, and a new frame begins.

[0187] See also Figure 8 If the third frame rate is 45 FPS, then at time T4, which is approximately 22.22 ms away from time T3, a frame at 45 FPS ends, and the SoC can generate VSync9.

[0188] See also Figure 9 If the third frame rate is 60 FPS, then at time T6, which is approximately 16.67 ms away from time T3, a 60 FPS frame ends, and the SoC can generate VSync9.

[0189] See also Figure 10 If the third frame rate is 30 FPS, then at time T9, which is approximately 33.33 ms away from time T3, a 30 FPS frame ends, and the SoC can generate VSync9.

[0190] As explained above, the reduction to the second frame rate when entering IDLE mode and the increase to the third frame rate when exiting IDLE mode are both triggered by the underlying layer. The upper layers (including the application layer and application framework layer) do not require frame rate adjustment and continue to run at the first frame rate. After running one frame at the third frame rate, the SoC reverts to the first frame rate, allowing the underlying layer to resume processing and displaying the image at the first frame rate. This ensures a frame rate consistent with the upper layers.

[0191] After the above S707 process restores the frame rate to the first frame rate and completes other operations to exit IDLE mode, exiting IDLE mode can be achieved.

[0192] In addition, in the case of "first type, the third frame rate is greater than or equal to the first frame rate" in S706 mentioned above, the third frame rate may be equal to the first frame rate. The SoC will switch the frame rate to the third frame rate, which means restoring the frame rate to the first frame rate. Therefore, after one frame of the third frame rate ends, there is no need to restore it again. The "After one frame of the third frame rate ends, the SoC restores the first frame rate" in S707 can be omitted. Furthermore, after setting the frame rate to the third frame rate in S706, you can exit IDLE mode.

[0193] S708: After a new frame begins, the SoC sends new image data to the display.

[0194] See also Figures 8-10 When "VSync9" arrives, a new frame begins. The underlying layer can send new image data to the display screen after "VSync9", such as starting to send new image data after the SOF signal following "VSync9". For details about the SOF signal, please refer to the previous section "2. Transmission Signals of Image Data in Video Mode", which will not be repeated here.

[0195] S709, the display shows a new image.

[0196] Therefore, it can be seen that adopting Figure 7 The proposed solution, after entering IDLE mode at the underlying level, upon receiving new image data, can increase the frame rate of the current frame to a third frame rate based on the duration of the current frame up to the current moment. This allows the current frame to end and a new frame to begin more quickly after the current moment, thus facilitating the timely delivery of new image data for display. Furthermore, the third frame rate is a frame rate supported by the display screen, ensuring display performance after switching frame rates.

[0197] In addition, in the above Figure 7 In one implementation, the VSync signal is used to determine the start of a frame. In other implementations, the SOF signal can also be used. In these implementations, the first moment in S705 above refers to the moment corresponding to the most recent SOF signal before the current moment. In S707 above, the end moment of a frame can be the moment when the last line of VBP signal transmission ends, which can be approximately understood as the moment when the next SOF signal arrives. That is, after waiting for the SOF signal, the SoC resumes the first frame rate.

[0198] At this point, it should be noted that the aforementioned Figure 7The proposed solution, illustrated by the interaction between the SoC and the display, refers to actions within the SoC that are actually implemented, such as actions of the display driver. For example, S701-S707 can all be executed by the display driver.

[0199] The following further combines Figure 4 and Figure 8 The above scenarios, with the display driver as the main execution entity, are introduced. Figure 7 Implementation process of the plan:

[0200] The first phase runs at 60 FPS.

[0201] Before VSync1, the display driver receives the first image, that is, it receives new image data. If the duration of not receiving new image data in S701 exceeds the first duration, the first image is sent to the display and the IDLE mode is not entered.

[0202] Before VSync2, the display driver receives the second image, that is, it receives new image data. If the duration for which S701 has not received new image data exceeds the first duration, the second image will be sent to the display and the IDLE mode will not be entered.

[0203] From VSync2 to VSync4, the display driver did not receive any new image data, but the duration did not exceed the first duration. Since the duration of not receiving new image data in S701 does not exceed the first duration, the second image can be sent to the display repeatedly without entering IDLE mode.

[0204] Before VSync5, the display driver had not received any new image data, and the duration of this delay exceeded the first duration. This satisfies the requirement that the duration of the delay in receiving new image data in S701 exceeds the first duration. In VSync5, the second image is repeatedly sent to the display, and the system enters IDLE mode, switching the frame rate to 30FPS.

[0205] Among them, the time interval between adjacent VSyncs from VSync1 to VSync5 is 16.67ms, which means that the trigger display is 60FPS.

[0206] The second phase runs at 30 FPS.

[0207] From VSync5 to VSync8, the display driver did not receive any new image data. The S702's detection result was: no new image data exists. The second image was repeatedly sent to the display without exiting IDLE mode.

[0208] Among them, the time interval between adjacent VSyncs from VSync5 to VSync8 is 33.33ms, which means that the trigger display is 30FPS.

[0209] The third stage runs at 45 FPS.

[0210] After VSync8, the display driver receives the third image. The S702's detection result is: new image data exists. Therefore, it executes the operation of exiting IDLE mode and switching to 45FPS, as follows:

[0211] When executing S703, the display driver detected that the frame rate needs to be switched after the current frame ends.

[0212] Execute S704 to retrieve the list of frame rates supported by the display driver {120FPS, 90FPS, 60FPS, 55FPS, 45FPS, 40FPS, 30FPS}.

[0213] Execute S705 at time T0, and calculate the second duration of T0-T3 as 19.23ms.

[0214] The execution value S706, 19.23ms, falls between the frame lengths of 18.18ms (corresponding to 55FPS) and 22.22ms (corresponding to 45FPS) in the frame rate list above. The display driver can switch the frame rate to 45FPS, for example, by reducing the number of rows in the VBLK region to achieve 45FPS. After setting it to 45FPS, the frame rate of the current frame is switched to 45FPS, and correspondingly, the frame length of the current frame is only 22.22ms, instead of the 33.33ms frame length corresponding to 30FPS, meaning the display is triggered at 45FPS.

[0215] The third stage runs at 60 FPS.

[0216] Executing S707, after switching to 45FPS, at time T4, the time interval between time T3 and time T4 is 22.22ms, which meets the frame length corresponding to 45FPS. At this point, one frame at 45FPS ends, and the display driver can switch the frame rate to 60FPS, such as by further reducing the number of lines in the VBLK region to achieve 60FPS. Subsequently, the image is processed and displayed at the speed corresponding to 60FPS.

[0217] Executing S708, at time T4, a frame at 45 FPS ends, and VSync9” is generated, indicating the start of the first frame at 60 FPS, triggering the start of the display of the third image. For example, after the SOF signal arrives after VSync9”, the display of the third image begins. It can be understood that the timing of the SOF signal after VSync9” matches 60 FPS.

[0218] Execute S709, and the display shows the third image.

[0219] Subsequently, when VSync10” arrives, it indicates the start of the second frame at 60FPS, triggering the start of the fourth image's display. When VSync11” arrives, it indicates the start of the third frame at 60FPS, triggering the start of the fifth image's display... This process is similar to the display process for VSync1 and VSync2 mentioned above, and will not be repeated here.

[0220] Among them, the time interval between adjacent VSyncs from VSync9” to VSync11” is 16.67ms, which means that the display is restored to 60FPS trigger.

[0221] This application also provides an electronic device, which may include a memory and one or more processors (such as a CPU, GPU, NPU, etc.). The memory and processor are coupled. The memory is used to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device can perform various functions or steps performed by the device in the above method embodiments.

[0222] This application also provides a chip system including at least one processor and at least one interface circuit. The processor and the interface circuit are interconnected via lines. For example, the interface circuit can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit can be used to send signals to other devices (e.g., the processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instructions are executed by the processor, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application does not specifically limit this.

[0223] This embodiment also provides a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the image processing method described above.

[0224] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the image processing method described above.

[0225] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the image processing methods in the above-described method embodiments.

[0226] In this embodiment, the electronic device, computer storage medium, computer program product or chip are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0227] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0228] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0229] The unit described as a separate component may or may not be physically separate. The component shown as a unit can be one physical unit or multiple physical units, that is, it can be located in one place or distributed in multiple different places. Some or all of the units can be selected to achieve the purpose of the solution in this embodiment according to actual needs.

[0230] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0231] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0232] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. An image display method, characterized in that, Applied to an electronic device, the electronic device including a system-on-a-chip (SoC) and a display screen, the method includes: The SoC sends image data to the display screen at a first frame rate; After the SoC has not sent new image data to the display for a continuous period of time exceeding a first duration, the SoC sends image data to the display at a second frame rate; After the SoC processes and obtains the first image, the SoC switches to the third frame rate. After one frame of the third frame rate ends, it switches back to the first frame rate and sends the first image to the display screen. The first image is a new frame of image data. Wherein, the second frame rate is less than the first frame rate, the third frame rate is greater than the second frame rate, and the first frame rate, the second frame rate and the third frame rate are all frame rates supported by the display screen.

2. The method according to claim 1, characterized in that, After displaying the first image, the method further includes: The SoC sends image data to the display screen at the first frame rate.

3. The method according to claim 1 or 2, characterized in that, The frame length corresponding to the third frame rate is greater than or equal to the second duration, the second duration is the time interval between the first moment and the second moment, the second moment is the moment after the first image is obtained, and the first moment is the start moment of the current frame in which the first moment is located.

4. The method according to claim 3, characterized in that, If the second duration is less than or equal to the frame length corresponding to the first frame rate, the third frame rate is greater than or equal to the first frame rate.

5. The method according to claim 3, characterized in that, When the second duration is greater than the frame length corresponding to the first frame rate and less than or equal to the frame length corresponding to the fourth frame rate, the third frame rate is greater than the second frame rate and less than the first frame rate. The fourth frame rate is the minimum frame rate supported by the display screen that is greater than the second frame rate.

6. The method according to claim 3, characterized in that, After the SoC processes and obtains the first image, the method further includes: When the second duration is greater than the frame length corresponding to the fourth frame rate but less than or equal to the frame length corresponding to the second frame rate, the SoC switches to the first frame rate after one frame of the second frame rate ends and sends the first image to the display screen. The fourth frame rate is the minimum frame rate supported by the display screen that is greater than the second frame rate.

7. The method according to any one of claims 1-6, characterized in that, The SoC switches to the third frame rate, including: Upon obtaining the first identifier, the SoC switches to the third frame rate; The first identifier is used to indicate that the frame rate should be switched after the current frame ends.

8. The method according to claim 7, characterized in that, The method further includes: Upon receiving the second identifier, the SoC switches to the first frame rate and sends the first image to the display screen.

9. The method according to any one of claims 1-8, characterized in that, Before the SoC switches to the third frame rate, the method further includes: The SoC calculates the time interval between the first and second moments to obtain the second duration; The SoC determines a third frame rate from the frame rates supported by the display, wherein the frame length is greater than or equal to the second duration; Wherein, the second moment is the moment after the first image is obtained through processing, and the first moment is the start time of the frame in which the first moment is located.

10. The method according to claim 9, characterized in that, The SoC determines a third frame rate with a frame length greater than the second duration from the frame rates supported by the display, including: If there are multiple frame rates with a frame length greater than the second duration, the SoC will determine the third frame rate as the maximum frame rate among the multiple frame rates.

11. The method according to any one of claims 1-9, characterized in that, After the SoC has not sent new image data to the display for a continuous period exceeding a first duration, the method further includes: The SoC switches to the second frame rate and enters idle mode. The SoC sends image data to the display screen at a second frame rate, including: In the IDLE mode, the SoC sends a display to the screen at a second frame rate.

12. The method according to claim 11, characterized in that, After switching to the first frame rate, the method further includes: The SoC exits the IDLE mode; Sending the first image to the display screen includes: After exiting the IDLE mode, the SoC sends the first image to the display screen.

13. The method according to any one of claims 1-12, characterized in that, The first image includes: the first frame of image data processed by the SoC after the SoC has not sent new image data to the display screen for a continuous period exceeding the first duration.

14. An electronic device, characterized in that, include: A display screen, one or more processors, and one or more memories; the one or more processors are coupled to the display panel and the one or more memories; the one or more memories are used to store computer instructions that, when executed by the one or more processors, cause the electronic device to perform the method as described in any one of claims 1-13.

15. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the computer instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-13.

16. A computer program product comprising computer instructions, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-13.

Images