Image processing method and electronic equipment

Abstract

The invention discloses an image processing method. The method comprises the steps that an application program draws and renders after a first software Vsync signal to obtain first image data; the image synthesis system monitors a first PresentFen within a first duration after the second software Vsync signal, first image data is synthesized to obtain a first image, and the first software Vsync signal and the second software Vsync signal are software Vsync signals generated at a first frame rate; the application program draws and renders after the third software Vsync signal to obtain second image data; the image synthesis system monitors a second PresentFen within a second duration after the fourth software Vsync signal and synthesizes second image data to obtain a second image, the third software Vsync signal and the fourth software Vsync signal are software Vsync signals generated at a second frame rate, when the first frame rate is greater than the second frame rate, the first duration is smaller than the second duration, and when the first frame rate is greater than the second frame rate, the second frame rate is smaller than the second duration, the second frame rate is greater than the second frame rate. When the first frame rate is smaller than the second frame rate, the first duration is larger than the second duration. By implementing the technical scheme provided by the invention, picture jamming caused by frame loss is avoided.

Description

Technical Field

[0001] This application relates to the field of display technology, and more particularly to an image processing method and an electronic device. Background Technology

[0002] With the development of computer technology, electronic devices such as mobile phones and tablets have become an indispensable part of people's daily lives. The screens of electronic devices can display various application windows, providing users with a good visual and operational experience.

[0003] To improve the smoothness of the display, Vsync signal calibration is usually enabled. However, in certain optimization scenarios, to reduce power consumption and improve system performance, Vsync signal calibration is disabled, and a software Vsync signal is generated using a software Vsync model. But because software Vsync signals have inherent time deviations, the generated software Vsync signal may differ from the hardware Vsync signal, ultimately causing screen stuttering. Summary of the Invention

[0004] This application provides an image processing method and an electronic device that enables an image synthesis system to synthesize image data after listening to the PresentFence for a longer period of time, thereby avoiding frame drops and screen stuttering.

[0005] In a first aspect, this application provides an image processing method applied to an electronic device, the electronic device including an application program, an image compositing system, and a display system. The method includes: the application program drawing and rendering first image data after a first software Vsync signal; the image compositing system listening to a first PresentFence within a first time period after a second software Vsync signal, compositing the first image data to obtain a first image, wherein the first software Vsync signal and the second software Vsync signal are software Vsync signals generated at a first frame rate, the second software Vsync signal is adjacent to the first software Vsync signal, and the second software Vsync signal is located after the first software Vsync signal; The program draws and renders the second image data after the third software Vsync signal to obtain the second image data; the image compositing system listens for the second PresentFence within a second duration after the fourth software Vsync signal, and composites the second image data to obtain the second image. The third and fourth software Vsync signals are software Vsync signals generated at the second frame rate. The third software Vsync signal is located after the second software Vsync signal, and the fourth software Vsync signal is adjacent to and located after the third software Vsync signal. When the first frame rate is greater than the second frame rate, the first duration is less than the second duration. When the first frame rate is less than the second frame rate, the first duration is greater than the second duration.

[0006] By adopting the above technical solution, even if the software Vsync signal generated by the software Vsync model is inconsistent with the hardware Vsync signal when Vsync signal calibration is turned off, the image compositing system can be given more time to complete image processing tasks by setting an appropriate waiting time for PresentFence according to the current frame rate after the software Vsync signal. This allows the image compositing system to listen to PresentFence within this time limit and then perform image data compositing, thereby avoiding frame drops and screen stuttering.

[0007] In conjunction with some embodiments of the first aspect, in some embodiments, after the image compositing system detects the first PresentFence within a first duration following the second software Vsync signal, composites the first image data, and obtains the first image, the system further includes: after the page is idle for a period of time, the image compositing system controls the switching from a first frame rate to a second frame rate, wherein the first frame rate is greater than the second frame rate.

[0008] By adopting the above technical solution, when the page maintains the first frame rate and remains still for a period of time without any user operation, the power consumption can be reduced while ensuring performance by lowering the frame rate.

[0009] In conjunction with some embodiments of the first aspect, in some embodiments, after the image synthesis system detects the first PresentFence within a first duration following the second software Vsync signal, synthesizes the first image data, and obtains the first image, the system further includes: performing a click operation on the page, and the image synthesis system controls the switching from a first frame rate to a second frame rate, wherein the first frame rate is less than the second frame rate.

[0010] By adopting the above technical solution, when the page maintains a low initial frame rate for a period of time, and the user performs a click operation, such as clicking the play button, the frame rate needs to be increased to meet the playback requirements in order to better display the page or play the video.

[0011] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes: when switching to a second frame rate, the image synthesis system determines a second duration based on the second frame rate.

[0012] By adopting the above technical solution, determining the second duration based on the second frame rate ensures that the processing time for each frame matches the current frame rate, helping to reduce screen stuttering and jitter, and providing a smoother visual experience. Simultaneously, the precise calculation of the second duration allows the system to rationally allocate computing resources according to actual needs, avoiding resource waste or excessive consumption. For example, when an extremely high frame rate is not required, the system can appropriately reduce the processing frequency, saving computing resources.

[0013] In conjunction with some embodiments of the first aspect, in some embodiments, the image synthesis system determines the second duration based on the second frame rate, including: the image synthesis system obtains the second frame interval corresponding to the second frame rate, calculates the product of the second frame interval and a preset threshold, and obtains the second duration.

[0014] By adopting the above technical solution, the calculated second duration provides a precise time reference for the processing of each frame, ensuring that the processing time of each frame meets expectations and avoiding frame rate fluctuations caused by excessively long or short processing times. Furthermore, the introduction of a preset threshold allows the image synthesis system to flexibly adjust the processing time of each frame according to actual conditions, avoiding unnecessary resource waste.

[0015] In conjunction with some embodiments of the first aspect, in some embodiments, the first frame rate is 60 FPS and the second frame rate is 1 FPS; or the first frame rate is 1 FPS and the second frame rate is 60 FPS.

[0016] By adopting the above technical solutions, the page can switch from a long-term playback state to a static state, from a high frame rate to a low frame rate, and vice versa.

[0017] In conjunction with some embodiments of the first aspect, in some embodiments, the image synthesis system detects the second PresentFence within a second duration after the fourth software Vsync signal and synthesizes the second image data, including: the image synthesis system reads the second duration at the start of the fourth software Vsync signal and detects the second PresentFence at the start of the fourth software Vsync signal; when the image synthesis system detects the second PresentFence within the second duration, it synthesizes the second image data.

[0018] By adopting the above technical solution, the second duration is read at the start of the fourth software Vsync signal. Only the second duration calculated based on the second frame rate during the last frame rate switch needs to be read, without recalculation, which can save computing resources. At the same time, the second PresentFence is monitored within the second duration, and the duration of waiting for the second PresentFence can be dynamically adjusted according to actual needs to avoid the time being too long or too short, which would affect the smoothness of the screen display.

[0019] In conjunction with some embodiments of the first aspect, in some embodiments, the method further includes: before switching to the second frame rate, the image synthesis system determines a first duration based on the first frame rate.

[0020] By adopting the above technical solution, determining the first duration based on the first frame rate ensures that the processing time for each frame matches the current frame rate, helping to reduce screen stuttering and jitter, and providing a smoother visual experience. Simultaneously, the precise calculation of the first duration allows the system to rationally allocate computing resources according to actual needs, avoiding resource waste or excessive consumption. For example, when an extremely high frame rate is not required, the system can appropriately reduce the processing frequency, saving computing resources.

[0021] In conjunction with some embodiments of the first aspect, in some embodiments, the image synthesis system determines the first duration based on the first frame rate, including: the image synthesis system obtains the first frame interval corresponding to the first frame rate, calculates the product of the first frame interval and a preset threshold, and obtains the first duration.

[0022] By adopting the above technical solution, the calculated first duration provides a precise time reference for the processing of each frame, ensuring that the processing time of each frame meets expectations and avoiding frame rate fluctuations caused by excessively long or short processing times. Furthermore, the introduction of a preset threshold allows the image synthesis system to flexibly adjust the processing time of each frame according to actual conditions, avoiding unnecessary resource waste.

[0023] In conjunction with some embodiments of the first aspect, in some embodiments, the image compositing system listens to the first PresentFence within a first duration after the second software Vsync signal and composites the first image data, including: the image compositing system reads the first duration at the start time of the second software Vsync signal and listens to the first PresentFence at the start time of the second software Vsync signal; when the image compositing system listens to the first PresentFence within the first duration, it composites the first image data.

[0024] By adopting the above technical solution, the first duration is read at the start of the second software Vsync signal. Only the first duration calculated based on the first frame rate during the last frame rate switch needs to be read, without recalculation, which can save computing resources. At the same time, the first PresentFence is listened to within the first duration, and the duration of waiting for the first PresentFence can be dynamically adjusted according to actual needs to avoid the time being too long or too short, which would affect the smoothness of the screen display.

[0025] In conjunction with some embodiments of the first aspect, in some embodiments, the first PresentFence is the PresentFence corresponding to the previous frame of the first image; the second PresentFence is the PresentFence corresponding to the previous frame of the second image.

[0026] By employing the above technical solution, PresentFence ensures that each frame is displayed strictly in sequence, avoiding frame out-of-order or overlapping issues, which helps maintain the continuity and smoothness of the image. Simultaneously, PresentFence prevents system anomalies caused by incomplete previous frames. By ensuring that the composition of each frame depends on the display of the previous frame, the system can better manage resources and task scheduling, improving overall stability and reliability.

[0027] In a second aspect, embodiments of this application provide an electronic device comprising: one or more processors and a memory; the memory is coupled to the one or more processors and is used to store computer program code, the computer program code including computer instructions, wherein the one or more processors invoke the computer instructions to cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0028] Thirdly, embodiments of this application provide a computer program product containing instructions that, when run on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0029] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect and any possible implementation thereof.

[0030] It is understood that the electronic device provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0031] Figure 1 A schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application; Figure 2 This is a schematic block diagram of the software structure of an electronic device according to an embodiment of this application; Figure 3 This is a schematic diagram of the image processing process of electronic devices in related technologies; Figure 4 This is a schematic diagram illustrating the relationship between signals in related technologies; Figure 5 This is a schematic diagram of the image display process after Vsync signal calibration is enabled in related technologies; Figure 6 This is a schematic diagram of the image display process after calibration with the Vsync signal turned off in related technologies; Figure 7 This is a schematic diagram of the image display process after Vsync signal calibration is turned off in an embodiment of this application; Figure 8 This is a schematic flowchart of an image processing method in an embodiment of this application; Figure 9 This is a schematic diagram illustrating one scenario in an embodiment of this application; Figure 10 This is another flowchart illustrating the image processing method in an embodiment of this application. Detailed Implementation

[0032] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0033] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0034] The image processing method provided in this application can be applied to electronic devices with display functions.

[0035] Electronic devices include terminal devices, which can also be referred to as terminals, user equipment (UE), mobile stations (MS), mobile terminals (MT), etc. Electronic devices can be mobile phones, smart TVs, wearable devices, tablets, computers with wireless transceiver capabilities, virtual reality (VR) electronic devices, augmented reality (AR) electronic devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, and so on. The embodiments in this application do not limit the specific technologies or device forms used in the electronic devices.

[0036] To better understand the embodiments of this application, the structure of the electronic device of this application is described below: Figure 1 This is a schematic diagram of the structure of the electronic device 100 provided in an embodiment of this application. The embodiment will be described in detail below using the electronic device 100 as an example. It should be understood that the electronic device 100 may have a more... Figure 1 The more or fewer components shown can be combined into two or more components, or they can have different component configurations. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits.

[0037] Electronic device 100 may include: processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0038] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0039] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0040] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0041] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0042] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0043] The charging management module 140 is used to receive charging input from the charger. The charger can be a wireless charger or a wired charger.

[0044] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, internal memory 121, external memory, display 194, camera 193, and wireless communication module 160, etc.

[0045] The wireless communication function of electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0046] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0047] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0048] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through audio devices (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0049] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0050] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies. The GNSS may include Global Positioning System (GPS), Global Navigation Satellite System (GLONASS), BeiDou Navigation Satellite System (BDS), Quasi-Zenith Satellite System (QZSS), and / or Satellite Based Augmentation Systems (SBAS).

[0051] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0052] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display panel can also be manufactured using organic light-emitting diodes (OLEDs), active-matrix organic light-emitting diodes (AMOLEDs), flexible light-emitting diodes (FLEDs), miniled, microled, micro-OLEDs, quantum dot light-emitting diodes (QLEDs), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0053] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0054] Internal memory 121 may include one or more random access memory (RAM) and one or more non-volatile memory (NVM).

[0055] Random access memory can include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, fifth generation DDR SDRAM is generally called DDR5 SDRAM), etc. Non-volatile memory can include disk storage devices and flash memory.

[0056] Flash memory can be classified according to its operating principle, including NOR FLASH, NAND FLASH, 3D NAND FLASH, etc.; according to the level of the storage cell, including single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc.; and according to the storage specification, including universal flash storage (UFS) and embedded multimedia card (eMMC), etc.

[0057] The random access memory can be directly read and written by the processor 110. It can be used to store executable programs (such as machine instructions) of the operating system or other running programs, as well as user and application data.

[0058] Non-volatile memory can also store executable programs and user and application data, and can be pre-loaded into random access memory for direct reading and writing by the processor 110.

[0059] The external memory interface 120 can be used to connect to external non-volatile memory, thereby expanding the storage capacity of the electronic device 100. The external non-volatile memory communicates with the processor 110 through the external memory interface 120 to perform data storage functions. For example, music, video, and other files can be stored in the external non-volatile memory.

[0060] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0061] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0062] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or make hands-free calls through the speaker 170A.

[0063] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0064] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0065] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0066] Pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 180A can be disposed on display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to pressure sensor 180A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 194, electronic device 100 detects the intensity of the touch operation based on pressure sensor 180A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 180A. In some embodiments, touch operations applied to the same touch position but with different touch operation intensities can correspond to different operation commands. For example: when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS is executed.

[0067] Touch sensor 180K, also known as a "touch panel," can be located on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a "touch screen." Touch sensor 180K detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be located on the surface of electronic device 100, in a different position than display screen 194.

[0068] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0069] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can be corresponding to touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations applied to different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0070] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0071] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and detach from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, and other SIM cards. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication.

[0072] In addition, an operating system runs on top of the aforementioned components. Examples include iOS, Android (open-source operating system), and Windows. Applications can be installed and run on this operating system. The operating system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses the layered architecture of the Android system as an example to illustrate the software structure of the electronic device.

[0073] like Figure 2 As shown, the software layer is divided into several layers, which communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: applications, application framework, hardware abstraction layer (HAL), and kernel.

[0074] The application layer may include a series of applications, such as e-book applications, browser applications, navigation applications, game applications, video applications, etc., and this application embodiment does not impose any limitations on this.

[0075] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0076] The application framework layer may include a refresh rate management module, an image synthesizer SurfaceFlinger, and a context recognition module. The refresh rate management module can notify the SurfaceFlinger module of the latest refresh rate, which in turn sends the refresh rate to the display driver. The display driver then sends the refresh rate to the DDIC, which controls the monitor to refresh and display the image at that refresh rate.

[0077] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0078] Among them, the context recognition module runs either resident or in a low-power mode, and has the ability to perceive external facts or the environment. The context recognition module can detect relevant events and obtain their states through APIs from the application layer, hardware abstraction layer, or kernel layer, such as detecting page switching and user touch operations. In this embodiment, the main function of the context recognition module is to listen for motion effect events. That is, based on the user-triggered click or swipe operation, it determines that the current application scenario is a dynamic display scenario. The context recognition module then notifies the refresh rate management module to determine a suitable screen refresh rate for this dynamic display scenario. Furthermore, the refresh rate management module switches the current screen refresh rate to the screen refresh rate suitable for this dynamic display scenario, realizing dynamic changes in the screen refresh rate.

[0079] The Hardware Abstraction Layer (HAL layer) can include HWC to provide hardware support for SurfaceFlinger.

[0080] SurfaceFlinger collects all layers that need to be displayed and determines the compositing strategy for each layer to HWC. This involves sending the layer attributes to HWC, including content, position, size, and opacity. After receiving these attributes, HWC categorizes the layers. Typically, HWC identifies which layers are suitable for direct processing and which require GPU processing. HWC layer compositing is suitable for static layers or layers with simple transformations, characterized by low latency and low power consumption. GPU layer compositing is suitable for dynamic layers or layers requiring complex transformations, characterized by high flexibility and high performance. By appropriately allocating layer processing methods, the overall system performance and display effect can be improved. After identification, SurfaceFlinger sends layers marked for HWC processing to HWC for compositing and layers marked for GPU processing to the GPU. The GPU composites the content of each layer and returns it to SurfaceFlinger. These layer processing steps typically include rotation, scaling, and alpha blending. HWC sorts all layers according to their Z-axis order to ensure correct layer order, allowing layers to be stacked in the correct order during final compositing. Furthermore, HWC obtains the GPU-composited layers from SurfaceFlinger and reads the layers that HWC directly processes, then overlays all the layers together in a specified order and size to form the final frame image.

[0081] The kernel layer is the layer between hardware and software. The kernel layer contains at least a display driver. Specifically, in the context of this application, the hardware involved may include a display driver integrated circuit (DDIC), a display screen, and a GPU. The display driver is used to drive the DDIC to complete the display processing and implementation.

[0082] To facilitate understanding, the relevant terms and concepts involved in the embodiments of this application will be introduced below.

[0083] Frame rate, also known as frame rate, is measured in frames per second, denoted as FPS. Frame rate refers to the number of image frames a graphics card generates per unit of time (per second), meaning the number of frames the graphics card can render and display per second. For example, 60 FPS means 60 frames are displayed per second, which can be calculated using the formula FPS = 1 / frame time (seconds). For instance, if the frame time is 0.0167 seconds, then the frame rate is 60 FPS. In video and image display, frame rate is a crucial performance indicator, directly impacting the smoothness and visual effects of the image. A higher frame rate results in smoother visuals and better visual effects. Low frame rates can lead to stuttering and choppy visuals, negatively affecting the user experience. Frame rate is limited by hardware performance and the display device. For example, if the display device's refresh rate is 60 Hz, the maximum frame rate can only reach 60 FPS. Furthermore, frame rate is also affected by system load and resource allocation. For instance, if the CPU or GPU load is too high, the frame rate may decrease.

[0084] Refresh rate refers to the number of times a display device updates the screen content per second, usually expressed in Hertz (Hz). For example, 60 Hz means the screen content is updated 60 times per second. It can be calculated using the formula refresh rate (Hz) = 1 / refresh time (seconds). For example, if the refresh time is 0.0167 seconds, then the refresh rate is 60 Hz. The higher the refresh rate, the smoother the picture and the better the visual effect.

[0085] Typically, when applications on the display switch or users interact with the displayed page, the refresh rate changes, and the frame rate changes accordingly. For example, when the refresh rate switches from 120Hz to 90Hz, the frame rate switches from 120FPS to 90FPS.

[0086] The following is combined Figures 3-6 Explain the image processing process in related technologies. For example... Figure 3 As shown, in the Android system, the image processing process includes image drawing, rendering, compositing, and display, which requires the collaborative work of the application (App), the image compositing system (SurfaceFlinger, Hwcomposer, HWC, GPU, and display driver), and the display system (display driver integrated circuit (DDIC) and screen).

[0087] First, the App draws and renders the page to obtain image data, which is then sent to SurfaceFlinger. SurfaceFlinger sends the rendered image data to HWC to inquire about the image data compositing strategy. After HWC determines the compositing strategy, HWC and / or the GPU composit the image data, and HWC performs the final compositing to obtain the image. The image is then processed by the display driver and sent to DDIC through the Mobile Industry Processor Interface (MIPI). DDIC stores the image in a buffer and controls the display screen to refresh and display the image by scanning the image in the buffer.

[0088] SurfaceFlinger is one of the core components in the Android system responsible for managing and compositing display content. SurfaceFlinger manages image data across multiple layers, each layer representing an application window or a single display element. Image data can be static images, dynamic video streams, or part of the application's interface. SurfaceFlinger uses HWC to merge this image data into a complete frame image.

[0089] To avoid screen tearing, DDIC can output a Vsync signal according to the screen's frame rate. When SurfaceFlinger detects the rising edge of the Vsync signal or a high level of the Vsync signal and has synthesized the next frame of image data, DDIC starts displaying the next frame of image.

[0090] It is understandable that the Vsync signal is a periodic signal, and the Vsync signal period can be set according to the display's frame rate. For example, when the frame rate is 60 FPS, the Vsync signal period can be 16.67 ms, that is, the electronic device generates a control signal every 16.67 ms to trigger the Vsync signal period.

[0091] It should be noted that Vsync signals can be divided into software Vsync signals and hardware Vsync signals. Software Vsync signals include Vsync-APP and Vsync-SF. Vsync-APP is used to trigger the drawing and rendering process. Vsync-SF is used to trigger the compositing process. Hardware Vsync signals, also known as TE signals, are emitted by the DDIC at regular intervals and are used to trigger the display refresh process.

[0092] The relationship between the hardware Vsync signal, Vsync-APP, and Vsync-SF is as follows: Figure 4As shown, Vsync-APP lags behind the hardware Vsync signal phase-APP by a certain time, Vsync-SF lags behind the hardware Vsync signal phase-SF by a certain time, and phase-SF > phase-APP. Typically, the interval between phase-APP and phase-SF is short, and this embodiment does not focus on the error between Vsync-SF and Vsync-APP. For ease of description, in this embodiment, Vsync-SF and Vsync-APP are considered to be synchronous.

[0093] Understandably, the process involves collecting the timestamps of at least six software Vsync signals prior to the current moment. Based on these timestamps and the sequence number of each software Vsync signal, a software Vsync model is constructed, represented by the linear equation y = kx + b, where x is the sequence number of the software Vsync signal and y is its timestamp. Substituting the sequence number of the next software Vsync signal into this equation yields its corresponding timestamp. The software Vsync model is updated every certain period.

[0094] For example, when Vsync signal synchronization is enabled (i.e., the hardware Vsync signal and the software Vsync signal are periodically synchronized). Figure 5 This is a schematic diagram of an electronic device interface display processing flow in a feasible implementation. The images displayed by the electronic device correspond sequentially to Frame 1 and Frame 2, according to time sequence.

[0095] Specifically, the second frame rate remains constant throughout the three Vsync cycles shown in the diagram. After the first software Vsync signal, the app renders frame 1. Once frame 1 is rendered, the app submits the rendered frame 1 to the image compositing system (including SurfaceFlinger, GPU, and HWC). After the second software Vsync signal, the app renders frame 2 and submits the rendered frame 2 to the image compositing system.

[0096] After the second software Vsync signal, the image compositing system composites the rendered frame 1 and generates PresentFence1, which is then submitted to the display system (DDIC and screen). After the start of the third hardware Vsync signal, the display system displays the content corresponding to frame 1, and releases PresentFence1 at time t0 after the display is complete.

[0097] After the third software Vsync signal, SurfaceFlinger synthesizes the rendered frame 2 and generates PresentFence2, which is then submitted to the display system (DDIC and screen).

[0098] It should be noted that, theoretically, PresentFence1 is released within 1ms after the third hardware Vsync signal. In related technologies, SurfaceFlinger waits for the PresentFence of the previous frame for up to 1ms at the start of each frame synthesis (the start time of the software Vsync signal), and begins synthesizing the next frame at the moment that PresentFence is received. That is, frame 2 is synthesized only at time t0. Therefore, even if the display system takes a long time to process the display of frame 1, as long as the release time of PresentFence1 is between the start time of the third hardware signal and t1, SurfaceFlinger can successfully wait for the release of PresentFence1 of frame 1 before synthesizing frame 2, and thus synthesize frame 2.

[0099] PresentFence is a synchronization object used to mark the occurrence of a specific event. In a display system, PresentFence is typically used to mark that the current frame has been fully displayed on the screen. PresentFence is created by SurfaceFlinger after a frame is composed. When submitted to the display system, the system sets PresentFence to the triggered state and releases it when the current frame is finished displaying. When SurfaceFlinger detects that PresentFence has been released, it indicates that the current frame has been fully displayed and begins compositing the next frame. If it does not detect that PresentFence has been released, it will not compose the next frame and will continue to wait, causing screen stuttering.

[0100] However, in certain optimization scenarios (such as low-power scenarios with stable frame rates below 120Hz), to reduce power consumption and improve performance, the system disables Vsync signal calibration. The software Vsync model predicts the timestamp of the next software Vsync signal based on the timestamps of the software Vsync signals generated in the previous few cycles. While this method can save power to some extent, due to the inherent time deviation of the software Vsync signal, the predicted software Vsync signal may be ahead of the actual hardware Vsync signal. In extreme cases, this deviation may be just close to the Vsync signal calibration threshold, but not exceeding it, causing SurfaceFlinger to be unable to wait for the PresentFence of the previous frame to be released within the expected time, thus failing to re-integrate into the current frame, ultimately resulting in screen stuttering.

[0101] For example, with Vsync signal calibration disabled, Figure 6 This is a schematic diagram of a possible implementation of an electronic device interface display process.

[0102] The second frame rate is kept constant throughout the three Vsync cycles shown in the diagram. The specific process is as follows: Figure 5 The difference lies in the fact that the simulated software Vsync signal arrives earlier than the actual hardware Vsync signal due to inherent bias. If PresentFence1 is released at time t2, then the release time of PresentFence1 is delayed by more than 1ms compared to the software Vsync signal. Even if SurfaceFlinger waits another 1ms after the software Vsync signal, it cannot successfully wait for PresentFence1 of frame 1 to be released, and thus frame 2 is not synthesized, resulting in screen stuttering.

[0103] To solve the above Figure 6 To address the problems in the process shown, this application provides an image processing method. This method calculates a second duration for SurfaceFlinger to wait for the PresentFence of the previous frame image to be released based on the changed frame rate (second frame rate) when switching the frame rate (from the first frame rate to the second frame rate). Then, with the subsequent frame rate unchanged, Vsync signal calibration is turned off. After the software Vsync signal, SurfaceFlinger calls the second duration, and within the second duration, it can successfully wait for the PresentFence of the previous frame to be released, thereby starting to synthesize the current frame image and avoiding screen stuttering.

[0104] For example, with Vsync signal calibration disabled, Figure 7 This is a schematic diagram of a possible implementation of an electronic device interface display process.

[0105] The second frame rate is kept constant throughout the three Vsync cycles shown in the diagram. The specific process is as follows: Figure 6 The difference lies in the fact that the simulated software Vsync signal, due to inherent bias, releases PresentFence1 earlier than the actual hardware Vsync signal, at time t3. At the start of the third software Vsync signal, SurfaceFlinger calls the second duration calculated based on the second frame rate during the previous switch from the first to the second frame rate and begins waiting for PresentFence1 to be released, meaning it waits until at least time t4. In fact, t3 is earlier than t4, allowing SurfaceFlinger to successfully wait for PresentFence1 of frame 1 to be released and synthesize frame 2 at time t3, avoiding screen stuttering.

[0106] The image processing method in the embodiments of this application will be described below with reference to the above hardware and software structure diagrams.

[0107] Please see Figure 8 This is an interactive diagram of an image processing method provided in an embodiment of this application.

[0108] Figure 8 The image processing method shown can be executed by an electronic device, which includes an application (APP), an image compositing system (SurfaceFlinger, HWC, GPU), and a display system (DDIC, display screen). Figure 8 As shown, each moment corresponds to a timestamp of a software Vsync signal.

[0109] Figure 8 The method shown includes steps S101 to S110, which will be described in detail below.

[0110] S101, The APP draws and renders the first image data after the first software Vsync signal; The first software Vsync signal is a software Vsync signal generated at the first frame rate. That is, within one cycle following the first software Vsync signal, the frame rate is the first frame rate. The value of the first frame rate differs depending on the page's state.

[0111] The frame interval corresponding to the first frame rate is obtained by taking the reciprocal of the first frame rate.

[0112] Taking an electronic device maintaining a frame rate of 1 FPS when the video playback page is paused, and then playing the video at a frame rate of 60 FPS for an extended period after the user clicks the play button as an example, the aforementioned first frame rate is 1 FPS. In this case, in S101, the APP performs drawing and rendering at a frame interval of 1000 ms corresponding to 1 FPS after the first software Vsync signal to obtain the first image data.

[0113] The first image data may include image data of at least one layer, which may be rendered by different apps or by the same app.

[0114] Understandably, the first image data may include the layer's position coordinates (x, y) on the display screen; the layer's width and height; the layer's transparency, used to control the layer's visibility; transformation information such as layer rotation, scaling, and translation; the layer's stacking order, which determines the layers' order in the final composition; the way the layer is blended with other layers, such as normal mode, additive mode, etc.; and the layer's color format, such as RGB, RGBA, etc.

[0115] S102, The display system displays the previous frame of the first image after the first software Vsync signal and releases the first PrensentFence.

[0116] S103. The image synthesis system detects the first PresentFence within a first time period after the second software Vsync signal, synthesizes the first image data, and obtains the first image.

[0117] Specifically, SurfaceFlinger reads the first duration at the start of the second software Vsync signal and listens for the first PresentFence at the start of the second software Vsync signal. When the first PresentFence is detected within the first duration, the second image data is synthesized. The first PresentFence is the PresentFence corresponding to the previous frame of the first image.

[0118] The second software Vsync signal is generated at the first frame rate, meaning that the frame rate is the first frame rate for one period following the second software Vsync signal. The second software Vsync signal is adjacent to the first software Vsync signal and follows the first software Vsync signal.

[0119] The first PresentFence is the PresentFence corresponding to the previous frame of the first image. Understandably, after the first software Vsync signal, the image compositing system begins compositing the data from the previous frame of the first image and generates the first PresentFence. After compositing, the composite image data and the first PresentFence are submitted to the DDIC. After the second software Vsync signal, the DDIC completes the display preparation for the previous frame and begins controlling the display screen to show the previous frame. After display, it resets the state of the first PresentFence, i.e., releases the first PresentFence. Simultaneously, after the second software Vsync signal, SurfaceFlinger monitors the state changes of the first PresentFence. When it detects that the first PresentFence is in a released state within the first time period, it is considered that the first PresentFence has been released, indicating that the previous frame has been correctly displayed. Then, the first image data is composited to obtain the first image.

[0120] Furthermore, the image synthesis system sends the first image to the display system for display.

[0121] In one feasible implementation, such as Figure 8 As shown, the synthesis process of the previous frame image data mainly includes steps S1 to S4, which will be described in detail below.

[0122] S1. After the second software Vsync signal, Surfaceflinger queries HWC for the synthesis strategy of the previous frame image data. SurfaceFlinger sends the previous frame's image data and the desired compositing method for the corresponding layers to HWC via API, and queries HWC for the optimal compositing strategy for the image data corresponding to each layer, such as GPU compositing or HWC compositing.

[0123] For example, the previous frame image data includes image data from four layers: Surface1, Surface2, Surface3, and Surface4. Surfaceflinger sends a first message to HWC, which is: Surface1, H1cm, W1cm, (x1, y1), scale, rotate, HWC; Surface2, H2cm, W2cm, (x2, y2), scale, rotate, HWC; Surface3, H3cm, W3cm, (x3, y3), scale, rotate, HWC; Surface4, H4cm, W4cm, (x4, y4), no scaling, no rotation, HWC.

[0124] The first message indicates that the desired compositing method for the video data corresponding to Surface1, the bullet screen data corresponding to Surface2, the program window data corresponding to Surface3, and the logo data corresponding to Surface4 is HWC compositing.

[0125] S2 and HWC obtain the current system state, and determine the synthesis strategy of the previous frame image data based on the first image data and the current system state, and mark it.

[0126] The current system status can include hardware status, system resource management status, software status, graphics stack status, user interaction status, network status, etc.

[0127] The hardware status may include, for example, the status of GPU load, the number, type and configuration of displays connected to the system, the amount of available system memory (RAM), the status of hardware accelerators such as the status of Vulkan or OpenGL ES drivers, and the status of other dedicated hardware accelerators (such as video decoders).

[0128] System resource management status may include, for example, battery level and charging status, CPU load, and device temperature status.

[0129] Software status can include, for example, a list of currently running applications and services and their priorities, the status of system services such as SurfaceFlinger itself, the Window Manager, the Activity Manager, etc., and system settings such as user interface (UI) animation settings and performance modes.

[0130] The graphics stack state may include, for example, all active Surfaces and their attributes, buffer queue state: the synchronization state of buffer producers and consumers, and the display nodes associated with each display device and their configuration.

[0131] User interaction states can include, for example, input events: whether a user input event is currently in progress, touch events: whether input on the touchscreen has been detected, and recent focus changes: which application or window currently has focus.

[0132] Network status can include, for example, connection status and bandwidth usage.

[0133] For HWC, the main focus is on state information related to image rendering, such as GPU load, memory status, the currently active Surface list, and display-related hardware status. This information helps HWC determine how to most effectively manage the compositing process of image data from each layer to achieve optimal performance and visual effects. For example, when system memory is low, it may prompt SurfaceFlinger to reduce cached image data; while when GPU load is high, it may choose to use software rendering to alleviate GPU pressure.

[0134] Specifically, HWC assesses current hardware status, system resource management status, and other graphics rendering-related status information, and analyzes the characteristics of each layer, such as whether it contains complex transformations or requires special blending modes, to determine the most suitable compositing strategy. For simple image data (such as still images or simple transformations), HWC can choose a direct display path after simple processing, meaning the image data is directly submitted to the display device after HWC compositing. For image data requiring complex transformations or blending, HWC may choose a GPU compositing path, meaning the compositing is performed by the GPU and submitted to HWC for display.

[0135] Furthermore, HWC tags each layer to indicate which compositing strategy should be used for the image data corresponding to that layer. The tag can be part of some internal data structure, such as a bit field or an enumeration value, used to indicate the compositing strategy for the image data.

[0136] For example, HWC received image data from two layers. Layer A's image data is a static image with its position unchanged and no complex transformations. After evaluation, HWC deemed it suitable for HWC processing and labeled it "HWC". Layer B's image data includes rotation and scaling transformations, and its color format is RGBA. After evaluation, HWC deemed it suitable for compositing using a GPU and labeled it "GPU".

[0137] In this way, HWC can select the optimal compositing strategy for each layer based on the current system state and the characteristics of the layer, and mark it accordingly, thereby achieving efficient and low-power graphics processing.

[0138] For example, in response to the first message mentioned above, HWC sends a second message to SurfaceFlinger. The second message could be: Surface1, H1cm, W1cm, (x1, y1), scale, rotate, HWC; Surface2, H2cm, W2cm, (x2, y2), scale, rotate, HWC; Surface3, H3cm, W3cm, (x3, y3), scale, rotate, HWC; Surface4, H4cm, W4cm, (x4, y4), no scaling, no rotation, HWC.

[0139] The second message indicates that the suggested synthesis method for the video data corresponding to Surface1, the bullet screen data corresponding to Surface2, the program window data corresponding to Surface3, and the logo data corresponding to Surface4 is HWC synthesis.

[0140] S3. If the previous frame image data is marked as being processed by HWC, then HWC synthesizes the previous frame image data to obtain the previous frame image.

[0141] If the previous frame image data is marked as being processed by HWC, SurfaceFlinger sends the previous frame image data to HWC. HWC then synthesizes the image data from each layer and finally synthesizes it into the previous frame image.

[0142] S4. After the previous frame of image data is synthesized, Surfaceflinger generates the first PresentFence and sends it to DDIC.

[0143] It should be noted that after the first PresentFence is sent to DDIC, Surfaceflinger begins to wait for the first PresentFence to be released.

[0144] In another feasible implementation, such as Figure 9 As shown, the process of synthesizing the previous frame image data mainly includes steps S1 to S6, which will be described in detail below.

[0145] S1. After the second software Vsync signal, Surfaceflinger queries HWC for the synthesis strategy of the previous frame image data. S2 and HWC obtain the current system state, and determine the synthesis strategy of the previous frame image data based on the previous frame image data and the current system state, and mark it. S3. If the previous frame image data is marked as being processed by the GPU, then SurfaceFlinger sends the previous frame image data to the GPU. S4, the GPU synthesizes the image data of the previous frame to obtain the previous frame image; S5, GPU sends the previous frame image to HWC via SurfaceFlinger; S6. After the previous frame of image data is synthesized, Surfaceflinger generates the first PresentFence and sends it to DDIC.

[0146] In another feasible implementation, such as Figure 10 As shown, the process of synthesizing the previous frame image data mainly includes steps S1 to S7, which will be described in detail below.

[0147] S1. After the second software Vsync signal, Surfaceflinger queries HWC for the synthesis strategy of the previous frame image data. S2 and HWC obtain the current system state, and determine the synthesis strategy of the previous frame image data based on the previous frame image data and the current system state, and mark it. For example, in response to the first message mentioned above, HWC sends a third message to SurfaceFlinger. The third message could be: Surface1, H1cm, W1cm, (x1, y1), scaling, rotation, GPU; Surface2, H2cm, W2cm, (x2, y2), scaling, rotation, GPU; Surface3, H3cm, W3cm, (x3, y3), scale, rotate, HWC; Surface4, H4cm, W4cm, (x4, y4), no scaling, no rotation, HWC.

[0148] The third message indicates that the recommended compositing method for the video data corresponding to Surface1 and the bullet screen data corresponding to Surface2 is GPU compositing, while the recommended compositing method for the program window data corresponding to Surface3 and the station logo data corresponding to Surface4 is HWC compositing. This can be understood as the previous frame's image data including multiple layers of image data, marked as being processed by both GPU and HWC.

[0149] S3. If the previous frame image data is marked as being processed by the GPU and HWC, then SurfaceFlinger will send the image data marked as being processed by the HWC to the HWC and the image data marked as being processed by the GPU to the GPU. Understandably, SurfaceFlinger sends the image data corresponding to Surface3 and Surface4 to HWC, while SurfaceFlinger sends the image data corresponding to Surface1 and Surface2 to the GPU.

[0150] S4 and HWC synthesize the received image data to obtain the first target image; Understandably, HWC synthesizes Surface3 and Surface4 to obtain the first target image.

[0151] S5 and GPU synthesize the received image data to obtain a second target image, and send the second target image to HWC through SurfaceFlinger; The GPU combines Surface1 and Surface2 to obtain a second target image, and sends the second target image to SurfaceFlinger, which then sends it to HWC.

[0152] In this process, the GPU sends the second target image to SurfaceFlinger, one possible method being through a shared memory mechanism. That is, both SurfaceFlinger and the GPU map to the same memory region. Once the GPU completes image processing and generates the second target image, it updates this shared memory and notifies SurfaceFlinger that the image is available. SurfaceFlinger can then access this shared memory to read the second target image.

[0153] Another feasible approach is to use a Buffer Queue to transfer image data. A Buffer Queue is an abstraction mechanism that allows producers (GPUs) and consumers (SurfaceFlinger) to exchange buffers. First, SurfaceFlinger creates a Buffer Queue instance. The GPU needs to connect to this Buffer Queue to place rendered image data into it. The GPU acquires a buffer by calling APIs provided by the Buffer Queue (such as `acquireBuffer()`), then renders the second target image into that buffer. After image processing is complete, the GPU commits the buffer by calling the Buffer Queue's `releaseBuffer()` method, notifying SurfaceFlinger that the second target image is ready. SurfaceFlinger listens for changes in the Buffer Queue's state, and when it determines that a new buffer is ready, it calls the `acquireBuffer()` method to acquire that buffer and then reads the second target image from it.

[0154] It should be noted that: GPU image data synthesis and HWC image data synthesis can be performed simultaneously, or GPU image data synthesis can be performed first and then HWC image data synthesis, or HWC image data synthesis can be performed first and then GPU image data synthesis. The embodiments of this application do not limit the order of the two synthesis.

[0155] S6 and HWC synthesize the first target image and the second target image to obtain the previous frame image.

[0156] The content on a display screen is usually composed of multiple different layers, each of which can be seen as part of the final image, such as application windows, system interface elements, notification bars, etc.

[0157] To display these layers, HWC needs to merge them into a single image, that is, to combine the first target image and the second target image to obtain the previous frame image. During the compositing process, operations such as scaling, rotating, and adjusting the opacity of the layers may be involved.

[0158] S7. After the previous frame image data is synthesized, Surfaceflinger generates the first PresentFence and sends it to DDIC.

[0159] It is understandable that the synthesis process of the first image data is similar to that of the previous frame image data, and will not be described in detail here.

[0160] Understandably, when the system's tolerable error is less than a preset threshold, the first duration is calculated during the last frame rate switch, which is the switch to the first frame rate. Therefore, the first duration is determined based on the first frame rate, and the calculation result is saved to SurfaceFlinger.

[0161] Where Percent represents the tolerable error of the first image relative to the previous frame image, and the corresponding calculation formula is as follows: Percent=(timestamp-validtimestamp)%mIdealPeriod*kMaxPercent / mIdealPeriod.

[0162] % indicates the remainder, timestamp is the timestamp for releasing the PresentFence corresponding to the first image, aValidaTimestamp is the timestamp for releasing the PresentFence corresponding to the previous frame, mIdealPeriod is the frame interval corresponding to the current frame rate, and kMaxPercent is the maximum percentage, i.e., 100.

[0163] kOutlierTolerancePercent / kMaxPercent are preset thresholds. These preset thresholds can be understood as the maximum tolerance error; that is, within this range, Vsync signal calibration is not required, while outside this range, Vsync signal calibration is required. Preferably, this preset threshold is 20%, but other values ​​such as 30% or 10% are also possible. The higher the accuracy requirement, the smaller the preset threshold should be.

[0164] Specifically, Surfaceflinger obtains the first frame interval corresponding to the first frame rate, calculates the product of the first frame interval and the preset threshold, and obtains the first duration. The corresponding calculation formula is: waitForFenceDuration=mIdealPeriod*(kOutlierTolerancePercent / kMaxPercent).

[0165] waitForFenceDuration is the first duration, mIdealPeriod is the first frame interval corresponding to the first frame rate, and kOutlierTolerancePercent / kMaxPercent are preset thresholds.

[0166] Assuming kOutlierTolerancePercent=20, kMaxPercent=100, and mIdealPeriod=16.67ms, then the first duration is 20*16.67ms / 100=3.33ms.

[0167] Assuming kOutlierTolerancePercent=20, kMaxPercent=100, and mIdealPeriod=33.33ms, then the first duration is 20*33.33ms / 100=6.67ms.

[0168] Under the same preset threshold, the lower the frame rate, the longer the first duration.

[0169] Assuming kOutlierTolerancePercent=30, kMaxPercent=100, and mIdealPeriod=16.67ms, then the first duration is 30*16.67ms / 100=5ms.

[0170] At the same frame rate, the larger the preset threshold, the longer the first duration.

[0171] Optionally, when the system's tolerable error is greater than or equal to a preset threshold, i.e., Percent ≥ kOutlierTolerancePercent, indicating that the error is too large, Vsync signal calibration is started after the second software Vsync signal, and Vsync signal calibration is turned off after calibration is completed.

[0172] Optionally, after step S103 above, Figure 8 The method shown also includes: S104. If the image synthesis system does not detect the first PresentFence within the first time period after the second software Vsync signal, it continues to wait and does not synthesize the first image data.

[0173] S105. When a click operation is performed on the page, the image compositing system controls the switch from the first frame rate to the second frame rate.

[0174] At any time after the second software Vsync, the user can click on the page to trigger a frame rate switch.

[0175] One feasible scenario is that the page is in a state where the video application is not playing a video, and in this state, the page maintains the first frame rate. When the user performs a click action, such as clicking the play button, the page switches to the second frame rate and plays the video at the second frame rate. The first frame rate is lower than the second frame rate.

[0176] Another feasible scenario is when the page is playing a video for an extended period of time, maintaining the first frame rate. When the user performs a click action, such as clicking the pause button, the page switches to the second frame rate and remains unchanged at that rate. In this case, the first frame rate is greater than the second frame rate.

[0177] Optionally, after the page has been idle for a period of time, the image compositing system switches from a first frame rate to a second frame rate. The first frame rate is greater than the second frame rate.

[0178] One feasible scenario is that the page remains on the e-book page for a long time without any user clicks or activities, and then after a period of time, the frame rate is switched to a lower frame rate.

[0179] S106. When switching to the second frame rate, the image synthesis system determines the second duration based on the second frame rate.

[0180] SurfaceFlinger obtains the second frame interval corresponding to the second frame rate, calculates the product of the second frame interval and a preset threshold to obtain the second duration, and saves the calculation result to SurfaceFlinger. The specific calculation process can be found in the calculation process of the first duration described above, and will not be repeated here.

[0181] When the first frame rate is greater than the second frame rate, the first duration is less than the second duration; when the first frame rate is less than the second frame rate, the first duration is greater than the second duration.

[0182] S107, The APP draws and renders the second image data after the third software Vsync signal; The third software Vsync signal is a software Vsync signal generated at the second frame rate. That is, within one cycle following the third software Vsync signal, the frame rate is the second frame rate. The third software Vsync signal is located after the second software Vsync signal.

[0183] The frame interval corresponding to the second frame rate is obtained by taking the reciprocal of the second frame rate. For example, if an electronic device is paused on a video playback page and the user clicks the play button to play the video at 60 FPS for an extended period, then the second frame rate is 60 FPS. In this case, the app performs drawing and rendering based on a frame interval of 16.67 ms corresponding to 60 FPS after the third software Vsync signal, obtaining the second image data.

[0184] The rendering process for the second image data is similar to that for the first image data, and will not be described in detail here.

[0185] S108. The display system displays the frame image preceding the second image after the third software Vsync signal and releases the second PresentFence.

[0186] S109. The image synthesis system detects the second PresentFence within a second time period after the fourth software Vsync signal, synthesizes the second image data, and obtains the second image.

[0187] The fourth software Vsync signal is generated at the second frame rate. That is, within one period following the fourth software Vsync signal, the frame rate is the second frame rate. The fourth software Vsync signal is adjacent to and follows the third software Vsync signal.

[0188] Specifically, SurfaceFlinger reads the second duration at the start of the fourth software Vsync signal and listens for the second PresentFence at the start of the fourth software Vsync signal. When the second PresentFence is detected within the second duration, the second image data is synthesized. The second PresentFence is the PresentFence corresponding to the previous frame of the second image.

[0189] The process of synthesizing the second image data is similar to the process of synthesizing the previous frame image data of the first image, and will not be described again here.

[0190] S110. The image synthesis system does not detect the second PresentFence within the second time period after the fourth software Vsync signal, and continues to wait without synthesizing the second image data.

[0191] By adopting the above technical solution, even when Vsync signal calibration is disabled, if the software Vsync signal generated by the software Vsync model is inconsistent with the hardware Vsync signal, setting an appropriate PresentFence waiting time for the image compositing system based on the current frame rate after the software Vsync signal allows more time for the display driver, GPU, CPU, and other processing units to complete image processing tasks. This ensures that the image compositing system can listen to the PresentFence within this time limit and then perform image data compositing, thus avoiding screen stuttering. Especially in frame-limiting scenarios caused by thermal throttling, where thermal throttling restricts the frequency of the display driver, CPU, GPU, and other processing units, increasing processing time, the above process allows these units more time to process tasks. This, in turn, gives the image compositing system more time to listen to the PresentFence and perform image data compositing, thus avoiding frame drops and screen stuttering.

[0192] The following is combined with Figures 9-10 The image processing method of this application embodiment will be described in detail using an electronic device playing video for a long time at a frame rate of 60 FPS as an example.

[0193] Figure 9 This is a schematic diagram illustrating an application scenario of an embodiment of this application.

[0194] Figure 9 Figure (a) shows the playback page for the video's "Media Title." On this page, the frame rate is 1 FPS, keeping the page unchanged. Further, on the video playback page shown in Figure (b), when the user clicks the play button, the page begins playing the video at 60 FPS, displaying the images in Figures (c) and (d) sequentially. During the user's click of the play button, a frame rate switch occurs, meaning the frame rate changes from (b) to (c). Subsequently, during normal video playback, the frame rate remains constant at 60 FPS. That is, the frame rates for (a) and (b) are both 1 FPS, and the frame rates for (c) and (d) are both 60 FPS.

[0195] It's important to note that during frame rate switching, calibration between the hardware Vsync_ signal and the software Vsync signal is initiated. At this time, SurfaceFlinger calculates the corresponding duration `waitForFenceDuration` using the formula `waitForFenceDuration = mIdealPeriod * (kOutlierTolerancePercent / kMaxPercent)`. If `kOutlierTolerancePercent = 20`, `kMaxPercent = 100`, and `mIdealPeriod = 1000ms`, then the first duration is 1000 * 20 / 100 = 200ms; if `mIdealPeriod = 16.67ms`, then the second duration is 16.67 * 20 / 100 = 3.33ms. It can be understood that (b) shows the first image, (a) shows the frame preceding the first image, (d) shows the second image, and (c) shows the frame preceding the second image.

[0196] Figure 10 The steps include S201 to S208, which will be described in detail below.

[0197] S201, the APP draws and renders to obtain the first image data after the first software Vsync signal.

[0198] The first software Vsync signal can be understood as the app's drawing and rendering. Figure 9 (b) shows the trigger signal for the first image. Within one cycle following the first software Vsync signal, the frame rate is 1 FPS, corresponding to a frame interval of 1000 ms.

[0199] Before switching to 60FPS, SurfaceFlinger's initial calculation time was 200ms.

[0200] S202, the DDIC control display screen shows the output after the first software Vsync signal. Figure 9 (a) shows the screen, and releases the first PresentFence.

[0201] S203 and SurfaceFlinger detect the first PresentFence within 200ms after the second software Vsync signal, and synthesize the first image data to obtain... Figure 9 (b) shows the first image.

[0202] The second software Vsync signal can be understood as SurfaceFlinger starting to wait. Figure 9(a) shows the trigger signal for the first PresentFence in the image. One period following the second software Vsync signal, the frame rate is 1 FPS, corresponding to a frame interval of 1000 ms.

[0203] S204. On the page shown in the first image, click the play button, and the SurfaceFlinger control will switch from 1FPS to 60FPS.

[0204] S205. When switching to 60FPS, SurfaceFlinger calculates the second duration as 3.33ms.

[0205] S206, the APP draws and renders the second image data after the third software Vsync signal.

[0206] The third software Vsync signal can be understood as the APP's drawing and rendering. Figure 9 (d) shows the trigger signal for the second image. Within one cycle following the third software Vsync signal, the frame rate is 60 FPS, corresponding to a frame interval of 16.67 ms.

[0207] S207, the DDIC control display shows the output after the second software Vsync signal. Figure 9 (c) shows the screen, and releases the second PresentFence.

[0208] S208 and SurfaceFlinger detect the second PresentFence within 3.33ms after the fourth software Vsync signal, and synthesize the second image data to obtain... Figure 9 The second image shown in (d).

[0209] The fourth software Vsync signal can be understood as SurfaceFlinger starting to wait. Figure 9 (c) The second PresentFence trigger signal. In one cycle following the fourth software Vsync signal, the frame rate is 60 FPS, and the corresponding frame interval is 16.67 ms.

[0210] In possible implementations, the computer execution instructions in the embodiments of this application may also be referred to as application code, and the embodiments of this application do not specifically limit this.

[0211] The frame rate switching device provided in this application embodiment is used to execute the frame rate switching method of the above embodiment. The technical principle and technical effect are similar, and will not be described again here.

[0212] This application provides an electronic device, the structure of which is described below. Figure 1The electronic device's memory can be used to store at least one program instruction, and the processor is used to execute at least one program instruction to implement the technical solution of the above-described method embodiments. Its implementation principle and technical effects are similar to those of the related embodiments of the above-described method, and will not be repeated here.

[0213] This application provides a chip. The chip includes a processor, which is used to call a computer program in memory to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those in the related embodiments described above, and will not be repeated here.

[0214] This application provides a computer program product that, when run on an electronic device, causes the electronic device to execute the technical solutions described in the above embodiments. Its implementation principle and technical effects are similar to the related embodiments described above, and will not be repeated here.

[0215] This application provides a computer-readable storage medium storing program instructions. When executed by an electronic device, the program instructions cause the electronic device to perform the technical solution described in the above embodiments. Its implementation principle and technical effects are similar to the related embodiments described above, and will not be repeated here.

[0216] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0217] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that, Applied to an electronic device, the electronic device including an application program, an image compositing system, and a display system, the method includes: The application draws and renders to obtain the first image data after the first software Vsync signal; The image synthesis system detects the first PresentFence within a first time period after the second software Vsync signal, and synthesizes the first image data to obtain the first image. The first software Vsync signal and the second software Vsync signal are software Vsync signals generated at a first frame rate. The second software Vsync signal is adjacent to the first software Vsync signal and is located after the first software Vsync signal. The application draws and renders the second image data after the third software Vsync signal; The image synthesis system detects the second PresentFence within a second duration following the fourth software Vsync signal, and synthesizes the second image data to obtain the second image. The third and fourth software Vsync signals are software Vsync signals generated at the second frame rate. The third software Vsync signal is located after the second software Vsync signal, and the fourth software Vsync signal is adjacent to and follows the third software Vsync signal. When the first frame rate is greater than the second frame rate, the first duration is less than the second duration; when the first frame rate is less than the second frame rate, the first duration is greater than the second duration.

2. The method according to claim 1, characterized in that, The image synthesis system detects the first PresentFence within a first time period after the second software Vsync signal, synthesizes the first image data to obtain the first image, and then further includes: After the page has been idle for a period of time, the image compositing system controls the switching from the first frame rate to the second frame rate, where the first frame rate is greater than the second frame rate.

3. The method according to claim 1, characterized in that, The image synthesis system detects the first PresentFence within a first time period after the second software Vsync signal, synthesizes the first image data to obtain the first image, and then further includes: When a click is made on the page, the image compositing system controls the switching from the first frame rate to the second frame rate, where the first frame rate is less than the second frame rate.

4. The method according to claim 2 or 3, characterized in that, The method further includes: When switching to the second frame rate, the image synthesis system determines the second duration based on the second frame rate.

5. The method according to claim 4, characterized in that, The image synthesis system determines the second duration based on the second frame rate, including: The image synthesis system obtains the second frame interval corresponding to the second frame rate, calculates the product of the second frame interval and a preset threshold, and obtains the second duration.

6. The method according to claim 4, characterized in that, The image synthesis system determines the second duration based on the second frame rate, including: The image synthesis system obtains the second frame interval corresponding to the second frame rate, calculates the product of the second frame interval and a preset threshold, and obtains the second duration.

7. The method according to claim 1, characterized in that, The image synthesis system detects the second PresentFence within a second time period following the fourth software Vsync signal, and synthesizes the second image data, including: The image synthesis system reads the second duration at the start of the fourth software Vsync signal and listens to the second PresentFence at the start of the fourth software Vsync signal. When the image synthesis system detects the second PresentFence within the second time period, it synthesizes the second image data.

8. The method according to claim 2 or 3, characterized in that, The method further includes: Before switching to the second frame rate, the image synthesis system determines a first duration based on the first frame rate.

9. The method according to claim 8, characterized in that, The image synthesis system determines a first duration based on a first frame rate, including: The image synthesis system obtains the first frame interval corresponding to the first frame rate, calculates the product of the first frame interval and the preset threshold, and obtains the first duration.

10. The method according to claim 1, characterized in that, The image synthesis system detects the first PresentFence within a first time period following the second software Vsync signal, and synthesizes the first image data, including: The image synthesis system reads the first duration at the start of the second software Vsync signal and listens to the first PresentFence at the start of the second software Vsync signal. When the image synthesis system detects the first PresentFence within the first time period, it synthesizes the first image data.

11. The method according to any one of claims 1-10, characterized in that, The first PresentFence is the PresentFence corresponding to the previous frame of the first image; the second PresentFence is the PresentFence corresponding to the previous frame of the second image.

12. An electronic device, characterized in that, The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1-11.

13. A computer program product containing instructions, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1-11.

14. A computer-readable storage medium comprising instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-11.

Images