Frame rate control method and related apparatus

By controlling the frame rate of SurfaceFlinger through the frame stabilization service module, and using delay and frame interpolation techniques, the problem of increased power consumption in high-performance applications of electronic devices is solved, achieving both power reduction and performance assurance.

WO2024244631A1PCT designated stage expired Publication Date: 2024-12-05HONOR DEVICE CO LTD
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2024/082700
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-29
Filing Date
2024-03-20
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

When electronic devices use high-performance applications, such as gaming applications, power consumption and heat generation increase, affecting the user experience.

Method used

The frame rate of SurfaceFlinger is controlled by the frame stabilization service module, which reduces the frame rate of the frame sending commands generated by the application. Delay and frame interpolation techniques are used to ensure that power consumption is reduced while meeting the frame control conditions, and to avoid screen freezes.

Benefits of technology

When frame control conditions are met, reduce the power consumption of electronic devices, avoid frame drops, ensure application performance, and reduce system resource usage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2024082700_05122024_PF_FP_ABST
    Figure CN2024082700_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Provided in the present application are a frame rate control method and a related apparatus. When a frame control condition is not met, SurfaceFlinger is controlled by a frame stabilization service module to receive, at a first frame rate, a frame sending instruction issued by an application; and when the frame control condition is met, SurfaceFlinger is controlled by the frame stabilization service module to receive, at a second frame rate lower than the first frame rate, the frame sending instruction issued by the application. In this way, when a frame control condition is met, the frame rate at which a frame sending instruction, which is generated by an application, is sent to SurfaceFlinger is reduced, such that the number of times that rendering of a picture frame is triggered by the application and the number of times that the frame is sent within a certain time are reduced, thus reducing the power consumption of an electronic device.
Need to check novelty before this filing date? Find Prior Art

Description

A frame rate control method and related device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 29, 2023, with application number 202310622478.7 and application name “A Frame Rate Control Method and Related Devices”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a frame rate control method and related devices. Background Art

[0003] As the application market for electronic devices continues to expand, some application manufacturers (for example, game application manufacturers) have begun to use technologies such as real physics engines and high-precision rendering in their applications, which has also placed higher requirements on the performance of electronic devices. Therefore, electronic devices will also allocate more system resources such as processing power and storage capacity to complete the rendering and synthetic display of application screen frames. This also brings more power consumption to electronic devices, resulting in increased heat generation of electronic devices and affecting the user experience of using applications.

[0004] Summary of the Invention

[0005] The present application provides a frame rate control method and related devices, which can reduce the frame rate of the application's frame sending instructions sent to SurfaceFlinger when the frame control conditions are met, thereby reducing the power consumption of the electronic device.

[0006] In a first aspect, the present application provides a frame rate control method, which is applied to an electronic device, the electronic device including a stable frame service module, the stable frame service module being located between an application layer and a layer synthesizer SurfaceFlinger, and the electronic device being installed with a first application, the method comprising: the first application generating a frame sending instruction; when the electronic device does not meet the frame control condition, the frame sending instruction generated by the first application is forwarded to the SurfaceFlinger at a first frame rate through the stable frame service module; wherein the frame sending instruction is used to provide a layer buffer buffer of the picture frame to the SurfaceFlinger, the buffer being used to store the picture frame rendered by the GPU, and the SurfaceFlinger being used to synthesize and display the picture frame in the buffer indicated by the frame sending instruction after obtaining the frame sending instruction; when the electronic device meets the frame control condition, the frame sending instruction generated by the first application is forwarded to the SurfaceFlinger at a second frame rate through the stable frame service module, wherein the first frame rate is greater than the second frame rate, and the frame control condition includes that the picture frame rate set by the first application is a full frame rate, the first frame rate is less than or equal to the full frame rate, and the full frame rate is the maximum refresh rate of the picture frame supported by the first application.

[0007] The frame rate control method provided in this application can achieve that when the frame control conditions are not met, the frame stabilization service module controls SurfaceFlinger to receive the frame sending instructions issued by the application at a first frame rate. When the frame control conditions are met, the frame stabilization service module controls SurfaceFlinger to receive the frame sending instructions issued by the application at a second frame rate lower than the first frame rate. In this way, when the frame control conditions are met, the frame rate of the frame sending instructions generated by the application and sent to SurfaceFlinger can be reduced, thereby reducing the number of times the application triggers the rendering and sending of screen frames within a certain period of time, thereby reducing the power consumption of the electronic device.

[0008] In one possible implementation, the method further includes: obtaining a frame sending instruction generated by the first application from a graphics interface through a stable frame service module; wherein the graphics interface includes any one of the following: an open graphics library OpenGL, an open graphics library for embedded systems OpenGL ES, and a drawing application program interface Vulkan.

[0009] In this way, there is no need to improve the first application, but to adapt the frame stabilization service module so that the first application can control the frame rate through the frame stabilization service module without any perception, thereby reducing the power consumption of the electronic device.

[0010] In one possible implementation, the frame control condition also includes: the temperature of the electronic device is higher than a specified temperature threshold, and / or the power of the electronic device is lower than a specified power, and / or the occupancy rate of the central processing unit (CPU) of the electronic device is greater than a specified occupancy rate, and / or the first application is a game application.

[0011] In this way, the electronic device can also control the frame rate of the frame sending instructions obtained by SurfaceFlinger when the temperature is too high, and / or the battery is too low, and / or the CPU occupancy is too high, thereby reducing the power consumption of the electronic device in multiple scenarios.

[0012] In one possible implementation, when the frame sending instruction issued by the first application is forwarded to the SurfaceFlinger at the second frame rate through the stable frame service module, the method further includes: synthesizing and displaying N picture frames at the first frame rate through the SurfaceFlinger, the N picture frames including M predicted picture frames, and the M predicted picture frames are generated based on historical picture frames.

[0013] In this way, after lowering the frame rate at which SurfaceFlinger obtains the frame sending instruction to the second frame rate, SurfaceFlinger can synthesize and display the picture frames at the first frame rate by filling in frames, thereby reducing the power consumption of the electronic device and avoiding frame loss by SurfaceFlinger.

[0014] In a possible implementation, M is a difference between the first frame rate and the second frame rate.

[0015] In this way, after lowering the frame rate at which SurfaceFlinger obtains the frame sending instruction to the second frame rate, the frame rate of SurfaceFlinger's synthetic display remains unchanged at the first frame rate, ensuring the stability of the frame rate during synthetic display.

[0016] In one possible implementation, the frame sending instruction generated by the first application is forwarded to SurfaceFlinger at a first frame rate through the stable frame service module, specifically including: when the first frame sending instruction generated by the first application is obtained through the stable frame service module, the first frame sending instruction is forwarded to the SurfaceFlinger through the stable frame service module; after the first frame sending instruction is forwarded to the SurfaceFlinger through the stable frame service module, when the stable frame service module obtains the second frame sending instruction generated by the first application, the second frame sending instruction is forwarded to the SurfaceFlinger through the stable frame service module.

[0017] In this way, when the frame control conditions are not met, the frame stabilization service module can immediately forward the frame sending instruction to SurfaceFlinger when it obtains the frame sending instruction generated by the first application, thereby ensuring that the frame sending instruction can reach SurfaceFlinger in time.

[0018] In one possible implementation, the frame sending instruction issued by the first application is forwarded to SurfaceFlinger at a second frame rate through the stable frame service module, specifically including: when the third frame sending instruction generated by the first application is obtained through the stable frame service module, the third frame sending instruction is provided to the SurfaceFlinger through the stable frame service module; after the third frame sending instruction is forwarded to the SurfaceFlinger through the stable frame service module, the fourth frame sending instruction generated by the first application is obtained through the stable frame service module; after the stable frame service module obtains the fourth frame sending instruction generated by the first application, the stable frame service module delays forwarding the fourth frame sending instruction to the SurfaceFlinger.

[0019] In this way, when the frame control conditions are met, the frame stabilization service module can delay forwarding certain frame sending instructions to SurfaceFlinger when it obtains certain frame sending instructions generated by the first application, thereby reducing the frame rate of the frame sending instructions reaching SurfaceFlinger and reducing the power consumption of the electronic device.

[0020] In one possible implementation, after the stable frame service module obtains the fourth frame send instruction issued by the first application, the stable frame service module delays forwarding the fourth frame send instruction to the SurfaceFlinger through the stable frame service module, specifically including: determining, through the stable frame service module, the first time interval between the stable frame service module obtaining the fourth frame send instruction and obtaining the third frame send instruction; if the first time interval is less than the first frame send interval, after the stable frame service module obtains the fourth frame send instruction, the stable frame service module delays the fourth frame send instruction for the first time length to forward it to the SurfaceFlinger, wherein the first frame send interval is a control frame period, the control frame period is the inverse of the second frame rate, and the first time length is the difference between the first frame send interval and the first time interval.

[0021] In this way, the delay duration for delaying forwarding the frame sending instruction to SurfaceFlinger can be determined according to the frame control period corresponding to the second frame rate, thereby controlling the frame rate at which SurfaceFlinger obtains the frame sending instruction to the second frame rate.

[0022] In a possible implementation, the method further includes: if the first time interval is greater than or equal to the first frame sending interval, when the stable frame service module obtains the fourth frame sending instruction, forwarding the fourth frame sending instruction to the SurfaceFlinger through the stable frame service module.

[0023] In one possible implementation, the method further includes: after the stable frame service module delays forwarding the fourth frame sending instruction to the SurfaceFlinger, obtaining the fifth frame sending instruction generated by the first application through the stable frame service module; determining, through the stable frame service module, a second time interval between the stable frame service module obtaining the fifth frame sending instruction and obtaining the fourth frame sending instruction; if the second time interval is less than the second frame sending interval, after the stable frame service module obtains the fifth frame sending instruction, delaying the fifth frame sending instruction to the SurfaceFlinger by the second time length through the stable frame service module, wherein the second time length is the difference between the second frame sending interval and the second time interval, and the second frame sending interval is determined by the first time interval, the first frame sending interval and the control frame period.

[0024] In this way, since the timing of the application generating the frame sending instruction will fluctuate and is not strictly generated and issued periodically, the delay duration for delaying the forwarding of the frame sending instruction to SurfaceFlinger can be dynamically adjusted to stabilize the frame rate of the frame sending instruction obtained by SurfaceFlinger at the second frame rate.

[0025] In a possible implementation, the second frame sending interval is determined by the following formula:

[0026] Wherein, Ta is the first time interval, T1 is the frame control period, TNa is the first frame sending interval, and TNb is the second frame sending interval.

[0027] In this way, through the above formula, it is possible to dynamically determine whether each frame sending instruction is delayed and how long the delay is, and to stabilize the frame rate of the frame sending instruction obtained by SurfaceFlinger at the second frame rate.

[0028] In a possible implementation, the method further includes: after the stable frame service module forwards the third frame sending instruction to the SurfaceFlinger, when the SurfaceFlinger obtains the first vertical synchronization vsync signal, the picture frames in the buffer indicated by the third frame sending instruction are synthesized and displayed by the SurfaceFlinger; after the stable frame service module forwards the fourth frame sending instruction to the SurfaceFlinger, when the SurfaceFlinger obtains the second VSYNC signal, the picture frames in the buffer indicated by the third frame sending instruction are synthesized and displayed by the SurfaceFlinger, and the second vsync signal is the next vsync signal of the first vsync signal.

[0029] In a possible implementation, the method further includes: if it is determined through the stable frame service module that when the SurfaceFlinger obtains the third vsync signal, there is no buffer in the SurfaceFlinger that has not been synthesized to send the display frame, and the estimated time when the stable frame service module obtains the fifth frame sending instruction generated by the first application is later than the time when the SurfaceFlinger obtains the third vsync signal, before the SurfaceFlinger obtains the third vsync signal, the stable frame service module generates a first predicted picture frame based on the picture frame in the buffer indicated by the third frame sending instruction and the picture frame in the buffer specified by the fourth frame sending instruction; and generates a first predicted picture frame through the stable frame service module. The module provides the first predicted picture frame to the SurfaceFlinger; when the SurfaceFlinger obtains the third vsync signal, the first predicted picture frame is synthesized and displayed by the SurfaceFlinger; wherein the third vsync signal is the next vsync signal of the second vsync signal, and the difference between the time when the SurfaceFlinger obtains the third vsync signal and the time when the SurfaceFlinger obtains the second vsync signal is equal to the difference between the time when the SurfaceFlinger obtains the second vsync signal and the time when the SurfaceFlinger obtains the first vsync signal.

[0030] In this way, the timing of frame loss can be predicted in advance, and before the frame loss, the predicted frame can be supplemented based on the historically rendered frame and given to SurfaceFlinger to prevent SurfaceFlinger from dropping frames when synthesizing and displaying.

[0031] In a second aspect, the present application provides an electronic device comprising: one or more processors and one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the method in any possible implementation of the above-mentioned first aspect is executed.

[0032] In a third aspect, the present application provides another electronic device, comprising one or more functional modules, which are used to execute the method in the above-mentioned first aspect and any possible implementation of the first aspect.

[0033] In a fourth aspect, the present application provides a chip system for use in an electronic device, the chip system comprising a processing circuit and an interface circuit, the interface circuit being used to receive instructions and transmit them to the processing circuit, the processing circuit being used to run instructions to execute the method in the above-mentioned first aspect and any possible implementation of the first aspect.

[0034] In a fifth aspect, the present application provides a computer-readable storage medium comprising instructions, characterized in that when the instructions are executed on an electronic device, the method in the above-mentioned first aspect and any possible implementation of the first aspect is executed.

[0035] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when running on a computer, enables the computer to execute the method in the above-mentioned first aspect and any possible implementation of the first aspect.

[0036] The beneficial effects of the second and sixth aspects can be referred to the beneficial effects of the first aspect and any possible implementation method of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG1 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0038] FIG2 is a timing diagram of a picture frame from rendering to display provided in an embodiment of the present application;

[0039] FIG3 is a timing diagram of a picture frame from rendering to display under frame rate control provided in an embodiment of the present application;

[0040] FIG4 is a schematic diagram of a frame rate control method provided in an embodiment of the present application;

[0041] FIG5 is a schematic diagram of a software system architecture of an electronic device provided in an embodiment of the present application;

[0042] FIG6A is a schematic diagram of a software system architecture interaction flow of an electronic device provided in an embodiment of the present application;

[0043] FIG6B is a schematic diagram of a software system architecture interaction flow of an electronic device provided in another embodiment of the present application;

[0044] FIG7 is a schematic diagram of a frame control process provided in an embodiment of the present application;

[0045] FIG8 is a frame rate control timing diagram provided in an embodiment of the present application;

[0046] FIG9 is a schematic diagram of a frame loss prediction process provided in an embodiment of the present application;

[0047] FIG10 is a frame rate control timing diagram provided in an embodiment of the present application;

[0048] FIG11 is a schematic diagram of a frame supplementation process provided in an embodiment of the present application;

[0049] FIG12 is a schematic diagram of frame supplementation provided in an embodiment of the present application;

[0050] FIG13 is a flow chart of the frame rate control method provided in the implementation of this application. DETAILED DESCRIPTION

[0051] The following is a clear and detailed description of the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship between related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0052] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0053] The following describes the hardware structure of an electronic device provided in an embodiment of the present application.

[0054] FIG1 shows a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.

[0055] It should be understood that the electronic device 100 shown in FIG1 is merely an example, and the electronic device 100 may have more or fewer components than shown in FIG1, may combine two or more components, or may have a different component configuration. The various components shown in FIG1 may be implemented in hardware, including one or more signal processing and / or application specific integrated circuits, software, or a combination of hardware and software.

[0056] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a gyroscope sensor 180B, an acceleration sensor 180E, and a touch sensor 180K.

[0057] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0058] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0059] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0060] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0061] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0062] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0063] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0064] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0065] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0066] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0067] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0068] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD). The display screen panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniLED, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0069] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0070] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0071] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0072] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0073] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0074] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0075] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0076] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0077] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0078] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0079] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0080] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0081] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0082] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0083] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0084] Accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in all directions (generally three axes). It can also detect the magnitude and direction of gravity when electronic device 100 is stationary. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0085] The touch sensor 180K is also called a "touch panel." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, in a location different from that of the display screen 194.

[0086] Keys 190 include a power button, volume button, and other buttons. Keys 190 can be mechanical or touch-sensitive. Electronic device 100 can receive key inputs and generate key signal inputs related to user settings and function control of electronic device 100. Motor 191 can generate vibration prompts. SIM card interface 195 is used to connect a SIM card. A SIM card can be connected to and disconnected from electronic device 100 by inserting or removing it from SIM card interface 195.

[0087] In the embodiment of the present application, the device type of the electronic device 100 can be any of a mobile phone, a tablet computer, a handheld computer, a desktop computer, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), as well as smart home devices such as smart large screens and smart speakers, wearable devices such as smart bracelets, smart watches, and smart glasses, extended reality (XR) devices such as augmented reality (AR), virtual reality (VR), and mixed reality (MR), in-vehicle devices or smart city devices, etc.

[0088] The following describes the process from rendering to displaying the picture frame.

[0089] First, some terminology and concepts involved in the process from rendering to display of picture frames in the embodiments of this application are introduced.

[0090] 1. Screen refresh rate: This refers to the number of times a display screen on an electronic device refreshes a frame per second. Generally, the refresh rate of an electronic device's display screen is a fixed value related to the display's fixed parameters. For example, the refresh rate can be 60 Hz.

[0091] 2. Rendering frame rate (framerate): Indicates the number of frames rendered by the application's rendering thread calling the GPU in 1 second. This rendering frame rate can be controlled by the application.

[0092] 3. Vertical synchronization (vsync) signal: Since the application needs to call the GPU to complete the rendering of the picture frame before it can store the rendered picture frame in the layer buffer (buffer) and provide it to the layer compositor (SurfaceFlinger) for layer synthesis. After SurfaceFlinger takes out the rendered picture frame from the buffer for layer synthesis, it will send the layer synthesized picture frame to the display for refresh. However, when SurfaceFlinger sends the layer synthesized picture frame to the display during the screen refresh process, it is easy for the two different picture frames before and after to overlap and display, causing the screen to tear (tearing) on ​​the display. Therefore, in order to prevent this screen tearing phenomenon, the hardware module corresponding to the display can generate a vsync signal with the same period as the refresh rate and give it to SurfaceFlinger. After receiving the vsync signal each time, SurfaceFlinger can take out the picture frame from the buffer for layer synthesis and send it to the display for refresh.

[0093] FIG2 shows a timing diagram of a picture frame from rendering to display provided in an embodiment of the present application.

[0094] As shown in Figure 2, an application (such as a game application) can send rendering instructions to the GPU to render the screen frame. After sending the rendering instructions, the application can send a frame sending instruction to SurfaceFlinger. After the GPU renders the screen frame, the GPU can store the screen frame in the buffer indicated by the rendering instruction. After SurfaceFlinger obtains the frame sending instruction, it can obtain the screen frame buffer. After SurfaceFlinger detects that the screen frame buffer is filled with the rendered screen frame, it can wait for the arrival of the next vsync signal, take out the screen frame from the screen frame buffer for layer synthesis, and send it to the display screen for refresh display.

[0095] For example, between vertical synchronization signal 0 (VSYNC0) and vertical synchronization signal 1 (VSYNC1), the application can send rendering instruction A to the GPU and frame instruction A to SurfaceFlinger. After receiving rendering instruction A, the GPU can render screen frame A and store the rendered screen frame A in the buffer of screen frame A indicated by rendering instruction A. SurfaceFlinger can obtain the buffer of screen frame A by sending frame instruction A, and after detecting that the buffer of screen frame has been filled with rendered screen frame A, it waits until VSYNC1 arrives, takes out screen frame A from the buffer of screen frame A, performs layer synthesis, and sends it to the display screen for refresh display.

[0096] Between vertical synchronization signal 1 (VSYNC1) and vertical synchronization signal 2 (VSYNC2), the application can send rendering instruction B to the GPU and send frame instruction B to SurfaceFlinger. After obtaining rendering instruction B, the GPU can render screen frame B and store the rendered screen frame B in the buffer of screen frame B indicated by rendering instruction B. SurfaceFlinger can obtain the buffer of screen frame B by sending frame instruction B, and after detecting that the buffer of screen frame B is filled with the rendered screen frame B, it can wait until VSYNC2 arrives, take out screen frame B from the buffer of screen frame B, perform layer synthesis, and send it to the display screen for refresh display.

[0097] Between vertical synchronization signal 2 (VSYNC2) and vertical synchronization signal 3 (VSYNC3), the application can send rendering instruction C to the GPU and send frame instruction C to SurfaceFlinger. After obtaining rendering instruction C, the GPU can render screen frame C and store the rendered screen frame C in the screen frame C buffer indicated by rendering instruction C. SurfaceFlinger can obtain the screen frame C buffer by sending frame instruction C, and after detecting that the rendered screen frame C is filled in the screen frame C buffer, it waits until VSYNC3 arrives, takes out the screen frame C from the screen frame C buffer for layer synthesis, and sends it to the display screen for refresh display.

[0098] Between vertical synchronization signal 3 (VSYNC3) and vertical synchronization signal 4 (VSYNC4), the application can send rendering instruction D to the GPU and frame instruction D to SurfaceFlinger. After receiving rendering instruction D, the GPU can render screen frame D and store the rendered screen frame D in the buffer of screen frame D indicated by rendering instruction D. SurfaceFlinger can obtain the buffer of screen frame D by sending frame instruction D, and after detecting that the buffer of screen frame D has been filled with the rendered screen frame D, it can wait until VSYNC4 arrives, take out screen frame D from the buffer of screen frame D, perform layer synthesis, and send it to the display screen for refresh display.

[0099] FIG3 shows a timing diagram of a picture frame from rendering to display under frame rate control provided in an embodiment of the present application.

[0100] As shown in Figure 3, to reduce power consumption in electronic devices, the frame rate of rendered frames can be controlled to reduce the frame rate of rendered frames. The frame rate control method can be as follows: when the frame rate needs to be reduced, the data processing time of the frame can be increased to delay the issuance of rendering instructions to the GPU. In this way, the frame rate of the frame can be controlled.

[0101] However, because the application delays sending rendering instructions to the GPU, the GPU may render a certain frame late. The application sends the frame buffer to SurfaceFlinger after the vertical sync signal arrives. As a result, SurfaceFlinger cannot obtain the frame for synthesis when the vertical sync signal arrives, causing the frame displayed on the display to be stuck.

[0102] For example, between vertical synchronization signal 0 (VSYNC0) and vertical synchronization signal 1 (VSYNC1), the application can perform frame rate control and delay sending rendering instruction A to the GPU. After receiving rendering instruction A, the GPU can render frame A. After the GPU renders frame A, the application can pass the buffer of frame A to SurfaceFlinger. After receiving the buffer of frame A, SurfaceFlinger can wait until VSYNC1 arrives, extract frame A from the buffer of frame A, perform layer synthesis, and send it to the display for refresh display.

[0103] Between vertical synchronization signal 1 (VSYNC1) and vertical synchronization signal 2 (VSYNC2), the application can perform frame rate control and delay sending rendering instruction B to the GPU. After receiving rendering instruction B, the GPU can render screen frame B. However, when VSYNC2 arrives, the GPU has not yet finished rendering screen frame B. After VSYNC2 arrives, the application sends instruction B to send the buffer of screen frame B to SurfaceFlinger. When VSYNC2 arrives, SurfaceFlinger does not obtain the screen frame for layer synthesis and cannot send the latest screen frame to the display. Therefore, if the display does not obtain the new screen frame, it will display screen frame A for two consecutive refresh cycles, resulting in a stuck display frame.

[0104] Between vertical synchronization signal 2 (VSYNC2) and vertical synchronization signal 3 (VSYNC3), the application sends the buffer of frame B to SurfaceFlinger using the frame send instruction B. After the application sends the buffer of frame B to SurfaceFlinger using the frame send instruction B, the application can control the frame rate and delay sending rendering instruction C to the GPU. After receiving rendering instruction C, the GPU can render frame C. After receiving the buffer of frame B, SurfaceFlinger can wait until VSYNC3 arrives, extract frame B from the buffer of frame B, perform layer synthesis, and send it to the display for refresh display.

[0105] The GPU renders frame C between vertical synchronization signals 3 (VSYNC3) and 4 (VSYNC4). After the GPU renders frame C, it sends the frame C buffer to SurfaceFlinger via command C. After receiving the frame C buffer, SurfaceFlinger waits until VSYNC4 arrives, extracts frame C from the buffer, performs layer synthesis, and sends it to the display for refresh.

[0106] Through the above-mentioned frame control method, the frame control time is longer than the original rendering cycle, that is, the rendering frame rate after frame control is lower than the original frame rate, which reduces the GPU's rendering frame rate of the picture frame and reduces the number of GPU rendering times within a certain period of time, thereby reducing the power consumption of the electronic device. However, it will also cause the picture to be stuck.

[0107] An embodiment of the present application provides a frame rate control method, whereby an electronic device can control the frame rate of the picture frames of an application (e.g., a game application) to be slightly lower than the original frame rate specified by the application (e.g., controlled from the original 60Hz to 58Hz), thereby making the load of rendering and frame sending lighter, thereby reducing the use of system resources on the electronic device and reducing the power consumption of the electronic device. Optionally, the electronic device can predict the timing of frame loss in advance during the process of controlling the frame rate of the picture frames of an application (e.g., a game application). Before the timing of frame loss arrives, a predicted picture frame is generated using multiple previously rendered picture frames through an active interpolation method that is lighter in load than the process of generating picture frames by the application, and is inserted into a queue of picture frames. In this way, the display frame rate of the picture on the display screen can also reach the original frame rate, ensuring application performance while reducing the load and power consumption of the electronic device.

[0108] FIG4 shows a schematic diagram of a frame rate control method in an embodiment of the present application.

[0109] As shown in Figure 4, the electronic device can obtain the rendering instructions and frame sending instructions issued by the application through the stable frame service module, and delay the frame sending instructions issued by the application to SurfaceFlinger for a certain period of time. In this way, the frame rate of the rendered screen frame can be controlled to reduce the GPU's rendering frame rate of the screen frame. Since the stable frame service module delays the application's frame sending instructions to SurfaceFlinger, it will result in the GPU rendering a certain screen frame later, so that within a certain vsync signal cycle, SurfaceFlinger cannot obtain the screen frame synthesis layer, and SurfaceFlinger loses the screen frame (abbreviated as frame loss). The electronic device can also predict the timing of frame loss during the frame rate control process, and before the frame loss timing arrives, generate a predicted screen frame based on the historically rendered screen frame, and send the predicted screen frame to SurfaceFlinger before the vsync signal arrives. In this way, the display frame rate of the screen can reach the original frame rate, ensuring application performance while reducing the load and power consumption of the electronic device.

[0110] For example, between vertical synchronization signal 0 (VSYNC0) and vertical synchronization signal 1 (VSYNC1), the application can sequentially send rendering instruction A and frame sending instruction A of picture frame A to the stable frame service module, and the stable frame service module can send rendering instruction A to the GPU. Since the stable frame service module controls the frame rate of the rendered picture frame, the stable frame service module can delay sending frame instruction A to SurfaceFlinger. After obtaining rendering instruction A, the GPU can render picture frame A, and after rendering picture frame A, store picture frame A in the buffer of picture frame A. After obtaining frame sending instruction A, SurfaceFlinger can obtain the buffer of picture frame A. After SurfaceFlinger detects that the buffer of picture frame A is filled with the rendered picture frame A, SurfaceFlinger can wait until VSYNC1 arrives, take out picture frame A from the buffer of picture frame A, perform layer synthesis, and send it to the display screen for refresh display.

[0111] Between vertical synchronization signal 1 (VSYNC1) and vertical synchronization signal 2 (VSYNC2), the application can send rendering instruction B and frame instruction B for frame B to the stable frame service module, which can then pass rendering instruction B to the GPU. Because the stable frame service module can control the frame rate of rendered frames, it can delay sending frame instruction B to SurfaceFlinger. After receiving rendering instruction B, the GPU can render frame B and store it in the frame B buffer after rendering. The stable frame service module can recognize that SurfaceFlinger cannot obtain frame B between vertical synchronization signal 1 (VSYNC1) and vertical synchronization signal 2 (VSYNC2). Therefore, the stable frame service module can generate frame 1, store it in the frame 1 buffer, and send it to SurfaceFlinger. However, SurfaceFlinger cannot obtain frame B when VSYNC2 arrives. However, before VSYNC2 arrives, the frame stabilization service module has already given frame 1 to SurfaceFlinger. Therefore, when VSYNC2 arrives, SurfaceFlinger can take out frame 1 for layer synthesis and send it for display.

[0112] Between vertical synchronization signals 2 (VSYNC2) and 3 (VSYNC3), the frame stabilization service module sends frame instruction B to SurfaceFlinger. After obtaining the buffer of frame B, SurfaceFlinger can wait until VSYNC3 arrives, extract frame B from the buffer of frame B, perform layer synthesis, and send it to the display for refresh display.

[0113] The application can send rendering instruction C for frame C to the frame stabilization service module, which can then pass rendering instruction C to the GPU. After receiving rendering instruction C, the GPU can render frame C and store it in the buffer of frame C after rendering.

[0114] Between vertical synchronization signal 3 (VSYNC3) and vertical synchronization signal 4 (VSYNC4), the application can send the frame sending instruction C of the screen frame C to the stable frame service module. Since the stable frame service module controls the frame rate of the rendered screen frame, the stable frame service module can delay sending the frame instruction C to SurfaceFlinger. After obtaining the frame sending instruction C, SurfaceFlinger can obtain the buffer of the screen frame C. After SurfaceFlinger detects that the buffer of the screen frame C is filled with the rendered screen frame C, it can wait until VSYNC4 arrives, take out the screen frame C from the buffer of the screen frame C, perform layer synthesis, and send it to the display screen for refresh display.

[0115] By using the frame rate control method provided in the embodiment of the present application, the total power consumption (FPS) of the picture frame generated after the frame rate control can be achieved. real-limit ) and the power consumption of the compensation frame (FPS ligntload ) and the total power consumption of the frame below the original frame rate (FPS real-max ), that is, Power(FPS real-limit )+Power(FPS ligntload ) <Power(FPS real-max ).

[0116] The following describes a software system architecture of an electronic device provided in an embodiment of the present application.

[0117] FIG5 shows a schematic diagram of a software system architecture of an electronic device provided in an embodiment of the present application.

[0118] As shown in FIG5 , the software system architecture of the electronic device 100 may include an application (APP) layer, an application framework (FWK) layer, a native service layer, and a kernel layer.

[0119] The application layer includes one or more applications, such as game applications, video applications, etc. The application can be used to provide the picture content of the picture frame, and call the graphics interface through the stable frame service module to configure the GPU to render the picture frame based on the picture content of the picture frame.

[0120] The application framework layer may include a layer synthesizer (SurfaceFlinger) and a frame stabilization service module. SurfaceFlinger is used to perform layer synthesis processing on the screen frames rendered by the GPU, generate an interface including the screen frames, and send the interface including the screen frames to the display screen for display. The frame stabilization service module can be used to control the frame rate of the application calling the GPU to render the screen frames, and pre-identify the opportunity for frame loss, and generate a predicted screen frame before the frame loss opportunity and send it to SurfaceFlinger, thereby avoiding frame jams when the screen displays the screen frames.

[0121] The local service layer may include one or more graphics interfaces. The one or more graphics interfaces may include one or more of the following: OpenGL (Open Graphics Library), OpenGL ES (Open Graphics Library for Embedded Systems), and Vulkan (Vulkan Application Programming Interface). The graphics interfaces may be used to configure a GPU to render image frames.

[0122] The kernel layer may include a graphics processing unit driver (GPU Driver), etc. The GPU Driver is used to drive the GPU to perform upper layer operations (eg, operations of rendering picture frames configured by a graphics interface).

[0123] The following uses a game application as an example to describe the interaction process of each module in the above software system architecture.

[0124] FIG6A shows a schematic diagram of the software system architecture interaction flow of an electronic device provided in an embodiment of the present application.

[0125] As shown in Figure 6A, the software system architecture interaction process is specifically described using a game application as an example. Among them, the game application may include a main thread (Main Thread) and a rendering thread (Render Thread). The frame stabilization service module may include a frame rate control module, a frame loss prediction module, a frame supplement control module, and a frame stabilization cache module. The layer compositor (SurfaceFlinger) may include a SurfaceFlinger information acquisition module.

[0126] The interaction process of each module can be as follows:

[0127] 1. The main thread of the game application can synchronize the game screen content to the rendering thread.

[0128] 2. After the rendering thread obtains the game screen content, it can send rendering instructions to the graphics interface.

[0129] The graphics interface may include any one of OpenGL, OpenGL ES, and Vulkan.

[0130] 3. The graphics interface can forward rendering instructions to the frame rate control module.

[0131] 4. After obtaining the rendering instruction, the frame rate control module can send the rendering instruction to the GPU driver.

[0132] After receiving the rendering instructions, the GPU driver sends them to the GPU to configure the GPU to render frames based on the game screen content, and then stores the rendered frames in the buffer specified by the rendering instructions.

[0133] After rendering the picture frame, the GPU can store the picture frame in the buffer specified by the rendering instruction.

[0134] 5. After issuing the rendering instruction, the rendering thread can send the frame instruction to the graphics interface.

[0135] 6. The graphics interface can forward frame instructions to the frame rate control module.

[0136] 7. The frame rate control module can choose to delay or immediately send frame instructions to SurfaceFlinger.

[0137] After obtaining the frame sending instruction, the frame rate control module can determine whether it is necessary to delay sending the frame instruction and the delay time when the frame sending instruction needs to be delayed based on the control frame rate.

[0138] After receiving the frame send instruction, SurfaceFlinger can obtain the buffer address of the picture frame from the frame send instruction. Among them, when the picture frame rendered by the GPU is stored in the buffer, SurfaceFlinger can promptly know that the picture frame has been stored in the buffer through the fence mechanism. Therefore, when the next vsync signal arrives, SurfaceFlinger can take the picture frame from the picture frame buffer for layer synthesis to synthesize the interface including the picture frame, and send the interface including the picture frame to the display screen for display.

[0139] After receiving the frame delivery instruction for the current frame, the frame rate control module can determine the rendering time of the current frame based on the time difference between the frame delivery instruction for the previous frame and the frame delivery instruction for the current frame. The frame rate control module can also determine whether to delay forwarding the frame delivery instruction for the current frame to SurfaceFlinger based on the rendering time of the current frame.

[0140] The frame rate control module can also record the frame sending time of the current picture frame and determine the fastest frame sending time of the next picture frame.

[0141] 8. The frame rate control module can send the frame sending time of the current picture frame and the fastest frame sending time of the next picture frame to the frame loss prediction module.

[0142] 9. After obtaining the buffer of the picture frame, SurfaceFlinger can send the buffer information of the picture frame and the timestamp of the vsync signal to the stable frame cache module for storage.

[0143] The buffer information of the picture frame may include the buffer address of the picture frame and the buffer accumulation number of the picture frame.

[0144] 10. The stable frame buffer module can provide the timestamp of the vsync signal and the buffer accumulation number of the picture frame to the frame loss prediction module.

[0145] The frame loss prediction module can determine the period of the vsync signal and predict whether the next frame will be lost based on the timestamp of the latest vsync signal, the number of frames buffered, the fastest frame transmission time of the next frame, and the vsync period.

[0146] 11. When the frame loss prediction module determines that there is a frame loss in the next picture frame, it can send a frame filling instruction to the frame filling control module.

[0147] 12. The frame supplement control module can obtain historical picture frames from the stable frame buffer module.

[0148] After acquiring the historical picture frames, the frame interpolation control module can determine the timing of frame interpolation, and when the timing of frame interpolation arrives, generate a predicted picture frame based on the historical picture frames and store the predicted picture frame in a buffer of predicted picture frames.

[0149] 13. The frame interpolation control module can give the buffer of the predicted picture frame to SurfaceFlinger.

[0150] After obtaining the buffer of the predicted picture frame, SurfaceFlinger can wait until the next vsync signal arrives, take out the predicted picture frame from the buffer of the predicted picture frame for layer synthesis, to synthesize the interface including the predicted picture frame, and send the interface including the predicted picture frame to the display screen for display.

[0151] Among them, the vertical synchronization signal thread (Vsync Thread) can periodically send a vsync signal to SurfaceFlinger.

[0152] In an embodiment of the present application, the frame stabilization service module can control the frame rate of any application when rendering a picture frame and perform frame interpolation at an appropriate time.

[0153] In one possible implementation, the frame stabilization service module may also control the frame rate of the rendered picture frame and perform frame interpolation at an appropriate interpolation time when it recognizes that a game application calls the GPU to render a picture frame, and / or the frame rate of the picture frame set by the application is greater than or equal to the preset frame rate, and / or the temperature of the electronic device is higher than a specified temperature threshold, and / or the power of the electronic device is lower than a specified power level, and / or the CPU occupancy rate of the electronic device is greater than a specified occupancy rate. The preset frame rate may be the maximum picture refresh rate (i.e., full frame rate) supported by the application, and the full frame rate may be equal to the refresh rate of the display screen of the electronic device.

[0154] Through the software system architecture interaction process shown in Figure 6A above, the frame stabilization service module can control the rendering frame rate of the application without the application's awareness, thereby reducing the power consumption of the electronic device.

[0155] FIG6B shows a schematic diagram of a software system architecture interaction flow of an electronic device provided in another embodiment of the present application.

[0156] As shown in Figure 6B, the software system architecture interaction process is specifically described using a game application as an example. Among them, the game application may include a main thread (Main Thread) and a rendering thread (Render Thread). The frame stabilization service module may include a frame rate control module, a frame loss prediction module, a frame supplement control module, and a frame stabilization cache module. The layer compositor (SurfaceFlinger) may include a SurfaceFlinger information acquisition module.

[0157] The interaction process of each module can be as follows:

[0158] 1. The main thread of the game application can synchronize the game screen content to the rendering thread.

[0159] 2. After the rendering thread obtains the game screen content, it can send rendering instructions to the frame rate control module.

[0160] 3. After obtaining the rendering instructions issued by the rendering thread, the frame rate control module can forward the rendering instructions to the graphics interface.

[0161] The graphics interface may include any one of OpenGL, OpenGL ES, and Vulkan.

[0162] 4. The graphics interface can forward rendering instructions to the GPU driver.

[0163] After receiving the rendering instructions, the GPU driver sends them to the GPU to configure the GPU to render frames based on the game screen content, and then stores the rendered frames in the buffer specified by the rendering instructions.

[0164] 5. After issuing the rendering instruction, the rendering thread can send a frame instruction to the frame rate control module.

[0165] 6. The frame rate control module can choose to delay or immediately send frame instructions to SurfaceFlinger.

[0166] After receiving the frame send instruction, SurfaceFlinger can obtain the buffer address of the picture frame from the frame send instruction. Among them, when the picture frame rendered by the GPU is stored in the buffer, SurfaceFlinger can promptly know that the picture frame has been stored in the buffer through the fence mechanism. Therefore, when the next vsync signal arrives, SurfaceFlinger can take the picture frame from the picture frame buffer for layer synthesis to synthesize the interface including the picture frame, and send the interface including the picture frame to the display screen for display.

[0167] After receiving the frame delivery instruction for the current frame, the frame rate control module can determine the rendering time of the current frame based on the time difference between the frame delivery instruction for the previous frame and the frame delivery instruction for the current frame. The frame rate control module can also determine whether to delay forwarding the frame delivery instruction for the current frame to SurfaceFlinger based on the rendering time of the current frame.

[0168] The frame rate control module can also record the frame sending time of the current picture frame and determine the fastest frame sending time of the next picture frame.

[0169] 7. The frame rate control module can send the frame sending time of the current picture frame and the fastest frame sending time of the next picture frame to the frame loss prediction module.

[0170] 8. After obtaining the buffer of the picture frame, SurfaceFlinger can send the buffer information of the picture frame and the timestamp of the vsync signal to the stable frame cache module for storage.

[0171] The buffer information of the picture frame may include the buffer address of the picture frame and the buffer accumulation number of the picture frame.

[0172] 9. The stable frame buffer module can provide the timestamp of the vsync signal and the buffer accumulation number of the picture frame to the frame loss prediction module.

[0173] The frame loss prediction module can determine the period of the vsync signal and predict whether the next frame will be lost based on the timestamp of the latest vsync signal, the number of frames buffered, the fastest frame transmission time of the next frame, and the vsync period.

[0174] 10. When the frame loss prediction module determines that there is a frame loss in the next picture frame, it can send a frame filling instruction to the frame filling control module.

[0175] 11. The frame supplement control module can obtain historical picture frames from the stable frame buffer module.

[0176] After acquiring the historical picture frames, the frame interpolation control module can determine the timing of frame interpolation, and when the timing of frame interpolation arrives, generate a predicted picture frame based on the historical picture frames and store the predicted picture frame in a buffer of predicted picture frames.

[0177] 12. The frame interpolation control module can give the buffer of the predicted picture frame to SurfaceFlinger.

[0178] After obtaining the buffer of the predicted picture frame, SurfaceFlinger can wait until the next vsync signal arrives, take out the predicted picture frame from the buffer of the predicted picture frame for layer synthesis, to synthesize the interface including the predicted picture frame, and send the interface including the predicted picture frame to the display screen for display.

[0179] Among them, the vertical synchronization signal thread (Vsync Thread) can periodically send a vsync signal to SurfaceFlinger.

[0180] In an embodiment of the present application, the frame stabilization service module can control the frame rate of any application when rendering a picture frame and perform frame interpolation at an appropriate time.

[0181] In one possible implementation, the frame stabilization service module may also control the frame rate of the rendered picture frame and perform frame interpolation at an appropriate interpolation time when it recognizes that a game application calls the GPU to render a picture frame, and / or the frame rate of the picture frame set by the application is greater than or equal to the preset frame rate, and / or the temperature of the electronic device is higher than a specified temperature threshold, and / or the power of the electronic device is lower than a specified power level, and / or the CPU occupancy rate of the electronic device is greater than a specified occupancy rate. The preset frame rate may be the maximum picture refresh rate (i.e., full frame rate) supported by the application, and the full frame rate may be equal to the refresh rate of the display screen of the electronic device.

[0182] The following describes the frame control process of the frame rate control module provided in the embodiment of the present application.

[0183] FIG7 shows a schematic diagram of the frame control process provided in an embodiment of the present application.

[0184] As shown in FIG7 , the control process may include the following steps:

[0185] S701: The frame rate control module may obtain a rendering instruction issued by an application to render the current picture frame.

[0186] S702: The frame rate control module may forward the rendering instruction to the GPU and configure the GPU to render the current picture frame.

[0187] The frame rate control module may obtain a rendering instruction for rendering the current frame from the graphics interface in the manner shown in FIG6A , and the frame rate control module may send the rendering instruction for the current frame to the GPU through the GPU driver.

[0188] The frame rate control module can also directly obtain the rendering instruction for rendering the current frame from the application in the manner shown in FIG6B . The frame rate control module can sequentially send the rendering instruction for the current frame to the GPU through the graphics interface and the GPU driver.

[0189] The graphics interface may include any one of OpenGL, OpenGL ES, and Vulkan.

[0190] S703: The frame rate control module may obtain a frame sending instruction for the current picture frame.

[0191] Among them, the frame rate control module can obtain the frame sending instruction of the current picture frame from the graphics interface in the manner shown in Figure 6A above, and the frame rate control module then delays or immediately gives the frame sending instruction of the current picture frame to SurfaceFlinger.

[0192] The frame rate control module can also directly obtain the frame sending instruction for rendering the current screen frame from the application in the manner shown in FIG6B , and then the frame rate control module delays or immediately sends the frame sending instruction of the current screen frame to SurfaceFlinger.

[0193] S704: The frame rate control module calculates the rendering time T of the current frame.

[0194] The frame rate control module can determine the rendering time T of the current frame based on the timestamp of the frame sending instruction of the previous frame and the timestamp of the frame sending instruction of the current frame. The rendering time T of the current frame can be the difference between the timestamp of the frame sending instruction of the current frame obtained by the frame rate control module and the timestamp of the frame sending instruction of the previous frame obtained by the frame rate control module. For example, the timestamp of the frame sending instruction of the previous frame can be Ma, and the timestamp of the frame sending instruction of the current frame can be Mb. Therefore, the rendering time T of the current frame = Mb-Ma.

[0195] S705: The frame rate control module determines whether the rendering time T of the current picture frame is less than the fastest frame sending time of the current picture frame.

[0196] The fastest frame sending time of the current frame can be used to indicate the minimum time interval between the frame sending instruction of the current frame and the frame sending instruction of the previous frame when the frame rate of the GPU rendering frame is reduced to the control frame rate.

[0197] Among them, if the previous picture frame is the first frame of the application, the frame rate control module immediately forwards the frame sending instruction of the previous picture frame. The frame rate control module can determine the fastest frame sending time TN of the next picture frame as the control frame period T1 based on the control frame rate F1. Among them, the control frame period T1 is the reciprocal of the control frame rate F1, that is, T1 = 1 / F1. F1 is less than the full frame rate F0. Among them, the full frame rate F0 can be equal to the refresh rate of the display screen. For example, the full frame rate F0 is 60 frames / second, and the control frame rate F1 can be 58 frames / second.

[0198] If the previous frame is not the first frame of the application, the frame rate control module can determine the fastest frame sending time TN' of the current frame according to the frame control period T1, the rendering time T0 of the previous frame, and the fastest frame sending time TN0 of the previous frame.

[0199] When T0 is less than T1 and TN0 is less than T1, TN' may be MIN(T1, TN0+(T1-T0)), that is, the value of TN' is the minimum value of T1 and (TN0+(T1-T0)).

[0200] When T0 is less than T1 and TN' is not less than T1, the value of TN can remain unchanged and continue to be TN0.

[0201] When T0 is not less than T1, TN' may be MAX(0, TN0-(T0-TN0)), that is, the value of TN' is the maximum value between 0 and (TN0-(T0-TN0)).

[0202] If the rendering time T of the current frame is less than the fastest frame-sending time of the current frame, it indicates that the frame rate instructed by the game application to render the frame by the GPU is greater than the control frame rate, the GPU load is too high, and the GPU frame rate needs to be reduced. Therefore, if the rendering time T of the current frame is less than the fastest frame-sending time of the current frame, steps S706 and S707 can be executed.

[0203] If the rendering time T of the current frame is not less than the fastest frame sending time of the current frame, it means that the frame rate instructed by the game application to render the frame by the GPU is not greater than the control frame rate, the GPU load is not high, and there is no need to reduce the frame rate of the GPU rendering the frame. Then, step S707 is directly executed.

[0204] S706 : The frame rate control module delays forwarding the frame sending instruction of the current screen frame to SurfaceFlinger.

[0205] When the frame rate control module determines that the frame rate of the GPU rendering frame needs to be reduced, the frame sending instruction of the current frame can be delayed to SurfaceFlinger. The delay duration can be the difference between the fastest frame sending time of the current frame and the rendering time T of the current frame.

[0206] Specifically, the frame rate control module can sleep for a certain period of time and then wake up automatically after obtaining the frame sending instruction of the current screen frame. Since the application can only render the next screen frame after the frame sending instruction is sent to SurfaceFlinger, and the frame rate control module delays issuing the frame sending instruction of the current screen frame, the interval between the application rendering screen frames is increased. In other words, the interval between the GPU rendering two adjacent screen frames is reduced. In this way, the number of rendering times of the GPU within a certain period of time is reduced, thereby achieving the purpose of reducing the load of the GPU and reducing the power consumption of electronic equipment.

[0207] S707: Determine the fastest frame sending time for the next picture frame.

[0208] Among them, if the current picture frame is the first frame of the application, the frame sending instruction of the current picture frame is forwarded immediately. The frame rate control module can determine the fastest frame sending time TN of the next picture frame as the control frame period T1 based on the control frame rate F1. Among them, the control frame period T1 is the reciprocal of the control frame rate F1, that is, T1 = 1 / F1. F1 is less than the full frame rate F0. Among them, the full frame rate F0 can be equal to the refresh rate of the display screen. For example, the full frame rate F0 is 60 frames / second, and the control frame rate F1 can be 58 frames / second.

[0209] If the current frame is not the first frame of the application, the frame rate control module can determine the fastest frame sending time TN for the next frame based on the frame control period T1, the rendering time T of the current frame, and the fastest frame sending time TN' of the current frame.

[0210] Specifically, the strategy for determining the fastest frame transmission time TN for the next picture frame can be shown in Table 1 below:

[0211] Table 1

[0212] As shown in Table 1 above, when T is less than T1 and TN' is less than T1, TN can be MIN(T1, TN'+(T1-T)), that is, the value of TN is the minimum value of T1 and (TN'+(T1-T)).

[0213] When T is less than T1 and TN' is not less than T1, the value of TN can remain unchanged and continue to be TN'.

[0214] When T is not less than T1, TN may be MAX(0, TN'-(T-TN')), that is, the value of TN is the maximum value between 0 and (TN'-(T-TN')).

[0215] After the frame rate control module readjusts the fastest frame sending time TN for the next frame, the frame rate control module may continue to wait for the rendering instruction for the next frame and re-execute steps S701 to S705 .

[0216] The frame control process of the frame rate control module will be described in detail below with reference to the frame rate control timing diagram exemplarily shown in FIG8 .

[0217] As shown in Figure 8, SurfaceFlinger can periodically receive vertical synchronization signal 0 (VSYNC0), vertical synchronization signal 1 (VSYNC1), vertical synchronization signal 2 (VSYNC2), vertical synchronization signal 3 (VSYNC3), vertical synchronization signal 4 (VSYNC4), and vertical synchronization signal 5 (VSYNC5) in sequence.

[0218] 1. The game application can sequentially issue rendering instruction A and frame send instruction A for frame A between the timestamps of VSYNC0 and VSYNC1. After receiving rendering instruction A for frame A, the frame rate control module can issue rendering instruction A to the GPU. After receiving rendering instruction A, the GPU can render frame A and store it in the frame A buffer.

[0219] When the frame sending instruction A is detected, the frame rate control module can determine the rendering time Ta of the picture frame A based on the time difference between obtaining the frame sending instruction of the previous picture frame and obtaining the frame sending instruction A. The frame rate control module can determine the fastest frame sending time TNb of the picture frame B. Picture frame B is the next frame of picture frame A. When Ta is less than T1 and TNa is less than T1, TNb can be MIN(T1, TNa+(T1-Ta)). That is, the value of TNb is the minimum value between T1 and (TNa+(T1-Ta)). When Ta is less than T1 and TNa is not less than T1, TNb can be TNa. When Ta is not less than T1, TNb can be the maximum value between 0 and (TNa-(Ta-TNa)).

[0220] Since the rendering time Ta of picture frame A is less than the fastest frame sending time TNa of picture frame A, the frame control condition is met. The frame rate control module can send the frame instruction A to SurfaceFlinger after obtaining the frame sending instruction A of picture frame A. The frame sending time of picture frame A can be Mqa. The fastest frame sending time TNa of picture frame A is confirmed when the frame sending instruction of the previous picture frame of picture frame A is obtained. If the previous picture frame of picture frame A is the first frame, TNa is the frame control period T1.

[0221] After SurfaceFlinger obtains the buffer of frame A through the frame sending instruction A, it waits for VSYNC1 to arrive, takes out frame A from the buffer of frame A for layer synthesis, and sends frame A to the display screen for display.

[0222] 2. The game application can send rendering instruction B for frame B to the frame rate control module between the timestamps of VSYNC1 and VSYNC2. However, when VSYNC2 arrives, the game application does not send frame instruction B for frame B.

[0223] Because SurfaceFlinger does not receive the frame send instruction B for frame B when VSYNC2 arrives, SurfaceFlinger cannot obtain the buffer for frame B when VSYNC2 arrives, resulting in frame loss. The display will display frame A for two consecutive refresh cycles.

[0224] 3. Between the timestamp of VSYNC2 and the timestamp of VSYNC3, the game application sends frame instruction B to the frame rate control module.

[0225] When the frame sending instruction B is detected, the frame rate control module can determine the rendering time Tb of the picture frame B based on the time difference between obtaining the frame sending instruction A and obtaining the frame sending instruction B. The frame rate control module can determine the fastest frame sending time TNc of the picture frame C. Picture frame C is the next frame of picture frame B. When Tb is less than T1, and TNb is less than T1, TNc can be MIN(T1, TNb+(T1-Tb)). That is, the value of TNc is the minimum value between T1 and (TNb+(T1-Tb)). When Tb is less than T1, and TNb is not less than T1, TNc can be TNb. When Tb is not less than T1, TNc can be the maximum value between 0 and (TNb-(Tb-TNb)).

[0226] Since the rendering time Tb of the picture frame B is less than TNb, the frame control condition is met. After obtaining the frame sending instruction B of the picture frame B, the frame rate control module can delay sending the frame instruction B to SurfaceFlinger by (TNb-Tb). The frame sending time of the picture frame B can be Mqb.

[0227] After SurfaceFlinger obtains the buffer of frame B through the frame sending instruction B, it waits for VSYNC3 to arrive, takes out frame B from the buffer of frame B for layer synthesis, and sends frame B to the display screen for display.

[0228] After sending frame instruction B, the game application can send rendering instruction C for frame C to the frame rate control module. After receiving rendering instruction C for frame C, the frame rate control module can send rendering instruction C to the GPU. After receiving rendering instruction C, the GPU can render frame C and store it in the frame C buffer after rendering.

[0229] 4. Between the timestamp of VSYNC3 and the timestamp of VSYNC4, the game application sends the frame instruction C to the frame rate control module.

[0230] When the frame sending instruction C is detected, the frame rate control module can determine the rendering time Tc of the picture frame C based on the time difference between obtaining the frame sending instruction B and obtaining the frame sending instruction C. The frame rate control module can determine the fastest frame sending time TNd of the picture frame D. Picture frame D is the next frame of picture frame C. When Tc is less than T1 and TNc is less than T1, TNd can be MIN(T1, TNc+(T1-Tc)). That is, the value of TNd is the minimum value between T1 and (TNc+(T1-Tc)). When Tc is less than T1 and TNc is not less than T1, TNd can be TNc. When Tc is not less than T1, TNd can be the maximum value between 0 and (TNc-(Tc-TNc)).

[0231] Since the rendering time Tc of the picture frame C is greater than or equal to TNc, the frame control condition is not met. After obtaining the frame sending instruction C of the picture frame C, the frame rate control module immediately sends the frame instruction C to SurfaceFlinger.

[0232] After SurfaceFlinger obtains the buffer of the picture frame C through the frame sending instruction C, it can wait for VSYNC4 to arrive, take out the picture frame C from the buffer of the picture frame C for layer synthesis, and send the picture frame C to the display screen for display.

[0233] 5. Between the timestamps of VSYNC4 and VSYNC5, the game application can send rendering instruction D for frame D to the frame rate control module. After receiving rendering instruction D for frame D, the frame rate control module can send rendering instruction D to the GPU. After receiving rendering instruction D, the GPU can render frame D and store it in the buffer for frame D.

[0234] Between the timestamp of VSYNC4 and the timestamp of VSYNC5, the game application can send a frame instruction D to the frame rate control module. When the frame rate control module obtains the frame sending instruction D, it can determine the rendering time Td of the picture frame D based on the time difference between obtaining the frame sending instruction C and obtaining the frame sending instruction D. The frame rate control module can determine the fastest frame sending time TNe of the picture frame E (not shown in Figure 8). The picture frame E is the next picture frame of the picture frame D. When Td is less than T1 and TNd is less than T1, TNe can be MIN(T1, TNd+(T1-Td)). That is, the value of TNe is the minimum value between T1 and (TNd+(T1-Td)). When Td is less than T1 and TNd is not less than T1, TNe can be TNd. When Td is not less than T1, TNe can be the maximum value between 0 and (TNd-(Td-TNd)).

[0235] Since the rendering time Td of the picture frame D is greater than or equal to TNd, the frame control condition is not met. After obtaining the frame sending instruction D of the picture frame D, the frame rate control module immediately sends the frame instruction D to SurfaceFlinger.

[0236] After obtaining the buffer of the picture frame D through the frame sending instruction D, SurfaceFlinger can wait for VSYNC5 to arrive, take out the picture frame D from the buffer of the picture frame D for layer synthesis, and send the picture frame D to the display screen for display.

[0237] The example shown in FIG8 is only used to explain the present application and should not be construed as limiting.

[0238] The following describes the frame loss prediction process of the above-mentioned frame loss prediction module provided in the embodiments of the present application.

[0239] FIG9 shows a schematic diagram of a frame loss prediction process provided in an embodiment of the present application.

[0240] As shown in FIG9 , the frame loss prediction process may include the following steps:

[0241] S901: A frame loss prediction module may receive a first notification from a frame rate control module, where the first notification is used to indicate that a frame sending instruction for a current picture frame has been obtained.

[0242] The frame rate control module may send a first notification to the frame loss prediction module when obtaining a frame sending instruction for the current picture frame.

[0243] S902: The frame loss prediction module may obtain the latest vsync signal timestamp (Mm), the buffer accumulation number (bufferNum) of the picture frame, and the fastest frame transmission time (TN) of the next picture frame.

[0244] The frame loss prediction module can obtain the latest vsync signal timestamp (Mm) and the buffer accumulation number (bufferNum) of the picture frame from the stable frame buffer module. The frame loss prediction module can also obtain the fastest frame transmission time (TN) of the next picture frame from the frame rate control module.

[0245] The process of determining the fastest frame transmission time (TN) for the next picture frame can refer to the text portion of the embodiments shown in FIG. 7 and FIG. 8 , and will not be repeated here.

[0246] S903: The frame loss prediction module may determine whether the buffer accumulation number of the picture frame is greater than 0.

[0247] If the buffer accumulation number of the picture frame is greater than 0, it means that SurfaceFlinger will not drop the frame when the next vsync signal arrives. Therefore, the frame drop prediction module can execute step S904.

[0248] If the buffer accumulation number of the picture frame is 0, it means that SurfaceFlinger may drop a frame when the next vsync signal arrives. Therefore, the frame drop prediction module can execute steps S905 to S908.

[0249] S904: The frame loss prediction module may determine that no frame supplementation is required.

[0250] S905. The frame loss prediction module determines the frame sending time (Mq) and vsync period (Pv) of the current picture frame.

[0251] The frame rate control module may record the frame sending time (Mq) of the current frame when obtaining the frame sending instruction of the current frame. The frame loss prediction module may obtain the frame sending time (Mq) of the current frame from the frame rate control module.

[0252] The frame loss prediction module may determine the vsync period (Pv) based on the full frame rate F0, ie, Pv=1 / F0.

[0253] In a possible implementation, the frame loss prediction module may also determine the vsync period (Pv) based on the difference between the timestamps of two adjacent vsync signals.

[0254] S906: The frame loss prediction module may determine the fastest frame sending time (Mn) of the next picture frame based on the frame sending time (Mq) of the current picture frame and the fastest frame sending time (TN) of the next picture frame.

[0255] Among them, Mn=Mq+TN.

[0256] S907: The frame loss prediction module may determine the vsync signal timestamp (Mv) of the next picture frame to be displayed based on the latest vsync signal timestamp (Mm) and the vsync period (Pv).

[0257] Among them, Mv=Mm+2*Pv.

[0258] S908: The frame loss prediction module may determine whether the fastest frame sending moment (Mn) of the next picture frame is later than the vsync signal timestamp (Mv) of sending and displaying the next picture frame.

[0259] If the fastest frame sending time (Mn) of the next frame is later than the vsync signal timestamp (Mv) of the next frame, it means that SurfaceFlinger will drop the frame when the next vsync signal arrives. Therefore, the frame loss prediction module can execute step S909.

[0260] If the fastest frame sending time (Mn) of the next frame is not later than the vsync signal timestamp (Mv) of the next frame, it means that SurfaceFlinger will not drop frames when the next vsync signal arrives. Therefore, the frame loss prediction module can execute step S904.

[0261] S909: The frame loss prediction module determines that a frame supplement is required.

[0262] When the frame loss prediction module determines that frame supplementation is required, a frame supplementation instruction may be sent to the frame supplementation control module.

[0263] The following describes in detail the process of predicting frame loss by the frame loss prediction module with reference to the frame rate control timing diagram exemplified in FIG10 .

[0264] As shown in Figure 10, SurfaceFlinger can periodically receive vertical synchronization signal 0 (VSYNC0), vertical synchronization signal 1 (VSYNC1), vertical synchronization signal 2 (VSYNC2), vertical synchronization signal 3 (VSYNC3), vertical synchronization signal 4 (VSYNC4), and vertical synchronization signal 5 (VSYNC5) in sequence.

[0265] 1. The game application can sequentially issue rendering instruction A and frame send instruction A for frame A between the timestamps of VSYNC0 and VSYNC1. After receiving rendering instruction A for frame A, the frame rate control module can issue rendering instruction A to the GPU. After receiving rendering instruction A, the GPU can render frame A and store it in the frame A buffer.

[0266] When the frame sending instruction A is detected, the frame rate control module can determine the rendering time Ta of the picture frame A based on the time difference between obtaining the frame sending instruction of the previous picture frame and obtaining the frame sending instruction A. The frame rate control module can determine the fastest frame sending time TNb of the picture frame B. Picture frame B is the next frame of picture frame A. When Ta is less than T1 and TNa is less than T1, TNb can be MIN(T1, TNa+(T1-Ta)). That is, the value of TNb is the minimum value between T1 and (TNa+(T1-Ta)). When Ta is less than T1 and TNa is not less than T1, TNb can be TNa. When Ta is not less than T1, TNb can be the maximum value between 0 and (TNa-(Ta-TNa)).

[0267] Since the rendering time Ta of picture frame A is less than the fastest frame sending time TNa of picture frame A, the frame control condition is met. The frame rate control module can send the frame instruction A to SurfaceFlinger after obtaining the frame sending instruction A of picture frame A. The frame sending time of picture frame A can be Mqa. The fastest frame sending time TNa of picture frame A is confirmed when the frame sending instruction of the previous picture frame of picture frame A is obtained. If the previous picture frame of picture frame A is the first frame, TNa is the frame control period T1.

[0268] After SurfaceFlinger obtains the buffer of frame A through the frame sending instruction A, it waits for VSYNC1 to arrive, takes out frame A from the buffer of frame A for layer synthesis, and sends frame A to the display screen for display.

[0269] When the frame rate control module detects the frame sending instruction A, it can notify the frame loss prediction module to enter the frame loss prediction process.

[0270] The frame loss prediction module can obtain the frame sending time Mqa of the picture frame A and the latest vsync signal timestamp Mm (in this case, the timestamp M0 of VYSNC0) from the frame rate control module. Based on the timestamp M0 of VYSNC0 and the vsync signal period Pv, the frame loss prediction module can determine that the vsync signal timestamp Mv of the next picture frame sent for display is the timestamp M2 of VSYNC2, where M2 = M0 + 2*Pv.

[0271] The frame loss prediction module can determine the fastest frame sending time Mnb of the picture frame B based on the frame sending time Mqa of the picture frame A and the fastest rendering time TNb of the picture frame B. Wherein, Mnb=Mqa+TNb.

[0272] Among them, since the fastest frame sending moment Mnb of the picture frame B is later than the timestamp M2 of VSYNC2, the frame loss prediction module determines that a frame supplement is required before the vsync timestamp M2. The frame loss prediction module can send a frame supplement instruction to the frame supplement control module. After obtaining the frame supplement instruction, the frame supplement control module can complete the frame supplement before the timestamp M2 of VSYNC2, and store the generated compensation frame 1 in the buffer of compensation frame 1, and provide it to SurfaceFlingr. After obtaining the buffer of compensation frame 1, SurfaceFlinger can take out the compensation frame 1 from the buffer of compensation frame 1 for layer synthesis when receiving VSYNC2, so as to send the compensation frame 1 to the display screen for display.

[0273] The frame loss prediction module can also determine the timing for compensating frame 1 and send this timing to the frame replacement control module in a frame replacement instruction. The frame replacement control module can perform frame replacement when the timing arrives. For example, the timing for compensating frame 1 can be (Mqa + (T1) / 2).

[0274] 2. The game application can send rendering instruction B for frame B to the frame rate control module between the timestamps of VSYNC1 and VSYNC2. However, when VSYNC2 arrives, the game application does not send frame instruction B for frame B.

[0275] Because SurfaceFlinger has not received the frame send instruction B for frame B when VSYNC2 arrives, the frame supplement control module has already stored the generated frame 1 in the frame 1 buffer and provided it to SurfaceFlinger before the VSYNC2 timestamp arrives. When VSYNC2 arrives, SurfaceFlinger can take frame 1 from the frame 1 buffer for layer synthesis and send frame 1 to the display for display.

[0276] 3. Between the timestamp of VSYNC2 and the timestamp of VSYNC3, the game application sends frame instruction B to the frame rate control module.

[0277] When the frame rate control module detects the frame sending instruction B, it can determine the rendering time Tb of the picture frame B based on the time difference between obtaining the frame sending instruction A and obtaining the frame sending instruction B. The frame rate control module can determine the fastest frame sending time TNc of the picture frame C. When Tb is less than T1, and TNb is less than T1, TNc can be MIN(T1, TNb+(T1-Tb)). That is, the value of TNc is the minimum value between T1 and (TNb+(T1-Tb)). When Tb is less than T1, and TNb is not less than T1, TNc can be TNb. When Tb is not less than T1, TNc can be the maximum value between 0 and (TNb-(Tb-TNb)).

[0278] Since the rendering time Tb of the picture frame B is less than TNb, the frame control condition is met. After obtaining the frame sending instruction B of the picture frame B, the frame rate control module can delay sending the frame instruction B to SurfaceFlinger by (TNb-Tb). The frame sending time of the picture frame B can be Mqb.

[0279] After SurfaceFlinger obtains the buffer of frame B through the frame sending instruction B, it waits for VSYNC3 to arrive, takes out frame B from the buffer of frame B for layer synthesis, and sends frame B to the display screen for display.

[0280] After sending frame instruction B, the game application can send rendering instruction C for frame C to the frame rate control module. After receiving rendering instruction C for frame C, the frame rate control module can send rendering instruction C to the GPU. After receiving rendering instruction C, the GPU can render frame C and store it in the frame C buffer after rendering.

[0281] When the frame rate control module detects the frame sending instruction B, it can notify the frame loss prediction module to enter the frame loss prediction process.

[0282] The frame loss prediction module can obtain the frame sending time Mqb of the picture frame B and the latest vsync signal timestamp Mm (in this case, the timestamp M3 of VYSNC2) from the frame rate control module. Based on the timestamp M2 of VYSNC2 and the vsync signal period Pv, the frame loss prediction module can determine that the vsync signal timestamp Mv of the next picture frame sent for display is the timestamp M4 of VSYNC4, where M4 = M2 + 2*Pv.

[0283] The frame loss prediction module can determine the fastest frame sending time Mnc of the picture frame C based on the frame sending time Mqb of the picture frame B and the fastest rendering time TNc of the picture frame C. Here, Mnc=Mqb+TNc.

[0284] Since the fastest frame sending time Mnc of the picture frame C is earlier than the timestamp M4 of VSYNC4, the frame loss prediction module determines that it is before the timestamp M4 of VSYNC4 and no frame supplement is required.

[0285] 4. Between the timestamp of VSYNC3 and the timestamp of VSYNC4, the game application sends the frame instruction C to the frame rate control module.

[0286] When the frame sending instruction C is detected, the frame rate control module can determine the rendering time Tc of the picture frame C based on the time difference between obtaining the frame sending instruction B and obtaining the frame sending instruction C. The frame rate control module can determine the fastest frame sending time TNd of the picture frame D. Among them, when Tc is less than T1, and TNc is less than T1, TNd can be MIN(T1, TNc+(T1-Tc)). That is, the value of TNd is the minimum value between T1 and (TNc+(T1-Tc)). When Tc is less than T1, and TNc is not less than T1, TNd can be TNc. When Tc is not less than T1, TNd can be the maximum value between 0 and (TNc-(Tc-TNc)).

[0287] Since the rendering time Tc of the picture frame C is greater than or equal to TNc, the frame control condition is not met. After obtaining the frame sending instruction C of the picture frame C, the frame rate control module immediately sends the frame instruction C to SurfaceFlinger.

[0288] After SurfaceFlinger obtains the buffer of the picture frame C through the frame sending instruction C, it can wait for VSYNC4 to arrive, take out the picture frame C from the buffer of the picture frame C for layer synthesis, and send the picture frame C to the display screen for display.

[0289] When the frame rate control module detects the frame sending instruction C, it can notify the frame loss prediction module to enter the frame loss prediction process.

[0290] The frame loss prediction module can obtain the frame sending time Mqc of the picture frame C and the latest vsync signal timestamp Mm (in this case, the timestamp M3 of VYSNC3) from the frame rate control module. Based on the timestamp M3 of VYSNC3 and the vsync signal period Pv, the frame loss prediction module can determine that the vsync signal timestamp Mv of the next picture frame sent for display is the timestamp M5 of VSYNC5, where M5 = M3 + 2*Pv.

[0291] The frame loss prediction module can determine the fastest frame sending time Mnd of the picture frame D based on the frame sending time Mqc of the picture frame C and the fastest rendering time TNd of the picture frame D. Wherein, Mnd=Mqc+TNd.

[0292] Among them, since the fastest frame sending time Mnd of the picture frame D is earlier than the timestamp M5 of VSYNC4, the frame loss prediction module determines that it is before the vsync timestamp M5 and no frame supplement is required.

[0293] 5. Between the timestamps of VSYNC4 and VSYNC5, the game application can send rendering instruction D for frame D to the frame rate control module. After receiving rendering instruction D for frame D, the frame rate control module can send rendering instruction D to the GPU. After receiving rendering instruction D, the GPU can render frame D and store it in the buffer for frame D.

[0294] Between the timestamp of VSYNC4 and the timestamp of VSYNC5, the game application can send a frame instruction D to the frame rate control module. When the frame rate control module obtains the frame sending instruction D, it can determine the rendering time Td of the picture frame D based on the time difference between obtaining the frame sending instruction C and obtaining the frame sending instruction D. The frame rate control module can determine the fastest frame sending time TNe of the picture frame E (not shown in Figure 8), and the picture frame E is the next picture frame of the picture frame D. Among them, when Td is less than T1, and TNd is less than T1, TNe can be MIN(T1, TNd+(T1-Td)). That is, the value of TNe is the minimum value between T1 and (TNd+(T1-Td)). When Td is less than T1, and TNd is not less than T1, TNe can be TNd. When Td is not less than T1, TNe can be the maximum value between 0 and (TNd-(Td-TNd)).

[0295] Since the rendering time Td of the picture frame D is greater than or equal to TNd, the frame control condition is not met. After obtaining the frame sending instruction D of the picture frame D, the frame rate control module immediately sends the frame instruction D to SurfaceFlinger.

[0296] After obtaining the buffer of the picture frame D through the frame sending instruction D, SurfaceFlinger can wait for VSYNC5 to arrive, take out the picture frame D from the buffer of the picture frame D for layer synthesis, and send the picture frame D to the display screen for display.

[0297] The example shown in FIG10 is only used to explain the present application and should not be construed as limiting.

[0298] The following describes the frame interpolation process of the frame interpolation control module provided in the embodiment of the present application.

[0299] FIG11 shows a schematic diagram of the frame supplementation process provided in an embodiment of the present application.

[0300] As shown in FIG11 , the frame supplement process may include the following steps:

[0301] S1101. A frame supplement control module may detect a frame supplement instruction, where the frame supplement instruction may include a frame supplement timing.

[0302] S1102: The frame supplement control module may determine whether the current time has reached the frame supplement timing.

[0303] If the current time reaches the frame supplementation opportunity, the frame supplementation control module may execute step S1103 .

[0304] If the current time has not yet arrived at the frame supplement opportunity, the frame supplement control module may wait until the frame supplement opportunity arrives and execute step S1103.

[0305] Among them, due to the short-term multi-tasking concurrency that will cause high system load and some tasks cannot be run in time, system resources cannot be reasonably utilized. Therefore, in the embodiment of the present application, the middle time between the rendering time of two original picture frames can be used as the best time to add frames, thereby reasonably utilizing system resources. Among them, the best time to add frames can be obtained by the frame sending time Mq of the current picture frame and the frame control period T1. Among them, the best time to add frames can be (Mq+(T1 / 2)).

[0306] For example, as shown in FIG10 , the frame delivery time of picture frame A is Mqa, and the frame loss prediction module determines that picture frame 1 needs to be generated after picture frame A. Therefore, the optimal frame supplementation time of picture frame 1 may be (Mqa+(T1 / 2)).

[0307] S1103 , the frame supplement control module generates a predicted frame through historically rendered frames, stores the predicted frame in a buffer, and provides the buffer of the predicted frame to SurfaceFlinger.

[0308] In an exemplary embodiment, since the similarity of the picture contents in adjacent frames is relatively high, the frame compensation control module can identify the movement trajectory of the picture contents in frame N-1 and frame N through the rendered frame N-1 and frame N, and can adjust the display position of the picture content in frame N based on the movement trajectory of the picture content, thereby obtaining the compensation frame X.

[0309] After SurfaceFlinger obtains the buffer of the predicted frame from the interpolation frame control module, it can wait for the arrival of the next vsync signal. When SurfaceFlinger receives the next vsync signal, it can extract the predicted frame from the buffer of the predicted frame for layer synthesis, generate an interface including the predicted frame, and send the predicted frame interface to the display screen for display.

[0310] After the frame interpolation control module provides the buffer of the predicted picture frame to SurfaceFlinger, the frame interpolation control module can wait for the frame loss prediction module to send the next frame interpolation instruction.

[0311] As shown in Figure 12, the frame interpolation control module can generate the predicted picture frame by frame extrapolation. That is, the frame interpolation control module can generate the predicted picture frame by using multiple picture frames that have been rendered by the game application through the GPU before the frame interpolation.

[0312] For example, a game application has already rendered frame N-1 and frame N sequentially through the GPU. The frame loss prediction module determines that frame X needs to be generated after frame N. Therefore, the frame interpolation control module can generate frame X based on the rendered frame N-1 and frame N. After the frame interpolation control module generates frame X and sends it to SurfaceFlinger, the game application continues to render frame N+1 through the GPU.

[0313] For another example, as shown in FIG10 , the frame loss prediction module determines that frame 1 needs to be generated after frame A. Therefore, the frame supplement control module can generate frame 1 based on the rendered adjacent frames 0 and A.

[0314] Because the interpolation control module generates compensation frames using multiple frames already rendered by the game application on the GPU before the interpolation, the generation of the predicted frames does not require GPU rendering. Therefore, compared with the original frame rendering process, the generation of the predicted frames consumes less power.

[0315] In a possible implementation, when there is only one rendered frame before the frame supplementation opportunity, the frame supplementation control module may abandon the frame supplementation.

[0316] In a possible implementation, when there are two rendered frames before the frame supplementation opportunity, but there is an abnormality in the two rendered frames, the frame supplementation control module may abandon the frame supplementation.

[0317] FIG13 exemplarily shows a flow chart of a frame rate control method provided in an embodiment of the present application.

[0318] As shown in Figure 13, the frame rate control method can be applied to an electronic device, wherein the electronic device may include a first application, a stable frame service module and SurfaceFlinger. The stable frame service module is located between the application layer and SurfaceFlinger. The frame rate control method may include the following steps:

[0319] S1301. The first application may generate a frame sending instruction.

[0320] Among them, the frame send instruction can be used to provide the buffer of the picture frame to SurfaceFlinger.

[0321] The buffer can be used to store the frames rendered by the GPU.

[0322] SurfaceFlinger can be used to synthesize and display the screen frame in the buffer indicated by the frame sending instruction after obtaining the frame sending instruction.

[0323] Among them, for the rendering and frame sending process of the first application on the screen frame, reference can be made to the embodiments shown in Figures 6A and 6B above, and no further details will be given here.

[0324] S1302: Determine whether the electronic device meets the frame control condition.

[0325] If the frame control condition is not met, step S1303 is executed. If the frame control condition is met, step S1304 is executed.

[0326] S1303: Forward the frame sending instruction generated by the first application to SurfaceFlinger at the first frame rate through the frame stabilization service module.

[0327] S1304. Forward the frame sending instruction generated by the first application to SurfaceFlinger at a second frame rate through the frame stabilization service module, where the second frame rate is lower than the first frame rate.

[0328] For example, the second frame rate may be 50 Hz, and the first frame rate may be 60 Hz.

[0329] The frame control condition may include that the picture frame rate set by the first application is a full frame rate, and / or the temperature of the electronic device is higher than a specified temperature threshold, and / or the power of the electronic device is lower than a specified power, and / or the CPU occupancy rate of the electronic device is greater than a specified occupancy rate, and / or the first application is a game application.

[0330] The first frame rate is less than or equal to the full frame rate, and the full frame rate is the maximum refresh rate of the picture frame supported by the first application. For example, the maximum refresh rate of the screen of the electronic device is 60 Hz. The maximum refresh rate of the picture frame supported by the first application is 60 Hz.

[0331] In one possible implementation, the frame stabilization service module can obtain the frame sending instructions generated by the first application from a graphics interface. The graphics interface includes any of the following: the open graphics library OpenGL, the open graphics library for embedded systems OpenGL ES, and the graphics application programming interface Vulkan. This eliminates the need to modify the first application by adapting it to the frame stabilization service module. This allows the first application to control the frame rate through the frame stabilization service module without being aware of the situation, thereby reducing power consumption in the electronic device.

[0332] For specific details, please refer to the embodiment shown in FIG6A above, which will not be described in detail here.

[0333] In a possible implementation, the frame stabilization service module may directly obtain the frame sending instruction generated by the first application from the first application. For details, please refer to the embodiment shown in FIG6B above, which will not be described in detail here.

[0334] In one possible implementation, the electronic device can use the SurfaceFlinger to synthesize and display N frames at the first frame rate through the stabilization frame service module at the second frame rate. The N frames include M predicted frames, and the M predicted frames are generated based on historical frames. In this way, after reducing the frame rate at which SurfaceFlinger obtains the frame sending instruction to the second frame rate, SurfaceFlinger can synthesize and display the frames at the first frame rate by supplementing frames, thereby reducing the power consumption of the electronic device and avoiding frame loss of SurfaceFlinger.

[0335] Here, M can be the difference between the first frame rate and the second frame rate. For example, the first frame rate can be 60 Hz and the second frame rate can be 50 Hz. SurfaceFlinger receives 50 frame delivery instructions per second and can synthesize and display the picture frames corresponding to the 50 frame delivery instructions and the 10 predicted picture frames generated by the supplementary frames.

[0336] For specific details, please refer to the embodiments shown in Figures 9, 10, 11, and 12 above, which will not be described again here.

[0337] In one possible implementation, when the stable frame service module does not control frames, the stable frame service module can immediately forward the frame sending instruction obtained from the first application to SurfaceFlinger. Specifically, when the stable frame service module obtains the first frame sending instruction generated by the first application, the stable frame service module can immediately forward the first frame sending instruction to SurfaceFlinger. When the stable frame service obtains the second frame sending instruction generated by the first application, the stable frame service can immediately forward the second frame sending instruction to SurfaceFlinger.

[0338] In one possible implementation, when the stable frame service module does not control frames, the stable frame service module can delay forwarding certain frame sending instructions obtained from the first application to SurfaceFlinger. Specifically, when the stable frame service module obtains the third frame sending instruction generated by the first application, the stable frame service module provides the third frame sending instruction to the SurfaceFlinger through the stable frame service module; after the third frame sending instruction is forwarded to the SurfaceFlinger through the stable frame service module, the stable frame service module obtains the fourth frame sending instruction generated by the first application through the stable frame service module; after the stable frame service module obtains the fourth frame sending instruction generated by the first application, the stable frame service module delays forwarding the fourth frame sending instruction to the SurfaceFlinger.

[0339] Specifically, the frame stabilization service module can determine the first time interval between the frame stabilization service module obtaining the fourth frame sending instruction and obtaining the third frame sending instruction; if the first time interval is less than the first frame sending interval, after the frame stabilization service module obtains the fourth frame sending instruction, the frame stabilization service module delays the first time length and forwards the fourth frame sending instruction to the SurfaceFlinger, wherein the first frame sending interval is the frame control period, the frame control period is the inverse of the second frame rate, and the first time length is the difference between the first frame sending interval and the first time interval.

[0340] If the first time interval is greater than or equal to the first frame sending interval, the frame stabilization service module may immediately forward the fourth frame sending instruction to the SurfaceFlinger upon obtaining the fourth frame sending instruction.

[0341] After the stable frame service module delays forwarding the fourth frame send instruction to the SurfaceFlinger, the stable frame service module can obtain the fifth frame send instruction generated by the first application. The stable frame service module can determine the second time interval between the stable frame service module obtaining the fifth frame send instruction and obtaining the fourth frame send instruction; if the second time interval is less than the second frame send interval, after the stable frame service module obtains the fifth frame send instruction, the stable frame service module can delay forwarding the fifth frame send instruction to the SurfaceFlinger for a second time length, wherein the second time length is the difference between the second frame send interval and the second time interval, and the second frame send interval is determined by the first time interval, the first frame send interval, and the control frame period.

[0342] The second frame sending interval is determined by the following formula:

[0343] Wherein, Ta is the first time interval, T1 is the frame control period, TNa is the first frame sending interval, and TNb is the second frame sending interval.

[0344] Among them, the first time interval is the rendering time of the picture frame corresponding to the fourth frame sending instruction, and the first frame sending interval is the fastest frame sending time of the picture frame corresponding to the fourth frame sending instruction. The first duration is the delay time of the picture frame corresponding to the fourth frame sending instruction. The second time interval is the rendering time of the picture frame corresponding to the fifth frame sending instruction, and the second frame sending interval is the fastest frame sending time of the picture frame corresponding to the fifth frame sending instruction. The second duration is the delay time of the picture frame corresponding to the fifth frame sending instruction.

[0345] For specific explanations on the rendering time of the picture frame, the fastest frame sending time of the picture frame, the delay time of the picture frame, etc., please refer to the embodiments shown in Figures 6A, 6B, 7, and 8 above and no further details will be given here.

[0346] After the stable frame service module forwards the third frame sending instruction to the SurfaceFlinger, SurfaceFlinger can synthesize and display the picture frames in the buffer indicated by the third frame sending instruction through SurfaceFlinger when it obtains the first vertical synchronization vsync signal; after the stable frame service module forwards the fourth frame sending instruction to the SurfaceFlinger, SurfaceFlinger can synthesize and display the picture frames in the buffer indicated by the third frame sending instruction through SurfaceFlinger when it obtains the second vsync signal, and the second vsync signal is the next vsync signal of the first vsync signal.

[0347] If the stable frame service module determines that there is no buffer in SurfaceFlinger that has not been synthesized and displayed when SurfaceFlinger obtains the third vsync signal and the estimated time when the stable frame service module obtains the fifth frame sending instruction generated by the first application is later than the time when SurfaceFlinger obtains the third vsync signal, before SurfaceFlinger obtains the third vsync signal, the stable frame service module generates a first predicted frame based on the frame in the buffer indicated by the third frame sending instruction and the frame in the buffer specified by the fourth frame sending instruction; the first predicted frame is provided to the SurfaceFlinger through the stable frame service module; when SurfaceFlinger obtains the third vsync signal, SurfaceFlinger synthesizes and displays the first predicted frame; wherein the third vsync signal is the next vsync signal of the second vsync signal, and the difference between the time when SurfaceFlinger obtains the third vsync signal and the time when SurfaceFlinger obtains the second vsync signal is equal to the difference between the time when SurfaceFlinger obtains the second vsync signal and the time when SurfaceFlinger obtains the first vsync signal.

[0348] Among them, the vertical synchronization signal thread (Vsync Thread) can periodically send a vsync signal to SurfaceFlinger.

[0349] For details about the frame loss prediction process and the frame supplement process, please refer to the embodiments shown in Figures 9, 10, 11, and 12 above, which will not be described in detail here.

[0350] Through a frame rate control method provided by an embodiment of the present application, an electronic device can control the frame rate of the picture frames of an application (for example, a game application) to be slightly lower than the original frame rate specified by the application (for example, controlled from the original 60Hz to 58Hz), so that the load of rendering and frame sending becomes lighter, thereby reducing the use of system resources on the electronic device and reducing the power consumption of the electronic device. Optionally, the electronic device can predict the timing of frame loss in advance in the process of controlling the frame rate of the picture frames of an application (for example, a game application). Before the timing of frame loss arrives, a predicted picture frame is generated using multiple previously rendered picture frames through an active interpolation method that is lighter in load than the process of generating picture frames by the application, and is inserted into the queue of the picture frames. In this way, the display frame rate of the picture on the display screen can also reach the original frame rate, ensuring application performance while reducing the load and power consumption of the electronic device.

[0351] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0352] An embodiment of the present application further provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned various method embodiments.

[0353] The present application also provides a chip system, comprising a processor coupled to a memory, the processor executing a computer program stored in the memory to implement the steps of any method embodiment of the present application. The chip system can be a single chip or a chip module composed of multiple chips.

[0354] The term "user interface (UI), or interface for short" in the specification and drawings of this application refers to the media interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface of an application is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on the terminal device, and finally presented as content that the user can recognize, such as pictures, text, buttons and other controls. Controls, also known as widgets, are the basic elements of the user interface. Typical controls include toolbars, menu bars, text boxes, buttons, scroll bars, pictures and text. The properties and contents of controls in the interface are defined by tags or nodes, such as XML through <textview> 、 <imgview> 、 <videoview>The controls contained in the interface are specified by nodes such as <head> and <body>. A node corresponds to a control or attribute in the interface, and the node is presented as user-visible content after parsing and rendering. In addition, many applications, such as hybrid applications, usually also contain web pages in their interfaces. A web page, also known as a page, can be understood as a special control embedded in the application interface. A web page is a source code written in a specific computer language, such as hypertext markup language (HTML), cascading style sheets (CSS), JavaScript (JS), etc. The web page source code can be loaded and displayed as user-recognizable content by a browser or a web page display component with similar functions to a browser. The specific content contained in a web page is also defined by tags or nodes in the web page source code, such as HTML through <body>. 、 、 <video> 、 <canvas>To define the elements and attributes of a web page.

[0355] A common form of user interface is the graphical user interface (GUI), which refers to a user interface related to computer operations that uses graphics. It can be an icon, window, control, or other interface element displayed on the display of an electronic device. Controls can include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, and other visual interface elements.

[0356] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk).

[0357] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0358] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.< / canvas> < / video> < / videoview> < / imgview> < / textview>

Claims

1. A frame rate control method, characterized in that: Applied to an electronic device, the electronic device includes a stable frame service module, the stable frame service module is located between an application layer and a layer synthesizer SurfaceFlinger, the electronic device is installed with a first application, and the method includes: The first application generates a frame sending instruction; When the electronic device does not meet the frame control condition, forwarding the frame sending instruction generated by the first application to the SurfaceFlinger at a first frame rate through the stable frame service module; wherein the frame sending instruction is used to provide the layer buffer buffer of the picture frame to the SurfaceFlinger, and the buffer is used to store the picture frame rendered by the GPU, and the SurfaceFlinger is used to synthesize and display the picture frame in the buffer indicated by the frame sending instruction after obtaining the frame sending instruction; When the electronic device meets the frame control condition, the frame sending instruction generated by the first application is forwarded to the SurfaceFlinger at a second frame rate through the stable frame service module, wherein the first frame rate is greater than the second frame rate, and the frame control condition includes that the picture frame rate set by the first application is the full frame rate, the first frame rate is less than or equal to the full frame rate, and the full frame rate is the maximum refresh rate of the picture frame supported by the first application.

2. The method according to claim 1, characterized in that: The method further comprises: The frame sending instruction generated by the first application is obtained from the graphics interface through the stable frame service module; wherein the graphics interface includes any one of the following: an open graphics library OpenGL, an open graphics library OpenGL ES for embedded systems, and a drawing application program interface Vulkan.

3. The method according to claim 1, characterized in that The frame control conditions also include: the temperature of the electronic device is higher than a specified temperature threshold, and / or the power of the electronic device is lower than a specified power, and / or the occupancy rate of the central processing unit (CPU) of the electronic device is greater than a specified occupancy rate, and / or the first application is a game application.

4. The method according to any one of claims 1 to 3, characterized in that When forwarding the frame sending instruction sent by the first application to the SurfaceFlinger at the second frame rate through the stable frame service module, the method further includes: The SurfaceFlinger synthesizes and displays N picture frames at the first frame rate, wherein the N picture frames include M predicted picture frames, and the M predicted picture frames are generated based on historical picture frames.

5. The method according to claim 4, characterized in that M is the difference between the first frame rate and the second frame rate.

6. The method according to claim 4 or 5, characterized in that: The step of forwarding the frame sending instruction generated by the first application to SurfaceFlinger at a first frame rate through the frame stabilization service module specifically includes: When a first frame sending instruction generated by the first application is acquired through the stable frame service module, the first frame sending instruction is forwarded to the SurfaceFlinger through the stable frame service module; After the first frame sending instruction is forwarded to the SurfaceFlinger through the stable frame service module, when the stable frame service module obtains the second frame sending instruction generated by the first application, the second frame sending instruction is forwarded to the SurfaceFlinger through the stable frame service module.

7. The method according to any one of claims 4 to 6, characterized in that: The step of forwarding the frame sending instruction issued by the first application to SurfaceFlinger at the second frame rate through the frame stabilization service module specifically includes: When the third frame sending instruction generated by the first application is acquired through the stable frame service module, the third frame sending instruction is provided to the SurfaceFlinger through the stable frame service module; After forwarding the third frame sending instruction to the SurfaceFlinger through the stable frame service module, obtaining the fourth frame sending instruction generated by the first application through the stable frame service module; After the stable frame service module obtains the fourth frame sending instruction generated by the first application, the stable frame service module delays forwarding the fourth frame sending instruction to the SurfaceFlinger.

8. The method according to claim 7, characterized in that After the stable frame service module obtains the fourth frame sending instruction sent by the first application, delaying forwarding the fourth frame sending instruction to the SurfaceFlinger through the stable frame service module specifically includes: Determine, by the stable frame service module, a first time interval between when the stable frame service module obtains the fourth frame sending instruction and when the stable frame service module obtains the third frame sending instruction; If the first time interval is less than the first frame sending interval, after the stable frame service module obtains the fourth frame sending instruction, the stable frame service module delays the first duration and forwards the fourth frame sending instruction to the SurfaceFlinger, wherein the first frame sending interval is a frame control period, the frame control period is the inverse of the second frame rate, and the first duration is the difference between the first frame sending interval and the first time interval.

9. The method according to claim 8, characterized in that The method further comprises: If the first time interval is greater than or equal to the first frame sending interval, when the stable frame service module obtains the fourth frame sending instruction, the fourth frame sending instruction is forwarded to the SurfaceFlinger through the stable frame service module.

10. The method according to any one of claims 7 to 9, characterized in that: The method further comprises: After the stable frame service module delays forwarding the fourth frame sending instruction to the SurfaceFlingre, the stable frame service module obtains the fifth frame sending instruction generated by the first application; Determine, by the stable frame service module, a second time interval between when the stable frame service module obtains the fifth frame sending instruction and when it obtains the fourth frame sending instruction; If the second time interval is less than the second frame sending interval, after the stable frame service module obtains the fifth frame sending instruction, the stable frame service module delays the second time length and forwards the fifth frame sending instruction to the SurfaceFlinger, wherein the second time length is the difference between the second frame sending interval and the second time interval, and the second frame sending interval is determined by the first time interval, the first frame sending interval and the control frame cycle.

11. The method according to claim 10, characterized in that The second frame sending interval is determined by the following formula: Among them, Ta is the first time interval, T1 is the control frame period, TNa is the first frame sending interval, and TNb is the second frame sending interval.

12. The method according to any one of claims 7 to 11, characterized in that: The method further comprises: After the frame stabilization service module forwards the third frame sending instruction to the SurfaceFlinger, when the SurfaceFlinger obtains the first vertical synchronization vsync signal, the picture frame in the buffer indicated by the third frame sending instruction is synthesized and displayed by the SurfaceFlinger; After the stable frame service module forwards the fourth frame sending instruction to the SurfaceFlinger, when the SurfaceFlinger obtains the second vsync signal, the screen frame in the buffer indicated by the third frame sending instruction is synthesized and displayed through the SurfaceFlinger, and the second vsync signal is the next vsync signal of the first vsync signal.

13. The method according to claim 12, characterized in that The method further comprises: If it is determined by the stable frame service module that there is no buffer of unsynthesized display screen frames in the SurfaceFlinger when the SurfaceFlinger obtains the third vsync signal, and the estimated time when the stable frame service module obtains the fifth frame sending instruction generated by the first application is later than the time when the SurfaceFlinger obtains the third vsync signal, before the SurfaceFlinger obtains the third vsync signal, the stable frame service module generates a first predicted screen frame based on the screen frames in the buffer indicated by the third frame sending instruction and the screen frames in the buffer specified by the fourth frame sending instruction; Providing the first predicted picture frame to the SurfaceFlinger through the stable frame service module; When the SurfaceFlinger obtains the third vsync signal, the first predicted picture frame is synthesized and displayed by the SurfaceFlinger; Among them, the third vsync signal is the next vsync signal of the second vsync signal, and the difference between the time when the SurfaceFlinger obtains the third vsync signal and the time when the SurfaceFlinger obtains the second vsync signal is equal to the difference between the time when the SurfaceFlinger obtains the second vsync signal and the time when the SurfaceFlinger obtains the first vsync signal.

14. An electronic device, characterized in that: It comprises one or more processors and one or more memories; wherein, the one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, and the computer program code includes computer instructions, and when the one or more processors execute the computer instructions, the first electronic device executes the method as described in any one of claims 1-13.

15. A chip system, characterized in that: The chip system is applied to an electronic device, and the chip system includes a processing circuit and an interface circuit, the interface circuit is used to receive instructions and transmit them to the processing circuit, and the processing circuit is used to run the instructions to execute the method as described in any one of claims 1-13.

16. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 13.

Citation Information

Patent Citations

  • Frame rate control method and related device

    CN119034205A

  • Frame-rate optimization method, apparatus, terminal, and storage medium

    CN109104638A

  • Frame rate switching method and device and storage medium

    CN114461051A

  • Data processing method and related device

    CN114579075A

  • Processing method and device for reducing power consumption of terminal, terminal and readable storage medium

    CN114740965A