Image processing method and electronic equipment

Abstract

The invention provides an image processing method and electronic equipment, and the method comprises the steps that the electronic equipment distributes different numbers of Frame Time Lines for different applications, so that the applications can select proper synthetic Frame Time Lines based on different scene demands, and the synthesis of image frames of the applications is ensured to be uniformly distributed in different Frame Time Line signal periods, and meanwhile, the synthesis efficiency of the applications is improved. The overhead occupied by the Frame Time Line of a part of applications is reduced, and the resource utilization rate of the system is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing and display technology, and in particular to an image processing method and electronic equipment. Background Art

[0002] With the development of terminal equipment technology, terminal equipment has become more and more widely used, and users can watch videos and take photos through terminal equipment. Among them, the visual coherence of the display image of the terminal equipment is an important human-computer interaction performance. Summary of the invention

[0003] The present application provides an image processing method and an electronic device. In the method, the electronic device can allocate a corresponding number of Frame Time Lines to the electronic device according to different scene requirements of the application, so as to realize the integration of the synthesis of image frames.

[0004] In a first aspect, the present application provides an image processing method. The method includes: a first application draws a first image frame. After the first application draws the first image frame, the first application sends a first synthesis request to an image synthesis system. At a first moment, in response to the received first synthesis request, the image synthesis system allocates N1 frame time lines FrameTime Line to the first application, wherein any two adjacent Frame Time Lines in the N1 Frame Time Lines are separated by a Vsync cycle corresponding to the first application, and the start time of the first Frame Time Line in the N1 Frame Time Lines is after the first moment. The first application selects a first target Frame Time Line from the N1 Frame Time Lines, and the start time of the first target Frame Time Line is a second moment. Then, the image synthesis system obtains the first target FrameTime Line and the first image frame. At a second moment, the image synthesis system synthesizes the first image frame based on the first target Frame Time Line. The second application draws a second image frame. After the second application draws the second image frame, the second application sends a second synthesis request to the image synthesis system. At the third moment, the image synthesis system responds to the second synthesis request and allocates M1 Frame Time Lines to the second application, any two adjacent Frame Time Lines among the M1 Frame Time Lines are separated by the Vsync period corresponding to the second application, and the starting time of the first Frame Time Line among the M1 Frame Time Lines is after the third moment; wherein N1 is a positive integer within the first preset range, M1 is a positive integer within the second preset range, and the minimum value of the first preset range is greater than the maximum value of the second preset range. Then, the second application selects a second target Frame Time Line from the M1 Frame Time Lines, and the starting time of the second target Frame Time Line is the fourth moment. And, the image synthesis system obtains the second target Frame Time Line and the second image frame. At the fourth moment, the image synthesis system synthesizes the second image frame based on the second target Frame Time Line.In this way, the present application allocates different numbers of Frame Time Lines to applications according to the scenario requirements of different applications. While satisfying the need for more Frame Time Lines for the first application to ensure the uniform distribution of its synthesized frames and avoid display instability, the second application is allocated a smaller number of Frame Time Lines according to its demand characteristics, so as to reduce the overhead of Frame Time Line information transmission, avoid waste of resources, and reduce the computational pressure of the image synthesis system on the Frame Time Line.

[0005] Exemplarily, the first application and the second application may be displayed on the screen at the same time.

[0006] Exemplarily, the first image frame and the second image frame are drawn in no particular order.

[0007] Exemplarily, the Vsync period of the first application is the same as or different from the Vsync period of the second application.

[0008] Exemplarily, the first Frame Time Line among the multiple Frame Time Lines allocated by the image synthesis system to the first application is the Frame Time Line that is after the first moment and closest to the first moment.

[0009] Exemplarily, the first time may be any time within a Vsync period of the first application.

[0010] In a possible implementation, the method further includes: the first application draws the third image frame. After drawing the third image frame, the first application sends a third synthesis request to the image synthesis system. At the fifth moment, in response to the third synthesis request, the image synthesis system allocates N2 Frame Time Lines to the first application, any two adjacent Frame Time Lines in the N2 Frame Time Lines are separated by the Vsync period corresponding to the first application, and the starting time of the first Frame Time Line in the N2 Frame Time Lines is the same as the starting time of the first Frame Time Line in the N1 Frame Time Lines; wherein N2 is a positive integer within the first preset range. The first application selects a third target Frame Time Line from the N2 Frame Time Lines, and the starting time of the third target Frame Time Line is the sixth moment. The image synthesis system obtains the third target Frame Time Line and the third image frame. At the sixth moment, the image synthesis system synthesizes the third image frame based on the third target Frame Time Line. In this way, for the image frames drawn within the same signal cycle, since the image synthesis system provides multiple Frame Time Lines for the application to choose from, the image frames within the same signal cycle of the application can be evenly distributed to different Frame Time Lines for synthesis, so as to achieve the integration of the synthesis of the image frames.

[0011] Exemplarily, the first image frame may be Figure 8 In frame 5, the third image frame can be Figure 8 Frame 6 in.

[0012] In a possible implementation, the third target Frame Time Line is adjacent to the second target Frame Time Line. In this way, the application can select a suitable Frame Time Line for image frames in the same signal cycle, so that two adjacent image frames can be synthesized in two adjacent Frame Time Lines to avoid frame drop.

[0013] In a possible implementation, the method further includes: at a sixth moment, the screen displays the first image frame. At a seventh moment, the screen displays the third image frame, and the seventh moment and the sixth moment are separated by a Vsync period corresponding to the first application. In this way, the present application integrates the synthesis of image frames so that the synthesis of image frames is allocated to different Frame Time Lines, and further enables the screen to display image frames frame by frame in the order in which the image frames are drawn, thereby avoiding frame drops and freezes.

[0014] In a possible implementation, the method further includes: the third application draws the fourth image frame; after drawing the fourth image frame, the third application sends a fourth synthesis request to the image synthesis system; at the eighth moment, the image synthesis system responds to the fourth synthesis request and allocates P Frame Time Lines to the third application, any two adjacent Frame Time Lines in the P Frame Time Lines are separated by the Vsync cycle corresponding to the third application, and the start time of the first Frame Time Line in the P Frame Time Lines is after the eighth moment; wherein P is a positive integer taken within the third preset range, the minimum value of the first preset range is greater than the maximum value of the third preset range, and the minimum value of the third preset range is greater than the maximum value of the second preset range; the third application selects a fourth target Frame Time Line from the P Frame Time Lines, and the start time of the fourth target Frame Time Line is the ninth moment; the image synthesis system obtains the fourth target Frame Time Line and the fourth image frame; at the ninth moment, the image synthesis system synthesizes the fourth image frame based on the fourth target Frame Time Line. In this way, the present application can set different numbers of Frame Time Lines for different types of applications to meet the needs of different applications.

[0015] Exemplarily, the third application may optionally be a game application.

[0016] In a possible implementation, the method further includes: the second application draws the fifth image frame; after drawing the fifth image frame, the second application sends a fifth synthesis request to the image synthesis system; the image synthesis system obtains a sliding touch event; at the tenth moment, the image synthesis system responds to the fifth synthesis request and the sliding touch event, and allocates M2 Frame Time Lines to the second application, any two adjacent Frame Time Lines in the M2 Frame Time Lines are separated by the Vsync cycle corresponding to the second application, and the starting time of the first Frame Time Line in the M2 Frame Time Lines is after the tenth moment; wherein M2 is a positive integer taken within the first preset range; the second application selects a fifth target Frame Time Line from the M2 Frame Time Lines, and the starting time of the fifth target Frame Time Line is the eleventh moment; the image synthesis system obtains the fifth target Frame Time Line and the fifth image frame; at the eleventh moment, the image synthesis system synthesizes the fifth image frame based on the fifth target Frame Time Line. In this way, the drawing of image frames of the same application in different scenarios may be uniform or uneven. For example, multiple image frames may be drawn in the same signal cycle. Accordingly, by monitoring sliding touch events, the present application can allocate more Frame Time Lines to the application in the case of uneven image frame drawing that may be caused by sliding events, so as to avoid frame drops and freezes.

[0017] In a possible implementation, the method also includes: the second application draws the sixth image frame; after drawing the sixth image frame, the second application sends a sixth synthesis request to the image synthesis system; at the twelfth moment, the image synthesis system responds to the sixth synthesis request and detects whether the sixth image frame includes dynamic image data; if the sixth image frame includes dynamic image data, M3 Frame Time Lines are allocated to the second application, any two adjacent Frame Time Lines in the M3 Frame Time Lines are separated by a Vsync cycle corresponding to the second application, and the starting time of the first Frame Time Line in the M3 Frame Time Lines is after the twelfth moment; wherein M3 is an integer taken within the first preset range; if the sixth image frame does not include dynamic image data, M4 Frame Time Lines are allocated to the second application, any two adjacent Frame Time Lines in the M4 Frame Time Lines are separated by a Vsync cycle corresponding to the second application, and the starting time of the first Frame Time Line in the M4 Frame Time Lines is after the twelfth moment; wherein M4 is an integer taken within the second preset range; the second application selects a sixth target Frame Time Line from the M3 Frame Time Lines or the M4 Frame Time Lines, and the sixth target Frame Time Line The starting time of the Line is the thirteenth time; the image synthesis system obtains the sixth target Frame Time Line and the sixth image frame; at the thirteenth time, the image synthesis system synthesizes the sixth image frame based on the sixth target Frame Time Line. In this way, the application can allocate a corresponding number of Frame Time Lines to different image frames of the same application. If the image frames of the application include dynamic data and the image frames may be unevenly drawn, more Frame Time Lines are allocated to avoid frame drops and freezes.

[0018] In a possible implementation, the first application is a video application or a camera application, and the second application is a news application. In this way, the present application can allocate more FrameTime Lines to video stream applications such as video applications or camera applications to ensure the stability of image frame display. For applications such as chat applications and news applications whose image frames are mainly static data, fewer FrameTime Lines can be allocated to them to avoid wasting resources.

[0019] In a possible implementation, the minimum value of the first preset range is 7, and the maximum value of the second preset range is 3. Exemplarily, N1 may be 7, and M1 may be 3.

[0020] In a second aspect, the present application provides an electronic device, comprising: one or more processors, a memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device performs the following steps: drawing a first image frame of a first application; at a first moment, allocating N1 frame time lines Frame Time Lines to the first application, any two adjacent Frame Time Lines in the N1 Frame Time Lines are separated by a vertical synchronization Vsync period corresponding to the first application, and the starting time of the first Frame Time Line in the N1 Frame Time Lines is after the first moment; wherein a first synthesis request is sent after the first application draws the first image frame; a first target Frame Time Line is selected from the N1 Frame Time Lines, and the starting time of the first target Frame Time Line is a second moment; at a second moment, based on the first target Frame Time Line, synthesizing the first image frame; drawing a second image frame of a second application; at a third moment, allocating M1 Frame Time Lines to the second application, any two adjacent Frame Time Lines in the M1 Frame Time Lines are separated by a Vsync period corresponding to the second application, and the starting time of the first Frame Time Line in the M1 Frame Time Line is after the first moment. The starting time of the Line is after the third time; wherein N1 is a positive integer within the first preset range, M1 is a positive integer within the second preset range, and the minimum value of the first preset range is greater than the maximum value of the second preset range; the second target Frame Time Line is selected from the M1 Frame Time Lines, and the starting time of the second target Frame Time Line is the fourth time; at the fourth time, the second image frame is synthesized based on the second target Frame Time Line.

[0021] In one possible implementation, when the computer program is executed by one or more processors, the electronic device performs the following steps: drawing a third image frame of a first application; at a fifth moment, allocating N2 Frame Time Lines to the first application, any two adjacent Frame Time Lines among the N2 Frame Time Lines are separated by a Vsync period corresponding to the first application, and the starting time of the first Frame Time Line among the N2 Frame Time Lines is the same as the starting time of the first Frame Time Line among the N1 Frame Time Lines; wherein N2 is a positive integer within a first preset range, and a third target Frame Time Line is selected from the N2 Frame Time Lines, and the starting time of the third target Frame Time Line is the sixth moment; at the sixth moment, synthesizing a third image frame based on the third target Frame Time Line.

[0022] In a possible implementation manner, the third target Frame Time Line is adjacent to the second target Frame Time Line.

[0023] In one possible implementation, when the computer program is executed by one or more processors, the electronic device performs the following steps: at the sixth moment, display the first image frame; at the seventh moment, display the third image frame, and the seventh moment is separated from the sixth moment by a Vsync period corresponding to the first application.

[0024] In one possible implementation, when the computer program is executed by one or more processors, the electronic device performs the following steps: drawing a fourth image frame of the second application; obtaining a sliding touch event; at the eighth moment, allocating M2 Frame Time Lines to the second application, any two adjacent Frame Time Lines among the M2 Frame Time Lines are separated by a Vsync cycle corresponding to the second application, and the starting time of the first Frame Time Line among the M2 Frame Time Lines is after the eighth moment; wherein M2 is a positive integer within a first preset range; a fourth target Frame Time Line is selected from the M2 Frame Time Lines, and the starting time of the fourth target Frame Time Line is the ninth moment; at the ninth moment, synthesizing a fourth image frame based on the fourth target Frame Time Line.

[0025] In one possible implementation, when the computer program is executed by one or more processors, the electronic device performs the following steps: drawing the fifth image frame of the second application; at the tenth moment, detecting whether the fifth image frame includes dynamic image data; if the fifth image frame includes dynamic image data, allocating M3 Frame Time Lines to the second application, any two adjacent Frame Time Lines among the M3 Frame Time Lines are separated by a Vsync cycle corresponding to the second application, and the starting time of the first Frame Time Line among the M3 Frame Time Lines is after the tenth moment; wherein M3 is an integer taken within a first preset range; if the fifth image frame does not include dynamic image data, allocating M4 Frame Time Lines to the second application, any two adjacent Frame Time Lines among the M4 Frame Time Lines are separated by a Vsync cycle corresponding to the second application, and the starting time of the first Frame Time Line among the M4 Frame Time Lines is after the tenth moment; wherein M4 is an integer taken within a second preset range; a fifth target Frame Time Line is selected from the M3 Frame Time Lines or the M4 Frame Time Lines, the starting time of the fifth target Frame Time Line is the eleventh moment; at the eleventh moment, based on the fifth target Frame Time Line Line, synthesize the fifth image frame.

[0026] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0027] In a third aspect, the present application provides a computer-readable medium for storing a computer program, wherein the computer program includes instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0028] In a fourth aspect, the present application provides a computer program, comprising instructions for executing the method in the first aspect or any possible implementation of the first aspect.

[0029] In a fifth aspect, the present application provides a chip, the chip comprising a processing circuit and a transceiver pin, wherein the transceiver pin and the processing circuit communicate with each other through an internal connection path, and the processing circuit executes the method in the first aspect or any possible implementation of the first aspect to control the receiving pin to receive a signal and control the sending pin to send a signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a schematic diagram of the hardware structure of an electronic device shown as an example;

[0031] Figure 2 A schematic diagram of the software structure of an electronic device is shown as an example;

[0032] Figure 3 A schematic diagram of an exemplary interface display processing flow;

[0033] Figure 4 A schematic diagram of an exemplary interface display processing flow;

[0034] Figure 5 is a schematic diagram of an image processing method flow chart shown as an example;

[0035] Figure 6 A schematic diagram of obtaining a Frame Time Line is shown as an example;

[0036] Figure 7 The following is a schematic diagram of an exemplary interface display processing flow chart;

[0037] Figure 8 The following is a schematic diagram of an exemplary interface display processing flow chart;

[0038] Fig. 9 A schematic diagram for selecting a Frame Time Line is shown as an example;

[0039] Fig.10 A schematic diagram for selecting a Frame Time Line is shown as an example;

[0040] Fig.11 A schematic diagram for selecting a Frame Time Line is shown as an example;

[0041] Fig.12 A schematic diagram for selecting a Frame Time Line is shown as an example;

[0042] Fig.13 A schematic diagram for selecting a Frame Time Line is shown as an example;

[0043] Fig.14A schematic diagram for selecting a Frame Time Line is shown as an example;

[0044] Fig.15 The following is a schematic diagram of an exemplary interface display processing flow chart;

[0045] Fig.16 The following is a schematic diagram of an exemplary interface display processing flow chart;

[0046] Fig.17 A schematic diagram for selecting a Frame Time Line is shown as an example;

[0047] Fig.18 is a schematic diagram of an image processing method flow chart shown as an example;

[0048] Fig.19 A schematic diagram of an exemplary user interface;

[0049] Fig. 20 The following is a schematic diagram of an exemplary interface display processing flow chart;

[0050] Fig.21 A schematic diagram of an exemplary user interface;

[0051] Fig. 22 Schematic diagram of the structure of the device shown as an example. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0053] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0054] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0055] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0056] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0057] The control method based on FrameTimeLine provided in the embodiment of the present application can be applied to electronic devices with display functions. Among them, the electronic device includes a terminal device, and the electronic device can also be called a terminal, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), etc. The electronic device can be a mobile phone, a smart TV, a wearable device, a tablet computer (Pad), a computer with wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiment of the present application does not limit the specific technology and specific device form adopted by the electronic device.

[0058] In order to better understand the embodiments of the present application, the structure of the electronic device in the embodiments of the present application is introduced below:

[0059] Figure 1 1 shows a schematic diagram of the structure of the electronic device 100. It should be understood that Figure 1 The electronic device 100 shown is only an example of an electronic device, and the structure shown in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0060] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0061] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0062] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0063] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0064] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0065] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the electronic device 100.

[0066] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0067] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to realize the function of answering calls via a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.

[0068] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0069] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the electronic device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the electronic device 100.

[0070] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0071] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 100, and can also be used to transfer data between the electronic device 100 and a peripheral device. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other electronic devices, such as AR devices, etc.

[0072] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0073] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0074] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0075] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0076] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0077] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0078] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0079] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0080] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0081] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0082] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0083] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0084] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0085] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0086] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0087] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0088] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0089] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0090] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0091] The sensor module 180 may include but is not limited to: a pressure sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, and the like.

[0092] Among them, the touch sensor is also called a "touch panel". The touch sensor can be set on the display screen 194, and the touch sensor and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0093] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.

[0094] Figure 2 It is a software structure block diagram of the electronic device 100 according to an embodiment of the present application.

[0095] The layered architecture of the electronic device 100 divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime and system library, the hardware abstraction layer, and the kernel layer.

[0096] The application layer can include a series of application packages.

[0097] like Figure 2 As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0098] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0099] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a resource manager, a notification manager, an image synthesis system, an Input module, and the like.

[0100] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0101] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0102] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0103] The image synthesis system is used to control image synthesis and generate a vertical synchronization (Vsync) signal.

[0104] The image synthesis system includes: synthesis thread, Vsync thread, and queue buffer thread. The synthesis thread is used to be awakened by the Vsync signal for synthesis. The Vsync thread is used to generate the next Vsync signal according to the Vsync signal request. The queue buffer thread is used to store the buffer, generate the Vsync signal request, and wake up the synthesis thread.

[0105] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0106] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0107] The Input module can be used to obtain touch events of the touch sensor from the sensor driver. Among them, the touch event includes but is not limited to: touch position (also known as touch coordinates), wherein the touch position is usually the corresponding position coordinate in the coordinate system established with the upper left corner of the screen as the origin. The data interaction process between the touch sensor, the sensor driver and the Input module can refer to the existing technology, and this application will not repeat it.

[0108] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0109] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0110] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0111] The system library can include multiple functional modules, such as image rendering library, image synthesis library, function library, media library, etc.

[0112] The image rendering library is used to render two-dimensional or three-dimensional images.

[0113] The image synthesis library is used for the synthesis of two-dimensional or three-dimensional images.

[0114] Optionally, the application renders the image through the image rendering library, and then the application sends the rendered image to the cache queue of the image synthesis system. Whenever the Vsync signal arrives, the image synthesis system (e.g., surface flinger) sequentially obtains a frame of image to be synthesized from the cache queue, and then synthesizes the image through the image synthesis library.

[0115] The function library provides macros, type definitions, string operation functions, mathematical calculation functions, and input and output functions used in the C language.

[0116] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0117] The hardware abstraction layer can contain multiple library modules, such as hardware synthesizer (hwcomposer, HWC), camera library module, etc. The Android system can load the corresponding library module for the device hardware, thereby enabling the application framework layer to access the device hardware. Device hardware can include LCD displays, cameras, etc. in electronic devices.

[0118] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, sensor driver (such as touch panel (TP) driver), etc.

[0119] Understandably, Figure 2The layers in the software structure shown and the components included in each layer do not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer layers than shown, and each layer may include more or fewer components, which is not limited in the present application.

[0120] To facilitate understanding, some of the examples given are provided for reference to the description of concepts related to the embodiments of the present application.

[0121] 1. Frame: refers to the smallest unit of a single picture in the interface display. A frame can be understood as a still picture. Displaying multiple connected frames quickly and continuously can create the illusion of motion. Frame rate refers to the number of frames that refresh an image in 1 second. It can also be understood as the number of times the graphics processor in an electronic device refreshes the screen per second. A high frame rate can produce smoother and more realistic animations. The more frames per second, the smoother the displayed action will be.

[0122] It should be noted that before the interface displays a frame, it usually needs to go through processes such as drawing, rendering, and synthesis.

[0123] 2. Frame drawing: refers to the picture drawing of the display interface. The display interface can be composed of one or more views, each view can be drawn by the visual control of the view system, each view is composed of subviews, and a subview corresponds to a small widget in the view, for example, one of the subviews corresponds to a symbol in the picture view.

[0124] 3. Frame rendering: It is to color the drawn view or add 3D effects, etc. For example, 3D effects can be lighting effects, shadow effects, and texture effects.

[0125] 4. Frame synthesis: It is the process of synthesizing multiple or more rendered views into a display interface.

[0126] For example, in order to improve the smoothness of display and reduce the phenomenon of display freeze, electronic devices generally display based on FrameTimeLine (such as Vsync signal) to synchronize the processes of image drawing, rendering, synthesis and screen refresh display. It should be noted that the Vsync signal described in the embodiments of the present application can be replaced by FrameTimeLine, which can be understood as a signal in FrameTimeLine.

[0127] It can be understood that the Vsync signal is a periodic signal, and the Vsync signal period can be set according to the screen refresh rate. For example, when the screen refresh rate is 60Hz, the Vsync signal period can be 16.6ms (or 16.67ms), that is, the electronic device generates a control signal every 16.6ms to trigger the Vsync signal period.

[0128] It should be noted that the Vsync signal can be divided into a software Vsync signal and a hardware Vsync signal. The software Vsync signal includes Vsync-APP and Vsync-SF. Vsync-APP is used to trigger the drawing and rendering process. Vsync-SF is used to trigger the synthesis process. The hardware Vsync signal (Vsync-HW) is used to trigger the screen display refresh process. Optionally, the Vsync signal periods corresponding to the applications in the embodiments of the present application may be the same or different, which is not limited in the present application.

[0129] Normally, the software Vsync signal and the hardware Vsync signal maintain periodic synchronization. Taking the change between 60Hz and 120Hz as an example, if Vsync-HW switches from 60Hz to 120Hz, Vsync-APP and Vsync-SF change synchronously, switching from 60Hz to 120Hz.

[0130] For example, Figure 3 The figure is a schematic diagram of an exemplary electronic device interface display processing flow. In chronological order, the images displayed by the electronic device correspond to frame 1, frame 2, and frame 3 in sequence.

[0131] Specifically, taking the display of frame 1 as an example, the application of the electronic device draws and renders frame 1 through the view system of the application framework layer. After the drawing of frame 1 is completed, the application of the electronic device sends the drawn frame 1 to the image synthesis system (for example, surface flinger). It should be noted that the "drawing" described in the embodiments of the present application can be understood as the drawing and rendering process, which will not be repeated below.

[0132] The image synthesis system synthesizes the drawn frame 1. After the synthesis of frame 1 is completed, the electronic device can start the display driver by calling the kernel layer to display the content corresponding to frame 1 on the screen (display screen). Frames 2 and 3 are synthesized and displayed in a similar process to frame 1, which will not be repeated here.

[0133] like Figure 3As shown, there is a Vsync signal cycle between the drawing and synthesis of each frame, that is, the synthesis lags behind the drawing by one Vsync signal cycle. There is a Vsync signal cycle between synthesis and display, that is, the display lags behind the synthesis by one Vsync signal cycle. In other words, from drawing to display, there is a lag of 2 Vsync signal cycles, and the electronic device has a lag in the process of displaying the image frame.

[0134] Please continue to refer to Figure 3 In some examples, the generation of image frames corresponding to some applications (such as setting applications, chat applications, etc.) in the electronic device or some subprograms (also called subfunctions) in the application is uniform, for example Figure 3 As shown in, taking the settings application as an example, when the user browses the display interface of the settings application, each image frame (such as frame 1) starts to be drawn when the Vsync signal cycle (such as Vsync0) is triggered, and starts to be synthesized when the next adjacent Vsync signal cycle (such as Vsync1) is triggered, and starts to be displayed when the next Vsync signal cycle (such as Vsync2) is triggered.

[0135] like Figure 4 As shown, in some examples, the generation of image frames corresponding to some applications of the electronic device (such as game applications, camera applications or video applications, etc.) or some subprograms of the application (such as the video function in the chat application) is non-uniform. Figure 4 , the electronic device draws frame 1 within the Vsync0 signal cycle, synthesizes frame 1 at the Vsync1 signal cycle triggering moment, and displays frame 1 at the Vsync2 signal cycle triggering moment. In this example, the synthesis of the image frame also lags behind the drawing by 1 signal cycle, and the display of the image frame lags behind the synthesis by 1 signal cycle. However, due to the size of the image data of the image frame and the processing power of the CPU (or GPU) (for example, the CPU is occupied), the drawing of the image frame may be uneven. In some examples, the drawing of the image frame may appear to have multiple frames drawn in the same signal cycle or the drawing of an image frame occupies two or more signal cycles (the specific situation will be described below). In order to ensure the continuity of the image displayed by the terminal device, the terminal device uses the Frame Time Line to control the processing flow of each image frame in the electronic device. How to reasonably set the Frame Time Line becomes one of the important factors for whether the image frame can be displayed coherently.

[0136] Figure 5 For an exemplary interface display processing flow diagram, please refer to Figure 5 , including but not limited to the following steps:

[0137] S401, the camera application sends a Frame Time Line request to the Vsync thread.

[0138] For example, please refer to Figure 2 After the camera application is started in response to the received user operation, it can call the camera driver to obtain the image data collected by the camera. The camera application periodically obtains image data from the camera driver, and the duration of the cycle can be the same as or different from the refresh rate of the display screen, which is not limited in this application.

[0139] After the camera application obtains the image data, it can send an image processing request to the image synthesis system. The request may include information such as the image data to request the image synthesis system to perform processes such as drawing, synthesis, and displaying the image data.

[0140] In response to a request from the camera application, the image synthesis system may call an image rendering library in the system library to draw (or draw and render) the image data.

[0141] For example Figure 4 As shown in , after the image rendering library completes drawing frame 1, it feeds back the completed frame 1 to the camera application. In response to the acquired frame 1, the camera application determines that the drawing of frame 1 is completed, and the camera application sends a Frame Time Line request to the Vsync thread of the image synthesis system, which is used to request the Vsync thread to allocate a Frame Time Line (e.g., a Vsync signal) for frame 1. Optionally, the request includes but is not limited to the identification information of the camera application (e.g., the application package name), etc., which is not limited in this application.

[0142] S402, the Vsync thread determines the number of Frame Time Lines.

[0143] Exemplarily, in an embodiment of the present application, the Vsync thread may maintain (also understood as save) a Frame Time Line correspondence table (in the embodiment of the present application, only the list form is used as an example for explanation, and in other embodiments, it may be in other forms, which is not limited by the present application). The Frame Time Line correspondence table may include, but is not limited to: the correspondence between the identification information of each application (such as the application package name) and the Frame Time Line selection range.

[0144] In the embodiment of the present application, the Frame Time Line selection range included in the Frame Time Line of Vsync may include but is not limited to three ranges, from small to large: [1, 3], [4, 6], [7, ∞). It should be noted that the number of selection ranges (for example, three ranges) and specific values ​​in the embodiment of the present application are only illustrative examples and can be set according to actual needs, and this application does not limit them.

[0145] In the embodiment of the present application, the displayed layer includes an image stream, and the drawing of the image frames in the image stream is non-uniform (for example Figure 4 ) is in the range of [7, ∞) to prevent the "frame drop" problem, avoid screen jitter, and improve the stability of image stream display (the specific effect analysis will be described in detail below). For example, the range corresponding to camera applications and video applications is [7, ∞). The displayed layer includes an image stream, and the image frames in the image stream are drawn more evenly. The corresponding range is [4, 6]. For example, the corresponding range for game applications is [4, 6]. The displayed image does not include the image stream, which can also be understood as the image displayed in the layer is static data, and its image frames are drawn evenly (for example Figure 3 As shown), the range corresponding to this type of application is (1, 3]. For example, the range corresponding to the setting application, chat application, etc. is (1, 3].

[0146] In this example, taking the camera application as an example (other applications will be described later), the range corresponding to the camera application is [7, ∞). Accordingly, the Vsync thread can query the Frame Time Line correspondence table based on the identification information of the camera application carried in the received Frame Time Line request, and determine that the range corresponding to the camera application is [7, ∞). The Vsync thread can select the number of Frame Time Lines for the camera application in the range (i.e., [7, ∞)).

[0147] In a possible implementation, the Vsync thread can select an appropriate number of Frame Time Lines within a preset range (i.e., [7, ∞)) based on the current CPU (and / or GPU) load status. For example, when the CPU load is greater than or equal to 30%, a smaller value (e.g., 7) within the preset range can be selected to minimize the CPU load pressure from the number of Frame Time Lines while meeting the camera application's requirements for the number of Frame Time Lines (e.g., greater than 7). For another example, when the CPU load is small, such as less than 30%, a larger value (e.g., 10) within the preset range can be selected.

[0148] In this example, it is taken that the Vsync thread allocates 7 Frame Time Lines to the camera application as an example for explanation.

[0149] S503, the Vsync thread obtains the Frame Time Line.

[0150] Exemplarily, after the Vsync thread determines the number of Frame Time Lines allocated to the camera application, the corresponding number of Frame Time Lines may be acquired. Figure 6 For an exemplary Frame Time Line acquisition diagram, please refer to Figure 6 In this example, the Vsync thread allocates 7 Frame Time Lines for frame 1 of the camera application. The Vsync thread takes the start time of the adjacent Frame Time Line (i.e., Vsync signal period) as the current time, and selects 7 Frame Time Lines based on the current frame rate (i.e., each signal period is 16.67ms). Figure 6 As shown, after Frame 1 is drawn, the Vsync thread receives the Frame Time Line request sent by the camera application, and the T1 moment is the moment corresponding to the start moment of the approaching (or understandable as about to arrive) signal cycle. The Vsync thread selects 7 Frame Time Lines based on the T1 moment and the frame rate. The moments corresponding to the 7 Frame Time Lines are: T1, T2, T3, T4, T5, T6, T7, and the interval between two adjacent moments (i.e., Frame Time Lines) is 16.67ms (the frame rate is only an illustrative example and is not limited in this application).

[0151] Exemplarily, the Vsync thread assigns corresponding identification information to each Frame Time Line. For example, the identification information corresponding to the Frame Time Line at time T1 is ID1 (for illustrative purposes only, and not limited in this application). The Vsync thread sequentially assigns identification information to each Frame Time Line, which are: ID2 (T2 moment), ID3 (T3 moment), ID4 (T4 moment), ID5 (T5 moment), ID6 (T6 moment), and ID7 (T7 moment).

[0152] It should be noted that the "time" in the embodiment of the present application can optionally be counted from the time the electronic device is turned on, that is, T can also be understood as the power-on time. For example, time T1 is the T1 time when the electronic device is turned on.

[0153] S504, the Vsync thread sends a Frame Time Line response to the camera application.

[0154] Exemplarily, the Vsync thread sends a Frame Time Line response to the camera application, and the response includes but is not limited to: identification information of each Frame Time Line and the corresponding start time and end time. The end time can also be understood as the start time of the next adjacent Frame Time Line. For example, the response includes: the correspondence between ID1, T1 and T2, to indicate that the start time of the Frame Time Line of ID1 is T1 and the end time is T2.

[0155] S505: The camera application selects a target Frame Time Line.

[0156] Exemplarily, the camera application receives the Frame Time Line response sent by the Vsync thread, and obtains the Frame Time Line allocated by the Vsync thread for the current frame (eg, Frame 1), including identification information (ie, ID) of each Frame Time Line as well as the start time and end time.

[0157] The camera application can select a suitable target Frame Time Line for the current frame based on the Frame Time Line corresponding to the synthesis process of the previous frame, so that the Frame Time Line corresponding to the synthesis process of the current frame is separated from the Frame Time Line corresponding to the synthesis process of the previous frame by 1 signal cycle, thereby realizing the integration of frame synthesis. Optionally, the target Frame Time Line is used to indicate the synthesis of the frame, such as the Vsync-SF described above, which is not limited in this application. Specific examples will be described in detail below.

[0158] Exemplarily, the camera application writes the camera application's identification information, relevant information of the selected target Frame Time Line (including ID, start time, end time, etc.), and the drawn frame (e.g., frame 1) into a designated area in the cache (hereinafter referred to as the Frame Time Line cache, also referred to as the drawing frame cache, which is not limited in this application).

[0159] It should be noted that this example only takes the camera application as an example. In actual adaptation, multiple applications can call the Frame Time Line at the same time, that is, execute the drawing, synthesis, display and other processes. Accordingly, after each application completes drawing and selects the corresponding target Frame Time Line, the application identification information and the target Frame Time Line (the target Frame Time Line refers to the one selected by the corresponding application. Each application selects the same or different target Frame Time Line for the frame after drawing. The chat application will be used as an example below for illustration, which will not be repeated here.

[0160] S506, the Vsync thread wakes up the synthesis thread.

[0161] Exemplarily, Vsync starts timing after the electronic device is turned on. As described above, the Vsync thread obtains information such as the Frame Time Line and its corresponding start time. The Vsync thread can generate a Vsync signal (including the Vsync-APP, Vsync-SF, Vsync-HW and other signals described above) based on the timing time when it detects that the start time of the Frame Time Line has been reached.

[0162] The synthesis thread is awakened after detecting the Vsync signal (specifically the Vsync-SF signal).

[0163] S507: The synthesis thread obtains the target Frame Time Line from the camera application.

[0164] Exemplarily, as described above, the camera application saves the target Frame Time Line in the Frame Time Line cache. After the synthesis thread is awakened, the synthesis thread reads the target Frame Time Line from the Frame Time Line cache, where, as described above, the target Frame Time Line selected by multiple applications may be the same, and only the camera application is used as an example for explanation here. The synthesis thread reads from the Frame Time Line cache that the time corresponding to the target Frame Time Line of the camera application (for example, the start time is T1) is the same as the current time (i.e., T1), then the synthesis thread obtains all target Frame Time Lines (including ID, start time, end time, etc.) and corresponding information (including application identification information and image frames, etc.) that are the same as the current time (e.g., T1) from the Frame Time Line cache, such as the identification information of the camera application and the drawn image frame (e.g., frame 1).

[0165] For example, assume that the Frame Time Line cache includes the target Frame Time Line 1 (including ID, start time (T1) and end time) corresponding to the camera application, the identification information of the camera application and frame A. And the target Frame Time Line 2 (including ID, start time (T3) and end time) of the chat application, the identification information of the chat application and frame B. After the synthesis thread is awakened in response to the Vsync-SF signal, it determines that the current time is T1. Based on the current time, the synthesis thread detects that the start time of the target Frame Time Line 1 in the Frame Time Line cache is the current moment, and determines that the current synthesis Frame Time Line should be allocated to the frame required to be synthesized by the target Frame Time Line 1, that is, frame A. The synthesis thread obtains the target Frame Time Line (including ID, start time, etc.) and corresponding information (including the identification information of the camera application, frame A, etc.) from the Frame Time Line cache.

[0166] S508: The synthesis thread performs synthesis based on the target Frame Time Line.

[0167] Specifically, after the synthesis thread obtains the target Frame Time Line of the camera application, the identification of the camera application and the image frame (for example, frame 1), the synthesis thread can call the hardware synthesizer or other modules to complete the synthesis of the image frame. The specific synthesis process can refer to the existing technology and is not limited in this application.

[0168] Exemplarily, after the synthesis thread completes synthesis of the image frame, it transmits the image frame to the display driver, which displays the image frame in the corresponding Frame Time Line signal cycle (ie, the next signal cycle adjacent to the Frame Time Line corresponding to the synthesis).

[0169] Exemplarily, the electronic device executes the process of S501 to S508 for each frame. That is, each time the camera application draws an image frame, it executes S501 to request the synthesis and display process for the image frame.

[0170] To better illustrate the impact of the number of Frame Time Lines on image flows such as camera applications, the following is combined with Figure 7 The interface shown in the figure shows the processing flow diagram, which explains the scenario where the Vsync thread allocates fewer (for example, 1) FrameTime Lines to the camera application. Figure 7As mentioned above, the image frames (frames for short) of camera applications may be irregular (i.e., non-uniform), that is, the starting point of frame drawing may not be at the triggering moment of the Vsync signal cycle (e.g., frame 1), and the duration of frame drawing may span at least two signal cycles (this solution will be explained below), and multiple frames may be drawn in the same signal cycle (e.g., frames 3 to 6). Due to the above characteristics of the frame, the problem of "dropped frames" may occur. Figure 7 As shown, frame 1 starts drawing (or rendering) after the triggering moment of the Vsync0 signal cycle, and is drawn within the Vsync0 signal cycle. As mentioned above, the synthesis and drawing of the image frame lag behind one signal cycle, and the display and synthesis of the image frame lag behind one signal cycle. Accordingly, the Vsync thread allocates one Frame TimeLine for frame 1 (hereinafter, the Vsync signal is taken as an example, and the description is not repeated), that is, the Vsync thread feeds back the identification information, start time and end time of the Vsync1 signal to the camera application. Vsync1 is the target Frame Time Line of frame 1. Accordingly, frame 1 is synthesized at the triggering moment of the Vsync1 signal cycle, and is displayed at the triggering moment of the Vsync2 signal cycle. Frame 2 starts drawing (or rendering) after the triggering moment of the Vsync1 signal cycle, and is drawn within the Vsync1 signal cycle. Frame 2 is synthesized at the triggering moment of the Vsync2 signal (that is, the target Frame Time Line of frame 2, the detailed description can refer to frame 1, which will not be repeated here, and the description will not be repeated below) cycle, and is displayed at the triggering moment of the Vsync3 signal cycle. Frame 3 starts drawing (or rendering) after the triggering moment of the Vsync2 signal cycle, and is completed within the Vsync2 signal cycle. After frame 3 is drawn, the Vsync thread determines that the Vsync3 signal cycle is one signal cycle away from the current signal cycle, and feeds back the identification information, start time, and end time of Vsync3 to the camera application. Vsync3 is the target Frame Time Line for frame 3. Accordingly, frame 3 will be synthesized at the triggering moment of the Vsync3 signal cycle. It should be noted that at this time, since the Vsync3 signal cycle has not arrived, the synthesis thread will not synthesize frame 3 for the time being, but wait for the Vsync3 signal cycle to arrive.

[0171] After frame 4 is drawn (drawn within the Vsync2 signal cycle), the Vsync thread determines that the Vsync3 signal cycle is one signal cycle away from the current signal cycle, and feeds back Vsync3's identification information, start time, end time and other information to the camera application. Vsync3 is the target Frame Time Line for frame 4. Accordingly, frame 4 will be synthesized at the moment when the Vsync3 signal cycle is triggered. It should be noted that at this time, since the Vsync3 signal cycle has not arrived, the synthesis thread will not synthesize frame 4 for the time being, but wait for the Vsync3 signal cycle to arrive. In this example, the Vsync thread assigns the Vsync3 signal cycle to frame 4 for synthesis, that is, replaces (or overwrites) the synthesis moment of frame 3. The processing of frames 5 and 6 is the same as that of frame 4, which will not be repeated here.

[0172] Exemplarily, the synthesis thread synthesizes frame 6 based on the target Frame TimeLine corresponding to frame 6 at the triggering moment of the Vsync3 signal cycle. After an interval of one signal cycle, that is, in the Vsync4 signal cycle, the screen displays frame 6. In other words, the screen displays frame 2 in the Vsync3 signal cycle, and displays frame 6 in the next adjacent signal cycle (that is, the Vsync4 signal cycle). This phenomenon is called "frame drop". For example, the display interface of the camera application displays the content in the Vsync3 signal cycle as follows: the first object is in the first position, but in reality, the original running trajectory of the first object is to slide from the first position to the second position, and thus appear in the third position and the fourth position. Due to "frame drop" (also known as "frame skipping"), after the display interface of the camera application displays the first object appearing in the first position, the first object directly appears in the fourth position in the content displayed in the next frame.

[0173] Please continue to refer to Figure 7 , frame 7 starts to be drawn in the Vsync6 signal cycle and is drawn in this signal cycle. After frame 6 is drawn, the Vsync thread determines that the Vsync7 signal cycle is one signal cycle away from the current signal cycle. Accordingly, at the moment when the Vsync7 signal cycle is triggered, the synthesis thread synthesizes frame 7. The screen displays frame 7 at the moment when the Vsync7 signal cycle is triggered.

[0174] Still refer to Figure 7 , in the Vsync5~Vsync7 signal cycle, since frame 7 has not been synthesized, the Vsync5~Vsync7 signal cycle always displays the "old" data, that is, frame 6. From the user's perspective, the same picture is displayed in multiple signal cycles, which is the "stuttering" phenomenon. For example, after the first object appears in the first position in the display interface of the camera application, the first object appears directly in the fourth position in the content displayed in the next frame. And the first object remains in the fourth position.

[0175] In an embodiment of the present application, the camera application can select a suitable Frame Time Line from the multiple (for example, 7) Frame Time Lines obtained to avoid the problem of frame synthesis being overlapped due to the synthesis of the same frames in the same signal cycle as described above, thereby reducing the probability of "frame drop".

[0176] Figure 8 For an exemplary interface display processing flow diagram, please refer to Figure 8 In this example, the signal period corresponding to the drawing of each image frame is Figure 7 Specifically, after frame 1 is drawn, the camera application and the Vsync thread execute S401 to S404. The specific details can be found above and will not be repeated here.

[0177] For example, the camera application obtains 7 Frame Time Lines allocated by the Vsync thread for frame 1, such as Fig. 9 As shown, it includes: the first Frame Time Line (ID11, starting time T1, ending time T2), the second Frame Time Line (ID12, starting time T2, ending time T3), the third Frame Time Line (ID13, starting time T3, ending time T4), the fourth Frame Time Line (ID14, starting time T4, ending time T5), the fifth Frame Time Line (ID15, starting time T5, ending time T6), the sixth Frame Time Line (ID16, starting time T6, ending time T7), and the seventh Frame Time Line (ID17, starting time T7, ending time T8).

[0178] Assuming that frame 1 is the start frame, the target Frame Time Line of frame 1 does not need to refer to the previous frame, and the first Frame Time Line can be selected as the target Frame Time Line. That is, the identification information of the target Frame Time Line of frame 1 is ID11, the start time is T1, and the end time is T2.

[0179] The camera application executes S505 , that is, writes the identification information of the target Frame Time Line of frame 1 as ID11 , the start time as T1 , and the end time as T2 into the Frame Time Line buffer.

[0180] Still refer to Figure 8, the Vsync thread generates a Vsync1 signal at time T1, and the synthesis thread is awakened in response to the Vsync1 signal.

[0181] After the synthesis thread is awakened, the synthesis thread determines that the current time is the synthesis Frame Time Line of frame 1 based on the current time (ie, T1). The synthesis thread reads the target Frame Time Line identification information ID11, the start time T1, the end time T2 and other information from the Frame Time Line cache, as well as the camera application identification and frame 1 and other information.

[0182] The synthesis thread synthesizes frame 1 within the Vscyn1 signal cycle, and saves the synthesized frame 1 into a cache (hereinafter referred to as screen cache).

[0183] Please continue to refer to Figure 8 After the camera application draws frame 2, the camera application and the Vsync thread execute S401 to S404. The specific details can be found above and will not be repeated here.

[0184] For example, the camera application obtains 7 Frame Time Lines allocated by the Vsync thread for frame 1, such as Fig.10 As shown. It includes: the first Frame Time Line (ID21, starting time T2, ending time T3), the second Frame Time Line (ID22, starting time T3, ending time T4), the third Frame Time Line (ID23, starting time T4, ending time T5), the fourth Frame Time Line (ID24, starting time T5, ending time T6), the fifth Frame Time Line (ID25, starting time T6, ending time T7), the sixth Frame Time Line (ID26, starting time T7, ending time T8), and the seventh Frame Time Line (ID27, starting time T8, ending time T9).

[0185] The camera application determines the target Frame Time Line of Frame 2 as ID21, i.e. the starting time as T2, based on the synthesized Frame Time Line of the previous frame (i.e. Frame 1) (i.e. the target Frame Time Line of Frame 1), so that the synthesized Frame Time Line of Frame 2 is adjacent to the Frame Time Line of Frame 1.

[0186] The camera application executes S505, and writes the identification information of the target Frame Time Line of frame 2 as ID21, the start time as T2, and the end time as T3 into the Frame Time Line cache. Optionally, the camera application can clear the relevant information of the target Frame Time Line of frame 1, and write the target Frame Time Line of frame 2. Optionally, the camera application can also write the target Frame Time Line of frame 2 in a continuous writing manner, and the synthesis thread reads the most recently written target Frame Time Line, which is not limited in this application.

[0187] Still refer to Figure 8 , the Vsync thread generates a Vsync2 signal at time T2. The synthesis thread is awakened in response to the Vsync2 signal.

[0188] After the synthesis thread is awakened, the synthesis thread determines that the current time is the synthesis Frame Time Line of frame 2 based on the current time (ie, T2). The synthesis thread reads the target Frame Time Line identification information ID21, the start time T2, the end time T3 and other information from the Frame Time Line cache, as well as the camera application identification and frame 2 and other information.

[0189] The synthesis thread synthesizes frame 2 within the Vsync2 signal period, and saves the synthesized frame 2 into a cache (hereinafter referred to as screen cache).

[0190] And, the display driver obtains frame 1 from the screen buffer in response to the Vsync2 signal, and displays frame 1 during the Vsync2 signal period.

[0191] Please continue to refer to Figure 8 After the camera application draws frame 3, the camera application and the Vsync thread execute S401 to S404. The specific details can be found above and will not be repeated here.

[0192] For example, the camera application obtains 7 Frame Time Lines allocated by the Vsync thread for frame 3, such as Fig.11As shown. Including: the first Frame Time Line (ID31, starting time T3, ending time T4), the second Frame Time Line (ID32, starting time T4, ending time T5), the first Frame Time Line (ID33, starting time T5, ending time T6), the first Frame Time Line (ID34, starting time T6, ending time T7), the first Frame Time Line (ID35, starting time T7, ending time T8), the first Frame Time Line (ID36, starting time T8, ending time T9), the first Frame Time Line (ID37, starting time T9, ending time T10).

[0193] The camera application determines the target Frame Time Line of Frame 3 as ID31, that is, the starting time is T3, based on the synthetic Frame Time Line of the previous frame (Frame 2) (that is, the target Frame Time Line of Frame 2), so that the synthetic Frame Time Line of Frame 3 is adjacent to the Frame Time Line of Frame 2.

[0194] The camera application executes S505 to write the identification information of the target Frame Time Line of frame 3 as ID31, the start time as T3, and the end time as T4 into the Frame Time Line buffer.

[0195] Still refer to Figure 8 , the Vsync thread generates a Vsync3 signal at time T3, and the synthesis thread is awakened in response to the Vsync3 signal.

[0196] After the synthesis thread is awakened, the synthesis thread determines that the current time is the synthesis Frame Time Line of frame 3 based on the current time (ie, T3). The synthesis thread reads the target Frame Time Line identification information ID31, the start time T3, the end time T4 and other information from the Frame Time Line cache, as well as the camera application identification and frame 3 and other information.

[0197] The synthesis thread synthesizes frame 3 within the Vsync3 signal period, and saves the synthesized frame 3 into a cache (hereinafter referred to as screen cache).

[0198] And, the display driver obtains frame 2 from the screen buffer in response to the Vsync3 signal, and displays frame 2 during the Vsync3 signal period.

[0199] Please continue to refer to Figure 8After the camera application draws frame 4, the camera application and the Vsync thread execute S401 to S404. The specific details can be found above and will not be repeated here.

[0200] For example, the camera application obtains 7 Frame Time Lines allocated by the Vsync thread for frame 4, such as Fig.12 As shown. Including: the first Frame Time Line (ID41, starting time T3, ending time T4), the second Frame Time Line (ID42, starting time T4, ending time T5), the first Frame Time Line (ID43, starting time T5, ending time T6), the first Frame Time Line (ID44, starting time T6, ending time T7), the first Frame Time Line (ID45, starting time T7, ending time T8), the first Frame Time Line (ID46, starting time T8, ending time T9), the first Frame Time Line (ID47, starting time T9, ending time T10).

[0201] The camera application determines the target Frame Time Line of Frame 4 as ID42, i.e. the starting time as T4, based on the synthesized Frame Time Line of the previous frame (i.e. Frame 3) (i.e. the target Frame Time Line of Frame 3), so that the synthesized Frame Time Line of Frame 4 is adjacent to the Frame Time Line of Frame 3.

[0202] The camera application executes S505 and writes the identification information of the target Frame Time Line of frame 4 as ID42, the start time as T4, and the end time as T5 into the Frame Time Line buffer.

[0203] Still refer to Figure 8 , the Vsync thread generates a Vsync4 signal at time T4, and the synthesis thread is awakened in response to the Vsync4 signal.

[0204] After the synthesis thread is awakened, the synthesis thread determines that the current time is the synthesis Frame Time Line of frame 4 based on the current time (i.e., T4). The synthesis thread reads the target Frame Time Line identification information ID42, the start time T4, the end time T5, and other information from the Frame Time Line cache, as well as the camera application identification and frame 3.

[0205] The synthesis thread synthesizes frame 4 in the Vsync4 signal cycle, and saves the synthesized frame 4 into a cache (hereinafter referred to as screen cache).

[0206] And, the display driver obtains frame 3 from the screen buffer in response to the Vsync4 signal, and displays frame 3 within the Vsync4 signal period.

[0207] Please continue to refer to Figure 8 After the camera application draws frame 5, the camera application and the Vsync thread execute S401 to S404. The specific details can be found above and will not be repeated here.

[0208] For example, the camera application obtains 7 Frame Time Lines allocated by the Vsync thread for frame 4, such as Fig.13 As shown. Including: the first Frame Time Line (ID51, starting time T3, ending time T4), the second Frame Time Line (ID52, starting time T4, ending time T5), the first Frame Time Line (ID53, starting time T5, ending time T6), the first Frame Time Line (ID54, starting time T6, ending time T7), the first Frame Time Line (ID55, starting time T7, ending time T8), the first Frame Time Line (ID56, starting time T8, ending time T9), the first Frame Time Line (ID57, starting time T9, ending time T10).

[0209] The camera application determines the target Frame Time Line of frame 5 as ID53, that is, the starting time is T5, based on the synthetic Frame Time Line of the previous frame (frame 4) (i.e., the target Frame Time Line of frame 4), so that the synthetic Frame Time Line of frame 5 is adjacent to the Frame Time Line of frame 4.

[0210] The camera application executes S505 and writes the identification information of the target Frame Time Line of frame 5 as ID53, the start time as T5, and the end time as T6 into the Frame Time Line buffer.

[0211] Still refer to Figure 8 , the Vsync thread generates a Vsync5 signal at time T5, and the synthesis thread is awakened in response to the Vsync5 signal.

[0212] After the synthesis thread is awakened, the synthesis thread determines that the current time is the synthesis Frame Time Line of frame 5 based on the current time (ie, T5). The synthesis thread reads the target Frame Time Line identification information ID53, the start time T5, the end time T6 and other information from the Frame Time Line cache, as well as the camera application identification and frame 3 and other information.

[0213] The synthesis thread synthesizes frame 5 within the Vsync5 signal cycle, and saves the synthesized frame 5 into a cache (hereinafter referred to as screen cache).

[0214] And, the display driver obtains frame 4 from the screen buffer in response to the Vsync5 signal, and displays frame 4 within the Vsync5 signal period.

[0215] The processing of frame 6 is similar to that of frame 5. The identification information of the target Frame Time Line corresponding to frame 6 is ID64 and the starting time is T6. The synthesis thread synthesizes frame 6 within the Vsync6 signal cycle. The screen displays frame 5 within the Vsync6 signal cycle. For specific details, please refer to the processing of frame 5, which will not be repeated here.

[0216] It should be noted that frames 3 to 6 are all drawn within the Vsync3 signal cycle. Therefore, when the Vsync thread obtains the Frame Time Line, the time of the first Frame Time Line determined is time T3.

[0217] Please continue to refer to Figure 8 After the camera application draws frame 7, the camera application and the Vsync thread execute S401 to S404. The specific details can be found above and will not be repeated here.

[0218] For example, the camera application obtains 7 Frame Time Lines allocated by the Vsync thread for frame 7, such as Fig.14As shown. Including: the first Frame Time Line (ID71, starting time T7, ending time T8), the second Frame Time Line (ID72, starting time T8, ending time T9), the first Frame Time Line (ID73, starting time T9, ending time T10), the first Frame Time Line (ID74, starting time T10, ending time T11), the first Frame Time Line (ID75, starting time T11, ending time T12), the first Frame Time Line (ID76, starting time T12, ending time T13), the first Frame Time Line (ID77, starting time T13, ending time T14).

[0219] The camera application determines the target Frame Time Line of frame 7 as ID71, that is, the starting time is T7, based on the synthetic Frame Time Line of the previous frame (frame 6) (i.e., the target Frame Time Line of frame 6), so that the synthetic Frame Time Line of frame 7 is adjacent to the Frame Time Line of frame 6.

[0220] The camera application executes S505 to write the identification information of the target Frame Time Line of frame 7 as ID71, the start time as T7, and the end time as T8 into the Frame Time Line buffer.

[0221] Still refer to Figure 8 , the Vsync thread generates a Vsync7 signal at time T7, and the synthesis thread is awakened in response to the Vsync7 signal.

[0222] After the synthesis thread is awakened, the synthesis thread determines that the current time is the synthesis Frame Time Line of frame 7 based on the current time (ie, T7). The synthesis thread reads the target Frame Time Line identification information ID71, the start time T7, the end time T8 and other information from the Frame Time Line cache, as well as the camera application identification and frame 3 and other information.

[0223] The synthesis thread synthesizes frame 7 within the Vsync7 signal cycle, and saves the synthesized frame 7 into a cache (hereinafter referred to as screen cache).

[0224] And, the display driver obtains frame 6 from the screen buffer in response to the Vsync7 signal, and displays frame 6 within the Vsync7 signal period.

[0225] Exemplarily, the Vsync thread generates a Vsync8 signal at time T8. The display driver responds to the Vsync8 signal, obtains frame 7 from the screen buffer, and displays frame 7 within the Vsync8 signal period.

[0226] In an embodiment of the present application, the Vsync thread allocates sufficient Frame TimeLine for each image frame of the camera application to meet the needs of the camera application, thereby integrating the synthesis of the image frames, so that the synthesis of each image frame can be evenly distributed to each signal cycle, and further enables the screen to display the image frames in the order in which the image frames are drawn, so as to avoid the "frame drop" problem.

[0227] In one possible implementation, Fig.15 As shown, the electronic device may also include a drawing frame buffer for storing at least two drawing frames. In this example, an example is given in which the storage limit is 2 completed frames. In this example, after frame 1 is drawn, it is saved in the drawing frame buffer, and the camera application does not trigger S401 for the time being. After frame 2 is drawn, it is written into the drawing frame buffer, and the camera application triggers S401. That is, in this example, when the Vsync thread obtains the Frame Time Line corresponding to frame 1, the starting time corresponding to the first Frame Time Line is the starting time of the adjacent Frame Time Line signal cycle after frame 2 is drawn, such as time T3. Correspondingly, time T3 is the starting time corresponding to the FrameTime Line of ID11. The processing of other frames is the same as that of frame 1, which will not be repeated here. In this example, if there is a drawing frame that spans multiple signal cycles, for example, the drawing process of frame 2 occupies the Vsync2 and Vsync3 signal cycles, the synthesis of frame 1 can start from time T3, thereby avoiding the stuttering problem (that is, if frame 1 is synthesized from time T2, and frame 2 has not been synthesized within the Vsync2 signal cycle, frame 1 will be displayed in both the Vsync1 and Vsync2 signal cycles).

[0228] The following explanation is based on the chat application as an example. It should be noted that the display processing flow of the chat application can be executed simultaneously with the camera application. For example, the camera application interface and the chat application interface are displayed simultaneously in the display interface of the mobile phone in a split screen or floating window manner. The image frames of the chat application and the camera application will both execute the drawing, synthesis and display processes. The display processing flow corresponding to the chat application is the same as that of the camera application. For details, please refer to Figure 5 The description in will not be repeated here.

[0229] like Fig.16 This is a schematic diagram of the interface display processing flow of the chat application shown as an example. Please refer to Fig.16After frame 1 is drawn, the chat application and the Vsync thread execute S401 to S404. The specific details can be found above and will not be repeated here.

[0230] For example, the Frame Time Line selection range corresponding to the chat application is (1, 3]. In this example, the Vsync thread allocates 3 Frame Time Lines for the chat application. Fig.17 As shown, the three Frame Time Lines allocated by the Vsync thread to frame 1 of the chat application are: the Frame Time Line corresponding to ID11 (starting time is T1, and ending time is T2), the Frame Time Line corresponding to ID12 (starting time is T2, and ending time is T3), and the Frame Time Line corresponding to ID13 (starting time is T3, and ending time is T4).

[0231] The chat application obtains the three Frame Time Lines allocated by the Vsync thread for frame 1, such as Fig.17 As shown above, the data of the chat application are all static data, that is, the signal cycles they occupy during the drawing process are uniform. Accordingly, the chat application can determine the first Frame Time Line (for example, ID11) among the three Frame Time Lines as the target Frame Time Line.

[0232] The chat application executes S505 and writes the identification information of the target Frame Time Line of frame 1 as ID11, the start time as T1, and the end time as T2 into the Frame Time Line cache.

[0233] Still refer to Fig.16 , the Vsync thread generates a Vsync1 signal at time T1, and the synthesis thread is awakened in response to the Vsync1 signal.

[0234] After the synthesis thread is awakened, the synthesis thread determines that the current time is the synthesis Frame Time Line of frame 1 based on the current time (ie, T1). The synthesis thread reads the target Frame Time Line identification information ID11, the start time T1, the end time T2 and other information from the Frame Time Line cache, as well as the camera application identification and frame 1 and other information.

[0235] The synthesis thread synthesizes frame 1 within the Vsync1 signal cycle, and saves the synthesized frame 1 into a cache (hereinafter referred to as screen cache).

[0236] The processing of frame 2 and frame 3 is the same as frame 1, and the first frame time line is selected as the target frame time line. The screen displays frame 1, frame 2 and frame 3 respectively in the Vsync2-Vsync4 signal cycle.

[0237] For example, as described above, the chat application and the camera application may be executed simultaneously. Figure 4 The process in the example above is different, and the number of obtained Frame Time Lines is different. The selected target Frame Time Lines may be the same or different.

[0238] For example, the Vsync thread receives the Frame Time Line request corresponding to frame 4 (frame 4 of the camera application) of the camera application and the Frame Time Line request corresponding to frame 3 (referring to frame 3 of the chat application) of the chat application. They may be received at the same time or one after another, but they are all received within the same signal cycle. The Vsync thread allocates 7 Frame Time Lines to the camera application and 3 Frame Time Lines to the chat application. The camera application selects the Frame Time Line at time T4 as the target Frame Time Line for frame 4, while the chat application selects the Frame Time Line at time T3 as the target Frame Time Line. Therefore, for requests received within the same signal cycle, the target Frame Time Lines selected by different applications may be the same or different, and this application does not limit this. For other undescribed parts, please refer to the above and will not be repeated here.

[0239] In summary, in the embodiment of the present application, the Vsync thread can allocate a corresponding number of FrameTime Lines to the application based on the application requirements. For example, for the camera application, the Vsync thread allocates a large number (for example, more than 7) of FrameTime Lines to it. The camera application can select a suitable Frame Time Line from a sufficient number of Frame Time Lines based on the drawing of the frame, thereby realizing the integration of the synthesis of the frame, so that the synthesis of each frame is evenly distributed in different signal cycles, and then the screen displays the frame in frame order to avoid the "drop frame" problem. For the chat application, its drawing frames are evenly distributed in each signal cycle, and correspondingly, the synthesis of the frame is also evenly distributed. Therefore, for such applications that may not have the "drop frame" problem, the Vsync thread allocates fewer (for example, less than 3) Frame Time Lines to it, thereby reducing the calculation and transmission overhead. For example, for the camera application, the Vsync thread calculates the relevant information of 7 Frame Time Lines for each frame, and feeds back the relevant information corresponding to more than 7 Frame Time Lines to the camera application. For the chat application, the Vsync thread only calculates the relevant information of less than 3 Frame Time Lines for each frame, and feeds back the relevant information corresponding to less than 3 Frame Time Lines to the chat application.

[0240] In an embodiment of the present application, a Vsync thread may be allocated a corresponding number of Frame Time Lines based on different usage scenarios of the application.

[0241] In a possible implementation, the Vsync thread determines the number of corresponding Frame Time Lines for the application based on the touch operation of the application. Fig.18 For an exemplary flow chart of the image processing method, please refer to Fig.18 , including but not limited to the following steps:

[0242] S1801, the chat application sends a Frame Time Line request to the Vsync thread.

[0243] For a specific description, please refer to S501, which will not be repeated here.

[0244] S1802, the Vsync thread obtains a touch event.

[0245] For example, the Vsync thread may pre-register with the Input module to monitor touch events, so that when the Input module obtains a sliding event, it sends a touch event to the Vsync thread to indicate that a sliding touch event occurs in the current interface (i.e., the chat application interface). Fig.19 As shown, the chat application interface is displayed in interface 2001, and the user can slide the interface up and down to browse different contents in the application. In the process of the user sliding the interface, the drawing of the corresponding image frame can be optionally unevenly distributed, for example, Figure 8 shown.

[0246] S1803, the Vsync thread determines the number of Frame Time Lines.

[0247] Exemplarily, the Frame Time Line correspondence table maintained by the Vsync thread may also be combined with touch events to determine the range of Frame Time Line quantities corresponding to different touch events when the application responds to them.

[0248] For example, in the Frame Time Line correspondence table, the selection range corresponding to the chat application is [1, 3], while the selection range corresponding to the chat application and the sliding event is [7, ∞). That is to say, during the display of the chat application interface, if no sliding event occurs, the selection range corresponding to the chat application is [1, 3], and if a sliding event occurs, the corresponding selection range is [7, ∞).

[0249] In this example, the Vsync thread determines that a sliding event occurs on the chat application interface. Accordingly, the Vsync thread allocates 7 Frame Time Lines for the image frames of the chat application.

[0250] The description of S1804 to S1809 can refer to S504 to S509 and will not be repeated here.

[0251] Fig.10 For an exemplary interface display processing flow diagram, please refer to Fig.10 At time T0-T2, the data of the chat application is static data, that is, the drawing of each frame is evenly distributed. Correspondingly, the processing flow of the chat application and the Vsync thread can be referred to Fig.16 For example, a sliding event occurs during the Vsync2 signal cycle to the Vsync7 signal cycle. Accordingly, after frame 4 is drawn, the Vsync thread responds to the chat application's Frame TimeLine request and, based on the obtained sliding event, allocates 7 Frame Time Lines to frames 4 to 8 of the chat application, so that the synthesis process of frames 4 to 8 is evenly distributed in different signal cycles. The specific processing process of frames 4 to 9 can be referred to Figure 8 The relevant description will not be repeated here.

[0252] In another possible implementation, some application interfaces may include image streams, for example, a news application interface may optionally include static data (e.g., text news) and dynamic data (e.g., dynamic advertisements or videos, etc.). In this example, the Frame Time Line request sent by the news application to the Vsync thread may include layer information, and the layer information is used to indicate that the content in the layer is dynamic data (i.e., image stream).

[0253] Exemplarily, the Vsync thread may determine, based on the layer information, that the image frames of the news application include an image stream, and accordingly, the Vsync thread may allocate a larger number (eg, 7) of Frame Time Lines to the image frames including the image stream.

[0254] For example, Fig.21 As shown in (1), the display interface 2101 includes layer 2102a and layer 2102b, wherein layer 2102a and layer 2102b include static data, namely text news. Exemplarily, after the frame 1 corresponding to the display interface 2101 is drawn, the news application sends a Frame Time Line request to the Vsync thread, and the request includes the first layer information of layer 2102a and the second layer information of layer 2102b, wherein the first layer information and the second layer information are both used to indicate that the corresponding layer includes static data. Accordingly, the Vsync thread allocates 3 Frame Time Lines for frame 1 of the news application based on the first layer information and the second layer information, which can be referred to Fig.16 The relevant description in will not be repeated here.

[0255] like Fig.21 As shown in (2), the news application displays a display interface 2103 in response to the received user operation, wherein the display interface 2103 includes but is not limited to: layer 2102c and layer 2102d. Layer 2102c includes static data, namely text news. Layer 2102d includes dynamic data, such as a video stream. Exemplarily, after the frame 2 corresponding to the display interface 2103 is drawn, the news application sends a Frame Time Line request to the Vsync thread, and the request includes the third layer information of layer 2102c and the fourth layer information of layer 2102d, wherein the third layer information is used to indicate that the corresponding layer includes static data, and the fourth layer information is used to indicate that the corresponding layer includes dynamic data. Accordingly, the Vsync thread determines that frame 2 of the news application includes dynamic data based on the third layer information and the fourth layer information, and the Vsync thread allocates 7 Frame Time Lines to frame 2, which can be referred to Figure 8 The relevant description in will not be repeated here.

[0256] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to be beyond the scope of the present application.

[0257] In one example, Fig. 22 A schematic block diagram of a device 2200 according to an embodiment of the present application is shown. The device 2200 may include: a processor 2201 and a transceiver / transceiver pin 2202 , and optionally, a memory 2203 .

[0258] The various components of the device 2200 are coupled together via a bus 2204, wherein the bus 2204 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, various buses are referred to as bus 2204 in the figure.

[0259] Optionally, the memory 2203 may be used for the instructions in the aforementioned method embodiment. The processor 2201 may be used to execute the instructions in the memory 2203, and control the receiving pin to receive a signal, and control the sending pin to send a signal.

[0260] The device 2200 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0261] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0262] This embodiment further provides a computer storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0263] This embodiment also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0264] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the methods in the above-mentioned method embodiments.

[0265] Among them, the electronic device, computer storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above and will not be repeated here.

[0266] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: include: The first application draws a first image frame; The first application sends a first synthesis request to the image synthesis system after drawing the first image frame; At a first moment, the image synthesis system allocates N1 frame time lines to the first application in response to the first synthesis request, any two adjacent frame time lines in the N1 frame time lines are separated by a vertical synchronization Vsync period corresponding to the first application, and a start time of the first frame time line in the N1 frame time lines is after the first moment; The first application selects a first target Frame Time Line from the N1 Frame Time Lines, and a start time of the first target Frame Time Line is a second time; The image synthesis system acquires the first target Frame Time Line and the first image frame; At the second moment, the image synthesis system synthesizes the first image frame based on the first target Frame Time Line; The second application draws a second image frame; The second application sends a second synthesis request to the image synthesis system after drawing the second image frame; At a third moment, the image synthesis system allocates M1 Frame Time Lines to the second application in response to the second synthesis request, any two adjacent Frame Time Lines among the M1 Frame Time Lines are separated by a Vsync period corresponding to the second application, and a start time of the first Frame Time Line among the M1 Frame Time Lines is after the third moment; wherein N1 is a positive integer within a first preset range, M1 is a positive integer within a second preset range, and a minimum value of the first preset range is greater than a maximum value of the second preset range; The second application selects a second target Frame Time Line from the M1 Frame Time Lines, and the start time of the second target Frame Time Line is a fourth time; The image synthesis system acquires the second target Frame Time Line and the second image frame; At the fourth moment, the image synthesis system synthesizes the second image frame based on the second target Frame Time Line.

2. The method according to claim 1, characterized in that The method further comprises: The first application draws a third image frame; The first application sends a third synthesis request to the image synthesis system after drawing the third image frame; At a fifth moment, the image synthesis system allocates N2 Frame Time Lines to the first application in response to the third synthesis request, any two adjacent Frame Time Lines among the N2 Frame Time Lines are separated by a Vsync period corresponding to the first application, and a start time of a first Frame Time Line among the N2 Frame Time Lines is the same as a start time of a first Frame Time Line among the N1 Frame Time Lines; wherein N2 is a positive integer within the first preset range; The first application selects a third target Frame Time Line from the N2 Frame Time Lines, and the start time of the third target Frame Time Line is the sixth time; The image synthesis system acquires the third target Frame Time Line and the third image frame; At the sixth moment, the image synthesis system synthesizes the third image frame based on the third target Frame Time Line.

3. The method according to claim 2, characterized in that The third target Frame Time Line is adjacent to the second target Frame Time Line.

4. The method according to claim 3, characterized in that The method further comprises: At the sixth moment, the screen displays the first image frame; At a seventh moment, the screen displays the third image frame, and the seventh moment is separated from the sixth moment by a Vsync period corresponding to the first application.

5. The method according to claim 1, characterized in that The method further comprises: The third application draws a fourth image frame; The third application sends a fourth synthesis request to the image synthesis system after drawing the fourth image frame; At an eighth moment, the image synthesis system allocates P Frame Time Lines to the third application in response to the fourth synthesis request, any two adjacent Frame Time Lines among the P Frame Time Lines are separated by a Vsync period corresponding to the third application, and a start time of a first Frame Time Line among the P Frame Time Lines is after the eighth moment; wherein P is a positive integer within a third preset range, a minimum value of the first preset range is greater than a maximum value of the third preset range, and a minimum value of the third preset range is greater than a maximum value of the second preset range; The third application selects a fourth target Frame Time Line from the P Frame Time Lines, and the starting time of the fourth target Frame Time Line is the ninth time; The image synthesis system acquires the fourth target Frame Time Line and the fourth image frame; At the ninth moment, the image synthesis system synthesizes the fourth image frame based on the fourth target Frame Time Line.

6. The method according to claim 1, characterized in that The method further comprises: The second application draws a fifth image frame; The second application sends a fifth synthesis request to the image synthesis system after drawing the fifth image frame; The image synthesis system acquires a sliding touch event; At the tenth moment, the image synthesis system allocates M2 Frame Time Lines to the second application in response to the fifth synthesis request and the sliding touch event, any two adjacent Frame Time Lines among the M2 Frame Time Lines are separated by a Vsync cycle corresponding to the second application, and a start time of the first Frame Time Line among the M2 Frame Time Lines is after the tenth moment; wherein M2 is a positive integer within the first preset range; The second application selects a fifth target Frame Time Line from the M2 Frame Time Lines, and the starting time of the fifth target Frame Time Line is the eleventh time; The image synthesis system acquires the fifth target Frame Time Line and the fifth image frame; At the eleventh moment, the image synthesis system synthesizes the fifth image frame based on the fifth target Frame Time Line.

7. The method according to claim 1, characterized in that The method further comprises: The second application draws a sixth image frame; The second application sends a sixth synthesis request to the image synthesis system after drawing the sixth image frame; At a twelfth moment, the image synthesis system detects whether the sixth image frame includes dynamic image data in response to the sixth synthesis request; If the sixth image frame includes dynamic image data, M3 Frame Time Lines are allocated to the second application, any two adjacent Frame Time Lines among the M3 Frame Time Lines are separated by a Vsync period corresponding to the second application, and a start time of the first Frame Time Line among the M3 Frame Time Lines is after the twelfth time; wherein M3 is an integer within the first preset range; If the sixth image frame does not include dynamic image data, M4 Frame Time Lines are allocated to the second application, any two adjacent Frame Time Lines among the M4 Frame Time Lines are separated by a Vsync period corresponding to the second application, and a start time of the first Frame Time Line among the M4 Frame Time Lines is after the twelfth time; wherein M4 is an integer within the second preset range; The second application selects a sixth target Frame Time Line from the M3 Frame Time Lines or the M4 Frame Time Lines, and a start time of the sixth target Frame Time Line is the thirteenth time; The image synthesis system acquires the sixth target Frame Time Line and the sixth image frame; At the thirteenth moment, the image synthesis system synthesizes the sixth image frame based on the sixth target Frame Time Line.

8. The method according to any one of claims 1 to 7, characterized in that: The first application is a video application or a camera application, and the second application is a news application.

9. The method according to any one of claims 1 to 8, characterized in that: The minimum value of the first preset range is 7, and the maximum value of the second preset range is 3.

10. An electronic device, characterized in that: include: One or more processors, memory; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and when the computer programs are executed by the one or more processors, the electronic device performs the following steps: Drawing a first image frame of a first application; At a first moment, N1 frame time lines are allocated to the first application, any two adjacent frame time lines among the N1 frame time lines are separated by a vertical synchronization Vsync period corresponding to the first application, and a start time of the first frame time line among the N1 frame time lines is after the first moment; Selecting a first target Frame Time Line from the N1 Frame Time Lines, wherein a start time of the first target Frame Time Line is a second time; At the second moment, synthesizing the first image frame based on the first target Frame Time Line; Drawing a second image frame of a second application; At a third moment, M1 Frame Time Lines are allocated to the second application, any two adjacent Frame Time Lines among the M1 Frame Time Lines are separated by a Vsync period corresponding to the second application, and a start time of the first Frame Time Line among the M1 Frame Time Lines is after the third moment; wherein N1 is a positive integer within a first preset range, M1 is a positive integer within a second preset range, and a minimum value of the first preset range is greater than a maximum value of the second preset range; Selecting a second target Frame Time Line from the M1 Frame Time Lines, wherein a start time of the second target Frame Time Line is a fourth time; At the fourth moment, the second image frame is synthesized based on the second target Frame Time Line.

11. The electronic device according to claim 10, characterized in that: When the computer program is executed by the one or more processors, the electronic device performs the following steps: Drawing a third image frame of the first application; At a fifth moment, N2 Frame Time Lines are allocated to the first application, any two adjacent Frame Time Lines among the N2 Frame Time Lines are separated by a Vsync period corresponding to the first application, and a start time of a first Frame Time Line among the N2 Frame Time Lines is the same as a start time of a first Frame Time Line among the N1 Frame Time Lines; wherein N2 is a positive integer within the first preset range; Selecting a third target Frame Time Line from the N2 Frame Time Lines, wherein the starting time of the third target Frame Time Line is the sixth time; At the sixth moment, the third image frame is synthesized based on the third target Frame Time Line.

12. The electronic device according to claim 11, characterized in that: The third target Frame Time Line is adjacent to the second target Frame Time Line.

13. The electronic device according to claim 12, characterized in that: When the computer program is executed by the one or more processors, the electronic device performs the following steps: At the sixth moment, displaying the first image frame; At a seventh moment, the third image frame is displayed, and the seventh moment is separated from the sixth moment by a Vsync period corresponding to the first application.

14. The electronic device according to claim 10, characterized in that: When the computer program is executed by the one or more processors, the electronic device performs the following steps: drawing a fourth image frame of the second application; Get the sliding touch event; At the eighth moment, M2 Frame Time Lines are allocated to the second application, any two adjacent Frame Time Lines among the M2 Frame Time Lines are separated by a Vsync period corresponding to the second application, and the start time of the first Frame Time Line among the M2 Frame Time Lines is after the eighth moment; wherein M2 is a positive integer within the first preset range; Selecting a fourth target Frame Time Line from the M2 Frame Time Lines, wherein the starting time of the fourth target Frame Time Line is the ninth time; At the ninth moment, the fourth image frame is synthesized based on the fourth target Frame Time Line.

15. The electronic device according to claim 10, characterized in that: When the computer program is executed by the one or more processors, the electronic device performs the following steps: drawing a fifth image frame of the second application; At a tenth moment, detecting whether the fifth image frame includes dynamic image data; If the fifth image frame includes dynamic image data, M3 Frame Time Lines are allocated to the second application, any two adjacent Frame Time Lines among the M3 Frame Time Lines are separated by a Vsync period corresponding to the second application, and a start time of the first Frame Time Line among the M3 Frame Time Lines is after the tenth time; wherein M3 is an integer within the first preset range; If the fifth image frame does not include dynamic image data, M4 Frame Time Lines are allocated to the second application, any two adjacent Frame Time Lines among the M4 Frame Time Lines are separated by a Vsync period corresponding to the second application, and a start time of the first Frame Time Line among the M4 Frame Time Lines is after the tenth time; wherein M4 is an integer within the second preset range; Selecting a fifth target Frame Time Line from the M3 Frame Time Lines or the M4 Frame Time Lines, wherein a starting time of the fifth target Frame Time Line is an eleventh time; At the eleventh moment, the fifth image frame is synthesized based on the fifth target Frame Time Line.

16. A computer storage medium, characterized in that: The method comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 9.

17. A computer program product, characterized in that When the computer program product is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 9.

Images