Image sending and displaying method, electronic equipment, storage medium and chip

Abstract

The invention provides an image sending and displaying method, electronic equipment, a storage medium and a chip, and relates to the technical field of image processing, and the method can be set as follows: after an upper layer application draws a buffer, a SurfaceFlinger (SF) does not need to wait for a Vsync signal, but immediately synthesizes the buffer drawn by the upper layer application, and after a display screen driver waits for the Vsync signal, the synthesized buffer is sent to a display screen for display; in order to avoid the accumulation of the synthesized buffer, the SF can be set to stop the synthesis and close a Vsync signal sent to the SF once after the buffer is drawn by the upper application and the synthesized buffer is not displayed, and the SF is set to take the latest buffer for synthesis and discard the old buffer when the buffer is synthesized; the method can reduce the delay of the content displayed by the display screen.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image display method, electronic equipment, storage medium and chip. Background Art

[0002] As electronic devices become more intelligent, they provide more and more functions. For example, some applications installed in electronic devices can provide a handwriting function, which can be a handwriting operation on the display screen of the electronic device by a stylus or a handwriting operation on the display screen of the electronic device by a user.

[0003] Important indicators of the handwriting function of electronic devices include touch delay, that is, the delay from the user's handwriting track on the display screen of the electronic device to the display of the handwriting in the area corresponding to the handwriting track. At present, the touch delay of the handwriting function of electronic devices is relatively long, resulting in a poor handwriting experience for users. Summary of the invention

[0004] The present application provides an image display method, an electronic device, a storage medium and a chip, which can reduce the touch delay of the handwriting function in the electronic device and improve the user experience.

[0005] In order to achieve the above-mentioned purpose, the first aspect of the present application adopts the following technical solutions:

[0006] The first application receives a first Vsync signal;

[0007] In response to the first Vsync signal, the first application draws a first buffer;

[0008] After the first application completes drawing the first buffer, the first application stores the first buffer in a first storage space, and the first application sends a first synthesis instruction to SurfaceFlinger;

[0009] In response to the first synthesis instruction, the SurfaceFlinger retrieves a second buffer from the first storage space;

[0010] The SurfaceFlinger synthesizes the second buffer;

[0011] The display screen driver receives a second Vsync signal;

[0012] In response to the second Vsync signal, the display screen driver sends the second buffer to the display screen, the second Vsync signal and the first Vsync signal are two adjacent Vsync signals, and the second Vsync signal is later than the first Vsync signal.

[0013] In the present application, after the first application completes drawing the buffer, a synthesis instruction is sent to SurfaceFlinger, so that SurfaceFlinger does not have to wait until the next Vsync signal arrives to synthesize the buffer, thereby shortening the time from drawing to synthesis (or display) of the buffer. In this way, when applied to an application with a handwriting function, the touch delay in the stylus application can be reduced.

[0014] In addition, during the image display process, the buffer drawn by the first application may be an image frame, that is, the first buffer, the second buffer, etc. in the present application may be a first image frame, a second image frame, etc.

[0015] As an implementation manner of the first aspect, the first buffer and the second buffer are the same buffer.

[0016] In the present application, the first buffer and the second buffer can be the same buffer, that is, after the first application draws the first buffer, it sends a synthesis instruction to SurfaceFlinger, and SurfaceFlinger can synthesize the buffer that has just been drawn, thereby shortening the time from drawing to synthesis (display) of the same buffer, thereby reducing the touch delay in the stylus application.

[0017] In addition, it should be noted that after the first application draws the first buffer, the process of SurfaceFlinger synthesizing the first buffer is to synthesize the first buffer with the buffers of other applications (for example, the status bar). For the convenience of description, since the synthesized buffer is still related to the first buffer, the image frame obtained after SurfaceFlinger synthesizes the first buffer can also be recorded as the first buffer, and the naming of other buffers is similar.

[0018] As another implementation manner of the first aspect, when the first buffer and the second buffer are the same buffer, the method further includes:

[0019] In response to the first synthesis instruction, the SurfaceFlinger retrieves a third buffer from the first storage space, where a storage time of the third buffer in the first storage space is earlier than a storage time of the first buffer in the first storage space.

[0020] In the present application, when the first storage space not only stores the first buffer that has just been drawn, but also has a third buffer stored earlier than the first buffer, SurfaceFlinger needs to take out the latest stored first buffer for synthesis, and clear the first storage space, that is, take out the third buffer and discard it. Of course, if there are other buffers in the first storage space that are stored earlier than the first buffer, buffers other than the first buffer need to be discarded. This process of discarding the old buffer and synthesizing the new buffer avoids delayed synthesis and display caused by buffer accumulation, so the touch delay can be reduced.

[0021] As another implementation manner of the first aspect, the first buffer and the second buffer are not the same buffer, and the method further includes:

[0022] In response to the first synthesis instruction, the SurfaceFlinger retrieves the first buffer from the first storage space.

[0023] In the present application, SurfaceFlinger can synthesize the buffer after taking out the buffer from the first storage space, and can also synthesize the buffer with the latest storage time and discard other buffers. In this way, after discarding other buffers, it is equivalent to clearing the first storage space, and can also reduce touch delay.

[0024] Of course, in actual applications, if the first storage space stores more buffers, any buffer among them can be taken for synthesis, and the other buffers are discarded, which can also reduce the touch delay.

[0025] As another implementation of the first aspect, before the first application receives the first Vsync signal, the method further includes:

[0026] The first application receives a second Vsync signal, where the second Vsync signal is a previous Vsync signal of the first Vsync signal;

[0027] In response to the second Vsync signal, the first application draws the third buffer;

[0028] After the first application completes drawing the third buffer, the first application stores the third buffer in the first storage space, and the first application sends a second synthesis instruction to the SurfaceFlinger;

[0029] When the fourth buffer is stored in the second buffer space, the SurfaceFlinger does not synthesize the third buffer in response to the second synthesis instruction, and the Vsync thread stops sending the next Vsync signal to the SurfaceFlinger.

[0030] In the present application, when there are still buffers in the second storage space that have not been sent to the display, SurfaceFlinger can stop the process of synthesizing the buffer this time when it receives the synthesis instruction, and the Vsync thread (the user sends a Vsync signal) can also stop the process of sending a Vsync signal to the SurfaceFlinger once, so that the buffers that may be accumulated in the second storage space can be accumulated in the first storage space, and when the first application synthesizes the buffer, the accumulated buffers can be cleared, thereby reducing the touch delay.

[0031] As another implementation of the first aspect, before the first application receives the first Vsync signal, the method further includes:

[0032] The first application receives a second Vsync signal;

[0033] In response to the second Vsync signal, the first application draws the third buffer;

[0034] After the first application completes drawing the third buffer, the first application stores the third buffer in the first storage space. When the fourth buffer is stored in the second buffer space, the first application stops sending the second synthesis instruction to the SurfaceFlinger, and the Vsync thread stops sending the next Vsync signal to the SurfaceFlinger. The next Vsync signal of the second Vsync signal is the first Vsync signal.

[0035] In the present application, when the fourth buffer is stored in the second buffer space, the first application stops sending the second synthesis instruction to the SurfaceFlinger, so that SurfaceFlinger stops the process of synthesizing the buffer once. Of course, the Vsync thread (used to send a Vsync signal) can also stop the process of sending a Vsync signal to the SurfaceFlinger once, so that the buffers that may be accumulated in the second storage space can be accumulated in the first storage space, and when the first application synthesizes the buffer, the accumulated buffers can be cleared, thereby reducing the touch delay.

[0036] As another implementation of the first aspect, before the first application receives the second Vsync signal, the method further includes:

[0037] The first application receives a third Vsync signal, where the third Vsync signal is a previous signal of the second Vaync signal;

[0038] In response to the third Vsync signal, the first application draws the fourth buffer;

[0039] After the first application completes drawing the fourth buffer, the first application stores the fourth buffer in the first storage space, and the first application sends a third synthesis instruction to the SurfaceFlinger;

[0040] In response to the third synthesis instruction, the SurfaceFlinger retrieves the fourth buffer from the first storage space;

[0041] The SurfaceFlinger synthesizes the fourth buffer;

[0042] After the SurfaceFlinger completes synthesis of the fourth buffer, the SurfaceFlinger stores the fourth buffer in the second storage space, and the time when the SurfaceFlinger completes synthesis of the fourth buffer is later than the time corresponding to the second Vsync signal and earlier than the time corresponding to the first Vsync signal.

[0043] In the present application, there is a situation that may cause the buffer to accumulate in the second storage space: because the time when SurfaceFlinger synthesizes the buffer is later than the time corresponding to the next Vsync signal, and earlier than the time corresponding to the next Vsync signal, the buffer that could have been sent for display by the next Vsync signal is not sent for display, so the buffer will be accumulated.

[0044] As another implementation manner of the first aspect, the first application is a stylus application.

[0045] In the present application, the handwriting operation in the stylus application has a high requirement for the touch delay, that is, the lower the touch delay, the better. Therefore, through the present application, the touch delay of the stylus application can be shortened by about 1 Vsync signal cycle, thereby improving the user experience.

[0046] In a second aspect, an electronic device is provided, comprising a processor, wherein the processor is used to call a computer program stored in a memory to implement any method of the first aspect of the present application.

[0047] In a third aspect, a chip is provided, including a processor, wherein the processor is coupled to a memory, and the processor executes a computer program stored in the memory to implement any method of the first aspect of the present application.

[0048] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer instructions are executed on an electronic device, the electronic device implements any one of the methods of the first aspect of the present application.

[0049] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a device, enables the device to execute any one of the methods in the first aspect of the present application.

[0050] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of a hardware structure of an electronic device provided in an embodiment of the present application;

[0052] Figure 2 A schematic diagram of a handwriting scene provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of touch delay in a handwriting scenario provided in an embodiment of the present application;

[0054] Figure 4 A schematic diagram of the time relationship of a handwriting image display process provided in an embodiment of the present application;

[0055] Figure 5 A schematic diagram of the time relationship of another image display process corresponding to handwriting provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of the time relationship of another image display process corresponding to handwriting provided in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of the time relationship of another image display process corresponding to handwriting provided in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of the time relationship of another image display process corresponding to handwriting provided in an embodiment of the present application;

[0059] Fig. 9A schematic diagram of the execution flow of a stylus application, SurfaceFlinger, and display driver provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In the following description, specific details such as specific system structures and technologies are provided for illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details.

[0061] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0062] It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more than two; "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0063] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", "fourth", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0064] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0065] The embodiment of the present application provides an image transmission and display method, and the electronic device can be a tablet computer, a mobile phone, a wearable device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc. The embodiment of the present application does not limit the specific type of the electronic device.

[0066] Figure 1 The schematic diagram of the structure of an electronic device is shown. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. Among them, the sensor module 180 may include a pressure sensor 180A, a touch sensor 180K, an ambient light sensor 180L, etc.

[0067] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0068] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a coprocessor (sensor coprocessor, SCP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. For example, the processor 110 is used to execute the image display method in the embodiment of the present application.

[0069] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0070] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0071] The USB interface 130 is an interface that complies with USB standard specifications, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 may be used to connect a charger to charge the electronic device 100, and may also be used to transmit data between the electronic device 100 and a peripheral device.

[0072] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0073] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0074] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and an application required for at least one function (such as a sound playback function, an image playback function, etc.).

[0075] In addition, the internal memory 121 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0076] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from the wired charger through the USB interface 130 .

[0077] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to power the processor 110, the internal memory 121, the external memory, the display screen 194, the camera 193, and the wireless communication module 160.

[0078] In some other embodiments, the power management module 141 may also be disposed in the processor 110. In some other embodiments, the power management module 141 and the charging management module 140 may also be disposed in the same device.

[0079] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0080] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0081] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0082] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0083] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0084] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0085] The audio module 170 is used to convert digital audio signals into analog audio signals for output, and is also used to convert analog audio inputs into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0086] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0087] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0088] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 170C with his mouth to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only monitor voice information but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, realization of directional recording functions, etc. The headphone jack 170D is used to connect wired headphones. The headphone jack 170D can be a USB interface 130, or it can be a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0089] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 180A.

[0090] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0091] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0092] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0093] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

[0094] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0095] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0096] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0097] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0098] The embodiment of the present application does not particularly limit the specific structure of the execution subject of the image transmission and display method, as long as the code of the image transmission and display method of the embodiment of the present application can be run to communicate according to the image transmission and display method provided by the embodiment of the present application. For example, the execution subject of the image transmission and display method provided by the embodiment of the present application can be a functional module in an electronic device that can call and execute a program, or a communication device applied to an electronic device, such as a chip.

[0099] With the popularity of touch screens in electronic devices, many third-party applications provide handwriting functions. For example, users can use their fingers or stylus to write on the touch screen to trigger the electronic device to generate handwriting corresponding to the handwriting operation. An important indicator of the handwriting function includes touch delay, which is the delay from the user's handwriting operation on the touch screen to the display of the handwriting at the location corresponding to the handwriting operation. This delay can be referred to Figure 2 Schematic diagram of the scene shown.

[0100] like Figure 2 As shown, the user holds a stylus pen and performs handwriting operation on the touch screen, and the electronic device generates handwriting traces based on the handwriting operation at the position where the handwriting operation slides on the touch screen. Due to the touch delay, the generated handwriting traces are obviously behind the handwriting operation of the stylus pen.

[0101] Figure 2 The scenario shown is only an example of touch delay. In actual applications, the user's handwriting operation on the touch screen will also produce Figure 2 Touch delay shown.

[0102] Reference Figure 3 , which is a schematic diagram of the cause of touch delay. The touch delay is the time from the start of a handwriting operation of a user on the touch screen of an electronic device to the display of the handwriting traces at the position corresponding to the handwriting operation on the touch screen.

[0103] The touch sensor can be set in the display screen, and the touch screen is composed of the touch sensor and the display screen. The touch sensor can detect the touch operation on the touch screen and send the touch operation to the stylus application through the InputService. The stylus application generally refers to any application that supports handwriting operation.

[0104] InputService, as a service in the system, can obtain user input and distribute the user input to the corresponding receivers. For example, some inputs are sent to the service in the application framework layer for processing, and some inputs are sent to the upper layer application (for example, the stylus application) for processing. In the embodiment of the present application, when the touch screen displays the handwriting interface of the stylus application, the handwriting operation acting on the handwriting interface area can be distributed to the stylus application as input.

[0105] SurfaceFlinger is used to implement interface synthesis. As an example, when an electronic device displays an interface through a touch screen, the content in the interface displayed on the touch screen may include one or more of the following: a status bar (a bar for displaying system time, power, signal strength, notifications, and other information), wallpapers, backgrounds, the interface of the application itself, a bottom virtual button bar (a bar for displaying back virtual buttons, home virtual buttons, and the like), etc. Each of the above items is a surface object, and the surface object can be cached in a buffer space as buffer data, so the surface can also be recorded as a buffer. In the embodiment of the present application, the buffer space storing the buffer drawn by the stylus application is recorded as the first buffer space. SurfaceFlinger synthesizes the surface objects corresponding to each content contained in the interface, and the synthesized content is also cached as buffer data in the second buffer space, so the synthesized content can also be recorded as a synthesized buffer.

[0106] For the stylus application, the stylus application draws the surface corresponding to the stylus application itself according to the touch coordinates (recorded as the stylus application buffer); the application or thread responsible for the status bar in the system draws the surface corresponding to the status bar (recorded as the status bar buffer), and SurfaceFlinger synthesizes the surface of the stylus application itself and the surface of the status bar to obtain a synthesized buffer, which is stored in the second buffer space.

[0107] The display driver is used to take out the synthesized buffer from the second buffer space and send it to the display for display.

[0108] The execution process in the electronic device corresponding to the touch delay includes: the touch sensor sends the touch point report to the InputService after receiving the handwriting operation; the InputService sends the touch point coordinates to the stylus application after receiving the touch point report; the stylus application draws the buffer according to the touch point coordinates after receiving the touch point coordinates; SurfaceFlinger obtains the buffer drawn by the stylus application and other buffers, and synthesizes the buffer drawn by the stylus application and other buffers; the display driver sends the synthesized buffer to the display for display.

[0109] Figure 3 In the example shown, after the touch sensor receives the handwriting operation, it can immediately send the touch point report to the InputService; after receiving the touch point report, the InputService can immediately send the touch point coordinates corresponding to the touch point report to the stylus application; the delay from the handwriting operation to the contact point coordinates being sent to the stylus application corresponds to segment 1. The stylus application, SurfaceFlinger, and display driver all need to perform corresponding operations based on the trigger of the Vsync signal, so the touch delay is mainly affected by the segment 2 process from the stylus application starting to draw the buffer to the display driver sending the synthesized buffer to the display. The embodiment of the present application can reduce the duration of segment 2.

[0110] Reference Figure 4 , which is a schematic diagram of the time taken from the stylus application starting to draw the buffer to the display screen driver sending it for display, provided in an embodiment of the present application.

[0111] The Vsync synchronization signals in this diagram mainly include hardware Vsync signals and software Vsync signals. The hardware Vsync signal is the TE signal that triggers the display screen to drive and display, and the software Vsync signal is the hardware Vsync signal simulated inside the system, which mainly includes two types: VsyncAPP and VsyncSF. Due to the different transmission processes of the above three Vsync signals, there may be a very small time difference between the VsyncAPP, VsyncSF and TE signals. In the subsequent embodiments, the time difference will be ignored, that is, it is roughly considered that the VsyncAPP, VsyncSF and TE signals are time-aligned, and Vsynci (i is a natural number) represents Vsync signals of different periods.

[0112] Reference Figure 4As shown, after the stylus application receives Vsync1 (actually VsyncAPP1), it starts to draw the buffer1 of the stylus application according to the received touch point coordinates (the touch point coordinates are the coordinates corresponding to the handwriting operation 1); after the stylus application finishes drawing buffer1, it caches the drawn buffer1 in the first buffer space. At this time, only buffer1 exists in the first buffer space. Of course, the drawing process in the embodiment of the present application is the process of the stylus application generating a buffer from the touch point coordinates.

[0113] After SurfaceFlinger receives Vsync2 (actually VsyncSF2), it takes out buffer1 from the first buffer space, synthesizes buffer1 with other buffers (for example, the buffer corresponding to the status bar), and caches the synthesized buffer1 in the second buffer space.

[0114] After receiving Vsync3 (actually TE3 signal), the display screen driver sends the synthesized buffer1 read from the second buffer space to the display screen for display, and the display screen displays the buffer1 sent by the display screen driver.

[0115] In the above embodiment, only buffer1 is used as an example to describe the tasks performed by the stylus application, SurfaceFlinger, and display driver. In actual applications, at the moment corresponding to a Vsync signal, the stylus application, SurfaceFlinger, and display driver may all perform tasks relative to different buffers. For details, please refer to Figure 4 Tasks are shown with dashed lines in the illustrated embodiment.

[0116] Therefore, the touch delay corresponding to the user's handwriting operation 1 includes a lot of waiting time: the waiting time of the stylus application, the waiting time of SurfaceFlinger, and the waiting time of the display driver, resulting in a long touch delay for the stylus application. For applications such as stylus applications that require lower latency, the user experience is poor.

[0117] An embodiment of the present application provides an image display method, which can shorten the time for waiting for the Vsync signal in the above embodiment, thereby shortening the touch delay of the handwriting function.

[0118] Reference Figure 5, which is an image display method provided in an embodiment of the present application. The method can be set to trigger SurfaceFlinger to perform buffer synthesis immediately after the stylus application draws the buffer based on the Vsync signal and puts the buffer into the first buffer space: take the buffer out of the first buffer space, synthesize the buffer with other buffers (for example, the buffer corresponding to the status bar), and put the synthesized buffer into the second buffer space.

[0119] Reference Figure 5 , is a schematic diagram of the time relationship between drawing a buffer and displaying the buffer on the screen in the image display method provided in an embodiment of the present application. The diagram only takes the process from drawing to displaying buffer 1 as an example, and other buffers in the process are not shown in the diagram.

[0120] After the stylus application receives Vsync1 (actually VsyncAPP1), it starts to draw the buffer1 of the stylus application according to the received touch point coordinates (the touch point coordinates are the coordinates corresponding to the handwriting operation 1). At time t1, the stylus application finishes drawing buffer1, and the stylus application caches the drawn buffer1 in the first buffer space. At this time, the number of buffers in the first buffer space changes from 0 to 1, and the buffer in the first buffer space is buffer1. Then, the stylus application sends a synthesis instruction to SurfaceFlinger to trigger SurfaceFlinger to perform buffer synthesis. The synthesis instruction can be sent to the rendering process (Render Thread) of the stylus application.

[0121] After receiving the synthesis instruction, SurfaceFlinger takes out buffer1 from the first buffer space, and the number of buffers in the first buffer space is updated from 1 to 0; SurfaceFlinger starts to synthesize buffer1 with other buffers (for example, the buffer corresponding to the status bar) at time t2, and at time t3, SurfaceFlinger ends the synthesis of buffer1, and SurfaceFlinger caches the synthesized buffer1 in the second buffer space, and the number of synthesized buffers stored in the second buffer space is updated from 0 to 1, and the synthesized buffer stored in the second buffer space is the synthesized buffer1. It can be understood that after time t1, the stylus application needs to store buffer1, and the stylus application needs to send a synthesis instruction. SurfaceFlinger needs to take out buffer1 from the first buffer space to start the synthesis of buffer1 at time t2, so time t2 is later than time t1.

[0122] After receiving Vsync2 (actually TE2 signal), the display screen driver takes out the synthesized buffer1 from the second buffer space and sends it to the display screen for display, and the display screen displays the synthesized buffer1.

[0123] Understandably, Figure 5 The image display method shown in the figure takes about one Vsync signal cycle from the time when the stylus application starts drawing buffer1 to the time when the synthesized buffer1 is displayed on the display screen. Figure 4 With the method shown, the touch delay is shortened by approximately the duration of one Vsync cycle.

[0124] The above embodiments all take the case where there is no buffer accumulation in the first buffer space and the second buffer space as an example. In actual applications, there may also be a case where there is buffer accumulation. For example, there may be two or more buffers in the first buffer space, and there may be two or more synthesized buffers in the second buffer space.

[0125] If there is buffer accumulation in the first buffer space, since the stylus application draws the buffer according to the Vsync signal, the buffers other than the latest buffer in the first buffer space may be delayed buffers. In view of the high touch delay requirements of the stylus application, it can be set that when there is buffer accumulation in the first buffer space, SurfaceFlinger only takes the latest stylus application buffer in the first buffer space and other application buffers (for example, the status bar buffer) for synthesis, and discards other buffers in the first buffer space.

[0126] If there is buffer accumulation in the second buffer space, in theory, the display driver can also be set to only take the latest synthesized buffer for display. However, since the buffer in the second buffer space is a synthesized buffer, if the other synthesized buffers before the latest synthesized buffer in the second buffer space are discarded, it will lead to invalid synthesis work of SurfaceFlinger and invalid drawing work of the stylus application, wasting system resources. Therefore, it can be set: before SurfaceFlinger synthesizes the buffer, if it is found that there are synthesized buffers in the second buffer space that have not been taken away and displayed on the display, SurfaceFlinger can stop the work of synthesizing the buffer this time. In this way, even if there is buffer accumulation, the buffers before synthesis will be accumulated in the first buffer space. As mentioned above, when there is buffer accumulation in the first buffer space, SurfaceFlinger only takes the latest buffer of the stylus application for synthesis, and discards other delayed buffers, thereby improving the image display speed and shortening the touch delay.

[0127] Reference Figure 6 , which is a delay time relationship diagram when there is buffer accumulation in the first buffer space provided in an embodiment of the present application.

[0128] After the stylus application receives Vsync5 (actually VsyncAPP5), it starts to draw the stylus application's buffer5 according to the received touch point coordinates. At time t4, the stylus application ends the drawing of buffer5 and obtains buffer5. The stylus application caches the drawn buffer5 in the first buffer space. At this time, the number of buffers cached in the first buffer space changes from 1 to 2, and buffer4 and buffer5 exist in order from early to late; the stylus application sends a synthesis instruction to SurfaceFlinger to trigger SurfaceFlinger to perform buffer synthesis.

[0129] After receiving the synthesis instruction, SurfaceFlinger takes out the latest buffer5 from the first buffer space, and discards the other buffers in the first buffer space: buffer4. At this time, there is no buffer in the first buffer space, and the number of buffers changes from 2 to 0. At time t5, SurfaceFlinger starts to synthesize the taken-out buffer5 and other application buffers (for example, the buffer corresponding to the status bar). At time t6, SurfaceFlinger completes the synthesis of buffer5 and obtains the synthesized buffer5. SurfaceFlinger caches the synthesized buffer5 in the second buffer space. The number of synthesized buffers in the second buffer space changes from 0 to 1. The synthesized buffer stored in the second buffer space is the synthesized buffer5. Time t5 is later than time t4.

[0130] After receiving Vsync6 (actually TE6 signal), the display driver takes out the synthesized buffer5 from the second buffer space and sends it to the display screen for display. The number of synthesized buffers in the second buffer space changes from 1 to 0, and the display screen displays the synthesized buffer5.

[0131] In actual applications, SurfaceFlinger can take out all buffers in the first buffer space, perform synthesis on buffer5 with the latest storage time, and discard other buffers. SurfaceFlinger can also take out buffer5 with the latest storage time in the first buffer space for synthesis, and then take out other buffers stored in the first buffer space and discard them.

[0132] It is understandable that if Figure 6In the scheme shown, the synthesized buffer5 will be sent to the display screen for display by the display screen driver at the time corresponding to Vsync6.

[0133] If the solution in which SurfaceFlinger takes out the buffers in time sequence for synthesis in the native process is adopted, after SurfaceFlinger receives the synthesis instruction, it takes out the earliest buffer4 from the first buffer space in time sequence, synthesizes buffer4 with the buffers of other applications (for example, the buffer corresponding to the status bar), and caches the synthesized buffer4 in the second buffer space; after the display driver receives Vsync6 (actually TE6 signal), it takes out the synthesized buffer4 from the second buffer space and sends it to the touch screen for display. Similarly, after the display driver receives Vsync7 (actually TE7 signal), it takes out the synthesized buffer5 from the second buffer space and sends it to the display screen for display. That is, the synthesized buffer5 will be sent to the display screen by the display driver at the time corresponding to Vsync7.

[0134] so, Figure 6 As shown, the solution in which SurfaceFlinger takes out the latest buffer from the first buffer space for synthesis and discards other buffers in the first buffer space can shorten the touch delay. Figure 6 The example shown takes the stacking of two buffers as an example. In actual applications, even if more than two buffers are stacked in the first buffer space, SurfaceFlinger takes the latest buffer from the first buffer space for synthesis and discards other buffers in the first buffer space.

[0135] Reference Figure 7 , which is a delay time relationship diagram for SurfaceFlinger to stop the current buffer synthesis work in order to avoid the accumulation of synthesized buffers in the second buffer space, provided in an embodiment of the present application.

[0136] After Vsync8, each module performs the work.

[0137] After receiving Vsync8 (actually VsyncAPP8), the stylus application starts to draw buffer8 of the stylus application according to the received touch point coordinates. The stylus application ends drawing buffer8 at time t7 to obtain buffer8. The stylus application caches buffer8 in the first buffer space. At this time, the number of buffers in the first buffer space changes from 0 to 1, and the buffer in the first buffer space is buffer8. The stylus application sends a synthesis instruction to SurfaceFlinger to trigger SurfaceFlinger to perform buffer synthesis.

[0138] After receiving the synthesis instruction, SurfaceFlinger takes out buffer8 from the first buffer space, and the number of buffers in the first buffer space changes from 1 to 0. At time t8, SurfaceFlinger starts to synthesize the taken-out buffer8 with the buffers of other applications (for example, the buffer corresponding to the status bar). Due to the system jam, the synthesis process is relatively long, and the synthesized buffer8 is not obtained before the arrival of Vsync9. In the embodiment of the present application, SurfaceFlinger finishes synthesizing buffer8 and obtains the synthesized buffer8 at time t9 (time t9 is later than Vsync9) as an example. SurfaceFlinger caches the synthesized buffer8 in the second buffer space, and the number of synthesized buffers in the second buffer space changes from 0 to 1. The synthesized buffer stored in the second buffer space is the synthesized buffer8. Time t8 is later than time t7.

[0139] After Vsync9, each module performs the work.

[0140] After the stylus application receives Vsync9 (actually VsyncAPP9), it starts to draw the stylus application's buffer9 according to the received touch point coordinates. The stylus application ends the drawing of buffer9 at time t10 and obtains buffer9. The stylus application caches the drawn buffer9 in the first buffer space. At this time, the number of buffers in the first buffer space changes from 0 to 1, and the buffer stored in the first buffer space is buffer9; the stylus application sends a synthesis instruction to SurfaceFlinger to trigger SurfaceFlinger to perform buffer synthesis.

[0141] After SurfaceFlinger receives Vsync9 (actually VsyncSF9), since there is no buffer in the first buffer space (it can be understood from the diagram that the number of buffers in the first buffer space from t8 to t10 is 0), SurfaceFlinger does not perform synthesis work based on Vsync9.

[0142] After SurfaceFlinger receives the synthesis instruction sent by the stylus application, it theoretically needs to take out buffer9 from the first buffer space, and then start synthesizing buffer9 at time t11. However, after SurfaceFlinger receives the synthesis instruction sent by the stylus application, it checks that there is buffer8 in the second buffer space that has not been sent to display, so it suspends the work of synthesizing the buffer based on the currently received synthesis instruction.

[0143] After the display driver receives Vsync9 (actually TE9), since Vsync9 is earlier than time t9, there is no synthesized buffer in the second buffer space, so there is no need to perform display sending work.

[0144] After Vsync10, each module performs the work.

[0145] After receiving Vsync10 (actually VsyncAPP10), the stylus application starts to draw the buffer10 of the stylus application according to the received touch point coordinates.

[0146] After SurfaceFlinger receives Vsync10 (actually VsyncSF10), SurfaceFlinger takes out buffer9 from the first buffer space and starts to synthesize the taken out buffer9 with the buffers of other applications (for example, the status bar buffer).

[0147] After receiving Vsync10 (actually TE10), the display driver takes out the synthesized buffer8 from the second buffer space and sends it to the display for display. The number of synthesized buffers stored in the second buffer space changes from 1 to 0, and the display displays the synthesized buffer8.

[0148] In actual applications, the display driver usually takes out the synthesized buffer in the second buffer space earlier than the Vsync signal, and sends the read synthesized buffer to the screen for display at the time corresponding to the Vsync signal. Therefore, after SurfaceFlinger receives Vsync10, it will determine that there is no undisplayed buffer in the second buffer. Therefore, SurfaceFlinger can perform the work of synthesizing buffer9.

[0149] At time t13, the stylus application finishes drawing buffer 10 and obtains buffer 10. After the stylus application stores buffer 10 in the first buffer space, the number of buffers stored in the first buffer space changes from 0 to 1, and the buffer stored in the first buffer space is buffer 10. The stylus application sends a synthesis instruction to SurfaceFlinger.

[0150] If t12 is earlier than t13, SurfaceFlinger performs the following work;

[0151] At time t12, SurfaceFlinger finishes the synthesis of buffer9, obtains the synthesized buffer9, stores the synthesized buffer9 in the second buffer space, and updates the number of synthesized buffers in the second buffer space from 0 to 1. The synthesized buffer stored in the second buffer space is the synthesized buffer9.

[0152] At time t13, after SurfaceFlinger receives the synthesis instruction, it determines that there is buffer9 in the second buffer space that has not been taken out and sent to the display screen for display, so SurfaceFlinger stops the work of synthesizing the buffer. That is, the work of synthesizing buffer10 that should have started at time t14 is stopped. t14 is later than t13.

[0153] If t12 is later than t13, SurfaceFlinger performs the following work;

[0154] At time t13, after SurfaceFlinger receives the synthesis, since the work of synthesizing the current buffer9 has not been completed, it waits until time t12 for SurfaceFlinger to complete the synthesis of buffer9 and obtain the synthesized buffer9, and stores the synthesized buffer9 in the second buffer space. The number of synthesized buffers in the second buffer space is updated from 0 to 1, and the synthesized buffer stored in the second buffer space is the synthesized buffer9.

[0155] SurfaceFlinger continues to respond to the synthesis command sent by the stylus application. However, since buffer9 in the second buffer space has not been taken out and sent to the display screen for display, SurfaceFlinger stops the work of synthesizing the buffer. That is, the work of synthesizing buffer10, which should have started at time t14, is stopped. t14 is later than t13.

[0156] Therefore, there is no restriction on the time between t12 and t13.

[0157] The work performed by each module after Vsync11 is similar to the work performed by each module after Vsync10, and the work performed by each module after Vsync10 may also be continued in the future.

[0158] After receiving Vsync11 (actually VsyncAPP11), the stylus application starts to draw buffer11 of the stylus application according to the received touch point coordinates.

[0159] After SurfaceFlinger receives Vsync11 (actually VsyncSF11), SurfaceFlinger takes out buffer10 from the first buffer space and starts to synthesize the taken out buffer10 with the buffers of other applications (for example, the status bar buffer).

[0160] After receiving Vsync11 (actually TE11), the display screen driver takes out the synthesized buffer9 from the second buffer space and sends it to the display screen for display, and the display screen displays the synthesized buffer9.

[0161] In actual applications, the display driver usually takes out the synthesized buffer in the second buffer space earlier than the Vsync signal, and sends the read-out synthesized buffer to the screen for display at the time corresponding to the Vsync signal. Therefore, after SurfaceFlinger receives Vsync11, it will determine that there is no undisplayed buffer in the second buffer. Therefore, SurfaceFlinger can perform the work of synthesizing buffer10.

[0162] At time t16, the stylus application finishes drawing buffer 11 and obtains buffer 11. After the stylus application stores buffer 11 in the first buffer space, the number of buffers stored in the first buffer space changes from 0 to 1, and the buffer stored in the first buffer space is buffer 11. The stylus application sends a synthesis instruction to SurfaceFlinger.

[0163] Taking t15 earlier than t16 as an example, the work performed by SurfaceFlinger is as follows;

[0164] At time t15, SurfaceFlinger finishes the synthesis of buffer10, obtains the synthesized buffer10, stores the synthesized buffer10 in the second buffer space, and updates the number of synthesized buffers in the second buffer space from 0 to 1. The synthesized buffer stored in the second buffer space is the synthesized buffer10.

[0165] At time t16, after receiving the synthesis, SurfaceFlinger determines that there is buffer10 in the second buffer space that has not been taken out and sent to the display screen for display, so SurfaceFlinger stops the work of synthesizing the buffer. That is, the work of synthesizing buffer11 that should have started at time t17 is stopped. t17 is later than t16.

[0166] When t15 is later than t17, no further description will be given, and the details may refer to the working description in the previous Vsync signal cycle.

[0167] It can be understood from the above process that buffer8 also needs about two Vsync signal cycles from the start of drawing to the display. Correspondingly, the subsequent buffer9 and buffer10 may also need about two Vsync signal cycles from drawing to the display.

[0168] Figure 7In the illustrated embodiment, in order to avoid the accumulation of synthesized buffers in the second buffer space, it is set that: after SurfaceFlinger receives the synthesis instruction sent by the stylus application, when it determines that there are buffers in the second buffer space that have not been sent to the display screen for display, SurfaceFlinger stops the process of synthesizing buffers based on the current synthesis instruction. At the same time, considering the possibility of buffer accumulation in the first buffer space, SurfaceFlinger is also set to take out the latest drawn buffer from the first buffer space each time and discard other buffers in the first buffer space. However, the buffers that should theoretically be accumulated in the first buffer space (for example, buffer 9) are synthesized by SurfaceFlinger after Vsync10 arrives. Therefore, SurfaceFlinger subsequently synthesizes the buffer after receiving the Vsync signal, rather than synthesizing the buffer after receiving the synthesis instruction from the stylus application. Therefore, it will continue to execute according to the processing method between Vsync10 and Vsync11, resulting in each buffer going from drawing to display and back to Vsync11. Figure 4 Approximately two Vsync cycles in the embodiment shown.

[0169] To solve this problem, you can set it to: every time SurfaceFlinger receives a synthesis instruction, if there is still a synthesized buffer in the second buffer space that has not been sent to the display, stop the process of synthesizing the buffer and turn off the VsyncSF signal once. In this way, you can restore to Figure 6 The method shown in the figure is then restored to Figure 5 The method shown.

[0170] Reference Figure 8 , which is a delay time relationship diagram when SurfaceFlinger stops the work of synthesizing the buffer and turns off the VsyncSF signal once when there is still a synthesized buffer that has not been displayed in the second buffer space provided by the embodiment of the present application.

[0171] After Vsync8 and after Vsync9, the work performed by each module is Figure 7 The same as in the embodiment shown, can refer to Figure 7 Description of the illustrated embodiment.

[0172] After Vsync10, the stylus application receives Vsync10 (actually VsyncAPP10) and starts to draw the buffer10 of the stylus application according to the received touch point coordinates.

[0173] SurfaceFlinger cannot receive Vsync10 (actually VsyncSF10) or can receive Vsync10 (actually VsyncSF10) but does not respond to Vsync10 (actually VsyncSF10), so the number of the first buffer space remains 1, and the stored buffer is buffer9.

[0174] After receiving Vsync10 (actually TE10), the display driver takes out the synthesized buffer8 from the second buffer space and sends it to the display for display. The number of synthesized buffers stored in the second buffer space changes from 1 to 0, and the display displays the synthesized buffer8.

[0175] At time t13, the stylus application finishes drawing buffer 10 and obtains buffer 10. After the stylus application stores buffer 10 in the first buffer space, the number of buffers stored in the first buffer space changes from 1 to 2, and the buffers stored in the first buffer space are buffer 9 and buffer 10. The stylus application sends a synthesis instruction to SurfaceFlinger.

[0176] After SurfaceFlinger receives the synthesis instruction sent by the stylus application, it determines that there is no unsent synthesized buffer in the second buffer space, so it can perform the work of synthesizing the buffer this time. SurfaceFlinger takes the latest buffer 10 from the first buffer space and discards buffer 9. The number of buffers stored in the first buffer space changes from 2 to 0. At time t14, SurfaceFlinger starts to perform the synthesis work of buffer 10.

[0177] At time t18, SurfaceFlinger finishes the synthesis of buffer10, obtains the synthesized buffer10, stores the synthesized buffer10 in the second buffer space, and updates the number of buffers stored in the second buffer space from 0 to 1. The synthesized buffer stored in the second buffer space is buffer10.

[0178] The work performed by each module after Vsync11 is similar to the work performed by each module after Vsync10, and the work performed by each module after Vsync10 may also be continued in the future.

[0179] After receiving Vsync11 (actually VsyncAPP11), the stylus application starts to draw buffer11 of the stylus application according to the received touch point coordinates.

[0180] After SurfaceFlinger receives Vsync11 (actually VsyncSF11), it checks that there is no buffer in the first buffer space and does not perform buffer synthesis.

[0181] After receiving Vsync11 (actually TE11), the display driver takes out the synthesized buffer10 from the second buffer space and sends it to the display for display. The number of synthesized buffers stored in the second buffer space changes from 1 to 0, and the display displays the synthesized buffer10.

[0182] At time t16, the stylus application finishes drawing buffer 11 and obtains buffer 11. After the stylus application stores buffer 11 in the first buffer space, the number of buffers stored in the first buffer space changes from 0 to 1, and the buffer stored in the first buffer space is buffer 11. The stylus application sends a synthesis instruction to SurfaceFlinger.

[0183] After SurfaceFlinger receives the synthesis instruction sent by the stylus application, it determines that there is no unsent synthesized buffer in the second buffer space, so it can perform the work of synthesizing the buffer this time. SurfaceFlinger takes buffer11 from the first buffer space, and the number of buffers stored in the first buffer space changes from 1 to 0. At time t17, SurfaceFlinger starts to perform the synthesis work of buffer11.

[0184] At time t19, SurfaceFlinger finishes the synthesis of buffer11, obtains the synthesized buffer11, stores the synthesized buffer11 in the second buffer space, and updates the number of buffers stored in the second buffer space from 0 to 1. The synthesized buffer stored in the second buffer space is buffer11.

[0185] In the above embodiment, at time t14, the buffer synthesized by SurfaceFlinger is buffer 10 with the latest storage time in the first buffer space, and buffer 9 is discarded; in actual applications, buffer 9 can also be synthesized and buffer 10 can be discarded. The difference is that at the Vsync11 moment, the display driver sends the synthesized buffer 9 to the display for display, but from the moment corresponding to the next Vsync12 signal, it is restored to display buffer 11, so the touch delay of subsequent buffers from drawing to display is also shortened.

[0186] The above embodiment only takes stacking two buffers as an example. In actual applications, multiple buffers may be stacked. The present application does not limit the number of stacked buffers.

[0187] In this way, although buffer8 goes through about two Vsync cycles from the start of drawing to the display screen display, after stopping VsyncSF10 or stopping responding to VsyncSF10 once, the execution of each internal module is restored. Figure 6 The execution process shown and Figure 5 The execution process, that is, after the subsequent buffer9 is discarded, buffer10 and buffer10 start drawing from the stylus application to being sent to the display for display is about 1 Vsync cycle, thereby shortening the touch delay.

[0188] In the above example, when SurfaceFlinger synthesizes the buffer in the first buffer space, it takes the latest buffer (for example, buffer 10) and discards the accumulated old buffer (for example, buffer 9), which may cause the handwriting corresponding to the discarded buffer (for example, buffer 9) to be missing. However, for the scenario of a stylus, the handwriting is triggered by the user's gesture operation, and the period of a Vsync signal is about 8.3 (120Hz refresh rate) or 16.7ms (60Hz refresh rate). The distance that the user's finger can move in a Vsync signal period is very small, so it is difficult to see the missing handwriting in the generated handwriting. Therefore, the image display method provided in the embodiment of the present application reduces the touch delay without causing the loss of notes in the user experience.

[0189] As another embodiment of the present application, since the stylus application can send synthesis instructions to SurfaceFlinger to trigger SurfaceFlinger to synthesize the buffer, it can also be set to turn off subsequent VsyncSF signals or SurfaceFlinger no longer respond to the VsyncSF signal after starting the stylus application (for example, the stylus application runs in the foreground and displays the handwriting interface of the stylus application).

[0190] The above embodiments take the stylus application scenario as an example. However, it can be understood from the above embodiments that the time shortened by the embodiments of the present application is the time from the start of drawing the buffer data to the time it is sent to the display screen for display. When the electronic device is in the screen-on state, the various modules inside the electronic device are continuously performing drawing, synthesis and display tasks. Therefore, the embodiments of the present application are not only applicable to the stylus application scenarios provided in the above embodiments, but also to scenarios where other applications are running and displayed in the foreground. This application will no longer give examples one by one.

[0191] In addition, in the above embodiments, the i in Vsynci in different embodiments (or drawings) does not necessarily mean that there is a sequence of precedence, and the i in Vsynci related to time in one embodiment (or drawing) has a sequence of precedence. For example, in the same embodiment (or drawing), the time of Vsync(i+1) is later than that of Vsynci. Vsync(i+1) and Vsynci in different embodiments (or drawings) do not have a unique time sequence. Similarly, the i in ti in different embodiments (or drawings) does not necessarily mean that there is a sequence of precedence, and the ti related to time in one embodiment (or drawing) is from early to late from left to right in the drawing.

[0192] by Figure 8 Taking the embodiment shown as an example, the continuous and uninterrupted Vsync signals include: Vsync8, Vsync9, Vsync10, and Vsync11. From Vsync8 to Vsync9 is a Vsync signal cycle, from Vsync9 to Vsync10 is a Vsync signal cycle, and from Vsync10 to Vsync11 is a Vsync signal cycle. The Vsync signals adjacent to Vsync9 are Vsync8 and Vsync10. That is, the time difference between the Vsync signal adjacent to the Vsynci signal and the Vsynci signal is a Vsync signal cycle.

[0193] Reference Fig. 9 , which is a flow chart of an embodiment of the present application from the perspective of the processes executed by each module.

[0194] The stylus application may execute steps S101 to S103 .

[0195] S101, after the stylus application receives the VsyncAPP signal, the stylus application starts drawing the buffer.

[0196] S102: The stylus application stores the drawn buffer in a first buffer space.

[0197] S103, the stylus application sends a synthesis instruction to SurfaceFlinger.

[0198] SurfaceFlinger may execute steps S201 to S208.

[0199] S201, SurfaceFlinger receives a synthesis instruction or a VsyncSF signal sent by a stylus application.

[0200] In an embodiment of the present application, SurfaceFlinger may receive a synthesis instruction issued by a stylus application, and may also receive a VsyncSF signal.

[0201] S202: After receiving the synthesis instruction or the VsyncSF signal, SurfaceFlinger determines whether there is a buffer in the first buffer storage space.

[0202] If there is no buffer in the first buffer storage space, SurfaceFlinger continues to wait for receiving a synthesis instruction or a VsyncSF signal issued by the stylus application.

[0203] S203: If no buffer exists in the first buffer storage space, determine whether there is a buffer in the second buffer storage space.

[0204] S204: If it is determined in S203 that there is a buffer in the second buffer space, SurfaceFlinger takes out all buffers from the first buffer space.

[0205] S205, SurfaceFlinger synthesizes the latest buffer among the buffers taken out and discards the other buffers.

[0206] S206, SurfaceFlinger caches the synthesized buffer in the second buffer space.

[0207] S207, if it is determined in S203 that there is a buffer in the second buffer space, SurfaceFlinger stops taking the buffer from the first buffer space and subsequent synthesis work.

[0208] S208, SurfaceFlinger turns off the next VsyncSF signal, and continues to wait for receiving a synthesis instruction or a VsyncSF signal issued by the stylus application.

[0209] The display screen driver may execute step S301.

[0210] S301, after receiving the TE signal, the display screen driver sends the buffer in the second buffer space to the display screen for display.

[0211] As another embodiment of the present application, an image display method includes:

[0212] The first application receives a first Vsync signal;

[0213] In response to the first Vsync signal, the first application draws a first buffer;

[0214] After the first application completes drawing the first buffer, the first application stores the first buffer in a first storage space, and the first application sends a first synthesis instruction to SurfaceFlinger;

[0215] In response to the first synthesis instruction, the SurfaceFlinger retrieves a second buffer from the first storage space;

[0216] The SurfaceFlinger synthesizes the second buffer;

[0217] The display screen driver receives a second Vsync signal;

[0218] In response to the second Vsync signal, the display screen driver sends the second buffer to the display screen, the second Vsync signal and the first Vsync signal are two adjacent Vsync signals, and the second Vsync signal is later than the first Vsync signal.

[0219] In the present application, after the first application completes drawing the buffer, a synthesis instruction is sent to SurfaceFlinger, so that SurfaceFlinger does not have to wait until the next Vsync signal arrives to synthesize the buffer, thereby shortening the time from drawing to synthesis (or display) of the buffer. In this way, when applied to an application with a handwriting function, the touch delay in the stylus application can be reduced.

[0220] In addition, during the image display process, the buffer drawn by the first application may be an image frame, that is, the first buffer, the second buffer, etc. in the present application may be a first image frame, a second image frame, etc.

[0221] As another embodiment of the present application, the first buffer and the second buffer are the same buffer.

[0222] This embodiment can refer to Figure 5 In the embodiment shown, the first buffer and the second buffer may be the same buffer, for example, buffer 1. Of course, reference may also be made to Figure 6 The embodiment shown, for example, buffer5. Figure 8 In the embodiment shown, for example, buffer 10, after the first application draws and obtains the first buffer, a synthesis instruction is sent to SurfaceFlinger, and SurfaceFlinger can synthesize the buffer just drawn, thereby shortening the time from drawing to synthesis (displaying) of the same buffer, and thus reducing the touch delay in the stylus application.

[0223] In addition, it should be noted that after the first application draws the first buffer, the process of SurfaceFlinger synthesizing the first buffer is to synthesize the first buffer with the buffers of other applications (for example, the status bar). For the convenience of description, since the synthesized buffer is still related to the first buffer, the image frame obtained after SurfaceFlinger synthesizes the first buffer can also be recorded as the first buffer, and the naming of other buffers is similar.

[0224] As another embodiment of the present application, when the first buffer and the second buffer are the same buffer, the method further includes:

[0225] In response to the first synthesis instruction, the SurfaceFlinger retrieves a third buffer from the first storage space, where a storage time of the third buffer in the first storage space is earlier than a storage time of the first buffer in the first storage space.

[0226] This embodiment can refer to Figure 6 In the embodiment shown, for example, the first buffer and the second buffer are buffer5, and the third buffer is buffer4. Figure 8 In the illustrated embodiment, for example, the first buffer and the second buffer are buffer 10 , and the third buffer is buffer 9 .

[0227] When the first storage space not only stores the first buffer that has just been drawn, but also has a third buffer stored earlier than the first buffer, SurfaceFlinger needs to take out the latest stored first buffer for synthesis and clear the first storage space, that is, take out the third buffer and discard it. Of course, if there are other buffers in the first storage space that are stored earlier than the first buffer, the buffers except the first buffer need to be discarded. This process of discarding the old buffer and synthesizing the new buffer avoids the delayed synthesis and display caused by buffer accumulation, so the touch delay can be reduced.

[0228] As another embodiment of the present application, the first buffer and the second buffer are not the same buffer, and the method further includes:

[0229] In response to the first synthesis instruction, the SurfaceFlinger retrieves the first buffer from the first storage space.

[0230] In the present application, SurfaceFlinger can synthesize the buffer after taking out the buffer from the first storage space, and can also synthesize the buffer with the latest storage time and discard other buffers. In this way, after discarding other buffers, it is equivalent to clearing the first storage space, and can also reduce touch delay.

[0231] Of course, in actual applications, if the first storage space stores more buffers, any buffer among them can be taken for synthesis, and the other buffers are discarded, which can also reduce the touch delay.

[0232] As another embodiment of the present application, before the first application receives the first Vsync signal, the method further includes:

[0233] The first application receives a second Vsync signal, where the second Vsync signal is a previous Vsync signal of the first Vsync signal;

[0234] In response to the second Vsync signal, the first application draws the third buffer;

[0235] After the first application completes drawing the third buffer, the first application stores the third buffer in the first storage space, and the first application sends a second synthesis instruction to the SurfaceFlinger;

[0236] When the fourth buffer is stored in the second buffer space, the SurfaceFlinger does not synthesize the third buffer in response to the second synthesis instruction, and the Vsync thread stops sending the next Vsync signal to the SurfaceFlinger.

[0237] This embodiment can refer to Figure 8 In the illustrated embodiment, the fourth buffer may be buffer8, the first buffer and the second buffer may be buffer10, and the third buffer may be buffer9.

[0238] When there is still buffer in the second storage space that has not been sent to the display, SurfaceFlinger can stop the process of synthesizing the buffer when it receives the synthesis instruction, and the Vsync thread (the user sends a Vsync signal) can also stop the process of sending a Vsync signal to the SurfaceFlinger. In this way, the buffer that may be accumulated in the second storage space can be accumulated in the first storage space, and when the first application synthesizes the buffer, the accumulated buffer can be cleared, thereby reducing the touch delay.

[0239] As another embodiment of the present application, before the first application receives the first Vsync signal, the method further includes:

[0240] The first application receives a second Vsync signal;

[0241] In response to the second Vsync signal, the first application draws the third buffer;

[0242] After the first application completes drawing the third buffer, the first application stores the third buffer in the first storage space. When the fourth buffer is stored in the second buffer space, the first application stops sending the second synthesis instruction to the SurfaceFlinger, and the Vsync thread stops sending the next Vsync signal to the SurfaceFlinger. The next Vsync signal of the second Vsync signal is the first Vsync signal.

[0243] In the present application, when the fourth buffer is stored in the second buffer space, the first application stops sending the second synthesis instruction to the SurfaceFlinger, so that SurfaceFlinger stops the process of synthesizing the buffer once. Of course, the Vsync thread (the user sends a Vsync signal) can also stop the process of sending the Vsync signal to the SurfaceFlinger once, so that the buffer that may be accumulated in the second storage space can be accumulated in the first storage space, and when the first application synthesizes the buffer, the accumulated buffer can be cleared, thereby reducing the touch delay.

[0244] As another embodiment of the present application, before the first application receives the second Vsync signal, the method further includes:

[0245] The first application receives a third Vsync signal, where the third Vsync signal is a previous signal of the second Vsync signal;

[0246] In response to the third Vsync signal, the first application draws the fourth buffer;

[0247] After the first application completes drawing the fourth buffer, the first application stores the fourth buffer in the first storage space, and the first application sends a third synthesis instruction to the SurfaceFlinger;

[0248] In response to the third synthesis instruction, the SurfaceFlinger retrieves the fourth buffer from the first storage space;

[0249] The SurfaceFlinger synthesizes the fourth buffer;

[0250] After the SurfaceFlinger completes synthesis of the fourth buffer, the SurfaceFlinger stores the fourth buffer in the second storage space, and the time when the SurfaceFlinger completes synthesis of the fourth buffer is later than the time corresponding to the second Vsync signal and earlier than the time corresponding to the first Vsync signal.

[0251] In the present application, there is a situation that may cause the buffer to accumulate in the second storage space: because the time when SurfaceFlinger synthesizes the buffer is later than the time corresponding to the next Vsync signal, and earlier than the time corresponding to the next Vsync signal, the buffer that could have been sent for display by the next Vsync signal is not sent for display, so the buffer will be accumulated.

[0252] As another embodiment of the present application, the first application is a stylus application.

[0253] In the embodiment of the present application, the handwriting operation in the stylus application has a high requirement for the touch delay, that is, the lower the touch delay, the better. Therefore, through this application, the touch delay of the stylus application can be shortened by about 1 Vsync signal cycle, thereby improving the user experience.

[0254] It should be noted that the "first", "second", "third", etc. in the embodiments of the present application are used to indicate different buffers, unused Vsync signals, different synthesis instructions, etc. Unless otherwise specified, they may indicate the same buffer.

[0255] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0256] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed on an electronic device, the steps in the above-mentioned method embodiments can be implemented.

[0257] The embodiment of the present application also provides a computer program product. When the computer program product is run on an electronic device or a wireless router, the electronic device can implement the steps in the above-mentioned various method embodiments.

[0258] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to the first device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0259] The embodiment of the present application also provides a chip, the chip includes a processor, the processor is coupled to a memory, and the processor calls a computer program stored in the memory to implement the steps of any method embodiment of the present application. The chip can be a single chip or a chip module composed of multiple chips.

[0260] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0261] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0262] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image display method, characterized in that: include: The first application receives a first Vsync signal; In response to the first Vsync signal, the first application draws a first buffer; After the first application completes drawing the first buffer, the first application stores the first buffer in a first storage space, and the first application sends a first synthesis instruction to SurfaceFlinger; In response to the first synthesis instruction, the SurfaceFlinger retrieves a second buffer from the first storage space; The SurfaceFlinger synthesizes the second buffer; The display screen driver receives a second Vsync signal; In response to the second Vsync signal, the display screen driver sends the second buffer to the display screen, the second Vsync signal and the first Vsync signal are two adjacent Vsync signals, and the second Vsync signal is later than the first Vsync signal.

2. The method according to claim 1, characterized in that The first buffer and the second buffer are the same buffer.

3. The method according to claim 2, characterized in that The method further comprises: In response to the first synthesis instruction, the SurfaceFlinger retrieves a third buffer from the first storage space, where a storage time of the third buffer in the first storage space is earlier than a storage time of the first buffer in the first storage space.

4. The method according to claim 1, characterized in that The first buffer and the second buffer are not the same buffer, and the method further includes: In response to the first synthesis instruction, the SurfaceFlinger retrieves the first buffer from the first storage space.

5. The method according to claim 3, characterized in that Before the first application receives the first Vsync signal, the method further includes: The first application receives a second Vsync signal, where the second Vsync signal is a previous Vsync signal of the first Vsync signal; In response to the second Vsync signal, the first application draws the third buffer; After the first application completes drawing the third buffer, the first application stores the third buffer in the first storage space, and the first application sends a second synthesis instruction to the SurfaceFlinger; When the fourth buffer is stored in the second storage space, the SurfaceFlinger does not synthesize the third buffer in response to the second synthesis instruction, and the Vsync thread stops sending the next Vsync signal to the SurfaceFlinger.

6. The method according to claim 3, characterized in that Before the first application receives the first Vsync signal, the method further includes: The first application receives a second Vsync signal; In response to the second Vsync signal, the first application draws the third buffer; After the first application completes drawing the third buffer, the first application stores the third buffer in the first storage space. When the fourth buffer is stored in the second storage space, the first application stops sending the second synthesis instruction to the SurfaceFlinger, and the Vsync thread stops sending the next Vsync signal to the SurfaceFlinger. The next Vsync signal of the second Vsync signal is the first Vsync signal.

7. The method according to claim 5 or 6, characterized in that Before the first application receives the second Vsync signal, the method further includes: The first application receives a third Vsync signal, where the third Vsync signal is a previous signal of the second Vsync signal; In response to the third Vsync signal, the first application draws the fourth buffer; After the first application completes drawing the fourth buffer, the first application stores the fourth buffer in the first storage space, and the first application sends a third synthesis instruction to the SurfaceFlinger; In response to the third synthesis instruction, the SurfaceFlinger retrieves the fourth buffer from the first storage space; The SurfaceFlinger synthesizes the fourth buffer; After the SurfaceFlinger completes synthesis of the fourth buffer, the SurfaceFlinger stores the fourth buffer in the second storage space, and the time when the SurfaceFlinger completes synthesis of the fourth buffer is later than the time corresponding to the second Vsync signal and earlier than the time corresponding to the first Vsync signal.

8. The method according to claim 1, characterized in that The first application is a stylus application.

9. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to call a computer program in a memory to execute the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method according to any one of claims 1 to 8.

11. A chip, characterized in that: The chip comprises a processor, and the processor is used to call a computer program in a memory to execute the method according to any one of claims 1-8.

Images