Display method, electronic device, and computer-readable storage medium

Abstract

The present application provides a display method, an electronic device, and a computer-readable storage medium. In this method, an electronic device can generate an input event based on the user's operation on the touch screen and then send the input event to an application without waiting for synchronization of a vertical synchronization signal. In this way, the electronic device can reduce the delay between the input management service generating the input event and the application receiving the input event. The application can process the input event more quickly to generate a rendering frame, and the electronic device can then display the rendering frame on the touch screen more quickly, thereby improving the hand tracking performance of the electronic device.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a display method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Electronic devices can receive user actions on a touchscreen and, in response to these actions, change the content displayed on the touchscreen to provide feedback on the user's actions. Sometimes, these changes can take too long, and feedback may not be provided on the user interface until long after the user has completed their actions, impacting the user experience. Summary of the Invention

[0003] The present application provides a display method, electronic device, and computer-readable storage medium. After an electronic device generates an input event based on a user operation, the electronic device can send the input event directly to an application without waiting for a vertical synchronization signal. In this method, the application does not need to wait for an input event for a long time and can process the input event as soon as possible. This effectively avoids the situation where the application reaches the rendering time but the input event has not yet been processed, thereby improving the hand tracking performance of the electronic device in the application.

[0004] In a first aspect, the present application provides a display method, which is applied to an electronic device, wherein the electronic device is installed with a first application and a second application, wherein the first application is a game application and the second application is a non-game application, and the method includes: receiving a first operation acting on the first application, and generating a first event based on the first operation; in response to generating the first event, providing the first event to the first application; generating a first layer based on the first event through the first application; displaying a first user interface based on the first layer; receiving a second operation acting on the second application, and generating a second event based on the second operation; providing a second event to the second application in response to a vertical synchronization VSync-app signal; generating a second layer based on the second event through the second application; and displaying a second user interface based on the second layer.

[0005] The first and second events are input events generated based on user operations. The electronic device can determine how to handle the input events based on the type of application. If the application is a gaming application, the electronic device can directly provide the generated input event to the gaming application. If the application is a non-gaming application, the electronic device can wait for the VSync-app signal to arrive before providing the input event to the application.

[0006] In combination with the first aspect, in some embodiments, the first application may include a logic thread and a rendering thread, the logic thread is used to generate first rendering data based on the first event, and the rendering thread is used to generate a first layer based on the first rendering data.

[0007] In conjunction with the first aspect, in some embodiments, the electronic device may include a touch screen manager and an input management service. The touch screen manager is configured to determine that a first application is an application that does not rely on VSync-app signals, and the input management service is configured to provide a first event to the first application in response to generating a first event. The touch screen manager is further configured to determine that a second application is an application that relies on VSync-app signals, and the input management service is further configured to provide a second event to the second application in response to a vertical synchronization (VSync-app) signal.

[0008] It is understandable that non-gaming applications often rely on VSync-app signals to trigger drawing, rendering, and generating rendering frames. Therefore, when electronic devices provide input events to non-gaming applications, they need to wait for VSync-app signals for synchronization. However, for gaming applications, the timing of generating rendering frames is controlled by their own engines. At this time, it is unnecessary for electronic devices to wait for the VSync-app signal to arrive before sending input events to gaming applications. Electronic devices can provide generated input events directly to gaming applications without having to wait for the VSync-app signal. This allows gaming applications to process input events as early as possible and are more likely to complete input event processing when the rendering timing arrives, allowing them to generate rendering frames based on the latest input events, thereby improving hand tracking.

[0009] In combination with the first aspect, in some embodiments, the first event is generated between a first VSync-app signal and a second VSync-app signal and provided to the first application, where the first VSync-app signal and the second VSync-app signal are two adjacent vertical synchronization signals.

[0010] It is understandable that since the electronic device does not wait for the arrival of the VSync-app signal when providing input events to the game application, the timing of sending the first event to the first application after it is generated between two adjacent VSync-app signals is also between the two adjacent VSync-app signals.

[0011] In combination with the first aspect, in some embodiments, before providing the first event to the first application, the method further includes: stopping sending the VSync-app signal; before providing the second event to the second application, the method further includes: resending the VSync-app signal.

[0012] In combination with the first aspect, in some embodiments, the first user interface does not include a layer for non-game applications.

[0013] The electronic device will traverse the layers contained in the first user interface. When the first user interface does not contain layers of non-gaming applications, the electronic device can directly stop sending VSync-app signals, which can reduce power consumption. When the electronic device determines that the second user interface contains layers of non-gaming applications, since non-gaming applications need to rely on VSync-app signals to trigger drawing, rendering and other processes, the electronic device needs to re-enable the sending of VSync-app signals so that the non-gaming applications can stably output rendering frames. Among them, the first user interface does not contain layers of non-gaming applications, which means that the first user interface displayed on the touch screen of the electronic device only contains the user interface of the gaming application; conversely, the second user interface contains both layers of gaming applications and layers of non-gaming applications, which means that the second user interface displayed on the touch screen contains user interfaces of both gaming applications and non-gaming applications, wherein the user interface of the gaming application and the user interface of the non-gaming application can be displayed in two split-screen user interfaces, or one user interface is displayed globally and the other user interface is displayed simultaneously on the touch screen in a windowed manner.

[0014] In combination with the first aspect, in some embodiments, the second user interface further includes a third layer of the first application, wherein the frame rate of the first application is a first frame rate, the frame rate of the second application is a second frame rate, and the first frame rate is greater than the second frame rate.

[0015] by Figure 8 Taking the embodiment shown as an example, the frame rate of the gaming application is twice that of the non-gaming application, wherein the gaming application is controlled by its own engine to output rendered frames, while the non-gaming application mainly relies on the VSync-app signal to output rendered frames. The second user interface can be image 1, the third layer can be layer 1, and the second layer can be layer 1'. When the electronic device determines that the second user interface contains layers of non-gaming applications in addition to the layers of gaming applications, the electronic device can control the frame rate of the non-gaming application by controlling the frequency of VSync-app, so that the frame rate of the non-gaming application is lower than the frame rate of the gaming application. The electronic device may include a surface synthesizer, which can be used to determine the layers (including the second layer and the third layer) contained in the second user interface and adjust the sending frequency of the VSync-app signal to the second frame rate.

[0016] When gaming applications output rendered frames at a high frame rate, the load on the electronic device is high. However, non-gaming applications do not need to refresh the display screen at the same high frame rate as gaming applications. Electronic devices can control the frequency of VSync-app to make non-gaming applications output rendered frames at a lower frame rate. This ensures that users can use both gaming and non-gaming applications normally, and also reduces the power consumption of the electronic device.

[0017] In combination with the first aspect, in some embodiments, before generating the first event according to the first operation, the method further includes: determining that the sampling rate of the touch screen is a multiple of the refresh rate, and determining that the first application is in the first list.

[0018] Optionally, before generating the first event according to the first operation, the method further includes: determining that the sampling rate of the touch screen is a multiple of the refresh rate, or determining that the first application is in the first list.

[0019] In combination with the first aspect, in some embodiments, before generating the second event according to the second operation, the method further includes: determining that the sampling rate of the touch screen is not a multiple of the refresh rate, and / or determining that the second application is not in the first list.

[0020] Among them, the electronic device can store the identifiers of one or more applications in the first list. Among them, the identifiers of applications that do not rely on the VSync-app signal may be stored in the first list. Not relying on the VSync-app signal here means that the application does not need to rely on VSync-app for synchronization when drawing, rendering, and generating rendering frames. The electronic device can ensure this type of application by sending input events to the application even without waiting for the VSync-app signal. Therefore, the electronic device can first determine whether the identifier of the application is in the first list, and then determine whether the application needs to rely on the VSync-app signal. Optionally, the identifier of the above application may be the package name of the application.

[0021] When the touchscreen sampling rate is a multiple of the refresh rate, the electronic device generates a consistent number of input events between displaying two adjacent image frames, meaning the timing of input event generation is more uniform. In this case, the application generates rendered frames based on the same number of input events each time. Even if the electronic device does not wait for the VSync-app signal when providing input events to the application, the application can still output stable rendered frames. However, if the touchscreen sampling rate is not a multiple of the refresh rate, the electronic device may generate an inconsistent number of input events between displaying two adjacent image frames. For example, the electronic device may generate two input events in one VSync-app cycle and one input event in the next. This means that the application generates rendered frames based on different numbers of input events each time. Without waiting for the VSync-app signal, the number of input events used by the application to generate each rendered frame is difficult to control, which can easily lead to uneven display. Therefore, before determining whether the touchscreen sampling rate is a multiple of the refresh rate to send input events to the application without waiting for the VSync-app signal, the electronic device can first determine whether the touchscreen sampling rate is a multiple of the refresh rate. This ensures that the application can output stable rendered frames.

[0022] The electronic device may provide input events to the application without relying on the VSync-app signal only if the touch screen sampling rate is a multiple of the refresh rate and / or the application's identifier is in the first list. This ensures that, even without relying on the VSync-app signal to provide input events to the application, the application can still output stable rendered frames, thereby ensuring a stable display of the user interface and a consistent viewing experience for the user.

[0023] In a second aspect, the present application provides a display method, which is applied to an electronic device, wherein the electronic device is installed with a first application, and the method includes: receiving a first operation acting on the first application, and generating a first event based on the first operation; providing the first event to the first application in response to generating the first event; generating a first layer based on the first event through the first application; and displaying a first user interface based on the first layer.

[0024] In combination with the second aspect, in some embodiments, the first event is generated between a first VSync-app signal and a second VSync-app signal and provided to the first application, where the first VSync-app signal and the second VSync-app signal are two adjacent vertical synchronization signals.

[0025] In combination with the second aspect, in some embodiments, only the first layer is included in the first user interface, and the method further includes: stopping sending the VSync-app signal.

[0026] In combination with the second aspect, in some embodiments, before generating the first event according to the first operation, the method further includes: determining that the sampling rate of the touch screen is a multiple of the refresh rate, and determining that the first application is in the first list.

[0027] In combination with the second aspect, in some embodiments, the electronic device is also installed with a second application, and the method also includes: receiving a second operation acting on the second application, and generating a second event based on the second operation; providing a second event to the second application in response to a vertical synchronization VSync-app signal; generating a second layer based on the second event through the second application; and displaying a second user interface based on the second layer.

[0028] In combination with the second aspect, in some embodiments, before providing the second event to the second application, the method further includes: resending the VSync-app signal.

[0029] In combination with the second aspect, in some embodiments, the second user interface further includes a third layer of the first application, wherein the frame rate of the first application is a first frame rate, the frame rate of the second application is a second frame rate, and the first frame rate is greater than the second frame rate.

[0030] In combination with the second aspect, in some embodiments, the electronic device includes a surface compositor, which is used to determine that the second user interface includes a second layer and a third layer; the surface compositor is also used to adjust the sending frequency of the VSync-app signal to a second frame rate.

[0031] In combination with the second aspect, in some embodiments, before generating the second event according to the second operation, the method further includes: determining that the sampling rate of the touch screen is not a multiple of the refresh rate, and / or determining that the second application is not in the first list.

[0032] In combination with the second aspect, in some embodiments, the electronic device includes a touch screen manager and an input management service, the touch screen manager is used to determine that the first application is an application that does not rely on the VSync-app signal, and the input management service is used to provide a first event to the first application in response to generating a first event.

[0033] In conjunction with the second aspect, in some embodiments, the input management service is further used to determine that the second application is an application that relies on the VSync-app signal, and to provide a second event to the second application in response to the vertical synchronization VSync-app signal.

[0034] In combination with the second aspect, in some embodiments, the first application includes a logic thread and a rendering thread, the logic thread is used to generate first rendering data based on the first event, and the rendering thread is used to generate a first layer based on the first rendering data.

[0035] In conjunction with the second aspect, in some embodiments, the first application is a game application.

[0036] In conjunction with the second aspect, in some embodiments, the second application is a non-game application.

[0037] In a third aspect, the present application provides an electronic device comprising a touch screen, a memory, and a processor coupled to the memory; the touch screen is used to receive user operations and display an interface, the memory stores a computer program, and when the processor executes the above-mentioned computer program, the electronic device implements the method described in any one of the above-mentioned first or second aspects.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program or computer instructions, and the aforementioned computer program or computer instructions are executed by a processor to implement the method described in any one of the first or second aspects above.

[0039] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is executed by a processor, the method described in any one of the first aspect or the second aspect above will be implemented.

[0040] In a sixth aspect, an embodiment of the present application provides a chip, which includes a processor and a memory, wherein the memory is used to store computer programs or computer instructions, and the processor is used to execute the computer programs or computer instructions stored in the memory, so that the chip executes the method described in any one of the first or second aspects above.

[0041] The solutions provided in the third to sixth aspects above are used to implement or cooperate with the corresponding methods provided in the first or second aspects above, and therefore can achieve the same or corresponding beneficial effects as the corresponding methods in the first or second aspects, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 1 is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application;

[0043] Figure 2 is a software structure block diagram of the electronic device provided in an embodiment of the present application;

[0044] Figure 3A-Figure 3B This is a user interface of some electronic devices in a gaming scenario provided by the embodiments of the present application;

[0045] Figure 4 This is a flow chart of an electronic device displaying a game screen according to an embodiment of the present application;

[0046] Figure 5 This is a flow chart of another electronic device displaying a game screen provided by an embodiment of the present application;

[0047] Figure 6 This is a flow chart of an electronic device controlling the VSync-app signal to be turned on or off, provided by an embodiment of the present application;

[0048] Figure 7A-7B It is a user interface displayed by some electronic devices provided in the embodiments of the present application;

[0049] Figure 8 This is a flow chart of an electronic device displaying a user interface at different frame rates in multiple windows provided by an embodiment of the present application;

[0050] Figure 9 This is a flow chart of a display method provided by an embodiment of the present application;

[0051] Figure 10 is a structural schematic diagram of a display device provided in an embodiment of the present application;

[0052] Figure 11 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The terms used in the following examples of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular expressions "a," "an," "said," "above," "the," and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and encompasses any or all possible combinations of one or more of the listed items.

[0054] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0055] Here, the hardware structure of the electronic device 100 provided in the embodiment of the present application is first introduced.

[0056] Figure 1 1 is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.

[0057] The following describes the embodiment in detail by taking the electronic device 100 as an example. It should be understood that the electronic device 100 may have more Figure 1 More or fewer components may be shown, two or more components may be combined, or the components may be arranged differently. Figure 1 The various components shown in the drawings may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0058] The electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a camera 193, a display screen 194, etc. The sensor module 180 may include a pressure sensor 180A, a touch sensor 180B, etc.

[0059] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0060] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0061] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0062] The charging management module 140 is configured to receive charging input from a charger, which may be a wireless charger or a wired charger.

[0063] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, etc.

[0064] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0065] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0066] The camera 193 is used to capture still images or videos. The electronic device 100 may include one or N cameras 193, where N is a positive integer greater than one.

[0067] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0068] Video codecs are used to compress or decompress digital videos. Electronic device 100 may support one or more video codecs. In this way, electronic device 100 can play or record videos in multiple encoding formats.

[0069] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0070] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0071] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store an operating system, applications required for at least one function (such as face recognition function, fingerprint recognition function, mobile payment function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as face information template data, fingerprint information template, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0072] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0073] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0074] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0075] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0076] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals.

[0077] The pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a device comprising at least two parallel plates with conductive material. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to the display screen 194, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions.

[0078] Touch sensor 180B, also known as a "touch panel," can be disposed on display screen 194. Touch sensor 180B and display screen 194 form a touch screen, also known as a touch screen. Touch sensor 180B is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via display screen 194. In other embodiments, touch sensor 180B can also be disposed on the surface of electronic device 100, at a location different from that of display screen 194.

[0079] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present application, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device.

[0080] Figure 2 This is a software structure block diagram of the electronic device provided in the embodiment of the present application.

[0081] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers: application layer, application framework layer, native library layer, and kernel layer, from top to bottom.

[0082] The application layer can include a series of application packages. Figure 2 As shown, the application layer may include applications such as games and instant messaging.

[0083] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions. The application framework layer can include game scene optimization services, input management services, and so on.

[0084] The game scene optimization service is used to schedule the system resources of electronic devices to optimize the gaming experience.

[0085] The input management service can perform resampling and other processing on touch events generated by the touch driver, and then distribute the processed input events to applications, which can then process the input events based on their own logic to generate new user interfaces. In some embodiments, the input management service can receive control information from the touch screen manager and, based on this information, determine whether to wait for a vertical synchronization signal before sending input events to applications. For more information on vertical synchronization signals, please refer to subsequent embodiments and will not be elaborated on here.

[0086] The native library can include multiple functional modules, such as graphics processing library, surface compositor (surfaceflinger), touch screen manager, etc.

[0087] The graphics processing library is used to implement graphics drawing, image rendering, synthesis, and layer processing.

[0088] The surface compositor is used to provide a fusion of two-dimensional (2D) and three-dimensional (3D) layers for multiple applications, thereby generating a user interface to be displayed. The user interface of an application can be called a layer. The user interface to be displayed on the touch screen may include the user interfaces of multiple applications. The surface compositor synthesizes the layers by combining the user interfaces of different applications into the same user interface according to certain rules, so as to facilitate subsequent display on the touch screen. In some embodiments, the surface compositor can control the turning on or off of the vertical synchronization signal based on the type and number of layers contained in the user interface to be displayed. For an introduction to the vertical synchronization signal, please refer to the subsequent embodiments and will not be expanded here.

[0089] The touchscreen manager can be used to obtain or adjust touchscreen parameters and control the touchscreen display. In some embodiments, the touchscreen manager can determine, based on an application identifier, whether the application's rendering frame generation requires a vertical synchronization signal. For more information on vertical synchronization signals, please refer to subsequent embodiments and will not be elaborated on here.

[0090] The kernel layer is the layer between hardware and software. It includes at least the GPU driver, touch driver, and display driver. The kernel layer receives instructions from the software layer and controls the hardware layer to perform corresponding tasks.

[0091] The GPU driver can be used to drive the GPU to further render the image drawn by the application program, thereby generating a user interface of the application program (ie, a layer of the application program).

[0092] The touch driver can receive the user's operation on the touch screen and process the touch operation into a touch event (including touch coordinates, touch operation timestamp and other information).

[0093] The display driver is used to push the user interface to be displayed to the frame buffer (FB) for storage. The frame buffer is a storage space in the electronic device 100, which can be located in the video memory or the internal memory. The content of the frame buffer corresponds to the interface display on the touch screen, and it can be simply understood as a cache corresponding to the content displayed on the touch screen. In other words, modifying the content in the frame buffer is to modify the picture displayed on the touch screen, and then the touch screen can display the user interface to be displayed. The touch screen is composed of a touch sensor 180B and a display screen 194.

[0094] It is understandable that Figure 2 The illustrated software architecture does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include Figure 2 More or fewer modules may be shown, some modules may be combined, some modules may be split, and so on.

[0095] Electronic devices typically modify some or all of their user interface content based on user actions to achieve human-computer interaction. The shorter the delay between an electronic device receiving a user action and changing its displayed content, the stronger its chirality. When an electronic device has poor chirality, it takes a long time for the user to respond to the user's action. This significantly impacts the user experience in scenarios where chirality is critical.

[0096] Here we introduce electronic devices and scenarios with poor chirality.

[0097] Figure 3A-Figure 3B This is the user interface of some electronic devices in a gaming scenario provided by the embodiments of the present application.

[0098] like Figure 3AAs shown, the electronic device 100 can display a game interface 301, which can include a wheel control 302 and a character sticker 303, wherein the wheel control 302 is used to control the movement of the character sticker 303. At time t1, the electronic device 100 can receive an operation in which the user presses the wheel control 302 and slides it to the right for a time A and then releases it. At time t2, in response to the above operation, the electronic device 100 controls the character sticker 303 to move to the right at a preset speed for a time A to reach the position shown in FIG. Figure 3B When the gap between time t1 and time t2 is large, the user will feel that the character map 303 cannot move in time according to the user's operation, which means that the hand tracking performance of the electronic device is poor.

[0099] To address this issue, embodiments of the present application provide a display method, an electronic device, and a computer-readable storage medium. In this method, after receiving an operation performed by a user on a touch screen, the electronic device can directly send an input event generated based on the user's operation to an application without having to wait for a vertical synchronization signal. This allows the application to more quickly obtain input events and generate rendering frames based on the input events. The electronic device can also more quickly display the rendering frames on the touch screen to provide feedback on the user's operation, thereby improving the hand-tracking performance of the electronic device.

[0100] Gaming is a scenario that places high demands on the chirality of electronic devices. When playing games, users often require that the electronic device provide timely feedback on the touch screen based on their operations. For ease of description and better understanding, the subsequent embodiments of this application use gaming scenarios as an example to introduce the display method provided by this application. This display method is not limited to gaming scenarios and can also be applied to other scenarios. This application does not limit the application scenarios of the above-mentioned display method in electronic devices.

[0101] When electronic devices display user interfaces, they usually need to coordinate with vertical synchronization (VSync) signals. VSync signals can include VSync-app signals, VSync-sf signals, and HW-VSync signals. The VSync-app signal is used to trigger the input management service to send input events to the application and notify the application to start drawing and generating a new frame of image. The VSync-sf signal is used to trigger the surface synthesizer to synthesize the user interface to be displayed based on the layers of one or more applications. The HW-VSync signal is used to trigger the display driver to display the user interface to be displayed. Among them, the VSync-app signal and the VSync-sf signal can be sent by the electronic device at preset time intervals.

[0102] The following describes in detail the process of displaying the user interface of the electronic device.

[0103] When the touch sensor receives a touch, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer's touch driver sends this raw input event to the input management service. The input management service performs resampling and other processing on the raw input event, and then sends the processed input event to the game application when the VSync-app signal arrives.

[0104] After receiving the input event, the game application can perform logical updates based on the input event, which may include updating the physical information in the game application (such as speed, position, collision handling), object state changes, etc. (for example, Figure 3A The position coordinates of the character map 303 after movement are shown). The game application can then draw and render the user interface to be displayed based on the result of the above-mentioned logical update (hereinafter referred to as rendering data) to generate a rendering frame. Among them, the threads that perform logical updates and generate rendering frames in the game application can be independent of each other. The thread responsible for logical updates is called a logic thread, and the thread responsible for generating rendering frames is called a rendering thread. In addition, the game application does not need to rely on the VSync-app signal to trigger the drawing and rendering of the layer. Instead, the game engine controls the frequency at which the rendering thread generates rendering frames (while most non-game applications need to wait for VSync-app to start drawing and rendering a frame of layer). Although the game application does not generate rendering frames in response to the VSync-app signal, the time interval between any two adjacent rendering frames generated by the game application is consistent. Among them, the time interval between the game application generating two adjacent rendering frames can be equal to the time interval between the touch screen refreshing two adjacent frames of the user interface, that is, the frequency at which the game application generates rendering frames can be consistent with the refresh rate of the touch screen, which can ensure that each rendering frame generated by the game application can be finally displayed. Optionally, the electronic device may include a synchronization signal that triggers the game application to generate a rendered frame. The timing of each transmission of the synchronization signal may differ from the timing of the VSync-app signal. After receiving an input event, the game application's logic thread is responsible for updating the logic, and then sends the updated logic results to the rendering thread for drawing and rendering. Each rendered frame generated by the application is also a layer of the application. After generating the layer, the application can send the layer to the surface compositor.

[0105] After receiving the VSync-sf signal, the surface compositor can composite the layers of different applications into the same screen, obtaining the user interface to be displayed on the touch screen, also known as the to-be-displayed user interface (or image to be displayed). After generating the to-be-displayed user image, Surfaceflinger can send the to-be-displayed image to the display driver.

[0106] The display driver can send the image to be displayed after receiving HW-VSync, and then the touch screen can display the image to be displayed.

[0107] by Figure 3A-Figure 3B Taking the embodiment shown as an example, after the electronic device receives the user's operation of pressing the wheel control 302 and sliding it to the right, the input management service will generate a press and right slide input event based on the above operation. Then, the input management event will send the generated input event to the game application after receiving the VSync-app signal. The game application can perform logic updates based on the above press and right slide input events. For example, the game application determines that after the above operation, the character map 303 should be changed from Figure 3A Move the position shown to Figure 3B At the position shown, the game application will complete the rendering of a frame of the user interface, which is called a rendering frame. The character texture 303 in the rendering frame is located at Figure 3B The game application can then send the generated rendering frame to the surfaceflinger. After receiving the VSync-sf signal, the surfaceflinger synthesizes the layers to generate a frame to be displayed. After receiving the HW-VSync signal, the display driver sends the frame to be displayed. Finally, the touch screen displays the frame to be displayed. Figure 3B The user interface shown.

[0108] The time interval between two adjacent VSync-app signals emitted by the electronic device, the time interval between two adjacent VSync-sf signals emitted by the electronic device, and the time interval between two adjacent HW-VSync signals emitted by the electronic device may be the same or different. Furthermore, the timing at which the electronic device emits the VSync-app signal, VSync-sf signal, and HW-VSync signal may be the same or different. In some embodiments, the timing at which the electronic device emits the VSync-app signal, VSync-sf signal, and HW-VSync signal may be offset, but this is not a limitation in the embodiments of the present application. For ease of description and better understanding, the subsequent embodiments of the present application will describe the display method provided by the embodiments of the present application, assuming that the electronic device emits the VSync-app signal, VSync-sf signal, and HW-VSync signal at the same time each time. Display methods when the electronic device emits the VSync-app signal, VSync-sf signal, and HW-VSync signal in other ways can refer to the method in which the electronic device emits the VSync-app signal, VSync-sf signal, and HW-VSync signal at the same time each time, and this embodiment of the present application will not be further described.

[0109] Here we introduce the scenario of electronic devices displaying game screens.

[0110] Figure 4 This is a flow chart of the electronic device displaying the game screen provided in an embodiment of the present application.

[0111] like Figure 4 As shown, the electronic device sends the first VSync-app signal ( Figure 4 During the time between the second VSync-app signal and the input management service generates input event 1 ( Figure 4 1 in the figure).

[0112] After the electronic device sends the second VSync-app signal, the input management service can send input event 1 to the gaming application in response to the VSync-app signal. The gaming application's logic thread can generate rendering data 1 based on event 1. The rendering thread then obtains the rendering data 1 from the logic thread and generates layer 1 based on rendering data 1. Between the second and third VSync-app signals, the input management service also generates input event 2.

[0113] In response to the third VSync-app signal, the input management service can send input event 2 to the game application. The logic thread of the game application can process input event 2 and generate rendering data 2. The rendering thread then generates layer 2 based on rendering data 2. The surface compositor can respond to the VSync-sf signal ( Figure 4 (not shown) composites image 1 based on layer 1. Between the third VSync-app signal and the fourth VSync-app signal, the input management service also generates input event 3.

[0114] In response to the fourth VSync-app signal, the input management service can send input event 3 to the game application. The method for the game application to process input event 3 can refer to the above-mentioned method for processing input event 1 and input event 2, which will not be repeated here. The surface compositor can respond to the VSync-sf signal ( Figure 4 (not shown) based on layer 2 to synthesize image 2. The display driver can be based on the HW-VSync signal ( Figure 4 (not shown) the synthesized image 1 is sent for display, and then the touch screen displays the image 1.

[0115] It should be noted that the game application will generate rendering frames based on its own logic even if it does not receive any input events. Figure 4 The figure only shows the process of the electronic device displaying the user interface after the user operation causes the user interface of the game application to be redrawn. Figure 4The frequency of input events generated by the input management service is only an example. The input management service can generate more than 1000 input events between each two adjacent VSync-apps. Figure 4 More or fewer input events are shown, and the number of input events generated in two adjacent VSync-app cycles can be the same or different, which is not limited in this embodiment of the present application.

[0116] Because the logic thread generates rendering data and the rendering thread generates rendered frames independently in game applications, the rendering thread doesn't wait for the logic thread to finish processing input events before rendering. Instead, it renders based on its preset cycle. If the logic thread can't finish processing input events before the rendering thread starts generating frames for the current cycle, the rendering thread will have to synchronize with the input events in the next frame layer, which can lead to poor hand tracking issues in electronic devices.

[0117] by Figure 4 For example, if the logic thread of the game application is still generating rendering data 1, and the rendering thread has reached the time to start generating a new rendering frame, the rendering thread will not be able to generate layer 1 based on the latest rendering data (i.e., rendering data 1). In this way, layer 1 cannot feedback the screen changes caused by input event 1. In this case, the rendering thread can only process rendering data 1 when generating the subsequent rendering frame (layer 2). As a result, the user interface displayed by the electronic device cannot provide timely feedback based on user operations, which means that the electronic device has poor hand tracking.

[0118] For most non-gaming applications, they need to rely on the VSync-app signal to trigger drawing and rendering to generate rendering frames. Therefore, the input management service needs to wait for the VSync-app signal for synchronization when sending input events to the application. For gaming applications, such applications do not rely on the VSync-app signal to trigger the generation of rendering frames. At this time, it is unnecessary for the input management service to wait for the VSync-app signal to arrive before sending input events to the gaming application. In order to improve the hand tracking performance of electronic devices, electronic devices can reduce the delay between the input management service generating input events and the gaming application receiving input events. The earlier the gaming application receives the input event, the sooner the gaming application's logic thread can process the input event, so that the rendering thread can receive the rendering data earlier and generate rendering frames based on the rendering data.

[0119] In some embodiments, the input management service in the electronic device can send the input event to the gaming application immediately after the input event is generated without having to wait for the VSync-app signal.

[0120] Figure 5 This is a flow chart of another electronic device displaying a game screen provided in an embodiment of the present application.

[0121] like Figure 5 As shown, after the input management service generates an input event, it no longer waits for the VSync-app signal, but sends it directly to the game application. Between VSync-app1 and VSync-2, the input management service generates input event 1. After the input management service generates input event 1, it sends it to the game application immediately without waiting for the arrival of VSync-app2. In this way, the game application can receive input events faster, and then the logic thread can generate rendering data 1 based on input event 1. The rendering thread can receive rendering data faster, and then generate layer 1 based on the rendering data. After receiving layer 1, the surface compositor can synthesize the image 1 to be displayed (including layer 1 generated by the game application) after receiving the next VSync-sf signal, and then the display driver sends the frame to be displayed, and the touch screen finally displays it.

[0122] Similarly, the input management service may also directly send other input events generated by the input management service to the game application. The method for the electronic device to process other input events may refer to the method for processing input event 1 above, which will not be repeated here.

[0123] It should be noted that Figure 5 This only demonstrates one of the situations in which a game application generates a rendered frame. The timing for a game application to generate a rendered frame based on input events in each VSync-app cycle may be earlier than Figure 5 The timing shown may also be later than Figure 5 The timing shown is not limited in the embodiments of the present application.

[0124] It can be seen that since the input management event no longer needs to wait for the VSync-app signal when sending the input event, the game application can receive the input event earlier, thus reducing the delay of the input event from the input management service to the game application, thereby avoiding the rendering thread's inability to generate rendering frames based on new input events due to the logical thread's inability to process the input event in time. Under this method, the game application's logical thread can process the input event earlier and then hand it over to the rendering thread to generate the rendering frame. The image frame corresponding to each input event can be displayed on the touch screen faster. In this way, the user can see the feedback made by the electronic device based on the user's operation on the touch screen more quickly, thereby improving the user's gaming experience.

[0125] Gaming applications can generate rendered frames at a certain frequency based on their own engines. Therefore, even if gaming applications do not rely on VSync-app signals, they can still output rendered frames relatively stably. However, some other applications require VSync-app signals for synchronization when generating rendered frames. In some embodiments, to ensure that applications can generate more stable rendered frames, electronic devices can determine whether to enable or disable the VSync-app signal based on the application type.

[0126] Figure 6 This is a flowchart of an electronic device controlling the VSync-app signal to be turned on or off, provided by an embodiment of the present application.

[0127] like Figure 6 As shown, the method for an electronic device to control the VSync-app signal to be turned on or off may include the following steps:

[0128] S601: Application A starts.

[0129] S602: Application A sends message A to the input management service. Message A indicates that application A has been started.

[0130] S603: The input management service sends the identifier of application A to the touch screen manager.

[0131] After application A is started, it may send message A to the input management service, wherein the input management service may send the identifier of application A to the touch screen manager based on message A. The identifier of application A may be the package name of application A. Optionally, application A may be a game application.

[0132] Optionally, the electronic device may add the identifiers of all game applications stored in the electronic device to the enabled list, or the electronic device may only store the identifiers of some game applications stored in the electronic device to the enabled list. The identifiers of the application programs stored in the enabled list may be added in response to user operations or may be added by the electronic device based on preset rules, and this embodiment of the application is not limited thereto.

[0133] The enabled list is not limited to gaming applications and may also include identifiers of other types of applications. For example, the enabled list may also include identifiers of other applications that do not rely on VSync-app signals to trigger drawing or rendering of user interfaces, and this embodiment of the application does not limit this.

[0134] S604: The touch screen manager determines, based on the identifier of application A, that application A is in the enabled list.

[0135] The touch screen manager can query whether the enabled list contains the identifier of application A. If the identifier of the application exists in the enabled list, it indicates that the application does not need to rely on the VSync-app signal for synchronization.

[0136] When the application's identifier does not exist in the enabled list, the touch screen manager may determine that the application needs to rely on the VSync-app signal. The application's reliance on the VSync-app signal means that the application needs to rely on the VSync-app signal to trigger drawing and rendering of the user interface.

[0137] For ease of description and better understanding, in the subsequent embodiments, applications that are identified as being in the enabled list will be referred to as in-list applications, and applications that are not identified as being in the enabled list will be referred to as non-list applications. In-list applications can generate rendering frames according to a preset period without relying on the VSync-app signal, while non-list applications need to wait for the VSync-app signal to arrive before starting to draw and render to generate rendering frames. When the electronic device determines that an application does not need to rely on the VSync-app signal, it may not send the VSync-app signal to the application, or it may still send the VSync-app signal to the application. The embodiments of the present application do not limit this.

[0138] S605: The touch screen manager queries the surface compositor for the touch screen refresh rate.

[0139] S606: The touch screen manager queries the touch screen driver for the touch screen sampling rate.

[0140] S607: The touch screen manager determines that the refresh rate of the touch screen is a multiple of the sampling rate.

[0141] When the sampling rate of the touch screen is not a multiple of the refresh rate, the number of input events generated by the input management service within the time interval of each two adjacent VSync-app signals may be different. For example, the input management service generates 1 input event between the Kth VSync-app signal and the K+1th VSync-app signal, and generates 2 input events between the K+1th VSync-app signal and the K+2th VSync-app signal, where K is an arbitrary positive integer. In this way, if the application does not rely on the VSync-app signal, the input events it relies on each time it generates a rendering frame may be uncontrollable, which may easily cause jitter in the display and affect the user's viewing experience. The refresh rate is used to indicate the number of user interfaces displayed per second on the touch screen of an electronic device.

[0142] In some embodiments, after the touchscreen manager determines that the application's identifier is in the enabled list, it also needs to further determine, based on the touchscreen's sampling rate and refresh rate, whether the input management service needs to rely on the VSync-app signal to send input events to application A under the current touchscreen state. Only when the application's identifier is in the application solution list and the touchscreen's sampling rate is a multiple of the refresh rate does the touchscreen manager determine that the input management service can send input events to application A without relying on the VSync-app signal, thereby improving the chirality of the electronic device running application A. This prevents the electronic device from blindly improving chirality while ignoring display stability, resulting in a reduced user experience.

[0143] S608: The touch screen manager sends control information A to the input management service. The control information A indicates that the application A does not need to rely on the VSync-app signal.

[0144] S609: The input management service determines, based on the control information A, that there is no need to wait for the VSync-app signal when sending an input event to the application A.

[0145] S610: The input management service sends control information B to the surface compositor, where the control information B is used to instruct the VSync-app signal to be turned off.

[0146] After receiving the control information A sent by the touch screen manager, the input management service can determine that it no longer needs to wait for the VSync-app signal when subsequently sending input events to the application.

[0147] Because periodic transmission of VSync-app signals by electronic devices consumes certain resources, in some embodiments, the input management service can send control information B to the surface compositor. After receiving control information B, the surface compositor can determine whether to disable the VSync-app signal based on the layers included in the user interface to be displayed. Optionally, control information B can also be sent by other modules in the electronic device, such as a touch screen manager, which is not limited in this embodiment of the present application.

[0148] S611: The surface compositor traverses the layers of the user interface to be displayed.

[0149] S612: The surface compositor determines, in the user interface to be displayed, only the layers of the applications included in the enabled list.

[0150] S613. The surface synthesizer turns off the VSync-app signal.

[0151] After receiving the control signal B, the surface compositor can traverse the layers included in the user interface to be displayed. When the surface compositor determines that only the layers of the application in the enable list are included, the VSync-app signal can be turned off (or the VSync-app signal remains turned off) to save resources. The reason why the layers of the user interface to be displayed include layers of multiple applications is because the user interfaces of multiple applications are compositely displayed on the touch screen. For example, the composite display may include, but is not limited to, split-screen display, windowed display, etc.

[0152] In some embodiments, the user interface to be displayed may include layers of applications outside the enabled list in addition to layers of applications in the enabled list. The surface compositor may keep the VSync-app signal enabled. The input management service does not need to wait for the VSync-app signal when sending input events to applications in the enabled list (e.g., application A), but still needs to wait for the VSync-app signal when sending input events to applications outside the enabled list.

[0153] When the user interfaces of multiple applications are displayed on the touch screen, the electronic device consumes a lot of power, and different applications require different frame rates. Take application A and application B as an example, where application A is an application in the list (such as a game application) and application B is a non-list application (such as an instant messaging application). When the user interface to be displayed on the touch screen contains layers of application A and application B, in order to ensure the user's gaming experience, application A may be displayed at a higher frame rate. Accordingly, the electronic device will also increase the refresh rate of the touch screen to ensure that each frame layer generated by application A can be displayed. At this time, if application B also generates layers at a higher frame rate, it is unnecessary for the user and this will also unnecessarily increase the power consumption of the electronic device. In some embodiments, the surface compositor can adjust the frequency of the VSync-app signal. For example, the surface compositor can reduce the frequency of the VSync-app signal to be lower than the frame rate of application A. In this way, application A still displays the user interface at a higher frame rate because it does not rely on the VSync-app signal, while application B relies on VSync-app to display the user interface at a lower frame rate, thereby reducing the power consumption of the electronic device while ensuring the user experience. The frame rate represents the number of user interfaces displayed per second by the application, that is, the number of rendered frames generated per second. The frame rate of an application can be different from the refresh rate of the electronic device, and the frame rates of different applications can be the same or different.

[0154] In some embodiments, steps S611-S613 may be executed in a loop. That is, the surface compositor traverses the layers of the user interface to be displayed when compositing each frame of the user interface. Furthermore, the surface compositor may control the VSync-app signal to be turned on or off when the layers of the user interface to be displayed change.

[0155] In some embodiments, the touchscreen manager may determine in step S604 that the application identifier is not in the enabled list, and / or in step S607 that the touchscreen refresh rate is not a multiple of the sampling rate. The touchscreen manager may then send control information C to the input management service, indicating that the application requires reliance on the VSync-app signal. The touchscreen manager may determine that reliance on the VSync-app signal is required when sending input events to the application. Furthermore, the touchscreen manager may send control information D to the surface compositor, instructing the surface compositor to enable the VSync-app signal. The surface compositor may then traverse the layers of the user interface to be displayed. If only layers of applications not in the enabled list are included, the surface compositor may change the VSync-app signal from off to on (or maintain the VSync-app signal in the enabled state). If layers of applications in the enabled list are also included in the layers of the user interface to be displayed, the surface compositor may, in addition to enabling the VSync-app signal, adjust the frequency of the VSync-app signal. In this way, the input management service does not need to wait for the VSync-app signal when sending input events to applications in the list, while sending input events to non-list applications is still triggered by VSync-app.

[0156] In some embodiments, steps S605 to S607 are optional, that is, the touch screen manager can determine whether to wait for the VSync-app signal when the input management service sends an input event to application A based solely on whether the identification of application A is in the enable list.

[0157] In some embodiments, step S604 is optional, that is, the touch screen manager can determine whether to wait for the VSync-app signal when the input management service sends an input event to application A based solely on whether the sampling rate of the touch screen is a multiple of the refresh rate.

[0158] When the user interface displayed on the touch screen includes both user interfaces of applications in the list and user interfaces of applications outside the list, the electronic device may adjust the frequency of the VSync-app signal to save power consumption.

[0159] The following describes some scenarios in which electronic devices display user interfaces provided by embodiments of the present application.

[0160] Figure 7A-7B It is a user interface displayed by some electronic devices provided in the embodiments of the present application.

[0161] like Figure 7A As shown, the electronic device displays a user interface 701 of a gaming application, where the gaming application is an application in the list. Since the user interface displayed by the electronic device only includes the user interface 701 of the gaming application, the electronic device can disable (or keep disabled) the VSync-app signal to improve the tracking performance of the gaming application, thereby improving the user's gaming experience.

[0162] like Figure 7B As shown, the electronic device 100 displays a user interface 701 of a gaming application and a user interface 702 of an instant messaging application, wherein the instant messaging application is a non-list application, which means that its frame rate is consistent with the frequency of the VSync-app signal. The entire user interface displayed on the touch screen (including the user interface 701 and the user interface 702) is synthesized by the surface synthesizer after receiving the corresponding layer of the user interface 701 sent by the gaming application and the corresponding layer of the user interface 702 sent by the instant messaging application. The electronic device can turn on (or keep on) the VSync-app and adjust the frequency of the VSync-app signal to control the power of the instant messaging application.

[0163] For example, to ensure the user's gaming experience, the game application refreshes the screen at frame rate A, where frame rate A is relatively high (for example, it can be 90Hz, 120Hz, etc.). The electronic device can adjust VSync-sf and HW-VSync to keep them consistent with frame rate A, ensuring that each layer generated by the game application can be displayed. The frame rate of the instant messaging application is called frame rate B. Since users do not have high requirements for the refresh rate of the instant messaging application, and the instant messaging application relies on the VSync-app signal to output rendered frames, the frequency of the VSync-app signal determines the frame rate B of the instant messaging application. The electronic device can set the VSync-app signal to be lower than frame rate A, so that the frame rate B of the instant messaging application is lower than frame rate A. The electronic device can not only ensure that the game application is displayed at a higher frame rate to ensure the user's viewing experience of the game screen, but also reduce the power consumption of the electronic device while ensuring that the frame rate of the instant messaging application can meet the user's usage requirements, thereby improving the user experience in a multi-application scenario.

[0164] Figure 8 This is a flow chart of an electronic device provided in an embodiment of the present application displaying a user interface at different frame rates in multiple windows.

[0165] For ease of description, we assume that the frame rate of listed applications is twice that of non-listed applications. For example, if the frame rate of listed applications is 120Hz and the frame rate of non-listed applications is 60Hz, the time interval between two consecutive rendered frames generated by listed applications is twice the time interval between two consecutive rendered frames generated by non-listed applications. Listed applications do not rely on the VSync-app signal when generating rendered frames, while non-listed applications only start generating rendered frames when they receive the VSync-app signal.

[0166] Between the first and second VSync-sf signals, the listed applications generate layer 1, and the non-list applications generate layer 1'. After receiving layer 1 and layer 1', the surface compositor can synthesize a frame of user interface to be displayed when the second VSync-sf signal arrives: image 1. This frame of user interface includes both the rendered frame generated by the listed applications (layer 1) and the rendered frame generated by the non-list applications (layer 1').

[0167] Between the second and third VSync-sf signals, the application in the list generates layer 2. At this time, since the non-list application is still waiting for VSync-app2, the non-list application does not generate a new rendering frame.

[0168] Between the third VSync-sf signal and the fourth VSync-sf signal, the application in the list generates layer 3, and the non-list application generates layer 2' in response to the second VSync-app signal. The surface compositor synthesizes the second frame user interface in response to the third VSync-sf signal: image 2, which consists of layer 2 and layer 1'. The display driver can generate layer 3 after receiving the HW-VSync signal ( Figure 8 (not shown) the user interface comprising layer 1+layer 1' synthesized by the surface synthesizer is sent for display, and then the touch screen can display the user interface.

[0169] After the fourth VSync-sf signal, the listed applications generate Layer 4, while non-listed applications are still waiting for VSync-app3. The surface compositor can then synthesize the user interface to be displayed: Image 3, which contains Layer 3 and Layer 2'. In response to the HW-VSync signal, the display driver can display the user interface synthesized by the surface compositor (including Layer 2 and Layer 1'), which is then displayed on the touchscreen.

[0170] The frame rate of the applications in the list is not limited to being higher than the frame rate of the applications outside the list. The frame rate of the applications in the list can also be lower than the frame rate of the applications outside the list. This embodiment of the present application does not impose any restrictions on this.

[0171] In some embodiments, Figure 8The applications in the list shown can be game applications, and the non-list applications can be applications other than game applications (such as instant messaging applications). Among them, the electronic device can maintain a high frame rate when displaying the user interface of the game application, while for applications other than the game application, a high frame rate is often not required. Therefore, the electronic device can display the user interface of the non-game application at a lower frame rate than the game application. In this way, the electronic device can not only ensure the user's gaming experience, but also avoid the non-game application in the multi-window composite scenario unnecessarily refreshing the user interface according to the frame rate of the game application, resulting in excessive power consumption of the electronic device.

[0172] The following describes the display method provided by the embodiments of the present application.

[0173] Figure 9 This is a flow chart of a display method provided by an embodiment of the present application. Figure 9 As shown, the method may include but is not limited to the following steps:

[0174] S901: Receive a first operation acting on a first application, and generate a first event according to the first operation.

[0175] The first application may be a game application. The electronic device may receive a first operation on a user interface of the first application, where the first operation may include, but is not limited to, a press, release, or slide operation. Furthermore, an input management service in the electronic device may generate a first event based on the first operation.

[0176] S902: In response to generating a first event, provide the first event to a first application.

[0177] In response to the generation of the first event, the input management service in the electronic device may directly send the first event to the first application. After generating the first event, the input management service does not wait for a VSync-app signal, but instead directly provides the signal to the first application. The first application may be a game application in the electronic device.

[0178] In some embodiments, the touch screen manager is configured to determine whether the first application is an application that relies on VSync-app signals. A VSync-app-reliant application is one that relies on the VSync-app synchronization mechanism to output rendered frames. Because gaming applications are controlled by their own engines to output rendered frames, gaming applications are considered applications that do not rely on VSync-app signals.

[0179] by Figure 5 Taking the embodiment shown as an example, the first event may be, for example, input event 1. After generating event 1, the input management service directly sends it to the game application. Figure 5The first VSync-app signal (i.e., VSync-app1 signal) can be referred to as a first VSync-app signal, and the second VSync-app signal (i.e., VSync-app2 signal) can be referred to as a second VSync-app signal. The first event is generated between the first VSync-app signal and the second VSync-app signal and provided to the first application. The first VSync-app signal and the second VSync-app signal are two adjacent vertical synchronization signals emitted by the electronic device.

[0180] S903: Generate a first layer based on the first event using the first application.

[0181] S904: Display a first user interface according to the first layer.

[0182] The first application in the electronic device can generate a first layer based on the first event. Figure 5 Taking the illustrated embodiment as an example, the first layer can be layer 1. The first application may include a logic thread and a rendering thread. After the first application receives a first event, the logic thread first generates first rendering data based on the first event. Here, the first rendering data may be rendering data 1. The rendering thread can then generate a first layer based on the first rendering data. The first layer may be, for example, layer 1. A surface compositor in the electronic device can then generate a first user interface based on the first layer. The display driver displays the first user interface, and the touch screen ultimately displays the first user interface. The first user interface may be, for example, image 1.

[0183] It is understandable that after the electronic device generates the first event, it can directly provide the first event to the first application without waiting for the VSync-app signal. This can reduce the delay between the generation of the input event and the application receiving the input event. The application receives the input event faster and can process the input event earlier, which can also avoid the situation where the rendering time has arrived but the rendering data corresponding to the latest input event has not yet been generated. This method can effectively improve the hand tracking performance of the electronic device. In addition, when the first application is a game application, it does not need to rely on the VSync-app signal to generate the rendering frame, and the input management service does not need to wait for the VSync-app signal to arrive before sending the input event to the first application. This can ensure that the first application still generates the rendering frame according to its cycle, while improving the hand tracking performance of the electronic device when running the first application.

[0184] In some embodiments, the first user interface includes only the first layer, and the method further includes: stopping sending the VSync-app signal. The surface compositor in the electronic device may determine the layers included in the first application interface before synthesizing the first application interface. When the surface compositor determines that only the first layer of the first application is included, it indicates that there is no layer of the application that relies on the VSync-app signal in the first application interface. In this way, the electronic device can stop sending (or keep it off) the VSync-app signal, thereby reducing the power consumption of the electronic device.

[0185] In some embodiments, before generating the first event according to the first operation, the method further includes: determining that the sampling rate of the touch screen is a multiple of the refresh rate, and determining that the first application is in the first list. Only when the sampling rate of the touch screen is a multiple of the refresh rate, the electronic device generates the same number of input events between refreshing each two frames of the user interface, so that it can be ensured that the application generates rendering frames based on the same number of input events each time, avoiding the problem of discontinuity in the display screen. The first list can be an enable list, wherein the first list is not limited to a list on a data structure, but is used to represent a list of identifiers storing one or more applications, which can be a list, a linked list, etc. The embodiment of the present application does not limit the data structure type of the first list. The first application in the first list means that the identifier of the first application is in the first list. The application corresponding to the identifier contained in the first list can be set by the user or preset by the electronic device, and the embodiment of the present application does not limit this. Optionally, the identifier of the application can be the package name of the application.

[0186] In some embodiments, the electronic device is also installed with a second application, and the method further includes: receiving a second operation acting on the second application, and generating a second event based on the second operation; providing a second event to the second application in response to a vertical synchronization VSync-app signal; generating a second layer based on the second event through the second application; and displaying a second user interface based on the second layer.

[0187] Among them, the second application can be an application outside the first list. The second application needs to rely on the VSync-app signal to trigger the drawing and rendering process, so the input management service still needs to wait for the VSync-app signal when sending the input event to the second application. Then, the second application can generate a second layer based on the second event, the surface compositor synthesizes the second user interface based on the second layer, the display driver sends the second user interface to the display, and then the touch screen displays the second user interface. Among them, the touch screen manager in the electronic device is used to determine that the second application is an application that relies on the VSync-app signal.

[0188] In some embodiments, before providing the second event to the second application, the method further includes: resending the VSync-app signal. Because the first application does not rely on the VSync-app signal, while the second application does rely on the VSync-app signal to trigger drawing and rendering processes, when the electronic device runs the second application, the VSync-app signal needs to be re-enabled to ensure that the second application can stably output image frames.

[0189] In some embodiments, the second user interface further includes a third layer of the first application, wherein the frame rate of the first application is a first frame rate, the frame rate of the second application is a second frame rate, and the first frame rate is greater than the second frame rate.

[0190] by Figure 7B Taking the illustrated scenario as an example, the second user interface can be the user interface displayed on the touch screen (including user interface 701 and user interface 702). The third layer can be the layer corresponding to user interface 701, and the second layer can be the layer corresponding to user interface 702. The electronic device can display the user interface of the first application at a first frame rate and the user interface of the second application at a second frame rate. To ensure the user's gaming experience, the first frame rate is typically higher, and the frame rate of gaming applications is not affected by the VSync-app signal. The electronic device can adjust the frequency of the VSync-app signal to control the frame rate of the second application so that the second frame rate is lower than the first frame rate. This prevents the second application from unnecessarily outputting rendered frames at the first frame rate, which could result in excessive power consumption in the electronic device. The adjustment of the VSync-app signal frequency can be performed by a surface compositor in the electronic device. When the surface compositor determines that the second user interface includes not only the third layer of the gaming application but also the second layer of the non-gaming application, it can adjust the transmission frequency of the VSync-app signal to the second frame rate.

[0191] In some embodiments, before generating the second event according to the second operation, the method further includes: determining that the sampling rate of the touch screen is not a multiple of the refresh rate, and / or determining that the second application is not in the first list. In the case that the sampling rate of the touch screen is not a multiple of the refresh rate, when the input management service sends input events to the application without relying on the VSync-app signal, it may cause the application to generate rendering frames based on a different number of input events each time, making the user interface that is ultimately displayed not smooth enough. Therefore, the electronic device can turn on (or keep on) the VSync-app signal when it is determined that the touch screen sampling rate is not a multiple of the refresh rate, and / or that the second application is not in the first list.

[0192] The following describes the apparatus for executing the above method provided in the embodiment of the present application. Figure 10 As shown, Figure 101 is a schematic diagram of the structure of a display device provided in an embodiment of the present application. The display device can be an electronic device in an embodiment of the present application, or a chip or chip system in an electronic device.

[0193] like Figure 10 As shown, the display device 1000 may include an operation acquisition module 1001 , a processing module 1002 , a display module 1003 and a storage module 1004 .

[0194] The storage module 1004 may store a first application, and the processing module 1002 is configured to periodically send a VSync-app signal. The operation acquisition module 1001 is configured to receive a first operation applied to the first application. The processing module 1002 is configured to generate a first event based on the first operation and, in response to generating the first event, provide the first event to the first application. The processing module 1002 is further configured to generate a first layer based on the first event by the first application; and the display module 1003 is configured to display a first user interface based on the first layer.

[0195] In some embodiments, when the first user interface only includes the first layer, the processing module 1002 is further configured to stop sending the VSync-app signal.

[0196] In some embodiments, before generating the first event according to the first operation, the processing module 1002 is further configured to determine whether the sampling rate of the touch screen is a multiple of the refresh rate, and determine that the first application is in the first list.

[0197] In some embodiments, the storage module 1004 further stores a second application. The operation acquisition module 1001 is further configured to receive a second operation applied to the second application. The processing module 1002 is further configured to generate a second event based on the second operation and provide the second event to the second application in response to a vertical synchronization (VSync-app) signal. The processing module 1002 is further configured to generate a second layer based on the second event by the second application. The display module 1003 is further configured to display a second user interface based on the second layer.

[0198] In some embodiments, before providing the second event to the second application, the processing module 1002 is further configured to resend the VSync-app signal.

[0199] In some embodiments, before generating the second event according to the second operation, the processing module 1002 is further configured to determine that the sampling rate of the touch screen is not a multiple of the refresh rate, and / or determine that the second application is not in the first list.

[0200] In some embodiments, the first application is a gaming application and the second application is a non-gaming application.

[0201] In some embodiments, the processing module 1002 and the storage module 1004 are connected via a circuit. The storage module 1004 may include one or more memories, which may be devices in one or more devices or circuits for storing programs or data. The storage module 1004 may exist independently and be connected to the processing module 1002 via a communication bus. The storage module 1004 may also be integrated with the processing module 1002.

[0202] The storage module 1004 can store computer-executable instructions for the method in the electronic device, so that the processing module 1002 can execute the method in the above embodiment. The storage module 1004 can be a register, a cache, or a random access memory (RAM), etc. The storage module 1004 can be integrated with the processing module 1002. The storage module 1004 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions. The storage module 1004 can be independent of the processing module 1002.

[0203] Figure 11 This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. Figure 11 As shown, the chip 1100 includes one or more (including two) processors 1101 , a communication line 1102 and a communication interface 1103 . Optionally, the chip 1100 also includes a memory 1104 .

[0204] In some embodiments, the memory 1104 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0205] The method described in the above embodiment of the present application can be applied to the processor 1101, or implemented by the processor 1101. The processor 1101 may be an integrated circuit chip with signal processing capabilities. During the implementation process, the steps of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1101. The above-mentioned processor 1101 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 1101 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0206] The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Among them, the software module can be located in a mature storage medium in the field such as random access memory, read-only memory, programmable read-only memory, or electrically erasable programmable read only memory (EEPROM). The storage medium is located in the memory 1104, and the processor 1101 reads the information in the memory 1104 and completes the steps of the above method in combination with its hardware.

[0207] The processor 1101 , the memory 1104 , and the communication interface 1103 can communicate with each other via the communication line 1102 .

[0208] In the above embodiment, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory or downloaded and installed in the memory in the form of software.

[0209] The present application embodiment also provides a computer program product, which includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrations. For example, available media can include magnetic media (e.g., floppy disk, hard disk or tape), optical media (e.g., digital versatile disc (DVD)), or semiconductor media (e.g., solid state disk (SSD)).

[0210] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program or instructions. When the computer program or instructions are executed by a processor, the above-mentioned method is implemented. The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0211] As a possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; computer-readable media may include magnetic disk storage or other magnetic disk storage devices. Moreover, any connecting line may also be appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of medium. Disks and optical discs as used herein include compact discs (CDs), laser discs, optical discs, DVDs, floppy disks and Blu-ray discs, where disks generally reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0212] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0213] As used in the above embodiments, the term “when…” may be interpreted to mean “if…” or “after…” or “in response to determining…” or “in response to detecting…”, depending on the context. Similarly, the phrases “upon determining…” or “if (stated condition or event) is detected” may be interpreted to mean “if determining…” or “in response to determining…” or “upon detecting (stated condition or event)” or “in response to detecting (stated condition or event)”, depending on the context.

[0214] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk).

[0215] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A display method, characterized in that: The method is applied to an electronic device, the electronic device including a touch screen, the electronic device having a first application and a second application installed thereon, wherein the first application is a game application and the second application is a non-game application, the method comprising: Start the first application and stop sending the vertical synchronization Vsync-app signal; receiving a first operation applied to the first application, and generating a first event according to the first operation; In response to generating the first event, providing the first event to the first application; generating, by the first application, a first layer based on the first event; Displaying a first user interface on the touch screen according to the first layer, wherein the first user interface does not include a layer of non-game applications, and the touch screen only includes the first user interface; Starting the second application and resuming the sending of the vertical synchronization Vsync-app signal; receiving a second operation applied to the second application, and generating a second event according to the second operation; In response to a vertical synchronization VSync-app signal, providing the second event to the second application; generating, by the second application, a second layer based on the second event; A second user interface is displayed on the touch screen according to the second layer.

2. The method according to claim 1, characterized in that The second user interface also includes a third layer of the first application, wherein the frame rate of the first application is a first frame rate, the frame rate of the second application is a second frame rate, and the first frame rate is greater than the second frame rate.

3. The method according to claim 2, characterized in that Before generating a first event according to the first operation, the method further includes: Determine the touchscreen's sampling rate as a multiple of its refresh rate, and Determine that the first application is in a first list.

4. The method according to claim 3, characterized in that Before generating a second event according to the second operation, the method further includes: Ensure that the touchscreen's sampling rate is not a multiple of the touchscreen's refresh rate, and / or, It is determined that the second application is not in the first list.

5. The method according to any one of claims 1 to 4, characterized in that The electronic device includes a touch screen manager and an input management service, the touch screen manager is used to determine that the first application is an application that does not rely on VSync-app signals, and the input management service is used to provide the first event to the first application in response to generating the first event.

6. The method according to claim 5, characterized in that The touch screen manager is further configured to determine that the second application is an application that relies on a VSync-app signal, and the input management service is further configured to provide the second event to the second application in response to a vertical synchronization VSync-app signal.

7. The method according to claim 2, characterized in that The electronic device includes a surface compositor, and the surface compositor is configured to determine that the second user interface includes the second layer and the third layer; The surface synthesizer is further configured to adjust a sending frequency of the VSync-app signal to the second frame rate.

8. The method according to any one of claims 1 to 4, characterized in that The first application includes a logic thread and a rendering thread. The logic thread is used to generate first rendering data based on the first event, and the rendering thread is used to generate the first layer based on the first rendering data.

9. A display method, characterized in that: The method is applied to an electronic device, the electronic device including a touch screen, the electronic device having a first application and a second application installed thereon, and the method includes: Starting the first application, determining that the first application is in the first list, and stopping sending a vertical synchronization Vsync-app signal; receiving a first operation applied to a first application, and generating a first event according to the first operation; In response to generating the first event, providing the first event to the first application; generating, by the first application, a first layer based on the first event; displaying a first user interface on the touch screen according to the first layer, wherein the first user interface only includes the layers of the applications in the first list, and the touch screen only includes the first user interface; Starting the second application, determining that the second application is not in the first list, and resuming sending the vertical synchronization Vsync-app signal; receiving a second operation applied to the second application, and generating a second event according to the second operation; In response to a vertical synchronization VSync-app signal, providing the second event to the second application; generating, by the second application, a second layer based on the second event; A second user interface is displayed on the touch screen according to the second layer.

10. The method according to claim 9, characterized in that Before generating a first event according to the first operation, the method further includes: Determine the touchscreen's sampling rate as a multiple of its refresh rate.

11. The method according to claim 9, characterized in that The second user interface also includes a third layer of the first application, wherein the frame rate of the first application is a first frame rate, the frame rate of the second application is a second frame rate, and the first frame rate is greater than the second frame rate.

12. The method according to claim 11, characterized in that The electronic device includes a surface compositor, and the surface compositor is configured to determine that the second user interface includes the second layer and the third layer; The surface synthesizer is further configured to adjust a sending frequency of the VSync-app signal to the second frame rate.

13. The method according to any one of claims 9 to 12, characterized in that Before generating a second event according to the second operation, the method further includes: Make sure the touchscreen's sampling rate is not a multiple of its refresh rate.

14. The method according to any one of claims 9 to 12, characterized in that The electronic device includes a touch screen manager and an input management service, the touch screen manager is used to determine that the first application is an application that does not rely on VSync-app signals, and the input management service is used to provide the first event to the first application in response to generating the first event.

15. The method according to claim 14, characterized in that The touch screen manager is further configured to determine that the second application is an application that relies on a VSync-app signal, and the input management service is further configured to provide the second event to the second application in response to a vertical synchronization VSync-app signal.

16. The method according to any one of claims 9 to 12, characterized in that The first application includes a logic thread and a rendering thread. The logic thread is used to generate first rendering data based on the first event, and the rendering thread is used to generate the first layer based on the first rendering data.

17. An electronic device, characterized in that: The electronic device includes: a touch screen, a memory, and a processor coupled to the memory; the touch screen is used to receive user operations and display an interface, the memory stores a computer program, and when the processor executes the computer program, the electronic device implements the method as described in any one of claims 1-8 or 9-16.

18. A computer-readable storage medium comprising computer instructions, characterized in that: When the computer instructions are executed on an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 8 or 9 to 16.

Images