Display processing method and electronic equipment

Abstract

The embodiment of the invention provides a display processing method and electronic equipment, and relates to the technical field of equipment. The problem that the chirality is reduced in the operation process of the electronic equipment is solved. According to the specific scheme, first operation of a user is received; at a first time point, storing a first rendering frame corresponding to the first operation into a first cache queue; wherein after the first rendering frame is stored in the first buffer queue, the first buffer queue comprises the first rendering frame and a second rendering frame, and the second rendering frame is arranged in front of the first rendering frame; in response to a first vertical synchronization signal, synthesizing a first graphical interface corresponding to the first rendering frame, the first vertical synchronization signal being a first vertical synchronization signal generated after the first time point; and displaying the first graphical interface.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of display technology, and more particularly to a display processing method and an electronic device. Background Art

[0002] With the continuous evolution of electronic devices, users are increasingly demanding higher efficiency in human-computer interaction. Screen smoothness is a key metric for users to assess the efficiency of human-computer interaction in electronic devices. This smoothness is significantly affected by the device's hand tracking.

[0003] It is understood that the chirality of an electronic device can be reflected in the delay time from "a user inputs an interactive operation to the electronic device" to "the electronic device displays the graphical interface corresponding to the interactive operation." The longer the delay time, the worse the chirality.

[0004] At present, during the actual operation of electronic devices, the problem of decreased hand tracking occurs, which can directly affect the smoothness of the electronic device's picture and reduce the human-computer interaction efficiency of the electronic device. Summary of the Invention

[0005] The embodiments of the present application provide a display processing method and an electronic device, which are used to solve the problem of decreased chirality during the operation of the electronic device and improve the human-computer interaction efficiency of the electronic device.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, an embodiment of the present application provides a display processing method, which is applied to an electronic device. The method includes: at a first time point after receiving a first operation from a user, storing a first rendering frame corresponding to the first operation into a first cache queue. The first cache queue is used to store rendering frames waiting to be synthesized. In the first cache queue, the rendering frames arranged in front are first taken out and synthesized into the corresponding graphical interface, and the rendering frames arranged in the back are taken out and synthesized into the corresponding graphical interface later.

[0008] After the first rendered frame is stored in the first cache queue, the first cache queue includes the first rendered frame and the second rendered frame, that is, there is a rendering frame backlog in the first cache queue. It is understandable that rendering frame backlog is a common phenomenon during the operation of electronic devices and is one of the factors that easily affect chirality.

[0009] In the event of rendering frame accumulation, the electronic device responds to the first vertical synchronization information (the first vertical synchronization information after the first rendering frame is stored in the first cache queue), skips processing the second rendering frame, and synthesizes and displays the first graphical interface corresponding to the first rendering frame.

[0010] In the above embodiment, after the electronic device stores the first rendered frame in the first cache queue, it skips processing the second rendered frame (e.g., skips synthesizing and displaying the graphical interface of the second rendered frame). This shortens the time the first rendered frame waits in the first cache queue for processing, thereby shortening the time between receiving the first operation and displaying the first graphical interface. In this way, in scenarios where rendered frames are accumulated, the degradation of hand tracking is avoided, and the efficiency of human-computer interaction of the electronic device is improved.

[0011] In some embodiments, after the first rendered frame is stored in the first cache queue, the electronic device retrieves the second rendered frame from the first cache queue and discards it. Furthermore, in response to a first vertical synchronization signal, the electronic device retrieves the first rendered frame from the first cache queue, thereby synthesizing and displaying a first graphical interface based on the first rendered frame.

[0012] In the above embodiment, by discarding the second rendering frame in the first cache queue, the problem of rendering frame accumulation in the first cache queue is eliminated, which not only solves the tracking problem between the first operation and the first graphical interface, but also shortens the waiting time for other rendering frames arranged after the first rendering frame to be processed, thereby enhancing the smoothness of the screen display.

[0013] In an exemplary scenario, the second rendered frame is an interpolated frame arranged before the first rendered frame. The second rendered frame includes a first tag, the first tag indicating that the second rendered frame is an interpolated frame. The first tag may be a tag added to the interpolated frame by the interpolation management module during interpolation.

[0014] In the above example, when the second rendered frame is an interpolated frame, the second rendered frame is skipped to ensure the hand tracking of the electronic device. This not only improves the human-computer interaction efficiency of the electronic device, but also ensures the image display quality.

[0015] In an exemplary scenario, the second rendered frames are all rendered frames arranged before the first rendered frame. For example, the number of second rendered frames can be one or more, and the second rendered frames can include interpolated frames or real frames. This prioritizes the hand tracking of the electronic device and improves the efficiency of human-computer interaction.

[0016] In some embodiments, the second rendered frame may be a rendered frame that precedes the first rendered frame in the first buffer queue. In other embodiments, the second rendered frame may be all rendered frames that precede the first rendered frame in the first buffer queue.

[0017] In some embodiments, before synthesizing the first graphical interface corresponding to the first rendered frame in response to the first vertical synchronization signal, the electronic device further needs to determine whether the first rendered frame includes a second tag, where the second tag indicates that the first rendered frame is a real frame. The second tag may be a tag added to the real frame by the interpolation frame management module.

[0018] In the above embodiments, in scenarios where chirality needs to be guaranteed, the problem of decreased chirality can be effectively avoided, thereby improving the human-computer interaction efficiency of the electronic device.

[0019] In another possible embodiment, after the first rendered frame is stored in the first cache queue, the first cache queue includes the first rendered frame, the second rendered frame, and the tenth rendered frame, with the second rendered frame and the tenth rendered frame arranged before the first rendered frame. In response to the first vertical synchronization signal, a seventh graphical interface corresponding to the tenth rendered frame is synthesized and displayed. In response to a seventh vertical synchronization signal, the first graphical interface corresponding to the first rendered frame is synthesized and displayed; the seventh vertical synchronization signal is the second vertical synchronization signal generated after the first time point.

[0020] In some embodiments, the method further includes: receiving a second operation of the user, wherein the time interval between a first receiving time point of the first operation and a second receiving time point of the second operation is less than a first threshold; or, the first receiving time point and the second receiving time point belong to the same enabling period, and the first receiving time point is earlier than the second receiving time point.

[0021] At the second time point, the third rendering frame corresponding to the second operation is stored in the first cache queue. After the third rendering frame is stored in the first cache queue, the first cache queue includes the third rendering frame and the fourth rendering frame, and the fourth rendering frame is arranged before the third rendering frame. In response to the second vertical synchronization signal, the second graphical interface corresponding to the fourth rendering frame is synthesized. The second vertical synchronization signal is the first vertical synchronization signal generated after the second time point; the second graphical interface is displayed; in response to the third vertical synchronization signal, the third graphical interface corresponding to the third rendering frame is synthesized; the third vertical synchronization signal is the second vertical synchronization signal generated after the second time point; and the third graphical interface is displayed.

[0022] In the above embodiment, when the receiving time point of the second operation is too close to the receiving time point of the first operation, even if there is a pile-up of rendering frames, the graphical interface of the rendering frames in the first cache queue will be processed in sequence to avoid skipping the processing of other rendering frames multiple times in a short period of time, thereby ensuring the display quality and frame rate of the electronic device.

[0023] In some embodiments, the method further includes: receiving a third operation of the user, the time interval between the third receiving time point of the third operation and the first receiving time point of the first operation is greater than or equal to a first threshold; or, the third receiving time point and the first receiving time point belong to different enable cycles; at the third time point, storing the fifth rendering frame corresponding to the third operation into the first cache queue; wherein, after the fifth rendering frame is stored in the first cache queue, the first cache queue includes the fifth rendering frame and the sixth rendering frame, and the sixth rendering frame is arranged before the fifth rendering frame; in response to a fourth vertical synchronization signal, synthesizing a fourth graphical interface corresponding to the fifth rendering frame; wherein, the fourth vertical synchronization signal is the first vertical synchronization signal generated after the third time point; and displaying the fourth graphical interface.

[0024] In some embodiments, before synthesizing the fourth graphical interface corresponding to the fifth rendering frame, the method further includes: taking out the sixth rendering frame from the first cache queue and discarding it.

[0025] In the above embodiment, by using the first threshold or the enabling period, the chirality is prevented from being reduced while other rendering frames are not frequently skipped, thereby improving the efficiency of human-computer interaction.

[0026] In some embodiments, the method further includes: receiving a fourth operation from the user, the time interval between the fourth receiving time point of the fourth operation and the first receiving time point of the first operation being greater than or equal to a first threshold; or, the fourth receiving time point and the first receiving time point belonging to different enabling cycles. At the fourth time point, storing the seventh rendering frame corresponding to the fourth operation into the first cache queue; wherein, after the seventh rendering frame is stored into the first cache queue, the first cache queue includes the seventh rendering frame, the eighth rendering frame, and the ninth rendering frame, and the eighth rendering frame and the ninth rendering frame are arranged before the seventh rendering frame. In response to the fifth vertical synchronization signal, synthesizing the fifth graphical interface corresponding to the ninth rendering frame. wherein the fifth vertical synchronization signal is the first vertical synchronization signal generated after the fourth time point; displaying the fifth graphical interface; in response to the sixth vertical synchronization signal, synthesizing the sixth graphical interface corresponding to the seventh rendering frame; wherein the sixth vertical synchronization signal is the second vertical synchronization signal generated after the fourth time point; displaying the sixth graphical interface.

[0027] In some embodiments, before synthesizing the fifth graphical interface corresponding to the ninth rendering frame, the eighth rendering frame may be taken out of the first cache queue and discarded.

[0028] In the above embodiment, by skipping the processing of only one rendering frame, the hand tracking and frame rate of the electronic device are guaranteed, and the efficiency of human-computer interaction is improved.

[0029] For example, in the first cache queue, the ninth rendered frame is arranged before the eighth rendered frame. After the first rendered frame is stored in the first cache queue, the processing of the rendered frame preceding the first rendered frame is skipped, thereby ensuring the hand tracking of the electronic device and improving the efficiency of human-computer interaction.

[0030] For example, the ninth rendered frame includes a second tag indicating that it is a real frame, and the eighth rendered frame includes a first tag indicating that it is an interpolated frame. After the first rendered frame is stored in the first cache queue, processing of only the interpolated frame is skipped, thereby ensuring the electronic device's hand tracking and display quality, and improving human-computer interaction efficiency.

[0031] In some embodiments, before receiving the first operation of the user, the method further includes: responding to the user's operation of opening the target application, running the target application in the foreground; and determining that the target running application meets the first condition.

[0032] The first condition includes: the application identifier of the target application is included in the pre-configured application list; or the application service provided by the target application belongs to a preset type.

[0033] In the above embodiment, only when a specific application is running will the skipping of other rendered frames arranged before the first rendered frame be triggered, so as to prioritize chirality in certain scenarios and prioritize display quality and frame rate in other scenarios.

[0034] In some embodiments, the electronic device includes an input event management module and a rendering library. After receiving the user's first operation, the method includes: the input event management module generates a first event corresponding to the first operation, and the first event includes type information and operation location information for describing the first operation; the input event management module reports the first event to the target application; the target application draws first graphic data and passes it to the rendering library; wherein the content of the first graphic data is related to the type information and / or operation location information of the first operation; the rendering library renders the first rendering frame based on the first graphic data; storing the first rendering frame corresponding to the first operation into the first cache queue includes: the rendering library stores the first rendering frame into the first cache queue.

[0035] In some embodiments, the electronic device also includes an image synthesizer, and the first event also includes a first receiving time point of the first operation; after storing the first rendering frame corresponding to the first operation into the first cache queue, the method also includes: when the first event meets the preset requirements, the rendering library sends first information to the image synthesizer, and the first information includes a target mark bit; the image synthesizer responds to the target mark bit in the first information, takes out the second rendering frame from the first cache queue, and discards it; wherein the preset requirements include: the time interval between the first receiving time point and the receiving time point of the adjacent previous interactive operation is greater than or equal to a first threshold.

[0036] In a second aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory is used to store code instructions; the processor is used to run the code instructions so that the electronic device executes the method described in the first aspect and any one of its implementations.

[0037] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed, the computer executes the method described in the first aspect and any one of its implementations.

[0038] In a fourth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, which, when executed, enables a computer to execute the method described in the first aspect and any one of its implementations.

[0039] It should be understood that the second to fourth aspects of the embodiments of the present application correspond to the technical solutions of the first aspect of the embodiments of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is an example diagram of the time delay T that affects the chirality of an electronic device provided in an embodiment of the present application;

[0041] Figure 2 This is an example diagram of a scene where rendered frames are piled up in the SF cache queue provided in an embodiment of the present application;

[0042] Figure 3 An example diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0043] Figure 4 An example diagram of the software and hardware structure of an electronic device provided in an embodiment of the present application;

[0044] Figure 5 A flowchart of a display processing method provided in an embodiment of the present application;

[0045] Figure 6 This is an example diagram of the process of processing rendered frames in the SF cache queue in the scenario where the hand tracking optimization function is not enabled and there is a backlog of rendered frames in the SF cache queue provided by the embodiment of the present application;

[0046] Figure 7 This is an example diagram of how to process rendered frames in the SF cache queue when the hand tracking optimization function is enabled and there is a backlog of rendered frames in the SF cache queue, according to an embodiment of the present application.

[0047] Figure 8 This is a second example of how to process rendered frames in the SF cache queue when the hand tracking optimization function is enabled and there is a backlog of rendered frames in the SF cache queue, according to an embodiment of the present application.

[0048] Figure 9 Figure 3 shows an example of a flow method for processing rendered frames in the SF cache queue when the hand tracking optimization function is enabled and there is a backlog of rendered frames in the SF cache queue according to an embodiment of the present application.

[0049] Figure 10 Figure 4 shows an example of a flow method for processing rendered frames in the SF cache queue when the hand tracking optimization function is enabled and there is a backlog of rendered frames in the SF cache queue according to an embodiment of the present application.

[0050] Figure 11 This is one of the principle example diagrams of determining interactive operation 1 provided in an embodiment of the present application;

[0051] Figure 12 This is a second diagram illustrating the principle of determining interactive operation 1 provided in an embodiment of the present application;

[0052] Figure 13 A signaling interaction diagram of a display processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "plurality" means two or more.

[0054] At the same time, in the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner to facilitate understanding.

[0055] With the development of technology, the performance of various electronic devices (such as mobile phones) is getting better and better. Correspondingly, users have higher and higher requirements for the human-computer interaction performance of electronic devices. For electronic devices equipped with display screens, chirality is one of the important factors affecting human-computer interaction performance. It can be understood that the above-mentioned chirality can be reflected as the time delay T between "the user's finger touches the touch screen of the electronic device" and "the electronic device actually displays the graphical interface corresponding to the touch operation." The shorter the length of the above-mentioned delay T, the better the hand-following performance of the electronic device. The better the hand-following performance of the electronic device, the better the user experience of controlling the electronic device through touch operation, and the smoother it feels.

[0056] Take the example of running a game application on an electronic device. Figure 1 As shown, the electronic device can display a game interface 101, which includes a game character 102 and a control 103 indicating a jump. According to the rules of the game application, a touch event (clicking the control 103 in the game interface 101) corresponds to multiple graphical interfaces showing the game character 102 jumping, such as the touch event corresponding to the game interface 104.

[0057] like Figure 1 As shown, at time t1, the user clicks the display area of ​​control 103 on the display. Between time t1 and time t2, the display's touch panel (TP) senses the user's touch operation, triggering the TP driver to scan and obtain the touch parameters corresponding to the touch (touch position, touch force, touch duration, etc.). The TP driver encapsulates the touch parameters corresponding to the detected touch as a raw input event and reports it to the Event Hub. The period 1 between time t1 and time t2 can be called the touch bottom layer processing latency.

[0058] Between time points t2 and t3, the inputreader thread of the input event management module can read the original input event uploaded by the TP driver from the Event Hub. Afterwards, the input event management module can translate, encapsulate, and process the original input event to obtain an input event containing more information (such as a touch event). Afterwards, the Input Dispatcher thread of the input management module passes the touch event to the application that has subscribed to the touch event. For example, when the game application running in the foreground subscribes to the touch event from the display screen through the input management module, the input management module can pass the touch event to the game application after determining the touch event. The above-mentioned period 2 between time points t2 and t3 can be called the touch input event processing delay.

[0059] Between time point t3 and time point t4, the game application can respond to the touch event and identify that the user clicks on the display area of ​​the control 103. Accordingly, according to the rules of the game, the game application can render one or more frames of rendering frames indicating that the game character 102 jumps. The above-mentioned rendering frame may include one or more rendering layers. For example, the game application can respond to the touch event and draw one or more layers contained in the graphical interface corresponding to the touch event, and then render the one or more layers to obtain one or more rendering layers, which can constitute a rendering frame. Afterwards, the game application can pass the rendering frame to the image synthesizer (surface flinger, SF) of the electronic device. The period 3 between the above-mentioned time point t3 and the time point t4 can be called the application processing delay.

[0060] Between time points t4 and t5, the SF receives rendered frames from the game application and synthesizes the corresponding graphical interfaces, such as game interface 104. Game interface 104 displays a scene of game character 102 jumping. The SF then transmits the synthesized graphical interface to a display panel, such as a liquid crystal display (LCD). The period 4 between time points t4 and t5 is referred to as the synthesis processing delay.

[0061] Between time points t5 and t6, the LCD receives the graphical interface synthesized by the SF and displays it, for example, the game interface 104. In addition, the period 5 between time points t5 and t6 can be referred to as a display processing delay.

[0062] Thus, at time t1, the user performs a touch operation of clicking on the control 103, and at time t6, the user can see the graphical interface corresponding to the touch operation on the display screen, such as the game interface 104. The time between time t1 and time t6 is the delay T.

[0063] In some embodiments, by upgrading the hardware of the electronic device, the above-mentioned delay T can be shortened to a certain extent. Under the same hardware conditions, the above-mentioned delay T will increase in some operating scenarios. For example, after the electronic device turns on the interpolation mode, the rendering frames waiting to be synthesized will be piled up during the process of SF performing image synthesis. When the period of the vertical synchronization (Vertical SyncSignal, VSYNC) signal remains unchanged and there is a pile of rendering frames to be synthesized, the actual synthesis time point of the graphical interface corresponding to the SF synthetic touch operation will be delayed relative to the expected synthesis time point. Among them, the expected synthesis time point refers to the time point when SF synthesizes the graphical interface under ideal conditions, such as when there is no accumulation of rendering frames. In this way, the actually displayed graphical interface will be delayed.

[0064] Gaming applications are often used in interpolation mode. Interpolation mode, for example, allows electronic devices to achieve a 120Hz refresh rate while using less power. However, enabling interpolation mode can also lead to a backlog of rendered frames waiting to be synthesized.

[0065] The following takes the running of a game application as an example to introduce the reason why the rendering frames waiting to be synthesized accumulate in the interpolation mode. It can be understood that if the interpolation mode is turned on during the running of other applications, it may also cause the rendering frames waiting to be synthesized to accumulate. The embodiment of the present application does not specifically limit the scenario.

[0066] like Figure 2 As shown, while the electronic device is running a game application in the foreground, the game application can render a rendering frame corresponding to the game interface. For example, rendering frame 1, rendering frame 2, and rendering frame 1.5 are rendered in sequence, wherein rendering frame 1 and rendering frame 2 are real frames, which are graphic data drawn by the game application. Rendering frame 1.5 is an inserted frame, which is the graphic data calculated by the insert frame management module based on rendering frame 1 and rendering frame 2, and can be used to connect rendering frame 1 and rendering frame 2. For example, the target object in rendering frame 1 is at position A, and the target object in rendering frame 2 is at position B. After predicting that the target object needs to pass through position C in the process of moving from position A to position B, the target object in rendering frame 1.5 is at position C.

[0067] Furthermore, before rendering Rendered Frame 2, the game application receives a touch event reported by the input management module. The game application can combine the touch event (e.g., the type and location of the touch operation) to render Rendered Frame 2. In other words, Rendered Frame 2 can be used to synthesize the graphical interface corresponding to the touch operation.

[0068] like Figure 2 As shown, the rendered frame 1 is buffered in the SF buffer queue before the SF receives the second VSYNC signal.

[0069] The SF cache queue is used to cache rendered frames waiting to be processed by the SF. The rendered frames in the SF cache queue are arranged in order. Every time the SF receives a VSYNC signal, it takes a rendered frame from the SF cache queue. The rendered frame that is arranged first in the SF cache queue is taken out first.

[0070] After receiving the second VSYNC signal, SF can take out the rendering frame 1 from the SF cache queue and synthesize the corresponding graphical interface (ie, the game interface) based on the rendering frame 1. Figure 2 As shown, rendered frames 1.5 and 2 are cached in the SF cache queue before the third VSYNC signal. Multiple rendered frames cached in the SF cache queue are called rendered frame accumulation in the SF cache queue.

[0071] Furthermore, the display time for Rendered Frame 1.5 needs to be between the display time for Rendered Frame 1 and the display time for Rendered Frame 2. In the SF cache queue, Rendered Frame 1.5 must be queued before Rendered Frame 2. For example, after Rendered Frame 2 is rendered, it is stored in the SF cache queue. After Rendered Frame 1.5 is rendered, it is stored in the SF cache queue, queued before Rendered Frame 2. For example, after Rendered Frame 1.5 is rendered, Rendered Frame 1.5 and Rendered Frame 2 are stored in the SF cache queue, sequentially.

[0072] It is understandable that the time interval between Rendered Frames 1.5 and 2 being cached in the SF cache queue is less than one VSYNC signal period. After the SF receives the third VSYNC signal, it removes Rendered Frame 1.5 from the SF cache queue and synthesizes the corresponding graphical interface (i.e., the game interface) based on Rendered Frame 1.5. Thereafter, it waits for the SF to receive the fourth VSYNC signal, removes Rendered Frame 2 from the SF cache queue, and synthesizes the corresponding graphical interface (i.e., the game interface) based on Rendered Frame 2.

[0073] It can be seen that Figure 2As shown, when there's no accumulation of rendered frames in the SF cache queue, the time interval from "the game application stores rendered frame 1 in the SF cache queue" to "SF performs synthesis based on rendered frame 1" is no more than one VSYNC signal cycle. When there's an accumulation of rendered frames in the SF cache queue, the time interval from "the game application stores rendered frame 2 in the SF cache queue" to "SF performs synthesis based on rendered frame 2" is nearly two VSYNC signal cycles. Because rendered frame 2 corresponds to a touch event, the synthesis processing latency of rendered frame 2 increases, and accordingly, the latency T also increases, resulting in poor hand tracking of the electronic device.

[0074] To improve the above-mentioned problem, an embodiment of the present application provides a display processing method for an electronic device. In this case, if a first rendered frame and a second rendered frame are stored in an SF cache queue of the electronic device, and the second rendered frame is arranged before the first rendered frame, if the first rendered frame corresponds to a user interaction operation (e.g., a touch operation), the SF of the electronic device can remove the second rendered frame from the SF cache queue and discard it, and remove the first rendered frame from the SF cache queue and synthesize a graphical interface corresponding to the interaction operation based on the first rendered frame.

[0075] In the above embodiment, by dropping frames, the time from when the first rendered frame enters the SF cache queue to when it is taken out of the SF cache queue by the SF is shortened, thereby delaying the synthesis processing delay in T and improving the hand tracking performance of the electronic device.

[0076] The electronic device in the embodiments of the present application may be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, as well as a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) or virtual reality (VR) device, etc., including a touch screen. The embodiments of the present application do not impose any special restrictions on the specific form of the electronic device.

[0077] Please refer to Figure 3 , Figure 3 A possible hardware structure diagram of an electronic device is shown below:

[0078] like Figure 3As shown, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc.

[0079] Among them, the above-mentioned sensor module 180 may include sensors such as pressure sensor, gyroscope sensor, air pressure sensor, magnetic sensor, acceleration sensor, distance sensor, proximity light sensor, fingerprint sensor, temperature sensor, touch sensor, ambient light sensor and bone conduction sensor.

[0080] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0081] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0082] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0083] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0084] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0085] It is understood that the interface connection relationship between the modules illustrated in this embodiment is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0086] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0087] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a touch layer and a display panel. The touch layer is used to sense the interaction between the user and display screen 194. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In the subsequent embodiments, the display panel of display screen 194 is mainly an LCD. As can be understood, in the subsequent embodiments, the LCD mentioned can also be replaced with other types of display screens.

[0088] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0089] The ISP is used to process data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element (image sensor). The light signal is converted into an electrical signal, which is then transmitted to the ISP for processing and converted into an image visible to the naked eye. The ISP can also perform algorithmic optimization on image noise, brightness, and skin color. The ISP can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be set in camera 193.

[0090] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include N cameras 193, where N is a positive integer greater than 1.

[0091] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0092] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0093] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU can enable intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0094] Figure 4 This is a block diagram of the software and hardware structure of the electronic device 100 provided in the embodiment of the present application. The layered architecture can divide the software structure into several layers, and each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the electronic device may include an application layer (abbreviated as application layer), a framework layer (framework), a native layer, a kernel layer and a hardware layer. Of course, the software layers divided in the electronic device may also include Figure 4 Layers not shown in the figure, such as the hardware abstraction layer, Android runtime (AndroidRuntime), etc.

[0095] Exemplarily, the application layer may include a series of application programs.

[0096] like Figure 4 As shown, the application layer may include game applications, live broadcast applications and other applications, and may also include Figure 4 The application is shown in FIG.

[0097] For example, the framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The framework layer includes some predefined functions.

[0098] like Figure 4As shown, the framework layer may include a game tool (gamekit), an input event (input) management module, an application type management module, and the like.

[0099] Among them, the game tool (gamekit) serves the optimization of game scenes. Its feature is that it can use different strategies to schedule system resources according to different game scenes and running states to achieve more refined optimization effects.

[0100] The input event management module can be used to translate, encapsulate, and process the original input events collected by various hardware modules to obtain input events containing more information and pass them to applications or services that subscribe to the events reported by the hardware modules. For example, the above-mentioned various hardware modules can be touch screens, buttons, etc. For example, a game application subscribes to touch practices from a touch screen. When the touch screen senses that the user touches the display screen, it can collect corresponding touch parameters (touch position, touch force, touch duration, etc.) as original input events and report them to the input event management module. The input event management module translates, encapsulates, and processes the original input events to obtain corresponding touch events. In addition, the above-mentioned touch events may include information such as touch type (click, slide, slide back and forth, click to slide, etc.), touch position, event time point, etc. Finally, the input event management module passes the processed touch events to the game application.

[0101] The application type management module is used to determine the type of application running in the foreground. For example, applications can be divided into games, videos, live broadcasts, etc. according to their service type. For another example, applications can be divided into pre-set or non-pre-set categories according to whether they belong to a preset application list.

[0102] For example, the local layer can also be called the service layer, which is used to provide basic capabilities such as graphics, video, and audio. Figure 4 As shown, the above-mentioned local layer may include a rendering library, SF, an interpolation management module, and a hand tracking management module.

[0103] Among them, the application can call the rendering library, use the processing power of the GPU to complete layer drawing and rendering, and output the rendered frame.

[0104] SF is a special service used to process graphics data (such as rendering frames) and synthesize the corresponding graphical interface based on the processed graphics data.

[0105] The interpolation management module can be used to calculate the corresponding interpolation frame based on two adjacent real frames. The interpolation management module can distinguish between real frames and interpolation frames during the SF processing of rendered frames by assigning specified tags to interpolation frames.

[0106] The hand tracking management module can query the type of application running in the foreground through the application type management module. In addition, it can also read the configuration file for optimizing hand tracking, which can include synchronization switch information, enable interval information, and initialization application scheme information.

[0107] The synchronization switch information includes rules for turning on and off the hand-tracking optimization function.

[0108] For example, if the application running in the foreground is a specified type (such as a game or a pre-installed application), the hand tracking optimization function is enabled. If the application running in the foreground changes from a specified type application to a non-specified type application, the hand tracking optimization function is disabled.

[0109] For example, the system configuration interface includes configuration item a corresponding to the hand-following optimization function. If configuration item a is selected, if an application of the specified type is running in the foreground, the hand-following optimization function is enabled. If configuration item a is selected, if the application running in the foreground is not of the specified type, the hand-following optimization function is disabled. If configuration item a is not selected, the hand-following optimization function is disabled. In addition, the electronic device can configure configuration item a to be selected or unselected in response to user operations.

[0110] It can be understood that when the hand tracking optimization function is turned on, the method provided in the embodiment of the present application can be executed to optimize the hand tracking performance by dropping frames. For specific details, please refer to the subsequent embodiments.

[0111] The aforementioned initialization application solution information may include configuration information for adapting the hand-following optimization function to various applications. After the initialization application solution information is loaded, the hand-following optimization function may be enabled during the running of the application.

[0112] The enabling interval information is the time interval for performing the optimization and chirality tracking. For example, if the enabling interval information is 5 seconds, the time interval between two consecutive optimization and chirality tracking operations should be no less than 5 seconds to avoid frequent frame drops that affect the image display quality.

[0113] For example, the kernel layer is located below the local layer and is the layer between hardware and software. In addition to the display driver, TP driver, and GPU driver mentioned above, the kernel layer may also include camera drivers, audio drivers, various sensor drivers, etc., which are not limited in this embodiment of the application.

[0114] In some embodiments, the hardware layer may include various hardware modules integrated into electronic devices, such as Figure 4 The GPU and touch screen shown. Of course, the hardware layer can also include Figure 4 Hardware modules not shown.

[0115] The methods in the following embodiments can all be implemented in a device having the above-mentioned hardware and software structures. The following describes the implementation details of the embodiments of the present application using a mobile phone as the electronic device.

[0116] like Figure 5 As shown, the method provided in the embodiment of the present application may include the following steps:

[0117] S101 , while a target application is running in the foreground of a mobile phone, in response to detecting a user interaction operation 1 , drawing and rendering a target rendering frame corresponding to the interaction operation 1 .

[0118] The target application is an application whose display content is affected by the interactive operation. In some examples, while the mobile phone displays the application interface of the target application, the application interface changes as the target application interacts with the user.

[0119] For example, the target application may be a game application. While the user is using the game application on their mobile phone, game interface 1 corresponding to game perspective 1 is displayed. Upon detecting interaction 1 indicating a user-indicated switch in game perspective, game interface 2 corresponding to game perspective 2 may be displayed. Game perspective 2 is different from game perspective 1.

[0120] For another example, the target application may be a live broadcast application. While the user is using the live broadcast application on their mobile phone to broadcast live, the mobile phone displays the live broadcast interface. Subsequently, upon detecting an interaction operation 1 in which the user indicates to add special effects, the live broadcast interface including the special effects may be displayed.

[0121] In addition, the above-mentioned interactive operation 1 can be a touch operation of the user on the touch screen of the mobile phone, such as sliding, clicking, and a combination of clicking and sliding. In other embodiments, the above-mentioned interactive operation 1 can also be other types of operations, such as eye movement interactive operations.

[0122] It is understood that in the scenario where interaction operation 1 is an eye movement interaction operation, after detecting the user's eye movement interaction operation, the input event (input) management module can report an eye movement interaction event to the target application. This eye movement interaction event is similar to a touch event corresponding to a touch operation. For example, the eye movement interaction event can include the location information of the gaze point (analogous to the touch location), the eye movement interaction type (analogous to the touch type), and the event time point (the time point when the eye movement interaction is reported).

[0123] In some embodiments, when the target application is running in the foreground of the mobile phone, the application interface corresponding to the target application can be displayed. After receiving the interaction operation 1, the target application can determine the interaction location point corresponding to the interaction operation 1 based on the touch event or eye movement interaction event corresponding to the interaction operation 1.

[0124] For example, the interaction point can be the touch position between the user and the touch screen, or the relative position on the application interface. For another example, the interaction point can also be the relative position of the user's gaze point on the application interface.

[0125] Then, based on the type of interaction operation 1, the interaction location, and the business rules of the target application, the mobile phone can draw and render one or more layers required to include the target graphical interface corresponding to interaction operation 1. In other words, it can obtain a target rendering frame (e.g., a first rendering frame) corresponding to interaction operation 1. The process of drawing the target rendering frame can be referred to in related technologies and will not be described in detail here.

[0126] S102: The mobile phone puts the target rendering frame into the SF cache queue.

[0127] Among them, the SF cache queue is used to cache the rendering frames that need to be synthesized by SF, and can also be called the first cache queue. In some embodiments, the rendering frames are arranged in the SF cache queue in the order of storage time. The earlier the storage time, the higher it is in the SF cache queue. In other possible embodiments, the mobile phone can also insert other rendering frames before a rendering frame in the SF cache queue. The embodiment of the present application does not specifically limit this. Every time SF receives a VSYNC signal, it takes out the rendering frame arranged in the first place from the SF cache queue, and performs image synthesis based on the rendering frame to generate the corresponding graphical interface.

[0128] S103 , in response to the VSYNC signal, the mobile phone takes out the target rendering frame from the SF buffer queue.

[0129] In some embodiments, the mobile phone executes the above S103 in different ways when the hand-following optimization function is turned on and when the hand-following optimization function is turned off.

[0130] In some embodiments, the hand-following optimization function is enabled when a specified type of application (e.g., a game or a pre-installed application) is running in the foreground of the mobile phone. In the above embodiment, if the target application belongs to the specified type of application, such as the target application satisfies the first condition, the hand-following optimization function is enabled when the target application is running in the foreground of the mobile phone. If the target application does not belong to the specified type of application, such as the target application does not satisfy the first condition, the hand-following optimization function is not enabled when the target application is running in the foreground of the mobile phone.

[0131] In an exemplary scenario, a mobile phone has a pre-installed application list. The first condition may include that the application identifier belongs to the above application list. In response to the user's instruction to run the target application in the foreground, the mobile phone obtains the application identifier (application name or package name) of the target application. If the application list includes the application identifier of the target application, the application is judged to be an application of the specified type, and the hand-following optimization function is enabled accordingly. If the application list does not include the application identifier of the target application, the application is judged to be not an application of the specified type, and the hand-following optimization function is not enabled accordingly.

[0132] In another exemplary scenario, the first condition may also include that the application service provided by the application belongs to a preset type. The preset type can be configured based on experience, for example, it can be a game type, a live broadcast type, etc., without specific limitation.

[0133] In response to a user's instruction to run a target application in the foreground, the mobile phone identifies the application service corresponding to the target application. If the application service corresponding to the target application belongs to the game category (or the live broadcast category), the mobile phone determines that the application is an application of the specified type and accordingly enables the hand-following optimization function. If the application service corresponding to the target application does not belong to the game category (or the live broadcast category), the mobile phone determines that the application is not an application of the specified type and accordingly does not enable the hand-following optimization function.

[0134] In other embodiments, if configuration item a corresponding to the mobile phone's hand-in-hand optimization function is selected and the target application is a specified type of application, the hand-in-hand optimization function is enabled. In the above embodiment, before S101, if the mobile phone has already set configuration item a to a selected state in response to the user's operation, the mobile phone will enable the hand-in-hand optimization function after determining that the target application running in the foreground is an application of the specified type. If configuration item a is not selected or the target application is not an application of the specified type, the hand-in-hand optimization function is not enabled.

[0135] When the mobile phone turns off the hand tracking optimization function, the SF in the mobile phone takes out the rendering frames one by one from the SF cache queue and synthesizes the graphical interface based on the taken-out rendering frames.

[0136] In an exemplary scenario, the SF cache queue only contains the target rendered frame. In this scenario, the mobile phone may execute S103 as follows: in response to the VSYNC signal, the SF of the mobile phone directly retrieves the target rendered frame from the SF cache queue, and the process proceeds to S104.

[0137] In another exemplary scenario, the SF cache queue contains a target rendered frame and rendered frame a. Rendered frame a is the rendered frame in the SF cache queue that precedes the target rendered frame. In this scenario, the mobile phone may execute S103 as follows: after synthesizing the graphical interface of rendered frame a, in response to the VSYNC signal, the mobile phone's SF retrieves the target rendered frame from the SF cache queue, and the process proceeds to S104.

[0138] like Figure 6 As shown, before the target rendered frame is stored in the SF cache queue, rendered frame (m), rendered frame (m+1), and rendered frame (m+2) are already cached in the SF cache queue. Rendered frame (m) and rendered frame (m+2) are interpolated frames, while rendered frame (m+1) is a real frame. m, m+1, and m+2 are serial numbers used to distinguish different rendered frames. The value of m is not specifically limited in this embodiment of the application.

[0139] Understandably, Figure 6 This is just an example. During actual operation, before the target rendering frame is stored in the SF cache queue, the SF cache queue may store more or fewer rendering frames. For example, there is no rendering frame in the SF cache queue. For another example, the SF cache queue only contains rendering frame (m+2).

[0140] In SF cache queue such as Figure 6 In the case shown, after the target rendering frame is stored in the SF cache queue, the SF cache queue includes rendering frame (m), rendering frame (m+1), rendering frame (m+2) and the target rendering frame.

[0141] In the SF cache queue, rendered frame (m) is ranked first. Rendered frame (m), rendered frame (m+1), and rendered frame (m+2) can be rendered frame a corresponding to the target rendered frame. If the SF cache queue contains multiple rendered frames, this is considered a rendered frame backlog in the SF cache queue.

[0142] like Figure 6 As shown in the figure, after receiving the i-th VSYNC signal, the SF takes the first rendered frame (m) from the SF cache queue and synthesizes the corresponding graphics interface. Where i is a positive integer greater than 1, and the i-th VSYNC signal is the first VSYNC signal received by the SF after the target rendered frame is stored in the SF cache queue.

[0143] In addition, after synthesizing the graphical interface corresponding to the rendering frame (m), the display screen of the mobile phone can display the graphical interface corresponding to the rendering frame (m).

[0144] Continue as Figure 6As shown, after rendering frame (m) is taken out from the SF cache queue, the SF cache queue includes rendering frame (m+1), rendering frame (m+2) and target rendering frame, wherein rendering frame (m+1) is arranged at the first position.

[0145] In response to the (i+1)th VSYNC signal, the SF extracts the first rendered frame (m+1) from the SF cache queue and synthesizes the corresponding graphical interface. Furthermore, after synthesizing the graphical interface corresponding to rendered frame (m+1), the phone's display can display the graphical interface corresponding to rendered frame (m+1). The (i+1)th VSYNC signal is the second VSYNC signal received after the target rendered frame is stored in the SF cache queue.

[0146] Continue as Figure 6 As shown, after rendering frame (m+1) is taken out from the SF cache queue, the SF cache queue includes rendering frame (m+2) and the target rendering frame. Among them, rendering frame (m+2) is ranked first. In response to the i+2th VSYNC signal, SF takes out the rendering frame (m+2) ranked first from the SF cache queue and synthesizes the corresponding graphical interface. Among them, the i+2th VSYNC signal is the third VSYNC signal received by SF after the target rendering frame is stored in the SF cache queue. In addition, after synthesizing the graphical interface corresponding to rendering frame (m+2), the mobile phone display can display the graphical interface corresponding to rendering frame (m+1).

[0147] Continue as Figure 6 As shown, after rendering frame (m+2) is removed from the SF cache queue, the SF cache queue includes the target rendering frame. The target rendering frame is ranked first. In response to the (i+3)th VSYNC signal, the SF removes the first-ranked target rendering frame from the SF cache queue, and the process proceeds to S104. The (i+3)th VSYNC signal is the fourth VSYNC signal received by the SF after the target rendering frame is stored in the SF cache queue.

[0148] in addition, Figure 6 Although only the process of storing the target rendered frame in the SF cache queue is shown, in actual operation, after the target rendered frame is stored in the SF cache queue, the mobile phone can also store rendered frame b in the SF cache queue. In this embodiment of the application, the process of storing rendered frame b in the SF cache queue is not described in detail. The display time of the above-mentioned rendered frame b is later than that of the target rendered frame.

[0149] In the scenario where the mobile phone has the hand tracking optimization function enabled, after the mobile phone stores the target rendering frame into the SF cache queue, the mobile phone may execute the above S103 in the following manner:

[0150] If the SF cache queue only contains the target rendering frame, that is, if there is no rendering frame accumulation problem in the SF cache queue, the SF directly takes out the target rendering frame from the SF cache queue in response to the VSYNC signal, and the process enters S104.

[0151] If the SF cache queue contains the target rendering frame and rendering frame a, that is, in the scenario where the SF cache queue has a rendering frame accumulation problem, the mobile phone can trigger the discarding of one or more rendering frames a in the SF cache queue, wherein the discarded rendering frame a can be called the second rendering frame.

[0152] In some embodiments, the mobile phone may discard the second rendered frame in the SF cache queue in response to the target rendered frame being stored in the SF cache queue.

[0153] In other embodiments, after the mobile phone stores the target rendered frame in the SF cache queue, it responds to the VSYNC signal to determine whether the SF cache queue contains the target rendered frame. If the target rendered frame is contained, the corresponding second rendered frame is discarded, and the rendered frame to be synthesized is retrieved from the SF cache queue (e.g., the target rendered frame). If the target rendered frame is contained but the second rendered frame is not, the discarding of the rendered frame in the SF cache queue is not triggered.

[0154] Understandably, SF only retrieves one rendered frame for synthesis within a VSYNC signal cycle. Obviously, the more rendered frames queued before the target rendered frame in the SF cache queue, the longer the waiting time for retrieval. In this embodiment of the present application, after the target rendered frame is stored in the SF cache queue, the second rendered frame is discarded, shortening the waiting time for the target rendered frame to be synthesized.

[0155] In some examples, the second rendered frame may be all interpolated frames in the rendered frame a.

[0156] like Figure 7 As shown, before the target rendered frame is stored in the SF cache queue, the SF cache queue already contains rendered frame (m), rendered frame (m+1), and rendered frame (m+2). After the target rendered frame is stored in the SF cache queue, the SF cache queue includes rendered frame (m), rendered frame (m+1), rendered frame (m+2), and the target rendered frame. In the SF cache queue, rendered frame (m) is ranked first. Rendered frame (m), rendered frame (m+1), and rendered frame (m+2) can be rendered frame a corresponding to the target rendered frame. Furthermore, rendered frame (m+2) and rendered frame (m) are the second rendered frames in rendered frame a.

[0157] like Figure 7 As shown, after the target rendering frame is stored in the SF cache queue, the SF can take out the rendering frame (m+2) and the rendering frame (m) from the SF cache queue and discard them.

[0158] Exemplarily, the SF may retrieve and discard the second rendered frame in response to the i-th VSYNC signal. In another exemplary embodiment, the SF may retrieve and discard the second rendered frame in response to the target rendered frame being stored in the SF cache queue.

[0159] After discarding rendered frame (m+2) and rendered frame (m), the SF cache queue contains rendered frame (m+1) and the target rendered frame, but does not contain rendered frame (m+2) and rendered frame (m). In addition, rendered frame (m+1) is arranged in the first place and the target rendered frame is arranged in the second place.

[0160] In addition, in response to the i-th VSYNC signal, the SF takes out the rendering frame (m+1) from the SF buffer queue and synthesizes the corresponding graphics interface.

[0161] Continue as Figure 7 As shown, after rendering frame (m+1) is removed from the SF cache queue, the SF cache queue includes the target rendering frame, and the target rendering frame is ranked first. In response to the (i+1)th VSYNC signal, the SF removes the first-ranked target rendering frame from the SF cache queue, and the process proceeds to S104.

[0162] contrast Figure 6 The scene of turning off the hand tracking optimization function is shown below. Figure 7 In the scenario shown in the figure, when the hand-tracking optimization function is enabled and there is a problem of rendering frame accumulation in the SF cache queue, the phone advances the VSYNC signal cycle by 2 and retrieves the target rendering frame from the SF cache queue after enabling the hand-tracking optimization function, thereby improving the latency T corresponding to interactive operation 1.

[0163] In other embodiments, if rendered frame a does not contain the second rendered frame, for example, if rendered frame a is entirely real, SF does not drop any frames, and SF retrieves rendered frames one by one from the SF cache queue and synthesizes the corresponding graphical interface to ensure display quality. After the target rendered frame is retrieved from the SF cache queue, the process proceeds to S104.

[0164] In other examples, the second rendering frame may be all rendering frames a.

[0165] like Figure 8As shown, before the target rendered frame is stored in the SF cache queue, the SF cache queue already contains rendered frame (m), rendered frame (m+1), and rendered frame (m+2). After the target rendered frame is stored in the SF cache queue, the SF cache queue includes rendered frame (m), rendered frame (m+1), rendered frame (m+2), and the target rendered frame. In the SF cache queue, rendered frame (m) is ranked first. Rendered frame (m), rendered frame (m+1), and rendered frame (m+2) can be rendered frame a corresponding to the target rendered frame, that is, they are all second rendered frames.

[0166] like Figure 8 As shown in FIG, after the target rendered frame is stored in the SF cache queue, the SF can retrieve rendered frame (m), rendered frame (m+1), and rendered frame (m+2) from the SF cache queue and discard them. After discarding rendered frame (m), rendered frame (m+1), and rendered frame (m+2), the SF cache queue contains the target rendered frame but does not contain rendered frame (m), rendered frame (m+1), and rendered frame (m+2). In addition, the target rendered frame is ranked first.

[0167] In response to the i-th VSYNC signal, the SF takes out the target rendering frame from the SF buffer queue, and the process enters S104.

[0168] contrast Figure 6 The scene of turning off the hand tracking optimization function is shown below. Figure 8 In the scenario shown below, when the hand-tracking optimization function is enabled and there is a problem of rendering frame accumulation in the SF cache queue, the phone advances the target rendering frame by 3 VSYNC signal cycles after enabling the hand-tracking optimization function, thereby improving the latency T corresponding to interactive operation 1.

[0169] In other examples, the second rendering frame may be an interpolated frame that precedes the target rendering frame.

[0170] like Figure 9 As shown, before the target rendering frame is stored in the SF cache queue, the SF cache queue has already cached the rendering frame (m+2), and the rendering frame (m+2) is an interpolated frame.

[0171] After the target rendered frame is stored in the SF cache queue, the SF cache queue now contains rendered frame (m+2) and the target rendered frame. In the SF cache queue, rendered frame (m+2) is ranked first, and the target rendered frame is ranked second. Rendered frame (m+2) precedes the target rendered frame, meaning that rendered frame (m+2) is the second rendered frame relative to the target rendered frame.

[0172] like Figure 9As shown, after the target rendered frame is stored in the SF cache queue, the SF can take out the rendered frame (m+2) from the SF cache queue and discard it. After discarding the rendered frame (m+2), the SF cache queue contains the target rendered frame and is ranked first.

[0173] In response to the i-th VSYNC signal, the SF takes out the target rendering frame from the SF buffer queue, and the process goes to S104.

[0174] like Figure 10 As shown, before the target rendering frame is stored in the SF cache queue, the SF cache queue has already cached the rendering frame (m+1), and the rendering frame (m+1) is a real frame.

[0175] After the target rendered frame is stored in the SF cache queue, the SF cache queue now contains rendered frame (m+1) and the target rendered frame. In the SF cache queue, rendered frame (m+1) is ranked first, and the target rendered frame is ranked second. Rendered frame (m+1) precedes the target rendered frame. Because rendered frame (m+1) is not an interpolated frame, it is not the second rendered frame corresponding to the target rendered frame.

[0176] like Figure 10 As shown, after the target rendering frame is stored in the SF cache queue, in response to the i-th VSYNC signal, the SF takes out the rendering frame (m+1) from the SF cache queue and synthesizes the corresponding graphics interface.

[0177] After rendering frame (m+1) is removed from the SF cache queue, the SF cache queue includes the target rendering frame, which is ranked first. In response to the (i+1)th VSYNC signal, the SF cache queue removes the target rendering frame ranked first from the SF cache queue, and the process proceeds to S104.

[0178] In other examples, the second rendering frame may also be a rendering frame that precedes the target rendering frame. Figure 9 In the scenario shown, after the target rendered frame is stored in the SF cache queue, the rendered frame (m+2) may be discarded. For another example, Figure 10 In the scenario shown, after the target rendered frame is stored in the SF cache queue, the discarded rendered frame (m+1) may also be triggered.

[0179] S104, the mobile phone synthesizes a target graphical interface based on the target rendering frame.

[0180] In some embodiments, after the mobile phone's SF retrieves the target rendered frame from the SF cache queue, the SF may synthesize a corresponding graphical interface based on the target rendered frame, such as a target graphical interface corresponding to the target rendered frame. The target graphical interface may be the graphical interface required to be displayed in response to interaction operation 1.

[0181] S105, the mobile phone displays the target graphical interface.

[0182] In some embodiments, the phone can send the synthesized target graphical interface to the display driver, which instructs the touch screen to display the target graphical interface. This approach allows for the SF cache queue to accumulate rendered frames, and by dropping frames, shortens the delay T between the user performing interaction 1 and actually seeing the target graphical interface, improving the phone's hand tracking.

[0183] In other embodiments, the aforementioned interactive operation 1 may be an interactive operation that meets preset conditions. The mobile phone can filter detected interactive operations based on the preset conditions and select interactive operation 1 suitable for hand-following optimization. This avoids frequent frame drops in scenarios where the user frequently interacts with the electronic device, ensuring display quality.

[0184] For example, the preset condition may be the first interactive operation within any enabling period. After the hand-following optimization function is enabled, the time period may be divided into multiple enabling periods according to the enabling interval information (e.g., the enabling interval is 5 seconds) until the mobile phone turns off the hand-following optimization function.

[0185] like Figure 11 As shown, after the hand-following optimization function is turned on, for example, it is detected that the target application is running in the foreground and the target application is an application of a specified type, or after the configuration item a corresponding to the hand-following optimization function is set to the selected state in response to a user operation, the timeline is divided into multiple enabling periods according to the enabling interval information (e.g., the enabling interval is 5S). Interaction operation a (e.g., the first operation, the third operation, or the fourth operation) and interaction operation b (e.g., the second operation) are interaction operations received in the same enabling period, and both can affect changes in the application interface. Among them, the time point when interaction operation a is detected is earlier than the time point when interaction operation b. In this scenario, interaction operation a meets the preset conditions and can be called interaction operation 1. Interaction operation b does not meet the preset conditions. Accordingly, after detecting interaction operation a, the mobile phone can execute the above steps S101 to S105. During the execution of the above steps, after the target rendering frame a corresponding to interaction operation a is stored in the SF cache queue, if there is a problem of rendering frame accumulation in the SF cache queue, the second rendering frame in the SF cache queue that is arranged before the target rendering frame a is discarded. After detecting interaction operation b, the mobile phone may also perform steps S101 to S105 above. The difference is that during the execution of the above steps, after storing the target rendered frame b corresponding to interaction operation b in the SF cache queue, if there is a problem of rendered frame accumulation in the SF cache queue, the rendered frames in the SF cache queue will not be discarded. The mobile phone can retrieve the rendered frames in the SF cache queue one by one and synthesize them into the corresponding graphical interface.

[0186] For example, the preset condition may also be that the time interval between the previous interaction operation 1 and the previous interaction operation 1 is greater than or equal to the enabling time interval.

[0187] like Figure 12 As shown, after the hand-following optimization function has been turned on, the mobile phone receives an interactive operation c (also referred to as the first operation) at time point 1. Time point 2 is a time point before time point 1. The time interval between time point 2 and time point 1 is equal to the enabling time interval. Between time point 2 and time point 1, the mobile phone does not receive other interactive operations, and it can be determined that the interactive operation c is an interactive operation 1 that meets the preset conditions. Accordingly, after detecting the interactive operation c, the mobile phone can execute the above S101 to S105. In the process of executing the above steps, after storing the target rendering frame c corresponding to the interactive operation c into the SF cache queue, if the SF cache queue has a problem of rendering frame accumulation, it triggers the discarding of the second rendering frame in the SF cache queue that is arranged before the target rendering frame c.

[0188] The mobile phone receives interactive operation d (e.g., a second operation) at time point 3. The time interval between time point 3 and time point 1 is less than the enabling time interval, and interactive operation d does not meet the preset conditions. After detecting interactive operation d, the mobile phone can also execute the above steps S101 to S105. The difference is that during the execution of the above steps, after the target rendering frame d corresponding to interactive operation d is stored in the SF cache queue, if there is a problem of rendering frame accumulation in the SF cache queue, the rendering frame in the SF cache queue is not discarded. The mobile phone can extract the rendering frames in the SF cache queue one by one and synthesize them into the corresponding graphical interface.

[0189] The mobile phone receives an interactive operation e (such as the third operation or the fourth operation) at time point 4. The time interval between time point 4 and time point 1 is greater than the enabling time interval, and the time interval between time point 4 and time point 3 is less than the enabling time interval. It can be understood that for interactive operation e, the previous interactive operation 1 is interactive operation c. At the same time, the time interval between receiving interactive operation e and receiving interactive operation c is greater than the enabling time interval. It can be determined that interactive operation c is also an interactive operation 1 that meets the preset conditions. Accordingly, after detecting interactive operation e, the mobile phone can execute the above S101 to S105. In the process of executing the above steps, after storing the target rendering frame e corresponding to interactive operation e into the SF cache queue, if there is a problem of rendering frame accumulation in the SF cache queue, it triggers the discarding of the second rendering frame in the SF cache queue that is arranged before the target rendering frame e.

[0190] As a way to implement Figure 13As shown, the mobile phone includes a hand tracking management module, an application type management module, an interpolation management module, an input event management module (input), a rendering library, a target application and SF.

[0191] For example, the hand tracking management module, rendering library, interpolation management module, and SF are functional modules running in the local layer. Furthermore, the phone can create an SF cache queue for SF. The input event management module (input) and application type management module are functional modules running in the framework layer, and the target application is the application running in the application layer.

[0192] like Figure 13 As shown, the mobile phone executes Figure 5 During the method shown, the signaling interactions between the various functional modules and applications in the mobile phone are as follows:

[0193] S201: The hand tracking management module queries the application type management module for type information of the application program running in the foreground.

[0194] In some embodiments, the hand tracking management module can send a query request to the application type management module in response to the user's instruction to run an application in the foreground. The application type management module responds to the query request to determine the type of application running in the foreground, such as game, video, live broadcast, etc., or preset, non-preset, etc.

[0195] Then, the application type management module sends the type information of the application running in the foreground to the hand tracking management module. The type information carries the type of the application running in the foreground. If the application running in the foreground is the target application, the process proceeds to S202.

[0196] S202: When the target application is running in the foreground, the application type management module sends type information corresponding to the target application to the hand tracking management module.

[0197] S203: The hand-following management module determines whether to enable the hand-following optimization function based on the type information corresponding to the target application.

[0198] In some embodiments, the hand tracking management module can determine whether to enable the hand tracking optimization function according to the preconfigured rules for enabling and disabling the hand tracking optimization function and the type information corresponding to the target application. The judgment process can be referred to the aforementioned embodiment.

[0199] If it is determined that the hand-following optimization function needs to be enabled, the process proceeds to S204. If it is determined that the hand-following optimization function does not need to be enabled, the process ends. Furthermore, after the hand-following optimization function is enabled, the hand-following management module can also disable the hand-following optimization function in response to the target application exiting the foreground.

[0200] S204: The hand-following management module instructs the input event management module (input) to execute an initialization process related to the hand-following optimization function.

[0201] In some embodiments, after the hand-following optimization function executes an initialization process related to the hand-following optimization function, the input event management module (input) may participate in the implementation process of the hand-following optimization function.

[0202] S205: The input event management module (input) registers a callback for obtaining information related to hand-following optimization with the hand-following management module, wherein the information related to hand-following optimization may include the on / off state and enabling time interval of the hand-following optimization function.

[0203] In some embodiments, the enabling time interval is derived from the enabling time interval information read by the hand-following management module. For example, the enabling time interval is 5 seconds. When the hand-following optimization function is enabled, the switch state of the hand-following optimization function is enabled. When the hand-following optimization function is disabled, the switch state of the hand-following optimization function may be disabled.

[0204] S206: The hand-following management module returns the hand-following optimization related information to the input event management module (input) through callback.

[0205] S207 , the interpolation frame management module adds a frame tag to the rendered frame rendered by the rendering library, wherein the frame tag is used for interpolation frames and real frames.

[0206] In some embodiments, if the rendering library renders rendered frame x based on graphics data from the interpolation frame management module, the interpolation frame management module assigns a tag indicating an interpolated frame to rendered frame x, such as a first tag. If the rendering library renders rendered frame y based on graphics data from the target application, the interpolation frame management module assigns a tag indicating a real frame to rendered frame y, such as a second tag.

[0207] In some embodiments, there is no necessary relationship between the above S207 and other steps.

[0208] Exemplarily, after the hand-following optimization function is enabled, the interpolation management module executes S207 once each time the rendering library renders a rendering frame, or the interpolation management module executes S207 once each time it instructs the rendering library to render a rendering frame.

[0209] As another example, during normal operation of the mobile phone, the frame insertion management module executes S207 once each time the rendering library renders a rendering frame, or the frame insertion management module executes S207 once each time it instructs the rendering library to render a rendering frame.

[0210] S208, the input event management module (input) detects the interactive operation and generates a corresponding interactive event 1.

[0211] Among them, the above-mentioned interaction event 1 can be a touch event or an eye movement interaction event, which will not be elaborated in the embodiment of the present application.

[0212] S209, the input event management module (input) reports the interaction event 1 corresponding to the interaction operation to the target application.

[0213] S210 , the target application draws a target rendering frame corresponding to the interaction event 1 .

[0214] In some embodiments, the interaction event 1 is information used to describe the interaction operation, and there is a corresponding relationship between the interaction event 1 and the interaction operation. The target rendering frame corresponding to the interaction event 1 can also be called the target rendering frame corresponding to the interaction operation.

[0215] S211 , the target application instructs the rendering library to render a target rendering frame.

[0216] S212, when the interactive event 1 meets the preset requirements, the input event management module (input) sends information 1 to the rendering library.

[0217] It is understandable that there is a corresponding relationship between the above-mentioned interaction event 1 and the interaction operation. When the interaction operation meets the preset conditions mentioned in the above embodiment, the corresponding interaction event 1 meets the preset requirements.

[0218] Exemplarily, when the interaction operation indicated by interaction event 1 is the first interaction operation in an enablement cycle, interaction event 1 meets the preset requirement. Furthermore, exemplarily, when the time interval between the interaction operation indicated by interaction event 1 and the previous interaction operation that meets the preset condition is not less than the enablement time interval, interaction event 1 meets the preset requirement.

[0219] After the input event management module (input) determines that the interaction event 1 meets the preset requirements, it triggers the start of the chirality optimization process by sending the above information 1 to the rendering library, for example, triggering the rendering library to execute S213.

[0220] In addition, there is no necessary order between the above S209 and S212, and the embodiments of the present application do not make specific limitations on this.

[0221] S213 : The rendering library adds a tag 1 to the rendered target rendering frame in response to the information 1 .

[0222] The mark 1 is used to distinguish the target rendering frame from other rendering frames, and is used to indicate that the target rendering frame is a rendering frame for which the synthesis processing delay needs to be shortened.

[0223] Exemplarily, the tag 1 can be added to the frame information of the target rendered frame. For example, before the tag 1 is added, the designated byte in the frame information of the target rendered frame is empty. Before the tag 1 is added, the designated byte in the frame information of the target rendered frame carries the designated character, wherein the type of the designated character is not specifically limited in this embodiment of the application.

[0224] S214: The rendering library stores the target rendered frame into the SF cache queue.

[0225] S215, the rendering library sends information 2 to SF.

[0226] The above information 2 (such as called the first information) may carry a Drop flag (such as called the target flag), which may be used to trigger the SF to detect the SF cache queue.

[0227] In some embodiments, SF may respond to information 2 and the process proceeds to S216. In other embodiments, S215 may be skipped. Accordingly, SF may respond to the VSYNC signal and the process proceeds to S216.

[0228] S216, the SF detects the SF cache queue.

[0229] In some embodiments, the above S216 may be: detecting whether there is a pile of rendering frames in the SF cache queue. If there is a pile of rendering frames, continue to detect whether there is a target rendering frame carrying a mark 1 in the SF cache queue.

[0230] It is understood that if the SF cache queue includes multiple rendered frames, there is a rendering frame backlog in the SF cache queue. If the SF cache queue includes only one rendered frame, there is no rendering frame backlog in the SF cache queue. As an implementation method, the SF can determine whether there is a rendering frame backlog in the SF cache queue by querying the number of rendered frames in the SF cache queue.

[0231] When it is detected that the SF cache queue has a pile of rendering frames and contains a target rendering frame carrying a flag 1, the process proceeds to S217.

[0232] In other embodiments, S216 may also be performed by simply checking whether there is any accumulation of rendered frames in the SF cache queue. If the SF cache queue is found to be accumulated, the process proceeds to S217. Alternatively, only the SF cache queue is checked for any target rendered frames carrying a flag of 1. If the SF cache queue contains any target rendered frames carrying a flag of 1, the process proceeds to S217. This is not specifically limited in this embodiment of the present application.

[0233] In a possible embodiment, S216 may be skipped, and after the SF receives the information 2, S217 is directly executed.

[0234] S217 : The SF determines a second rendering frame from the rendering frames in the SF cache queue based on the frame tag and the target rendering frame.

[0235] In some embodiments, when the frame tag of the target rendered frame indicates that the target rendered frame is a real frame, the SF determines the second rendered frame from the SF buffer queue.

[0236] If the second rendered frame is an interpolated frame, SF needs to identify the second rendered frame by combining the frame tag of rendered frame a (the frame preceding the target rendered frame in the SF cache queue) with the frame tag of rendered frame a. If the second rendered frame is either an interpolated frame or a real frame, SF can still identify the second rendered frame without combining the frame tag of rendered frame a. The process of identifying the second rendered frame can be referred to the previous embodiment and will not be further described here.

[0237] In other embodiments, if the SF buffer queue does not contain the second rendering frame, the process ends.

[0238] S218: The SF takes out the second rendered frame from the SF cache queue and discards it.

[0239] In some embodiments, through the above S214 , S215 , S216 , S217 and S218 , the above embodiment mentioned can be implemented: in response to the target rendered frame being stored in the SF cache queue, triggering the discarding of the second rendered frame.

[0240] In some embodiments, through the above S214 and in response to the VSYNC signal, the process enters S216, S217 and S218, which can achieve the above embodiment mentioned: triggering the discarding of the second rendering frame in response to the VSYNC signal.

[0241] After discarding the second rendered frame, the time the target rendered frame waits to be synthesized is shortened, and the latency T corresponding to the interactive operation is also shortened. This improves the problem of increased latency T and decreased chirality during mobile phone operation, thereby improving the human-computer interaction efficiency of the mobile phone.

[0242] Some embodiments of the present application further provide an electronic device, which may include a memory and one or more processors. The memory and processors are coupled. The memory is configured to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device may perform the functions or steps described above in the method embodiments.

[0243] In one exemplary scenario, the electronic device receives a first operation from a user. At a first time point after receiving the first operation, the electronic device stores a first rendered frame (also referred to as a target rendered frame) corresponding to the first operation into a first cache queue. After the first rendered frame is stored in the first cache queue, the first cache queue includes the first rendered frame and a second rendered frame, with the second rendered frame arranged before the first rendered frame.

[0244] The electronic device synthesizes and displays a first graphical interface corresponding to the first rendering frame in response to a first vertical synchronization signal, wherein the first vertical synchronization signal is the first vertical synchronization signal generated after the first time point, such as the i-th VSYNC signal.

[0245] As an implementation method, the electronic device can draw first graphic data in response to the first operation. The first graphic data may include one or more layers of data drawn. The content of the first graphic data is affected by the first operation. For example, when the operation type and / or operation position of the first operation is different, the generated first graphic data may also be different. Based on the first graphic data, the electronic device can render a first rendering frame. The first rendering frame includes one or more rendered layers of data. Thereafter, the electronic device can synthesize the corresponding first graphical interface based on the first rendering frame.

[0246] In another exemplary scenario, the electronic device receives a second operation from the user. The first reception time point of the first operation may be the event time point in the interaction event (first event) corresponding to the first operation. The first event may further include type information describing the first operation (eye movement interaction type or touch type) and operation location information (touch location or gaze point location information). The second reception time point of the second operation may be the event time point in the interaction event corresponding to the second operation.

[0247] In the above scenario, the time interval between the first receiving time point and the second receiving time point is less than the first threshold (that is, the enabling time interval mentioned in the above embodiment); or, the first receiving time point and the second receiving time point belong to the same enabling cycle, and the first receiving time point is earlier than the second receiving time point.

[0248] At a second time point after receiving the second operation, the electronic device stores the third rendering frame (also referred to as the target rendering frame) corresponding to the second operation into the first cache queue; wherein, after the third rendering frame is stored in the first cache queue, the first cache queue includes the third rendering frame and the fourth rendering frame, and the fourth rendering frame is arranged before the third rendering frame.

[0249] In response to the second vertical synchronization signal, the electronic device retrieves the fourth rendered frame from the first cache queue, synthesizes and displays the second graphical interface corresponding to the fourth rendered frame. The second vertical synchronization signal is the first vertical synchronization signal generated after the second time point, such as the i-th VSYNC signal.

[0250] In response to the third vertical synchronization signal, the electronic device retrieves the third rendered frame from the first cache queue, synthesizes, and displays a third graphical interface corresponding to the third rendered frame. The third vertical synchronization signal is a second vertical synchronization signal generated after the second time point, such as the (i+1)th VSYNC signal.

[0251] In another exemplary scenario, the electronic device receives a third user operation. The third reception time point of the third operation may be an event time point in the interaction event corresponding to the third operation. The time interval between the third reception time point and the first reception time point is greater than or equal to a first threshold; or the third reception time point and the first reception time point belong to different enabling cycles.

[0252] At a third time point after receiving the third operation, the fifth rendered frame (also referred to as the target rendered frame) corresponding to the third operation is stored in the first cache queue. After the fifth rendered frame is stored in the first cache queue, the first cache queue includes the fifth rendered frame and the sixth rendered frame. The sixth rendered frame is arranged before the fifth rendered frame. The sixth rendered frame is determined to be discardable. The method for determining the sixth rendered frame can be referenced to the method for determining the second rendered frame and is not further described here.

[0253] The electronic device may also retrieve a fifth rendered frame from the first cache queue in response to a fourth vertical synchronization signal. The fourth vertical synchronization signal is the first vertical synchronization signal generated after the third time point, such as the i-th VSYNC signal. The electronic device synthesizes and displays a fourth graphical interface corresponding to the fifth rendered frame.

[0254] Combining scenarios three and four, we can see that electronic devices will filter interactive operations. For example, after the first operation triggers frame drop, the time interval between the third operation and the first operation is long, and the third operation can also trigger frame drop. The time interval between the second operation and the first operation is short, and no frame drop is triggered. In this way, frequent frame drop is avoided, which affects the picture quality.

[0255] In another exemplary scenario, the electronic device receives a fourth user operation. The fourth reception time point of the fourth operation may be an event time point in the interaction event corresponding to the fourth operation. The time interval between the fourth reception time point and the first reception time point is greater than or equal to a first threshold; or the fourth reception time point and the first reception time point belong to different enabling cycles.

[0256] At a fourth time point after receiving the fourth operation, the seventh rendered frame corresponding to the fourth operation is stored in the first cache queue. The seventh rendered frame is the target rendered frame corresponding to the fourth operation. After the seventh rendered frame is stored in the first cache queue, the first cache queue includes the seventh rendered frame, the eighth rendered frame, and the ninth rendered frame. The eighth and ninth rendered frames are arranged before the seventh rendered frame and are both rendered frame a corresponding to the seventh rendered frame.

[0257] In response to the fifth vertical synchronization signal, the electronic device retrieves the ninth rendered frame from the first cache queue, synthesizes, and displays the fifth graphical interface corresponding to the ninth rendered frame. The fifth vertical synchronization signal is the first vertical synchronization signal generated after the fourth time point, such as the i-th VSYNC signal mentioned in the aforementioned embodiment.

[0258] In response to the sixth vertical synchronization signal, the electronic device retrieves the seventh rendered frame from the first cache queue, synthesizes, and displays a sixth graphical interface corresponding to the seventh rendered frame. The sixth vertical synchronization signal is a second vertical synchronization signal generated after the fourth time point, such as the (i+1)th VSYNC signal mentioned in the aforementioned embodiment.

[0259] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes the various functions or steps executed by the mobile phone in the above-mentioned method embodiment.

[0260] An embodiment of the present application further provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the functions or steps executed by the mobile phone in the above method embodiment.

[0261] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0262] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0263] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0264] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0265] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0266] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A display processing method, characterized in that: include: Receiving the user's first operation; At a first time point, storing a first rendered frame corresponding to the first operation into a first cache queue; wherein, after the first rendered frame is stored in the first cache queue, the first cache queue includes the first rendered frame and a second rendered frame, and the second rendered frame is arranged before the first rendered frame; In response to a first vertical synchronization signal, synthesizing a first graphical interface corresponding to the first rendering frame, wherein the first vertical synchronization signal is the first vertical synchronization signal generated after the first time point; The first graphical interface is displayed.

2. The method according to claim 1, characterized in that Before synthesizing the first graphical interface corresponding to the first rendering frame, the method further includes: Taking out the second rendering frame from the first cache queue and discarding it; In response to a first vertical synchronization signal, the first rendering frame is retrieved from the first buffer queue.

3. The method according to claim 1 or 2, characterized in that The second rendered frame includes a first tag, where the first tag indicates that the second rendered frame is an interpolated frame.

4. The method according to any one of claims 1 to 3, characterized in that Before synthesizing the first graphical interface corresponding to the first rendering frame in response to the first vertical synchronization signal, the method further includes: It is determined that the first rendered frame includes a second tag, where the second tag indicates that the first rendered frame is a real frame.

5. The method according to any one of claims 1 to 4, characterized in that The method further comprises: A second operation of the user is received; wherein the time interval between a first receiving time point of the first operation and a second receiving time point of the second operation is less than a first threshold; or, the first receiving time point and the second receiving time point belong to the same enabling period, and the first receiving time point is earlier than the second receiving time point; At a second time point, storing a third rendered frame corresponding to the second operation into the first cache queue; wherein, after the third rendered frame is stored in the first cache queue, the first cache queue includes the third rendered frame and a fourth rendered frame, and the fourth rendered frame is arranged before the third rendered frame; In response to a second vertical synchronization signal, synthesizing a second graphical interface corresponding to the fourth rendering frame; wherein the second vertical synchronization signal is the first vertical synchronization signal generated after the second time point; displaying the second graphical interface; In response to a third vertical synchronization signal, synthesizing a third graphical interface corresponding to the third rendering frame; wherein the third vertical synchronization signal is a second vertical synchronization signal generated after the second time point; The third graphical interface is displayed.

6. The method according to any one of claims 1 to 4, characterized in that The method further comprises: A third operation of the user is received, where the time interval between a third receiving time point of the third operation and a first receiving time point of the first operation is greater than or equal to a first threshold; or, the third receiving time point and the first receiving time point belong to different enabling cycles; At a third time point, storing a fifth rendered frame corresponding to the third operation into the first cache queue; wherein, after the fifth rendered frame is stored in the first cache queue, the first cache queue includes the fifth rendered frame and a sixth rendered frame, and the sixth rendered frame is arranged before the fifth rendered frame; In response to a fourth vertical synchronization signal, synthesizing a fourth graphical interface corresponding to the fifth rendering frame; wherein the fourth vertical synchronization signal is the first vertical synchronization signal generated after the third time point; The fourth graphical interface is displayed.

7. The method according to any one of claims 1 to 4, characterized in that The method further comprises: A fourth operation of the user is received, where the time interval between a fourth receiving time point of the fourth operation and a first receiving time point of the first operation is greater than or equal to a first threshold; or, the fourth receiving time point and the first receiving time point belong to different enabling periods; At a fourth time point, storing a seventh rendered frame corresponding to the fourth operation into the first cache queue; wherein, after the seventh rendered frame is stored in the first cache queue, the first cache queue includes the seventh rendered frame, the eighth rendered frame, and the ninth rendered frame, and the eighth rendered frame and the ninth rendered frame are arranged before the seventh rendered frame; In response to a fifth vertical synchronization signal, synthesizing a fifth graphical interface corresponding to the ninth rendering frame; wherein the fifth vertical synchronization signal is the first vertical synchronization signal generated after the fourth time point; displaying the fifth graphical interface; In response to a sixth vertical synchronization signal, synthesizing a sixth graphical interface corresponding to the seventh rendering frame; wherein the sixth vertical synchronization signal is a second vertical synchronization signal generated after the fourth time point; The sixth graphical interface is displayed.

8. The method according to claim 7, characterized in that When the first cache queue includes the seventh rendering frame, the eighth rendering frame, and the ninth rendering frame, the ninth rendering frame is arranged before the eighth rendering frame.

9. The method according to claim 7, characterized in that The ninth rendered frame includes a second tag, the second tag indicating that the ninth rendered frame is a real frame, and the eighth rendered frame includes a first tag, the first tag indicating that the eighth rendered frame is an interpolated frame.

10. The method according to any one of claims 1 to 9, characterized in that: Before receiving the first operation of the user, the method further includes: In response to a user's operation of opening a target application, running the target application in the foreground; It is determined that the target running program meets a first condition.

11. The method according to claim 10, characterized in that The first condition includes: the application identifier of the target application is included in a pre-configured application list; or the application service provided by the target application belongs to a preset type.

12. The method according to claim 10, characterized in that The method is applied to an electronic device, the electronic device including an input event management module and a rendering library. After receiving a first operation from a user, the method includes: The input event management module generates a first event corresponding to the first operation, wherein the first event includes type information and operation position information for describing the first operation; The input event management module reports the first event to the target application; The target application draws first graphic data and transmits the first graphic data to the rendering library; wherein the content of the first graphic data is related to the type information and / or operation position information of the first operation; The rendering library renders the first rendering frame based on the first graphics data; The storing the first rendering frame corresponding to the first operation into the first cache queue includes: the rendering library storing the first rendering frame into the first cache queue.

13. The method according to claim 12, characterized in that The electronic device further includes an image synthesizer, and the first event further includes a first receiving time point of the first operation; After storing the first rendering frame corresponding to the first operation into the first cache queue, the method further includes: When the first event meets a preset requirement, the rendering library sends first information to the image synthesizer, where the first information includes a target flag bit; The image synthesizer takes out the second rendering frame from the first cache queue in response to the target flag bit in the first information and discards the second rendering frame; The preset requirement includes: the time interval between the first receiving time point and the receiving time point of the previous adjacent interactive operation is greater than or equal to a first threshold.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 13.

15. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

16. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images