Data processing method and related apparatus

Abstract

The embodiments of the present application relate to the technical field of terminals. Provided are a data processing method and a related apparatus. The method comprises: synthesizing an interface of a first application according to a first synthesis policy, wherein the first synthesis policy is used for instructing to perform synthesis on images, the drawing for which has been completed, in a buffer queue; in response to a first operation, enabling a dynamic effect; during the dynamic effect, synthesizing an interface of the dynamic effect according to a second synthesis strategy, wherein the interface of the dynamic effect comprises an image of the first application and an image of a second application, the second synthesis policy is used for instructing to perform synthesis on second-type images when the rendering for first-type images has not been completed; when the drawing for the first-type images is completed and the rendering therefor is completed, performing synthesis on the first-type images and the second-type images, wherein the first-type images are images of a first-type application, and the second-type images are not the images of the first-type application; and at the end of the dynamic effect, synthesizing an interface of the second application according to the first synthesis policy. In this way, display lags caused by a long image rendering time of a first-type application in a dynamic effect scenario can be reduced.

Description

Data processing method and related apparatus

[0001] The present application claims priority to the Chinese patent application No. 202410904472.3, filed on July 5, 2024, and entitled "Data processing method and related apparatus", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of terminal, and in particular, to a data processing method and related apparatus. BACKGROUND

[0003] Currently, an electronic device can display an animation effect when switching between applications. For example, when displaying a home interface, the electronic device can display an animation effect of entering a first application in response to a triggering operation of a user on an icon of the first application, and display an interface of the first application after the animation effect ends. The animation effect refers to a dynamic change effect of elements in the interface.

[0004] However, during the display of the animation effect, the electronic device can have a lag phenomenon. SUMMARY

[0005] Embodiments of the present application provide a data processing method and related apparatus, which are applied to the technical field of terminal. The method is used to reduce display lag in an animation effect scenario.

[0006] In a first aspect, a data processing method is provided. The method includes: at a first time, receiving, at a desktop, an operation for starting a first application; at a second time, starting, by a desktop application, drawing of an Nth frame of an image of the desktop application, and starting, by the first application, drawing of an Mth frame of an animation effect image of the first application, N and M being integers greater than zero; at a third time, completing the drawing of the Nth frame of the image, completing the drawing of the Mth frame of the animation effect image, and not completing rendering of the Mth frame of the animation effect image, synthesizing the Nth frame of the image, and not synthesizing the Mth frame of the animation effect image; at a fourth time, completing, by the desktop application, drawing of an (N+i)th frame of the image and not completing rendering of the (N+i)th frame of the image, completing rendering of the Mth frame of the animation effect image, and synthesizing the (N+i)th frame of the image and the Mth frame of the animation effect image, i being an integer greater than zero; and at a fifth time, ending the animation effect of the first application, completing, by the first application, drawing of an Lth frame of an image at a running time of the first application and not completing rendering of the Lth frame of the image, and synthesizing the Lth frame of the image. The second time is later than the first time, the third time is later than the second time, the fourth time is later than the third time, and the fifth time is later than the fourth time.

[0007] The first application can be any application, for example, a game application, a map application, a video editing application, etc., which is not limited here. The frame rate of the first application can be the same as or different from the frame rate of the desktop application, which is not limited here. The desktop can correspond to the home interface in the following.

[0008] In some embodiments, the first application can be an application with a frame rate lower than the refresh rate of the motion effect scene; the first application can be an application with a long single-frame image rendering time (for example, a game application). Here, it is not limited.

[0009] It can be understood that the Nth frame image can correspond to the image of the desktop application in the following, for example, frame 1B, frame 2B, etc.; the Mth motion effect image can correspond to the image of the game application in the following, for example, frame 1A, frame 2A, etc.

[0010] It can be understood that during the motion effect process, the synthesis of the image of the first application is after the image rendering is completed, so that the display lag phenomenon caused by the long image rendering time of the first application can be reduced.

[0011] In a possible implementation, at a third moment, the rendering of the Nth frame image is not completed.

[0012] It can be understood that during the motion effect process, the rendering and synthesis of the image of the desktop application can be performed synchronously; the synthesis of the image of the first application is after the image rendering is completed.

[0013] In this way, the rendering and synthesis can be performed synchronously, which can reduce the image synthesis waiting time of the desktop application and improve the response speed of the image display of the desktop application.

[0014] In a possible implementation, the electronic device includes: an application list of the first type of application; the application list is used to indicate that in the motion effect scene, the images of the first type of application are all synthesized after the rendering is completed.

[0015] The first application is the first type of application, and the desktop application is not the first type of application.

[0016] In this way, the electronic device can control the synthesis mode of the images of different applications in the motion effect scene by setting the application list of the first type of application.

[0017] In a possible implementation, the electronic device comprises: an image synthesis system, the image synthesis system being configured to synthesize images; after the first time point, the method further comprises: before the third time point, the image synthesis system generates a first Vsync signal, the first Vsync signal being configured to trigger the image synthesis system to synthesize images; after the image synthesis system generates the first Vsync signal, the image synthesis system confirms, by using the application list, that the Nth image is not an image of the first type of application, the Mth motion effect image is an image of the first type of application, and the Mth motion effect image is not rendered; before the fourth time point, the image synthesis system generates a second Vsync signal, the second Vsync signal being configured to trigger the image synthesis system to synthesize images; after the image synthesis system generates the second Vsync signal, the image synthesis system confirms, by using the application list, that the N+i th image is not an image of the first type of application, the Mth motion effect image is an image of the first type of application, and the Mth motion effect image is rendered.

[0018] It can be understood that the first Vsync signal and the second Vsync signal can both be Vsync-SF signals.

[0019] In this way, the scheme can be applicable to a scenario in which the Vsync signal triggers synthesis.

[0020] In a possible implementation, the electronic device comprises: a GPU driver of an image processing unit (GPU), the GPU driver being configured to drive the GPU to render a motion effect image of a first application; after the first Vsync signal, the image synthesis system receives a first message from the GPU driver, the first message being configured to indicate that the Mth motion effect image is not rendered, and the third time point is later than a time point at which the image synthesis system receives the first message; after the second Vsync signal, the image synthesis system receives a second message from the GPU driver, the second message being configured to indicate that the Mth motion effect image is rendered, and the fourth time point is later than a time point at which the image synthesis system receives the second message.

[0021] In this way, the image synthesis system can confirm, by using the GPU driver, a rendering situation of the Mth motion effect image of the first application, to facilitate subsequent confirmation of whether to synthesize the Mth motion effect image.

[0022] In a possible implementation, the method further comprises: at a sixth time point, a desktop application transmits a third message to the image synthesis system, the third message being configured to indicate that a motion effect is started, the sixth time point being later than the first time point and earlier than the second time point; after a preset time length of the sixth time point, the desktop application transmits a fourth message to the image synthesis system, the fourth message being configured to indicate that the motion effect is ended, the fourth time point being earlier than the preset time length of the sixth time point, and a fifth time point being later than the preset time length of the sixth time point.

[0023] In this way, the image synthesis system can confirm the start and end of the dynamic effect, and further control different synthesis manners in the dynamic effect scene and the non-dynamic effect scene, reduce the lag of the dynamic effect scene, and improve the response speed of the interface in the non-dynamic effect scene.

[0024] In a possible implementation, at a seventh time point, the interface of the first application receives an operation for entering the multi-tasking interface; the multi-tasking interface includes: a first card, the first card being configured to display an image of the first application; at an eighth time point, the desktop application starts to draw a first A frame image of the desktop application, and the first application starts to draw a first B frame dynamic effect image of the first application, A and B being integers greater than zero; at a ninth time point, the drawing of the first A frame image is completed, the drawing of the first B frame dynamic effect image is completed and the rendering of the first B frame dynamic effect image is not completed, the first A frame image is synthesized, and the first B frame dynamic effect image is not synthesized; at a tenth time point, the desktop application completes the drawing of a first A+j frame image, and the rendering of the first A+j frame image is not completed, the rendering of the first B frame dynamic effect image is completed, and the first A+j frame image and the first B frame dynamic effect image are synthesized, j being an integer greater than zero.

[0025] It can be understood that the first A frame image can correspond to the image of the desktop application hereinafter; and the first B frame dynamic effect image can correspond to the image of the game application hereinafter.

[0026] In this way, in the dynamic effect process of entering the multi-tasking, the synthesis of the image of the first application is performed after the rendering of the image is completed, so that the display lag phenomenon caused by a long image rendering time of the first application can be reduced.

[0027] In some embodiments, at the ninth time point, the rendering of the first A frame image is not completed.

[0028] In this way, the rendering and synthesis of the image of the desktop application can be performed synchronously, the image synthesis waiting time of the desktop application can be reduced, and the response speed of the image display of the desktop application can be improved.

[0029] In a possible implementation, at an eleventh time point, the multi-tasking interface receives an operation for entering the first application; at a twelfth time point, the desktop application starts to draw a first C frame image of the desktop application, and the first application starts to draw a first D frame dynamic effect image of the first application, C and D being integers greater than zero; at a thirteenth time point, the drawing of the first C frame image is completed, the drawing of the first D frame dynamic effect image is completed and the rendering of the first D frame dynamic effect image is not completed, the first C frame image is synthesized, and the first D frame dynamic effect image is not synthesized; at a fourteenth time point, the desktop application completes the drawing of a first C+e frame image, and the rendering of the first C+e frame image is not completed, the rendering of the first D frame dynamic effect image is completed, and the first C+e frame image and the first D frame dynamic effect image are synthesized, e being an integer greater than zero.

[0030] It can be understood that the C-th frame image can correspond to the image of the desktop application hereinafter; the D-th frame animation image can correspond to the image of the game application hereinafter.

[0031] In this way, in the animation process of exiting the first application, the synthesis of the image of the first application is performed after the rendering of the image is completed, so that the display lag phenomenon caused by the long image rendering time of the first application can be reduced.

[0032] In some embodiments, at the thirteenth moment, the rendering of the C-th frame image is not completed.

[0033] In this way, the rendering and synthesis of the image of the desktop application can be performed synchronously, the image synthesis waiting time of the desktop application can be reduced, and the response speed of the image display of the desktop application can be improved.

[0034] In a possible implementation, at the fifteenth moment, the interface of the first application receives an operation of exiting the first application; at the sixteenth moment, the desktop application starts to draw a F-th frame image of the desktop application, and the first application starts to draw a G-th frame animation image of the first application, F and G are both integers greater than zero; at the seventeenth moment, the drawing of the F-th frame image is completed, the drawing of the G-th frame animation image is completed and the rendering of the G-th frame animation image is not completed, the F-th frame image is synthesized, and the G-th frame animation image is not synthesized; at the eighteenth moment, the desktop application completes the drawing of the F+h-th frame image, and the rendering of the F+h-th frame image is not completed, the rendering of the G-th frame animation image is completed, the F+h-th frame image and the G-th frame animation image are synthesized, and h is an integer greater than zero.

[0035] It can be understood that the F-th frame image can correspond to the image of the desktop application hereinafter; the G-th frame animation image can correspond to the image of the game application hereinafter.

[0036] In this way, in the animation process of exiting the first application, the synthesis of the image of the first application is performed after the rendering of the image is completed, so that the display lag phenomenon caused by the long image rendering time of the first application can be reduced.

[0037] In some embodiments, at the seventeenth moment, the rendering of the F-th frame image is not completed.

[0038] In this way, the rendering and synthesis of the image of the desktop application can be performed synchronously, the image synthesis waiting time of the desktop application can be reduced, and the response speed of the image display of the desktop application can be improved.

[0039] In a second aspect, an embodiment of the present application provides a data processing method, which comprises: synthesizing an interface of a first application according to a first synthesis strategy; the first synthesis strategy is used to instruct to synthesize an image that has completed drawing in a cache queue; in response to a first operation, starting an animation effect; synthesizing an interface of the animation effect according to a second synthesis strategy during the animation effect, the interface of the animation effect comprising: an image of the first application and an image of a second application; the second synthesis strategy is used to instruct to synthesize a second type of image in a case where all images of a first type have not completed rendering; in a case where the drawing of the image of the first type is completed and the rendering of the image of the first type is completed, synthesizing the image of the first type and the image of the second type; the image of the first type is an image of a first type of application, and the image of the second type is not an image of the first type of application; after the animation effect ends, synthesizing an interface of the second application according to the first synthesis strategy. The first application is the first type of application, and / or the second application is the first type of application.

[0040] In this way, in the animation effect scenario, the synthesis of the image of the first type of application is performed after the rendering of the image is completed, so that the display lag phenomenon caused by a long image rendering time of the first type of application can be reduced.

[0041] In a possible implementation, the first application is not the first type of application, and the second application is the first type of application, and the first operation is an operation for starting the first application.

[0042] In this way, the method can be applied to the animation effect scenario of application starting, and the lag phenomenon during application starting can be reduced.

[0043] In a possible implementation, synthesizing the interface of the animation effect according to the second synthesis strategy during the animation effect comprises:

[0044] At a second moment, the desktop application starts to draw an Nth frame of image of the desktop application, and the first application starts to draw an Mth frame of animation effect image of the first application, N and M are both integers greater than zero; at a third moment, the drawing of the Nth frame of image is completed, the drawing of the Mth frame of animation effect image is completed and the rendering of the Mth frame of animation effect image is not completed, the Nth frame of image is synthesized, and the Mth frame of animation effect image is not synthesized; at a fourth moment, the desktop application completes drawing of an N+i th frame of image, and the rendering of the N+i th frame of image is not completed, the rendering of the Mth frame of animation effect image is completed, and the N+i th frame of image and the Mth frame of animation effect image are synthesized, i is an integer greater than zero; the second moment is later than a moment of receiving the first operation, the third moment is later than the second moment, and the fourth moment is later than the third moment.

[0045] In some embodiments, the first application can be an application with a frame rate lower than a refresh rate of the animation effect scenario; the first application can be an application with a long single frame image rendering time (for example, a game application). Here, no specific limitation is made.

[0046] It can be understood that the Nth image can correspond to the image of the desktop application below; the Mth motion effect image can correspond to the image of the game application below.

[0047] It can be understood that during the motion effect process, the synthesis of the image of the first application is after the image rendering is completed, so that the display lag phenomenon caused by the long image rendering time of the first application can be reduced.

[0048] In a possible implementation, at a third moment, the rendering of the Nth image is not completed.

[0049] It can be understood that during the motion effect process, the rendering and synthesis of the image of the desktop application can be performed synchronously; and the synthesis of the image of the first application is after the image rendering is completed.

[0050] In this way, the rendering and synthesis can be performed synchronously, the image synthesis waiting time of the desktop application can be reduced, and the response speed of the image display of the desktop application can be improved.

[0051] In a possible implementation, after the motion effect ends, the interface of the second application is synthesized according to the first synthesis strategy, including: at a fifth moment, the motion effect of the first application is started, the first application completes the drawing of the Lth image during the running of the first application, and the rendering of the Lth image is not completed, and the Lth image is synthesized; the fifth moment is later than the fourth moment.

[0052] In a possible implementation, the first application is not the first type of application, the second application is the first type of application, and the first operation is an operation for exiting the multitasking interface.

[0053] The operation for exiting the multitasking interface can refer to the corresponding description below, and will not be described in detail here.

[0054] In this way, the motion effect scene of application exit can be applied, and the lag phenomenon during application startup can be reduced.

[0055] In a possible implementation, the first application is the first type of application, the second application is not the first type of application, and the first operation is an operation for entering the multitasking interface.

[0056] The operation for entering the multitasking interface can refer to the corresponding description below, and will not be described in detail here.

[0057] In this way, the motion effect scene of entering the multitasking interface can be applied, and the lag phenomenon during application startup can be reduced.

[0058] In a possible implementation, the first application is the first type of application, the second application is not the first type of application, and the first operation is an operation for exiting the first application.

[0059] The operation for exiting the first application can refer to the corresponding description below, and details are not described here.

[0060] In this way, the animation effect scenario can be applied to application exit, and the lag phenomenon during application startup can be reduced.

[0061] In a third aspect, an embodiment of the present application provides a data processing apparatus. The data processing apparatus can be an electronic device, or a chip or chip system in the electronic device. The data processing apparatus can include a display unit and a processing unit. When the data processing apparatus is an electronic device, the display unit can be a display screen. The display unit is configured to perform the display step, so that the electronic device implements a data processing method described in the first aspect, the second aspect, or any possible implementation of the two aspects. When the data processing apparatus is an electronic device, the processing unit can be a processor. The data processing apparatus can further include a storage unit, which can be a memory. The storage unit is configured to store instructions, and the processing unit executes the instructions stored in the storage unit, so that the electronic device implements a data processing method described in the first aspect, the second aspect, or any possible implementation of the two aspects. When the data processing apparatus is a chip or chip system in the electronic device, the processing unit can be a processor. The processing unit executes the instructions stored in the storage unit, so that the electronic device implements a data processing method described in the first aspect, the second aspect, or any possible implementation of the two aspects. The storage unit can be a storage unit (e.g., a register, a cache, etc.) in the chip, or a storage unit (e.g., a read-only memory, a random access memory, etc.) in the electronic device and located outside the chip.

[0062] Specifically, the display unit is configured to display an interface. The interface can be a desktop, an animation interface, or an application interface. The processing unit is configured to perform steps of drawing, rendering, and synthesizing an image.

[0063] For example, at a first time, the processing unit is configured to receive, at the desktop, an operation for starting a first application; at a second time, the processing unit is further configured to control the desktop application to start drawing an Nth frame image of the desktop application, and the first application to start drawing an Mth frame dynamic effect image of the first application, where N and M are integers greater than zero; at a third time, the drawing of the Nth frame image is completed and the rendering of the Nth frame image is not completed, and the drawing of the Mth frame dynamic effect image is completed and the rendering of the Mth frame dynamic effect image is not completed, the processing unit is further configured to synthesize the Nth frame image and not synthesize the Mth frame dynamic effect image; at a fourth time, the desktop application completes drawing of an N+i th frame image and the rendering of the N+i th frame image is not completed, and the rendering of the Mth frame dynamic effect image is completed, the processing unit is further configured to synthesize the N+i th frame image and the Mth frame dynamic effect image, where i is an integer greater than zero; at a fifth time, the dynamic effect of the first application ends, and the first application completes drawing of an Lth frame image during running of the first application and the rendering of the Lth frame image is not completed, the processing unit is further configured to synthesize the Lth frame image; the second time is later than the first time, the third time is later than the second time, the fourth time is later than the third time, and the fifth time is later than the fourth time.

[0064] For example, the processing unit is configured to synthesize an interface of a first application according to a first synthesis strategy, where the first synthesis strategy is configured to indicate to synthesize images that have been completed in drawing in a cache queue; the processing unit is configured to start a dynamic effect in response to a first operation; during the dynamic effect, synthesize an interface of the dynamic effect according to a second synthesis strategy, where the interface of the dynamic effect includes an image of the first application and an image of a second application, and the second synthesis strategy is configured to indicate to synthesize a second type of image in a case where the rendering of a first type of image is not completed, and to synthesize the first type of image and the second type of image in a case where the drawing and the rendering of the first type of image are completed, the first type of image is an image of a first type of application, and the second type of image is not an image of the first type of application; and the processing unit is configured to synthesize an interface of the second application according to the first synthesis strategy after the dynamic effect ends.

[0065] In a fourth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, the memory is configured to store code instructions, and the processor is configured to run the code instructions to execute the method described in the first aspect, the second aspect, or any possible implementation manner of the two aspects.

[0066] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, and the computer readable storage medium stores a computer program or instructions, when the computer program or instructions run on a computer, the computer program or instructions cause the computer to execute the method described in the first aspect, the second aspect, or any possible implementation manner of the two aspects.

[0067] In a sixth aspect, an embodiment of the present application provides a computer program product including a computer program, which, when executed on a computer, causes the computer to perform the method described in the first aspect, the second aspect, or any possible implementation manner of the two aspects.

[0068] In a seventh aspect, the present application provides a chip or a chip system, which includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected through a line, the at least one processor is configured to execute a computer program or an instruction to perform the method described in the first aspect or any possible implementation manner of the first aspect. The communication interface in the chip can be an input / output interface, a pin, or a circuit, etc.

[0069] In a possible implementation, the chip or the chip system described in the present application further includes at least one memory, and the at least one memory stores the instruction. The memory can be a storage unit inside the chip, for example, a register, a cache, etc., or a storage unit of the chip (for example, a read-only memory, a random access memory, etc.).

[0070] It should be understood that the second aspect to the seventh aspect of the present application correspond to the technical solution of the first aspect of the present application, and the beneficial effects obtained by each aspect and the corresponding possible implementation manner are similar, which will not be repeated. BRIEF DESCRIPTION OF DRAWINGS

[0071] FIG. 1 is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0072] FIG. 2 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0073] FIG. 3 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0074] FIG. 4 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0075] FIG. 5 is a flow diagram of interface display in a dynamic effect scenario in a possible design;

[0076] FIG. 6 is a flow diagram of interface display in a possible design;

[0077] FIG. 7 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application;

[0078] FIG. 8 is a schematic diagram of a software structure of an electronic device provided by an embodiment of the present application;

[0079] FIG. 9 is a flow diagram of a data processing method provided by an embodiment of the present application;

[0080] FIG. 10 is a display flow diagram provided by an embodiment of the present application;

[0081] FIG. 11 is a data processing method flow diagram provided by an embodiment of the present application;

[0082] FIG. 12 is a display flow diagram provided by an embodiment of the present application;

[0083] FIG. 13 is a display flow diagram provided by an embodiment of the present application. DETAILED DESCRIPTION

[0084] In order to clearly describe the technical solutions of the embodiments of the present application, the following briefly introduces some terms and technologies involved in the embodiments of the present application:

[0085] 1. Frame

[0086] Frame refers to the smallest unit of a single picture in interface display. One frame can be understood as a still picture, and fast and continuous display of multiple connected frames can form an illusion of object movement. Frame rate refers to the number of frames refreshed in 1 second, which can also be understood as the number of times the graphics processor in an electronic device refreshes a picture per second. High frame rate can obtain smoother and more realistic animation. The more frames per second, the smoother the displayed motion will be.

[0087] It should be noted that the frame in interface display usually needs to go through processes such as drawing, rendering, and composition.

[0088] 2. Frame drawing

[0089] Frame drawing refers to picture drawing of a display interface. The display interface can be composed of one or more views, and each view can be drawn by a visual control of a view system. Each view is composed of sub-views, and one sub-view corresponds to one widget in the view, for example, one sub-view corresponds to one symbol in the picture view.

[0090] 3. Frame rendering

[0091] Frame rendering is a coloring operation or addition of 3D effects on the drawn view. For example, 3D effects can be light effects, shadow effects, and texture effects, etc.

[0092] 4. Frame composition

[0093] Frame composition is a process of composing multiple rendered views as described above into a display interface.

[0094] 6. Vertical synchronization (Vsync) signal

[0095] The Vsync signal is a periodic signal, and the Vsync signal period can be set according to the screen refresh rate. For example, when the screen refresh rate is 60 Hz, the Vsync signal period can be 16.6 ms, that is, the electronic device generates a control signal every 16.6 ms to trigger the Vsync period.

[0096] It can be understood that the electronic device generally displays based on the Vsync signal to synchronize the processes such as image drawing, rendering, synthesis, and screen refresh display, so as to improve the smoothness of display and reduce the phenomenon of display lag.

[0097] It should be noted that the Vsync signal can be divided into software Vsync signal and hardware Vsync signal. The software Vsync signal includes Vsync-APP and Vsync-SF. The Vsync-APP is used to trigger the drawing and rendering process. The Vsync-SF is used to trigger the synthesis process. The hardware Vsync signal (for example, Vsync-HW signal, tear-effect signal) is used to trigger the screen display refresh process.

[0098] Generally, the software Vsync signal and the hardware Vsync signal keep period synchronization. Taking 60 Hz and 120 Hz as examples, if the Vsync-HW is switched from 60 Hz to 120 Hz, the Vsync-APP and the Vsync-SF are synchronously changed, and are both switched from 60 Hz to 120 Hz.

[0099] It can be understood that the Vsync-APP, the Vsync-SF, and the Vsync-HW can have a certain deviation. For example, the Vsync-SF can be later than the Vsync-APP, and the Vsync-HW can be later than the Vsync-SF. Here, no specific limitation is made.

[0100] In addition, because the frame rates corresponding to different applications can be different, different applications correspond to different Vsync-APP. The time of the Vsync-APP corresponding to different applications can be the same or different, and no specific limitation is made here.

[0101] 7. Other terms

[0102] In the embodiments of the present application, the same items or similar items with basically the same functions and effects are distinguished by using "first", "second", and the like. For example, the first chip and the second chip are only used to distinguish different chips, and do not limit the sequence. Those skilled in the art can understand that the "first", "second", and the like do not limit the quantity and execution sequence, and the "first", "second", and the like also do not necessarily mean different.

[0103] It should be noted that the terms "exemplary" and "for example" are used herein to mean "serving as an example, instance, or illustration," and not "preferred" or "advantageous over other examples." The usage of these terms in this application is not intended to convey any preference or advantage for the embodiments or examples described with such terms.

[0104] In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. The relationship between the associated objects described by "and / or" indicates that there can be three kinds of relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0105] 8. Electronic device

[0106] The electronic device of the embodiments of the present application can include a handheld device with display function, a vehicle-mounted device, etc. For example, some electronic devices are: a mobile phone, a tablet computer, a palm computer, a notebook computer, a mobile internet device (MID), a wearable device (for example, a smart watch, smart glasses, a smart bracelet, or smart jewelry, etc.), a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, a cellular phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (WLL) station, a personal digital assistant (PDA), a handheld device with wireless communication function, a computing device or other processing device connected to a wireless modem, a vehicle-mounted device, a terminal device in an internet of things (IoT) system, a terminal device in a 5G network, or a terminal device in a future evolved public land mobile network (PLMN), etc. The embodiments of the present application are not limited thereto.

[0107] The electronic device in the embodiments of the present application can also be referred to as: a terminal device, a user equipment (UE), a mobile station (MS), a mobile terminal (MT), an access terminal, a subscriber unit, a subscriber station, a mobile station, a remote station, a remote terminal, a mobile device, a user terminal, a terminal, a wireless communication device, a user agent, or a user apparatus, etc.

[0108] In the embodiments of the present application, the electronic device or each network device comprises a hardware layer, an operating system layer running above the hardware layer, and an application layer running on the operating system layer. The hardware layer comprises hardware such as a central processing unit (CPU), a memory management unit (MMU), and a memory (also referred to as main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a windows operating system. The application layer comprises applications such as a browser, an address book, word processing software, and instant messaging software.

[0109] The data processing method of the embodiments of the present application is applicable to a dynamic effect scenario. The dynamic effect scenario refers to a scenario in which the interface of the display screen of the electronic device changes. The dynamic effect scenario can be a scenario in which an application is started, a scenario in which an application is exited, a scenario in which an application is switched (for example, from one application to another application), a scenario in which a multi-task interface is entered, a scenario in which a multi-task interface is exited, a scenario in which an application is switched between recent tasks, and the like, which are not limited herein.

[0110] The following describes a dynamic effect scenario described in the embodiments of the present application by taking a mobile phone as an example of an electronic device. It should be understood that the scenarios described in the following embodiments are only some examples of the embodiments of the present application, and do not constitute a limitation on the present application, and other dynamic effect scenarios also belong to the protection scope of the embodiments of the present application.

[0111] For example, taking a game application as an application, and a scenario in which the game application is started as a dynamic effect scenario. FIG. 1 is a schematic diagram of an application scenario provided by the embodiments of the present application.

[0112] As shown in FIG. 1, when the electronic device is in the home interface shown in a of FIG. 1, and a click operation on the icon 101 of the game application is received, the electronic device can enter the game interface 102 shown in b of FIG. 1. The game interface 102 is the interface after the game application is started. In some embodiments, as shown in b of FIG. 1, the game interface 102 can be a loading interface of the game application. In the scenario in which the game application is started, the starting dynamic effect of the game application is the process from the user clicking the icon 101 of the game application to the mobile phone displaying the game interface 102.

[0113] It should be understood that the click operation on the icon 101 of the game application is one of the starting operations of the game application, and the starting operation can be a gesture operation, a voice operation, or any other type of operation, for example, a sliding operation, which is not limited herein.

[0114] It should be noted that the dynamic effect playing can have a fixed time length (for example, 400 ms, etc.). A series of image frames played continuously within the fixed time length is a dynamic effect scene. Taking the above starting dynamic effect as an example, the starting dynamic effect can be the process of the image displayed in the game interface 102 becoming larger. Specifically, when the user clicks the icon 101 of the game application, the starting dynamic effect starts, and the image displayed in the game interface 102 is played; when the image displayed in the game interface 102 fills the entire screen (or display screen), the starting dynamic effect ends.

[0115] For example, taking the mobile phone playing two frames of images continuously within the fixed time length of the dynamic effect as an example. When the user clicks the icon 101 of the game application, the mobile phone plays frame 1 as shown in Fig. 1c and frame 2 as shown in Fig. 1d continuously. The process of the mobile phone playing frame 1 and frame 2 is the process of starting the dynamic effect.

[0116] It should be noted that in the scene of starting the dynamic effect, the starting dynamic effect can be a dynamic effect played when the user starts the application for the first time, or a dynamic effect played when the user starts the application for the second time. The first time means that the application is neither running in the foreground nor running in the background. The second time means that the application is running in the background. In this case, the starting dynamic effect can also be understood as the process of switching the application from the background to the foreground.

[0117] For example, taking the application as a game application and the dynamic effect scene as the scene of exiting the game application as an example. Fig. 2 is another application scene diagram provided by an embodiment of the present application.

[0118] As shown in Fig. 2, when the electronic device is in the game interface 201 as shown in Fig. 2a, and receives a trigger operation of exiting the game application, the electronic device can display the interface 202 as shown in Fig. 2b. The game interface 201 can be any interface of the game application, and the interface 202 can be the main interface of the mobile phone.

[0119] In some embodiments, the exit operation can be a gesture operation, a voice operation, or any other type of operation, for example, a touch operation, etc. The touch operation can be, for example, a click operation, a sliding operation, etc. Taking the exit operation as a sliding operation as an example, as shown in Fig. 2b, the exit operation can be, for example, the operation of the user sliding up the interface of the game application.

[0120] In the scenario that the game application exits, the exit animation of the game application is the process from the user's upslide operation on the interface of the game application to the interface of the main screen of the mobile phone. In some embodiments, in combination with the above-mentioned embodiments, the exit animation can be the process that the image displayed in the game interface 201 changes from large to small. Specifically, when the user upslides on the interface of the game application, the exit animation starts; the image in the game interface 201 displayed by the mobile phone starts to shrink; when the image displayed in the game interface 201 completely exits and the mobile phone displays the main interface, the exit animation ends.

[0121] For example, the mobile phone continuously plays two frames of images in the fixed time length of the animation. When the mobile phone receives the trigger operation of exiting the game application, the mobile phone continuously plays frame 1 as shown in c of FIG. 2 and frame 2 as shown in d of FIG. 2. The process that the mobile phone plays frame 1 and frame 2 is the process of the exit animation.

[0122] For example, the animation scenario is the scenario that the mobile phone switches from the interface of the game application to the multi-task interface. FIG. 3 is another application scenario provided by the embodiments of the present application.

[0123] As shown in FIG. 3, when the electronic device is in the game interface 301 as shown in a of FIG. 3, the electronic device can display the multi-task interface 302 as shown in b of FIG. 3 after receiving the user's upslide operation from the bottom of the display screen. The multi-task interface 302 includes the cards of the application programs recently used by the user. For example, the card 303 of the game application and the card 304 of the gallery application.

[0124] It should be understood that the upslide operation from the bottom of the display screen is one of the trigger operations for entering the multi-task interface. The trigger operation can also be a gesture operation, a voice operation or any other type of operation, which is not limited here.

[0125] In some embodiments, the multi-task management interface further includes the identification (for example, the application program icon, the application program name, etc.) of the application program indicated by each preview interface. As shown in FIG. 3, the multi-task interface 302 can include the identification 305 of the game application and the identification 306 of the gallery application. The multi-task interface 302 further includes the control 307 for cleaning the background.

[0126] In some embodiments, the card of the application displayed in the multitasking interface can be arranged from right to left in order of application usage time from recent to old. For example, a user first uses a gallery application, then uses a game application, and when the phone displays the interface of the game, the user performs an upward swipe from the bottom edge of the screen and stops, and the phone displays the multitasking interface 302, which includes a card 303 of the game and a card 304 of the gallery application. The card 304 of the gallery application is located at the rightmost side and is partially displayed, and the card 303 of the game is located to the left of the card 304 of the gallery application and can be fully displayed.

[0127] In the scenario of entering the multitasking interface, the dynamic effect of entering the multitasking interface is the process from the user's upward swipe on the interface of the game application to the phone displaying the multitasking interface. In some embodiments, in combination with the above-mentioned embodiments, the dynamic effect of entering the multitasking interface can be the process of the image displayed in the interface of the game application changing from large to small. Specifically, when the user swipes upward on the interface of the game application, the dynamic effect of entering the multitasking interface starts, the image in the interface 301 of the game application displayed by the phone starts to shrink, and when the image displayed in the interface 301 of the game application shrinks to the size of the card in the multitasking interface, the dynamic effect of entering the multitasking interface ends.

[0128] For example, the phone continuously plays two frames of images in a fixed time length of the dynamic effect. When the phone receives the operation of entering the multitasking interface, the phone continuously plays frame 1 as shown in FIG. 3c and frame 2 as shown in FIG. 3d. The process of the phone playing frame 1 and frame 2 is the process of the dynamic effect of entering the multitasking interface.

[0129] For example, the dynamic effect scenario is the scenario of the phone switching from the interface of the game application to the multitasking interface. FIG. 4 is another application scenario provided by the embodiments of the present application.

[0130] As shown in FIG. 4, when the electronic device is in the multitasking interface 401 as shown in FIG. 4a, and receives a trigger operation on the card of the game application, the electronic device can display the game interface 402 as shown in FIG. 4b.

[0131] It should be understood that the trigger operation on the card of the game application is one of the trigger operations for exiting the multitasking interface, and the trigger operation can also be a gesture operation, a voice operation, or any other type of operation, which is not limited here.

[0132] In the scenario of exiting the multitasking interface, the dynamic effect of exiting the multitasking interface is the process from the user triggering the card of the game application to the phone displaying the game interface. In some embodiments, in combination with the above-mentioned embodiments, the dynamic effect of exiting the multitasking interface can be the process of the image displayed in the interface of the game application changing from small to large. In the scenario of entering the multitasking interface, the dynamic effect of entering the multitasking interface is the process from the user's upward swipe on the interface of the game application to the phone displaying the multitasking interface. In some embodiments, in combination with the above-mentioned embodiments, the dynamic effect of entering the multitasking interface can be the process of the image displayed in the interface of the game application changing from large to small. Specifically, when the user swipes upward on the interface of the game application, the dynamic effect of entering the multitasking interface starts, the image in the interface 301 of the game application displayed by the phone starts to shrink, and when the image displayed in the interface 301 of the game application shrinks to the size of the card in the multitasking interface, the dynamic effect of entering the multitasking interface ends.

[0133] Specifically, when the user clicks the card of the game application, the dynamic effect of entering the multitasking interface starts; the image in the preview interface of the game application displayed by the phone starts to enlarge; when the preview interface of the game application is enlarged to the size of the display screen, the game application is entered.

[0134] For example, the phone continuously plays two frames of images within a fixed time length of the dynamic effect. When the phone receives the operation of entering the multitasking interface, the phone continuously plays frame 1 as shown in FIG. 4c and frame 2 as shown in FIG. 4d. The process of the phone playing frame 1 and frame 2 is the process of the dynamic effect of entering the multitasking interface.

[0135] The dynamic effect scenarios shown in FIGS. 1-4 are examples only, and other dynamic effect scenarios are also possible, such as switching between different applications through a shortcut, which is not limited here.

[0136] In a possible design, the electronic device synchronously performs synthesis of image frames when rendering the image frames, so as to reduce the synthesis waiting time of the image frames and improve the response speed of the interface.

[0137] For example, FIG. 5 is a flowchart of the interface display of an application in a possible design. Taking the dynamic effect scenario of entering the multitasking interface from the game application as an example, the content displayed by the electronic device in time sequence corresponds to frame-1, frame 0, frame 1, frame 2, and frame 3.

[0138] Specifically, taking the display of frame 1 as an example, the game application performs drawing and rendering preprocessing on frame 1, and transmits frame 1A to the buffer queue B for storage to wait for synthesis. The game application also calls the GPU to render frame 1.

[0139] The image synthesis system (surface flinger) performs synthesis processing on frame 1 to obtain the synthesized frame 1. After frame 1 is completed and frame 1 is completed, the display driver displays the content corresponding to frame 1 on the screen.

[0140] Frame 2 and frame 3 are synthesized and displayed in a manner similar to that of frame 1, which is not described here.

[0141] In the 5 Vsync cycles from 0 ms to 132.8 ms, the interface is displayed as frame-1, frame 0, frame 1, frame 2, and frame 3 in sequence. The display time lengths of frame 0, frame 1, frame 2, and frame 3 are consistent, and the interface changes continuously and smoothly without lag.

[0142] As can be seen from FIG. 5, when the GPU renders frame 1, the surface flinger can perform composition processing on frame 1. In this way, GPU rendering and surface flinger composition are performed synchronously, so that the drawing rendering composition can be completed within two Vsync periods, the time of the drawing rendering composition is shortened, and the response speed of the interface is improved.

[0143] However, in the dynamic effect scenario, the display of the electronic device can appear to be stuck.

[0144] It can be understood that, in the dynamic effect scenario, the electronic device needs to render frames of at least two applications, compared with displaying the interface of a single application, the GPU needs to render frames of at least two applications of the composed interface, more images are rendered, and thus the rendering time is also longer, which can cause the rendering to time out, resulting in the stuck phenomenon.

[0145] In some embodiments, when the dynamic effect is triggered, the screen refresh rate can be increased, and thus the Vsync period time is shortened, while the image rendering time remains unchanged, resulting in the display stuck phenomenon.

[0146] In addition, the electronic device can also cause the image rendering time to be longer due to various reasons, and thus the stuck phenomenon occurs. The reasons include but are not limited to: the complexity of the graphics drawing, or the system load is affected by the fluctuation of the downloaded data, etc.

[0147] For example, FIG. 6 is a flow diagram of interface display in a dynamic effect scenario in a possible design. Taking the dynamic effect scenario of entering a multi-task interface as an example, in time sequence, the content displayed by the electronic device corresponds to frame-1, frame 0, frame 1, frame 2, frame 3, frame 4 and frame 5 in turn. The electronic device performs drawing, rendering, composition and display on frame 1, frame 2, frame 3, frame 4 and frame 5 in FIG. 6 respectively, and specific descriptions can be referred to the corresponding descriptions of FIG. 5.

[0148] In FIG. 6, frame 1A causes the rendering time to increase due to various reasons, and cannot complete drawing and rendering within 2 Vsync periods (33.2 ms). At 33.2 ms, frame 1A has not completed rendering, and cannot display frame 1. In the 8 periods from 0 ms to 122.8 ms in FIG. 6, the interface is displayed as frame-1, frame 0, frame 0, frame 1, frame 2, frame 3, frame 4 and frame 5 in turn. When the electronic device displays, the display time of frame 0 increases, causing the electronic device to appear stuck, and the user experience is reduced.

[0149] Therefore, the embodiments of the present application provide a data processing method and related apparatus. In the case of identifying the dynamic effect scenario, the composition of the image corresponding to part of the application is performed after the rendering is completed, so as to reduce the stuck phenomenon caused by the too long rendering time, and improve the fluency problem in the dynamic effect scenario.

[0150] In order to better understand the embodiments of the present application, the structure of the electronic device of the embodiments of the present application is introduced as follows. For example, FIG. 7 is a schematic diagram of a hardware structure of an electronic device provided by the embodiments of the present application.

[0151] The electronic device can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0152] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0153] The processor 110 can include one or more processing units. Different processing units can be independent devices or integrated into one or more processors. The processor 110 can also be provided with a memory for storing instructions and data. For example, the processor 110 is used to store instructions and data related to the data processing method provided by the embodiments of the present application.

[0154] In the embodiments of the present application, after the electronic device detects a trigger operation through the touch sensor 180K, the display screen of the electronic device can realize the switching display of the corresponding interfaces in FIGS. 1 to 4. When the electronic device detects a dynamic effect scene for indicating application opening, application exit, entering a multi-task interface, etc., the processor can realize the display of the interface in the dynamic effect scene by the data processing method provided by the embodiments of the present application.

[0155] The software system of the electronic device can employ a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture, and the like, which will not be described herein.

[0156] For example, FIG. 8 is a schematic diagram of a software structure of an electronic device according to an embodiment of the present application. As shown in FIG. 8,

[0157] The layered architecture divides the software into several layers, each of which has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0158] The application layer can include a series of application packages. In an embodiment of the present application, the application package can include: a launcher, a game, a camera, a gallery, and the like.

[0159] In some embodiments, the application package includes: a UI thread and a rendering thread. The UI thread is used for interface drawing, generating drawing instructions. The rendering thread is used for rendering preprocessing or rendering processing of the drawing instructions. The rendering preprocessing includes but is not limited to: fuzzy algorithm processing, and the like.

[0160] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the application programs of the application layer. The application framework layer includes some pre-defined functions. In an embodiment of the present application, the application framework layer can include: a surface flinger, an input manager, a window manager, and the like.

[0161] The surface flinger is used to control image synthesis and generate a vertical synchronization (Vsync) signal. Specifically, the surface flinger can receive graphical display data from multiple sources (e.g., different applications), and synthesize the display data.

[0162] For example, the surface flinger can include: a synthesis thread, a Vsync thread, and a quene buffer thread. The synthesis thread is used to be awakened by the Vsync signal for synthesis. The Vsync thread is used to request the generation of the next Vsync signal according to the Vsync signal. The quene buffer thread is used to store a cache, generate a Vsync signal request, and wake up the synthesis thread, and the like. There is one or more cache queues in the quene buffer thread, which are used to store caches corresponding to different applications.

[0163] The input manager is used to manage the program of the input device. For example, the input system can determine the input operation of the mouse click operation, the keyboard input operation and the touch sliding, etc.

[0164] The window manager is used to manage the window program. The window manager can acquire the display screen size, determine whether there is a status bar, lock the screen, intercept the screen, etc.

[0165] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0166] The core library contains two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

[0167] The application layer and the application program framework layer run in the virtual machine. The virtual machine executes the java file of the application layer and the application program framework layer into a binary file. The virtual machine is used to execute the management of the object life cycle, the stack management, the thread management, the security and the exception management, and the garbage collection, etc.

[0168] The system library can include multiple function modules. For example: image rendering library, image synthesis library, function library, media library and input processing library, etc.

[0169] The image rendering library is used for rendering two-dimensional or three-dimensional images. The image synthesis library is used for synthesizing two-dimensional or three-dimensional images.

[0170] In a possible implementation manner, the application draws and renders the image through the image rendering library, and then sends the image after the drawing and rendering to the cache queue of the image synthesis system. Whenever the Vsync signal arrives, the image synthesis system (for example, the surface flinger) obtains a frame of image to be synthesized from the cache queue in order, and then performs image synthesis through the image synthesis library.

[0171] The function library provides macros, type definitions, string operation functions, mathematical calculation functions and input / output functions used in C language, etc.

[0172] The media library supports multiple commonly used audio, video format playback and recording, and static image files, etc. The media library can support multiple audio / video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG and PNG, etc.

[0173] The input processing library is used to process the input device library, and can realize mouse, keyboard and touch input processing, etc.

[0174] The hardware abstraction layer can include a plurality of library modules, such as a hardware composer (HWC) library module, a camera library module, and the like. The Android system can load corresponding library modules for device hardware, thereby enabling the application framework layer to access the device hardware. The device hardware can include, for example, an LCD display screen, a camera, and the like in an electronic device.

[0175] The kernel layer is a layer between hardware and software. The kernel layer includes at least a touch panel (TP) driver, a display driver, a Bluetooth driver, a WIFI driver, a keyboard driver, a shared memory driver, and a camera driver, and the like.

[0176] The hardware can be an audio device, a Bluetooth device, a camera device, a sensor device, and the like.

[0177] The following describes, by way of example, the workflow of the software and hardware of an electronic device in the context of application startup or interface switching in an application.

[0178] When the touch sensor 180K in the touch panel receives a touch operation, the kernel layer processes the touch operation into a raw input event (including touch coordinates, touch force, a timestamp of the touch operation, and the like). The raw input event is stored in the kernel layer. The kernel layer reports the raw input event to the input manager of the application framework layer through an input processing library. The input manager of the application framework layer analyzes the information of the raw input event (including the operation type and the report point position, and the like) and determines the focus application according to the current focus, and sends the analyzed information to the focus application. The focus can be a touch point in a touch operation or a click position in a mouse click operation. The focus application is an application running in the foreground of the terminal device or an application corresponding to the touch position in the touch operation. The focus application determines the control corresponding to the raw input event according to the analyzed information of the raw input event (for example, the report point position).

[0179] Taking the touch operation as a touch sliding operation and the control corresponding to the touch sliding operation as a list control of a game application as an example, the game application draws an image and calls a rendering thread to perform rendering preprocessing on the drawn image. The rendering thread transmits the drawn image to a buffer queue of an image composition system and a GPU driver. The image composition system composes the drawn image in the image composition system into a game interface through an image composition library in the system library; the GPU driver renders the image after rendering preprocessing. After the image rendering and image composition are completed, the display driver of the kernel layer drives the screen (display screen) to display the corresponding interface of the game application.

[0180] In the embodiment of the present application, during the dynamic effect period of the game application starting, the image synthesis system synthesizes the image of the game application rendered in the image synthesis system and the image of the desktop application drawn into a dynamic effect interface through the image synthesis library in the system library after the GPU renders the preprocessed image.

[0181] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be implemented independently or in combination. For the same or similar concepts or processes, some embodiments can not be described in detail.

[0182] For example, FIG. 9 is a flowchart of a data processing method provided by the embodiment of the present application. Taking the game application starting as an example, as shown in FIG. 9, the method comprises:

[0183] S901, the desktop application receives a trigger operation on the icon of the game application, and calls the game application to start.

[0184] In some embodiments, S902, after receiving the trigger operation of S901, the desktop application transmits message A to the image synthesis system, and message A is used to indicate a dynamic effect starting message. Adaptively, the image synthesis system performs synthesis processing according to synthesis strategy A until the dynamic effect ends after receiving the message.

[0185] In the embodiment of the present application, synthesis strategy A is used to indicate that the cache of the first type of application is synthesized after rendering ends; and the cache of the second type of application is synchronized for rendering and synthesis.

[0186] In some embodiments, message A comprises: a type identifier A of the dynamic effect, and an identifier B indicating the dynamic effect starting.

[0187] It can be understood that the electronic device can divide the dynamic effect into multiple types, for example, a starting dynamic effect, a quitting dynamic effect, an entering multi-task dynamic effect, a quitting multi-task dynamic effect, etc. Different types of dynamic effects can correspond to different type identifiers, which are not limited here.

[0188] It can be understood that different types of dynamic effects can correspond to different interface display logics, for example, the image synthesis system can have different synthesis processing logics for different types of dynamic effects.

[0189] In this way, the image synthesis system can confirm the dynamic effect as a starting dynamic effect according to the type identifier A of the dynamic effect, and then confirm the synthesis processing logic of the interface. This will not be described in detail here.

[0190] In some embodiments, a whitelist is pre-stored in the image composition system, by which the cache of the first type of application and the cache of the second type of application can be confirmed.

[0191] For example, the whitelist includes the identification (e.g., application package name, application name, application corresponding layer number, etc.) of the first type of application. When the cached application is the application indicated by the whitelist (e.g., the cached application package name is the same as any application package name in the whitelist), the cache is synthesized after the rendering is completed. When the cached application is not the application indicated by the whitelist, the cache is rendered and synthesized synchronously.

[0192] For example, the whitelist can also indicate the second type of application. When the cached application is the application indicated by the whitelist (e.g., the cached application package name is the same as any application package name in the whitelist), the cache is rendered and synthesized synchronously. When the cached application is not the application indicated by the whitelist, the cache is synthesized after the rendering is completed.

[0193] For example, the whitelist can also indicate the first type of application and the second type of application at the same time. When the cached application is the first type of application indicated by the whitelist, the cache is rendered and synthesized synchronously. When the cached application is the second type of application indicated by the whitelist, the cache is synthesized after the rendering is completed.

[0194] It can be understood that the whitelist is only an example, and different synthesis strategies can also be determined by other means, for example, different identifiers are added to the images drawn by the first type of application and the images drawn by the second type of application, etc., which are not limited here.

[0195] S903, after the game application is started, a message for indicating the creation of a layer corresponding to the game application is transmitted to the image composition system. Adaptively, the surface flinger creates the layer A and allocates the cache queue A corresponding to the layer A. The cache queue A is used to store the cache of the application A.

[0196] S904, when the Vsync1 signal arrives, the UI thread B of the desktop application draws the frame 1B corresponding to the layer B to obtain the drawing instruction 1B.

[0197] The drawing instruction is used to indicate the drawing of the image. The drawing instruction can include one or more of the following information: vertex data, primitive type, etc. Among them, the vertex data is used to define the vertex position of the geometric shape, and the primitive type is used to indicate the basic primitive type of the drawing, such as point, line, triangle, etc.

[0198] In some embodiments, the drawing and rendering process of the desktop application can be as follows: the desktop application transmits a Vsync signal request to the image synthesis system to draw the image corresponding to the desktop application. After receiving the Vsync signal request, the image synthesis system generates a Vsync-APP signal corresponding to the desktop application according to the Vsync period corresponding to the desktop application; the UI thread B of the desktop application is awakened after receiving the Vsync-APP signal. After being awakened, the UI thread B requests the image synthesis system for caching to store frame 1B corresponding to the desktop application. After receiving the request caching command sent by the rendering thread B, the image synthesis system reserves the space for storing frame 1B, and sends an instruction for indicating the cache out-of-order to the UI thread B. After receiving the instruction, the UI thread B starts to draw frame 1B and generates drawing instruction 1B. The UI thread B transmits the drawing instruction 1B to the rendering thread B to render frame 1B. The rendering thread B calls the graphics processing unit (GPU) or the central processing unit (CPU) to render frame 1B.

[0199] In some embodiments, the UI thread B can also transmit a drawing request to the rendering thread 1B, and the rendering thread 1B draws frame 1B after receiving the drawing request to obtain drawing instruction 1B, which is not limited here.

[0200] S905, the UI thread B transmits the drawing instruction 1B to the rendering thread B of the desktop application. Adaptively, the rendering thread B renders frame 1B after receiving the drawing instruction 1B to obtain rendering instruction 1B.

[0201] The rendering preprocessing can include complex logical calculation and non-graphic related processing, for example, image blur processing, etc. Here, no specific limitation is made. The rendering preprocessing can be understood as the process of converting the drawing instruction into pixels on the screen using OpenGL ES or Vulkan and other graphic APIs.

[0202] The rendering instruction 1B is used to indicate the rendering image. The rendering instruction 1B can also be understood as a command for controlling the graphics processing unit (GPU) or the central processing unit (CPU) to perform a specific rendering task.

[0203] The rendering instruction 1B can include one or more of the following information: drawing commands, state setting commands, shader commands, texture commands, buffer commands, framebuffer commands, synchronization commands, or clear commands, etc. No specific limitation is made herein.

[0204] S906, the rendering thread B transmits the rendering instruction 1B to the GPU driver to render the frame 1B by the GPU.

[0205] In some embodiments, the GPU driver modifies the rendering status of the frame 1B to rendering completed after the rendering of the frame 1B is completed. For example, the GPU driver modifies the fence value corresponding to the frame 1A from dequeued to queued. No specific limitation is made herein.

[0206] S907, the rendering thread B transmits a message for indicating the rendering of the frame 1B to the image compositing system to wait for compositing (buffer enqueuing). The message includes the address of the buffer corresponding to the frame 1B. In this way, it is convenient for subsequent compositing (consumption) of the image compositing system.

[0207] S908, when the Vsync1 signal arrives, the UI thread A of the game application draws the frame 1A corresponding to the layer A to obtain the drawing instruction 1A.

[0208] It can be understood that the electronic device can execute S908 and S904 simultaneously or not simultaneously. No specific limitation is made herein for the sequence of S908 and S904.

[0209] S909, the UI thread A transmits the drawing instruction 1A to the rendering thread A of the game application. Adaptively, the rendering thread A performs rendering preprocessing on the frame 1A after receiving the drawing instruction 1A to obtain the rendering instruction 1A.

[0210] S910, the rendering thread A transmits the rendering instruction 1A to the GPU driver to render the frame 1A by the GPU.

[0211] In some embodiments, an identifier (e.g., a fence) is set in the GPU driver to indicate the rendering status. After the GPU finishes rendering, the identifier corresponding to the frame is modified to indicate that the rendering is completed. For example, the fence value corresponding to frame 1A is dequeued when the GPU renders frame 1A, and the fence value corresponding to frame 1A is queued when the GPU renders frame 1A. The specific implementation is not limited here.

[0212] In some embodiments, the GPU processes the rendering instructions according to the order in which the rendering instructions are received. For example, if rendering instruction 1A is received first and then rendering instruction 1B is received, rendering instruction 1A is processed first to render frame 1A, and then rendering instruction 1B is processed to render frame 1B. If rendering instruction 1B is received first and then rendering instruction 1A is received, rendering instruction 1B is processed first to render frame 1B, and then rendering instruction 1A is processed to render frame 1A.

[0213] In other embodiments, the GPU is provided with a priority, and the rendering instruction corresponding to the application with a higher priority is processed first. After receiving the rendering instruction of the application with a higher priority, the GPU can interrupt the rendering instruction of the application with a lower priority and process the rendering instruction of the application with a higher priority first.

[0214] For example, if rendering instruction 1A is received first and then rendering instruction 1B is received, rendering instruction 1A is processed first to render frame 1A. If rendering instruction 1B is received while processing rendering instruction 1A, the processing of rendering instruction 1A is interrupted, rendering instruction 1B is processed first to render frame 1B, and then rendering instruction 1A is processed to render frame 1A. In this way, the lag of the desktop application can be reduced.

[0215] S911, the rendering thread A transmits a message for indicating the rendering of frame 1A to the image synthesis system to wait for synthesis (buffering into a queue). The message includes the address of the buffer corresponding to frame 1A. In this way, the subsequent synthesis (consumption) of the image synthesis system is facilitated.

[0216] S912, when the Vsync2 signal arrives, the image synthesis system synthesizes frame 1B to synthesize frame 1.

[0217] It can be understood that since frame 1B is a frame of the second type of application, frame 1A is a frame of the first type of application, and frame 1A has not been rendered, the image synthesis system synthesizes frame 1 through frame 1B.

[0218] In some embodiments, the image compositing system can query the GPU driver for the rendering status of each frame when the Vsync2 signal arrives. For example, the fence value corresponding to each frame of the GPU can be queried.

[0219] In some embodiments, the GPU driver can also transmit a message to the image compositing system after the rendering of frame 1A is completed to facilitate the determination of the image compositing system. The specific implementation is not limited here.

[0220] S913, the image compositing system transmits the synthesized frame 1 to the display driver.

[0221] S914, the display driver receives the message B from the GPU driver, and the message B is used to indicate that the rendering of frame 1B is completed.

[0222] It can be understood that the message B can be obtained by the display driver querying the GPU driver when triggering the display, or can be transmitted by the GPU driver to the display driver when the rendering of frame 1B is completed. The triggering condition of the message B is not limited here.

[0223] In the embodiments of the present application, frame 1B can be completed before the Vsync2 signal, or can be completed before the Vsync3 signal. The specific implementation is not limited here.

[0224] S915, the display driver displays frame 1 after receiving the message B.

[0225] In this way, the display of frame 1 does not need to wait for the completion of the rendering of frame 1A, and the display lag caused by the long rendering time can be reduced.

[0226] S916, the drawing and rendering of frame 2B is started when the Vsync2 signal arrives.

[0227] The drawing and rendering process of frame 2B is similar to that of frame 1B. Specifically, the UI thread B of the desktop application draws frame 2B corresponding to layer B to obtain drawing instruction 2B. The UI thread B transmits the drawing instruction 2B to the rendering thread B. Adaptively, the rendering thread B performs rendering preprocessing after receiving the drawing instruction 2B to obtain rendering instruction 2B. The rendering thread B transmits the rendering instruction 2B to the GPU driver to render frame 2B through the GPU.

[0228] It can be understood that if the electronic device performs S920 before S916, the image compositing system composites frame 1A and frame 1B in S916 to obtain frame 1.

[0229] S918. When the Vsync3 signal arrives, the image compositing system composites frame 2B to create frame 2. Adaptively, the image compositing system transmits the composited frame 2 to the display driver.

[0230] It is understandable that, since frame 2B is a frame for the second type of application, and frames 1A and 2A are frames for the first type of application, and the image compositing system finds that frame 1A has not been rendered (rendering is finished), the image compositing system synthesizes frame 2B through frame 2B.

[0231] S919. When the Vsync3 signal arrives, begin drawing and rendering frames 3B and 2A.

[0232] The compositing and rendering of frame 3B can refer to the rendering and rendering process of frame 2B or frame 1B as described above, and will not be elaborated here; the compositing and rendering of frame 2A can refer to the rendering and rendering process of frame 1A as described above, and will not be elaborated here.

[0233] In S920 and after S918, after frame 1A is rendered, the GPU driver modifies the rendering status of frame 1A to "rendering complete". For example, the GPU driver changes the fence value corresponding to frame 1A from "dequeued" to "queued". No specific limitations are made here.

[0234] S921. When the Vsync4 signal arrives, frame 3B and frame 1A are combined to form frame 3. The image compositing system transmits the combined frame 3 to the display driver.

[0235] It is understandable that, since frame 3B is a frame for the second type of application, and frames 1A and 2A are frames for the first type of application, and the image compositing system finds that frame 1A has completed rendering (rendering finished), the image compositing system synthesizes frame 3 using frames 3B and 1A.

[0236] S922, The display driver receives message C from the GPU driver. Message C indicates that frame 2B rendering is complete.

[0237] It is understandable that message C could be obtained by the display driver querying the GPU driver when triggering display, or it could be transmitted by the GPU driver to the display driver at the end of frame 2B rendering. There are no specific restrictions on the triggering conditions for message C.

[0238] S923, The display driver displays frame 2 after receiving message C.

[0239] In this way, framing frame 1B after rendering is complete can reduce display stuttering caused by excessively long rendering time for frame 1B.

[0240] S924. After receiving message D, the display driver displays frame 3. Message C is used to indicate that the rendering of frame 3B is complete.

[0241] The trigger condition of message D is similar to that of message C, which will not be described herein. During the animation effect, the display process of other frames can refer to the display processes of frame 1, frame 2 and frame 3, which will not be described in detail herein.

[0242] S925, the desktop application transmits a message to the image synthesis system to indicate the end of the animation effect. Adaptively, the image synthesis system performs synthesis processing according to synthesis strategy B until the end of the animation effect after receiving the message.

[0243] In the embodiments of the present application, synthesis strategy B is used to indicate that the synchronization of the cache of each application is rendered and synthesized.

[0244] In some embodiments, the electronic device can also not perform S925. The electronic device can determine the animation effect technology based on any other manner. For example, the animation effect duration is also included in message A, and the electronic device can determine the end time of the animation effect based on the animation effect duration. For example, the animation effect starts at 8 ms and the animation effect duration is 100 ms, and the electronic device ends the animation effect at 108 ms.

[0245] S926, when the Vsync5 signal arrives, the UI thread A of the game application draws frame 3A corresponding to layer A to obtain drawing instruction 3A.

[0246] S927, the UI thread A transmits the drawing instruction 3A to the rendering thread A. Adaptively, the rendering thread A performs rendering preprocessing on frame 3A to obtain rendering instruction 3A after receiving the drawing instruction 3A.

[0247] S928, the rendering thread A transmits the rendering instruction 3A to the GPU driver to render frame 3A through the GPU.

[0248] S929, the rendering thread A transmits a message to the image synthesis system to indicate the rendering of frame 3A, so as to wait for synthesis (cache queuing). The message includes the address of the cache corresponding to frame 3A. In this way, it is convenient for subsequent synthesis (consumption) of the image synthesis system.

[0249] S930, since the animation effect ends after S925, the image synthesis system synthesizes frame 5 through frame 3A when receiving the Vsync6 signal.

[0250] S931, the image synthesis system transmits the synthesized frame 5 to the display driver.

[0251] S932, the display driver receives message D from the GPU driver, and message D is used to indicate that the rendering of frame 3A is completed.

[0252] S933, the display driver displays frame 5 after receiving message D.

[0253] In summary, the electronic device synthesizes according to the synthesis strategy A during the motion effect, the cache of the partial application is synthesized after the rendering is completed, which can reduce the display lag caused by long rendering time; and synthesizes according to the synthesis strategy B after the motion effect is completed, the cache of each application is synchronized for rendering and synthesis, which can reduce the synthesis waiting time and improve the response speed of interface display.

[0254] In addition, synthesizing according to the synthesis strategy A during the motion effect can also reduce the power consumption of the electronic device. This is because the electronic device can perform frequency adjustment according to the time points in the display flow, and when the display lags, the rendering time and the synthesis time can be shortened by frequency adjustment of the GPU and the CPU. In the embodiment of the present application, the display lag is reduced, and then the frequency adjustment processing of the GPU and the CPU is reduced, thereby reducing the power consumption of the electronic device.

[0255] For example, FIG. 10 is a display flow diagram provided by an embodiment of the present application. As shown in FIG. 10, at 0 ms, the Vsync1 signal arrives, the game application starts to draw and render pre-process frame 1A, and transmits frame 1A to the cache queue A for storage to wait for synthesis. The game application also calls the GPU to render frame 1A.

[0256] The desktop application starts to draw and render pre-process frame 1B, and transmits frame 1B to the cache queue B for storage to wait for synthesis. The desktop application also calls the GPU to render frame 1B.

[0257] At 6.6 ms, the Vsync2 signal arrives, the desktop application starts to draw and render pre-process frame 2B, and transmits frame 2B to the cache queue B for storage to wait for synthesis. The desktop application also calls the GPU to render frame 2B.

[0258] Since the desktop application is the second type of application, the image synthesis system synthesizes frame 1B to obtain frame 1 and transmits the synthesized frame 1 to the display driver. Since the game application is the first type of application and frame 1A has not been rendered, the image synthesis system does not synthesize frame 1A.

[0259] At 33.2 ms, the Vsync3 signal arrives. The display driver displays frame 1.

[0260] The game application starts to draw and render pre-process frame 2A, and transmits frame 2A to the cache queue A for storage to wait for synthesis. The game application also calls the GPU to render frame 2A.

[0261] The desktop application starts to draw and render pre-process frame 3B, and transmits frame 3B to the cache queue B for storage to wait for synthesis. The desktop application also calls the GPU to render frame 3B.

[0262] Since the desktop application is the second type of application, the game application is the first type of application, and frame 1A is not completed rendering, the image composition system composes frame 2B to obtain frame 2, and transmits the composed frame 2 to the display driver.

[0263] 49.8 ms, the Vsync4 signal arrives.

[0264] The desktop application starts to draw and render pre-process frame 4B, and transmits frame 4B to the buffer queue B for storage to wait for composition. The desktop application also calls the GPU to render frame 4B.

[0265] Since the desktop application is the second type of application, the game application is the first type of application, and frame 1A is completed rendering and frame 2A is not completed rendering, the image composition system composes frame 3B and frame 1A to obtain frame 3, and transmits the composed frame 3 to the display driver.

[0266] 66.4 ms, the Vsync5 signal arrives. Since frame 1A and frame 3B are both completed rendering, the display driver displays frame 3.

[0267] The game application starts to draw and render pre-process frame 3A, and transmits frame 3A to the buffer queue A for storage to wait for composition. The game application also calls the GPU to render frame 3A.

[0268] Since the animation effect ends, the desktop application does not draw and render.

[0269] Since the animation effect ends, the image composition system composes frame 2A and frame 4B to obtain frame 4.

[0270] 83 ms, the Vsync5 signal arrives. Since frame 2A and frame 4B are both completed rendering, the display driver displays frame 4.

[0271] Since the animation effect ends, the image composition system composes frame 3A to obtain frame 5.

[0272] 99.6 ms, the Vsync6 signal arrives. Since frame 3A is completed rendering, the display driver displays frame 5.

[0273] As can be seen from FIG. 10, in the 7 Vsync periods from 0 ms to 106.2 ms, the interface is displayed as frame-1, frame 0, frame 1, frame 2, frame 3, frame 4 and frame 5 in turn. The time length of frame-1, frame 0, frame 1, frame 2, frame 3, frame 4 and frame 5 is consistent, and the interface changes continuously and smoothly without lag.

[0274] The above FIG. 9 and FIG. 10 illustrate the display of the animation effect scene when the application starts, and the following FIG. 11 illustrates the display of the animation effect scene when entering the multi-task scene.

[0275] Illustratively, FIG. 11 is a flow diagram of a data processing method provided by an embodiment of the present application. Taking the game application opening as an example, as shown in FIG. 11, the method comprises:

[0276] S1101, the electronic device receives a trigger operation of entering the multi-task interface, and calls the desktop application to open.

[0277] In some embodiments, after the electronic device is opened, the desktop application continues to run, and the opening of the desktop application can also be referred to as waking up the desktop application.

[0278] S1102, after receiving the trigger operation of S1101, the desktop application transmits a message E to the image synthesis system, and the message E is used to indicate the opening of the animation effect. Adaptively, the surface flinger performs synthesis processing according to the synthesis strategy A until the animation effect ends after receiving the message.

[0279] In some embodiments, the message E comprises: a type identifier B of the animation effect, and an identifier B used to indicate the opening of the animation effect. In this way, the image synthesis system can confirm that the animation effect is the animation effect of entering the multi-task according to the type identifier of the animation effect, and then confirm the synthesis processing logic of the interface. Here, no detailed description is given.

[0280] The synthesis strategy A and the judgment of each buffer can refer to the corresponding description above, and no detailed description is given here.

[0281] S1104, when the Vsync1 signal arrives, the UI thread B of the desktop application draws the frame 1B corresponding to the layer B, and obtains the drawing instruction 1B.

[0282] The drawing instruction and the drawing and rendering process of the desktop application can refer to the corresponding description above, and no detailed description is given here.

[0283] S1105, the UI thread B transmits the drawing instruction 1B to the rendering thread B of the desktop application. Adaptively, the rendering thread B performs rendering preprocessing on the frame 1B after receiving the drawing instruction 1B, and obtains the rendering instruction 1B.

[0284] The rendering preprocessing can refer to the corresponding description above, and no detailed description is given here.

[0285] S1106, the rendering thread B transmits the rendering instruction 1B to the GPU driver to render the frame 1B through the GPU.

[0286] S1107, the rendering thread B transmits a message used to indicate the rendering of the frame 1B to the image synthesis system to wait for synthesis (buffer queuing). The message comprises: the address of the buffer corresponding to the frame 1B. In this way, it is convenient for subsequent synthesis (consumption) of the image synthesis system.

[0287] S1108, when the Vsyncl signal arrives, the UI thread A of the game application draws the frame 1A corresponding to the layer A to obtain drawing instruction 1A.

[0288] It can be understood that the electronic device can execute S1108 and S1104 simultaneously, or can not execute them simultaneously. The sequence of S1108 and S1104 is not limited here.

[0289] S1109, the UI thread A transmits the drawing instruction 1A to the rendering thread A of the game application. Adaptively, the rendering thread A performs rendering preprocessing on the frame 1A after receiving the drawing instruction 1A to obtain rendering instruction 1A.

[0290] S1110, the rendering thread A transmits the rendering instruction 1A to the GPU driver to render the frame 1A by the GPU.

[0291] In some embodiments, an identifier (for example, fence) for indicating the rendering condition is set in the GPU driver. After the GPU rendering is completed, the identifier corresponding to the frame is modified to indicate that the rendering is completed. For example, the fence value corresponding to the frame 1A is dequeued when the GPU renders the frame 1A, and the fence value corresponding to the frame 1A is queued when the GPU renders the frame 1A. The specific implementation is not limited here.

[0292] S1111, the rendering thread A transmits a message for indicating the rendering of the frame 1A to the image synthesis system to wait for synthesis (buffering into a queue). The message includes the address of the buffer corresponding to the frame 1A. In this way, the subsequent synthesis (consumption) of the image synthesis system is facilitated.

[0293] S1112, when the Vsync2 signal arrives, the image synthesis system synthesizes the frame 1B to synthesize the frame 1.

[0294] It can be understood that since the frame 1B is the frame of the second type of application, the frame 1A is the frame of the first type of application, and the rendering of the frame 1A is not completed, the image synthesis system synthesizes the frame 1 through the frame 1B.

[0295] In some embodiments, the image synthesis system can query the rendering condition of each frame from the GPU driver when the Vsync2 signal arrives. For example, the fence value corresponding to each frame of the GPU can be queried.

[0296] In some embodiments, the GPU driver can also transmit a message for indicating the completion of the rendering to the image synthesis system after the rendering of the frame 1A is completed to facilitate the judgment of the image synthesis system. The specific implementation is not limited here.

[0297] S1113, the image synthesis system transmits the synthesized frame 1 to the display driver.

[0298] S1114, the display driver receives a message B from the GPU driver, the message B being used to indicate that the frame 1 rendering is completed.

[0299] It can be understood that the message B can be queried by the display driver from the GPU driver when triggering the display, or can be transmitted by the GPU driver to the display driver when the frame 1B rendering is completed. The triggering condition of the message B is not limited.

[0300] S1115, the display driver displays the frame 1 after receiving the message B.

[0301] In this way, the display of the frame 1 does not need to wait for the frame 1A rendering to be completed, and the display lag caused by too long rendering time can be reduced.

[0302] S1116, when the Vsync2 signal arrives, the drawing and rendering of the frame 2B is started.

[0303] The drawing and rendering process of the frame 2B is similar to that of the frame 1B. Specifically, the UI thread B of the desktop application draws the frame 2B corresponding to the layer B, and obtains the drawing instruction 2B. The UI thread B transmits the drawing instruction 2B to the rendering thread B. Adaptively, the rendering thread B performs rendering preprocessing after receiving the drawing instruction 2B, and obtains the rendering instruction 2B. The rendering thread B transmits the rendering instruction 2B to the GPU driver, so as to render the frame 2B by the GPU. S1117, the rendering thread B transmits a message for indicating the frame 2B rendering to the image synthesis system, so as to wait for synthesis (buffering into a queue). The message includes the address of the buffer corresponding to the frame 2B. In this way, the subsequent synthesis (consumption) of the image synthesis system is facilitated.

[0304] It can be understood that if the electronic device performs S1120 before S1116, the image synthesis system synthesizes the frame 1A and the frame 1B in S1116, and obtains the frame 1.

[0305] S1118, when the Vsync3 signal arrives, the image synthesis system synthesizes the frame 2B to obtain the frame 2. Adaptively, the image synthesis system transmits the synthesized frame 2 to the display driver.

[0306] It can be understood that since the frame 2B is the frame of the second type of application, the frame 1A and the frame 2A are the frames of the first type of application, and the image synthesis system queries that the frame 1A is not completed rendering (the rendering is completed), the image synthesis system synthesizes the frame 2 by the frame 2B.

[0307] S1119, when the Vsync3 signal arrives, the drawing and rendering of the frame 3B and the frame 2A is started.

[0308] The composition rendering of frame 3B can refer to the rendering process of frame 2B or frame 1B described above, which will not be described in detail here; the composition rendering of frame 2A can refer to the rendering process of frame 1A described above, which will not be described in detail here.

[0309] S1120, after S1118, the GPU driver modifies the rendering state of frame 1A to rendering completed after the rendering of frame 1A is completed. For example, the GPU driver modifies the fence value corresponding to frame 1A from dequeued to queued. Here, no specific limitation is made.

[0310] S1121, when the Vsync4 signal arrives, frame 3B and frame 1A are composed to synthesize frame 3. The image composition system transmits the synthesized frame 3 to the display driver.

[0311] It can be understood that since frame 3B is a frame of the second type of application, frame 1A and frame 2A are frames of the first type of application, and the image composition system queries that frame 1A is rendered (rendering is completed), the image composition system synthesizes frame 3 by frame 3B and frame 1A.

[0312] S1122, the display driver receives message C from the GPU driver, and message C is used to indicate that frame 2B is rendered.

[0313] It can be understood that message C can be queried by the display driver from the GPU driver when triggering the display, or can be transmitted by the GPU driver to the display driver when frame 2B is rendered. The triggering condition of message C is not specifically limited.

[0314] S1123, the display driver displays frame 2 after receiving message C.

[0315] In this way, frame 1B is synthesized after rendering is completed, which can reduce the display lag caused by too long rendering time of frame 1B.

[0316] S1124, the display driver displays frame 3 after receiving message D, and message C is used to indicate that frame 3B is rendered.

[0317] The triggering condition of message D is similar to the triggering condition of message C described above, which will not be described in detail here. During the motion effect, the display process of other frames can refer to the display process of frame 1, frame 2 and frame 3 described above, which will not be described in detail here.

[0318] S1125, the desktop application transmits a message to the image composition system to indicate the end of the motion effect. Adaptively, the image composition system performs composition processing according to the composition strategy B until the motion effect ends.

[0319] In the embodiment of the application, the composition strategy B is used to indicate that the synchronization of the buffers of each application is rendered and composed.

[0320] In some embodiments, the electronic device can also not perform S1125. The electronic device can determine the animation effect technology based on any other manner. For example, the message A further includes an animation effect duration, and the electronic device can determine the animation effect end time based on the animation effect duration. For example, the animation effect start time is 8 ms, and the animation effect duration is 100 ms, and the electronic device ends the animation effect at 108 ms.

[0321] S1126, when the Vsync5 signal arrives, the UI thread B draws the frame 5B corresponding to the layer B to obtain the drawing instruction 3B.

[0322] S1127, the UI thread B transmits the drawing instruction 3B to the rendering thread B. Adaptively, the rendering thread B performs rendering preprocessing on the frame 5B after receiving the drawing instruction 3B to obtain the rendering instruction 3B.

[0323] S1128, the rendering thread B transmits the rendering instruction 3B to the GPU driver to render the frame 5B by the GPU.

[0324] S1129, the rendering thread B transmits a message for indicating the rendering of the frame 5B to the image synthesis system to wait for synthesis (buffer enqueuing). The message includes the address of the buffer corresponding to the frame 5B. In this way, it is convenient for subsequent synthesis (consumption) of the image synthesis system.

[0325] S1130, since the animation effect ends after S1125, the image synthesis system synthesizes the frame 5 by the frame 5B when receiving the Vsync6 signal.

[0326] S1131, the image synthesis system transmits the synthesized frame 5 to the display driver.

[0327] It can be understood that the multi-task interface shown in FIG. 11 also displays a corresponding card of the game application, and the card displays an image of the game application. The image can be the last frame image drawn for the game application in the animation effect, for example, the frame 2A, or a screenshot of the last frame image drawn for the game application, or other images related to the game application, which are not limited here.

[0328] In the embodiments of the present application, the multi-task interface can be obtained by synthesizing the last frame image drawn for the game application and the image drawn for the desktop application by the image synthesis system, for example, synthesizing the frame 2A and the frame 5B to obtain the frame 5. The multi-task interface can also be obtained by any other manner, for example, by superimposing the image drawn for the desktop application and the screenshot of the last frame image drawn for the game application, or the image drawn for the desktop application includes the screenshot of the last frame image drawn for the game application, which are not limited here.

[0329] S1132, the display driver receives a message D from the GPU driver, the message D is used to indicate that the frame 5B rendering is completed.

[0330] S1133, the display driver displays the frame 5 after receiving the message D.

[0331] In summary, the electronic device synthesizes according to the synthesis strategy A during the motion effect, and the cache of the partial application is synthesized after the rendering is completed, which can reduce the display lag caused by long rendering time; after the motion effect is completed, the synthesis is performed according to the synthesis strategy B, and the caches of the applications are synchronized for rendering and synthesis, which can reduce the synthesis waiting time and improve the response speed of the interface display.

[0332] In addition, synthesizing according to the synthesis strategy A during the motion effect can also reduce the power consumption of the electronic device. This is because the electronic device can perform frequency adjustment according to the time points in the display flow, and when the display lags, the rendering time and the synthesis time can be shortened by increasing the frequency of the GPU and the CPU. In the embodiment of the present application, the display lag is reduced, and then the frequency adjustment of the GPU and the CPU is reduced, and the power consumption of the electronic device is reduced.

[0333] For example, FIG. 12 is a display flow diagram provided by an embodiment of the present application. As shown in FIG. 12,

[0334] At 0ms, the Vsync1 signal arrives, the game application starts to draw and render the frame 1A, and transmits the frame 1A to the cache queue A for storage to wait for synthesis. The game application also calls the GPU to render the frame 1A.

[0335] The desktop application starts to draw and render the frame 1B, and transmits the frame 1B to the cache queue B for storage to wait for synthesis. The desktop application also calls the GPU to render the frame 1B.

[0336] At 6.6ms, the Vsync2 signal arrives, the desktop application starts to draw and render the frame 2B, and transmits the frame 2B to the cache queue B for storage to wait for synthesis. The desktop application also calls the GPU to render the frame 2B.

[0337] Since the desktop application is the second type of application, the image synthesis system synthesizes the frame 1B to obtain the frame 1, and transmits the synthesized frame 1 to the display driver. Since the game application is the first type of application, and the frame 1A is not completed, the image synthesis system does not synthesize the frame 1A.

[0338] At 33.2ms, the Vsync3 signal arrives. The display driver displays the frame 1.

[0339] The game application starts to draw and render preprocess for frame 2A, and transmits frame 2A to the buffer queue A for storage, waiting for composition. The game application also calls GPU to render frame 2A.

[0340] The desktop application starts to draw and render preprocess for frame 3B. And transmits frame 3B to the buffer queue B for storage, waiting for composition. The desktop application also calls GPU to render frame 3B.

[0341] Since the desktop application is the second type of application, the game application is the first type of application, and frame 1A is not completed rendering, the image composition system composes frame 2B to obtain frame 2, and transmits the composed frame 2 to the display driver.

[0342] 49.8ms, the Vsync4 signal arrives.

[0343] The desktop application starts to draw and render preprocess for frame 4B. And transmits frame 4B to the buffer queue B for storage, waiting for composition. The desktop application also calls GPU to render frame 4B.

[0344] Since the desktop application is the second type of application, the game application is the first type of application, and frame 1A is completed rendering, frame 2A is not completed rendering, the image composition system composes frame 3B and frame 1A to obtain frame 3, and transmits the composed frame 3 to the display driver.

[0345] 66.4ms, the Vsync5 signal arrives. Since frame 1A and frame 3B are both completed rendering, the display driver displays frame 3.

[0346] The desktop application starts to draw and render preprocess for frame 5B. And transmits frame 5B to the buffer queue B for storage, waiting for composition. The desktop application also calls GPU to render frame 5B.

[0347] Since the animation effect ends, the desktop application does not draw and render.

[0348] Since the animation effect ends, the image composition system composes frame 2A and frame 4B to obtain frame 4.

[0349] 83ms, the Vsync5 signal arrives. Since frame 2A and frame 4B are both completed rendering, the display driver displays frame 4.

[0350] Since the animation effect ends, the image composition system composes frame 5B to obtain frame 5.

[0351] 99.6ms, the Vsync6 signal arrives. Since frame 5B is completed rendering, the display driver displays frame 5.

[0352] As can be seen from FIG. 12, in the 7 Vsync cycles from 0 ms to 106.2 ms, the interface is displayed in turn as frame-1, frame 0, frame 1, frame 2, frame 3, frame 4 and frame 5. The time length of display of frame-1, frame 0, frame 1, frame 2, frame 3, frame 4 and frame 5 is consistent, and the interface changes continuously and smoothly without lag. In addition, the time length of display of frame 1B, frame 2B, frame 3B and frame 4B is consistent, and the interface changes continuously and smoothly without lag.

[0353] The above embodiments describe the starting of the application and the entering of the multi-task interface. It can be understood that the exiting of the application and the switching of the synthesis strategy in other application scenarios such as the application from the multi-task interface are similar to the switching of the synthesis strategy in the application starting and entering multi-task interface application scenario in the above embodiments, and will not be described in detail here.

[0354] It can be understood that when the dynamic effect is started, the identification of the dynamic effect type is transmitted. The image synthesis system can also confirm whether to switch the synthesis strategy according to the identification of the dynamic effect type. That is, in some dynamic effect scenarios (for example, starting dynamic effect, exiting dynamic effect, etc.), synthesis strategy B is adopted; in other dynamic effect scenarios (for example, dynamic effect under different scene switching of the application, etc.), synthesis strategy A is adopted. Here, no specific limitation is made. In some embodiments, the image synthesis system can also not be triggered by the Vsync signal.

[0355] For example, FIG. 13 is a display flow diagram for entering a multi-task interface provided by an embodiment of the present application. Taking the frame rate of the desktop application as 120 fps, the frame rate of the game application as 60 fps, and the screen refresh rate as 120 Hz as examples. As shown in FIG. 13,

[0356] At 0 ms, the Vsync signal arrives, and the game application starts to draw and render frame 1A for preprocessing, and transmits frame 1A to the buffer queue A for storage to wait for synthesis. The game application also calls the GPU to render frame 1A.

[0357] The desktop application starts to draw and render frame 1B for preprocessing. And transmits frame 1B to the buffer queue B for storage to wait for synthesis. The desktop application also calls the GPU to render frame 1B.

[0358] During 0 ms to 8.3 ms, since the desktop application is the second type of application, the image synthesis system synthesizes frame 1B to obtain frame 1 and transmits the synthesized frame 1 to the display driver. Since the game application is the first type of application and frame 1A has not completed rendering, the image synthesis system does not synthesize frame 1A.

[0359] At 8.3 ms, the Vsync signal arrives. The display driver displays frame 1.

[0360] The desktop application starts drawing and rendering preprocessing for frame 2B, and transmits frame 2B to the buffer queue B for storage to wait for composition. The desktop application also calls the GPU to render frame 2B.

[0361] During 8.3 ms to 16.6 ms, since the desktop application is the second type of application, the game application is the first type of application, and frame 1A is completed rendering, the image composition system composes frame 2B and frame 1A to obtain frame 2, and transmits the composed frame 2 to the display driver.

[0362] At 16.6 ms, the Vsync signal arrives. The display driver displays frame 2.

[0363] The game application starts drawing and rendering preprocessing for frame 2A, and transmits frame 2A to the buffer queue A for storage to wait for composition. The game application also calls the GPU to render frame 2A.

[0364] The desktop application starts drawing and rendering preprocessing for frame 3B, and transmits frame 3B to the buffer queue A for storage to wait for composition. The desktop application also calls the GPU to render frame 3B.

[0365] During 16.6 ms to 24.9 ms, since the desktop application is the second type of application, the game application is the first type of application, and frame 1A is completed rendering, the image composition system composes frame 3B and frame 1A to obtain frame 3, and transmits the composed frame 3 to the display driver.

[0366] At 24.9 ms, the Vsync signal arrives. The display driver displays frame 3.

[0367] The desktop application starts drawing and rendering preprocessing for frame 4B. Transmits frame 4B to the buffer queue B for storage to wait for composition. The desktop application also calls the GPU to render frame 4B.

[0368] During 24.9 ms to 33.2 ms, since the desktop application is the second type of application, the game application is the first type of application, and frame 2A is completed rendering, the image composition system composes frame 4B and frame 2A to obtain frame 4, and transmits the composed frame 4 to the display driver.

[0369] At 33.2 ms, the Vsync signal arrives. The display driver displays frame 6.

[0370] As can be seen from FIG. 13, in the 5 Vsync cycles from 0 ms to 58.1 ms, the interface is displayed in turn as frame 0, frame 1, frame 2, frame 3, and frame 4. The display time of frame 0, frame 1, frame 2, frame 3, and frame 4 is consistent, and the interface changes continuously and smoothly without lag.

[0371] In addition, the time length of frame 1A and frame 2A is consistent, the time length of frame 1B, frame 2B, frame 3B and frame 4B is consistent, and the interface change is continuous and smooth without lag.

[0372] In the above embodiment, the frame rate of the desktop application and the frame rate of the game application are different. For example, the frame rate of the desktop application is 120fps, and the frame rate of the game application is 60fps. In some embodiments, the frame rate of the first type of application and the second type of application can be the same or different, which is not limited here. In some embodiments, the second type of application can also be a non-first type of application.

[0373] It can be understood that the above embodiment is described by taking two applications as an example. In some embodiments, the interface can also be composed of three or more application layers, and the specific implementation manner is similar to the above manner, which is not described in detail here. The above embodiment is described by taking the desktop application as the second type of application and the game application as the first type of application. The second type of application and the first type of application can also include other applications, which are not limited here.

[0374] For example, the first type of application can be a game scene, a video editing application, a map application, and other applications with long rendering time. The second type of application can be a desktop application, and a system application (system server, system UI, etc.) for displaying a pop-up message, a volume bar, etc.

[0375] It should be understood that the above embodiment is described by taking GPU rendering as an example. The electronic device can also perform rendering through a CPU. When rendering through the CPU, the image synthesis system can query the rendering of each frame from the CPU, and the display driver can query the rendering of each frame from the CPU.

[0376] It should be noted that the module names involved in the embodiments of the present application can be defined as other names, as long as the functions of the modules can be realized, and the names of the modules are not limited.

[0377] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0378] The data processing method of the embodiments of the present application has been described above, and the device provided by the embodiments of the present application for executing the above method is described below. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the related device provided by the embodiments of the present application can execute the steps in the above method.

[0379] The data processing method provided by the embodiments of the present application can be applied in an electronic device with a data processing function. The electronic device includes a terminal device, and the specific device form of the terminal device can refer to the above related description, which will not be repeated here.

[0380] The embodiments of the present application provide an electronic device, which includes one or more processors and a memory. The memory is coupled with the one or more processors, and the memory is configured to store computer program codes including computer instructions. The one or more processors invoke the computer instructions to enable the electronic device to execute the above method.

[0381] The embodiments of the present application provide a chip or a chip system. The chip or the chip system includes one or more processors configured to invoke computer instructions to enable the electronic device to execute the technical solutions in the above embodiments. The implementation principles and technical effects are similar to those of the above related embodiments, which will not be repeated here.

[0382] The embodiments of the present application also provide a computer readable storage medium. The computer readable storage medium includes computer instructions, which, when executed on an electronic device, enable the electronic device to execute the above method. The method described in the above embodiments can be realized by software, hardware, firmware or any combination thereof, in whole or in part. If realized in software, the functions can be stored as one or more instructions or codes on or transmitted on a computer readable medium. The computer readable medium can include computer storage medium and communication medium, and can also include any medium that can transfer computer programs from one place to another. The storage medium can be any target medium accessible by a computer.

[0383] In a possible implementation, the computer readable medium can include a RAM, a ROM, a compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that is suitable for storing desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer readable media.

[0384] The embodiment of the present application provides a computer program product, which comprises computer program code, when the computer program code is executed, the computer program code causes the computer to execute the above method.

[0385] The embodiment of the present application is described with reference to the flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiment of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing devices produce a device for implementing the functions specified in one or more flows in the flowcharts and / or one or more blocks in the block diagrams.

[0386] The above detailed description of the specific implementation is further detailed for the purpose of the present application, technical solutions, and beneficial effects, and it should be understood that the above is only a specific implementation of the present application, and is not used to limit the protection scope of the present application, and any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present application should be included in the protection scope of the present application.

Claims

1. A data processing method, characterized by, The application is applied to an electronic device, comprising: A first time, a desktop receives an operation for starting a first application; A second time, a desktop application starts to draw an Nth frame image of the desktop application, the first application starts to draw an Mth frame dynamic effect image of the first application, and N and M are both integers greater than zero; A third time, the drawing of the Nth frame image is completed and the rendering of the Nth frame image is not completed, the drawing of the Mth frame dynamic effect image is completed and the rendering of the Mth frame dynamic effect image is not completed, the Nth frame image is synthesized, and the Mth frame dynamic effect image is not synthesized; A fourth time, the desktop application completes the drawing of an N+i th frame image and the rendering of the N+i th frame image is not completed, the Mth frame dynamic effect image is rendered, the N+i th frame image and the Mth frame dynamic effect image are synthesized, and i is an integer greater than zero; A fifth time, the dynamic effect of the first application is started, the first application completes the drawing of an Lth frame image during the running of the first application and the rendering of the Lth frame image is not completed, and the Lth frame image is synthesized. The second time is later than the first time, the third time is later than the second time, the fourth time is later than the third time, and the fifth time is later than the fourth time.

2. The method of claim 1, wherein, The electronic device comprises an application list of a first type of application, and the application list is used to indicate that the images of the first type of application are synthesized after the rendering is completed in a dynamic effect scene. The first application is the first type of application, and the desktop application is not the first type of application.

3. The method of claim 2, wherein, The electronic device comprises an image synthesis system used to synthesize images. After the first time, the method further comprises: Before the third time, the image synthesis system generates a first Vsync signal used to trigger the image synthesis system to synthesize images. After the image synthesis system generates the first Vsync signal, the image synthesis system confirms, through the application list, that the Nth frame image is not an image of the first type of application, the Mth frame dynamic effect image is an image of the first type of application, and the rendering of the Mth frame dynamic effect image is not completed. Before the fourth time, the image synthesis system generates a second Vsync signal used to trigger the image synthesis system to synthesize images. After the image synthesis system generates the second Vsync signal, the image synthesis system confirms, through the application list, that the N+i th frame image is not an image of the first type of application, the Mth frame dynamic effect image is an image of the first type of application and the rendering of the Mth frame dynamic effect image is completed.

4. The method of claim 3, wherein, The electronic device comprises a GPU driver of a GPU processing unit, and the GPU driver is used to drive the GPU to render a dynamic effect image of the first application. After the first Vsync signal, the image synthesis system receives a first message from the GPU driver, the first message is used to indicate that the rendering of the Mth frame dynamic effect image is not completed, and the third time is later than the time when the image synthesis system receives the first message. After the second Vsync signal, the image synthesis system receives a second message from the GPU driver, the second message being used to indicate that the rendering of the Mth frame of the dynamic effect image is completed, and the fourth time point is later than the time point at which the image synthesis system receives the second message.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: At a sixth time point, the desktop application transmits a third message to the image synthesis system, the third message being used to indicate that the dynamic effect is started, the sixth time point being later than the first time point and earlier than the second time point; After a preset time length at the sixth time point, the desktop application transmits a fourth message to the image synthesis system, the fourth message being used to indicate that the dynamic effect is ended, the fourth time point being earlier than the preset time length at the sixth time point, and the fifth time point being later than the preset time length at the sixth time point.

6. The method of any one of claims 1-5, wherein, At a seventh time point, an operation for entering a multi-task interface is received at the interface of the first application, and the multi-task interface comprises a first card used to display an image of the first application. At an eighth time point, the desktop application starts to draw an A th frame of image of the desktop application, and the first application starts to draw a B th frame of dynamic effect image of the first application, A and B being integers greater than zero. At a ninth time point, the drawing of the A th frame of image is completed and the rendering of the A th frame of image is not completed, the drawing of the B th frame of dynamic effect image is completed and the rendering of the B th frame of dynamic effect image is not completed, the A th frame of image is synthesized, and the B th frame of dynamic effect image is not synthesized. At a tenth time point, the desktop application completes the drawing of an A+j th frame of image and the rendering of the A+j th frame of image is not completed, the rendering of the B th frame of dynamic effect image is completed, the A+j th frame of image and the B th frame of dynamic effect image are synthesized, and j is an integer greater than zero.

7. An electronic device, comprising: The electronic device comprises one or more processors and a memory; The memory is coupled to the one or more processors, and the memory is configured to store computer program codes, the computer program codes comprising computer instructions, and the one or more processors are configured to invoke the computer instructions to enable the electronic device to perform the method of any one of claims 1 to 6.

8. A chip system, characterized by The chip system is applied to an electronic device, and the chip system comprises one or more processors configured to invoke computer instructions to enable the electronic device to perform the method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises computer instructions, and when the computer instructions run on an electronic device, the computer instructions enable the electronic device to perform the method of any one of claims 1 to 6.

10. A computer program product, characterised in that, The computer program product comprises computer program codes, and when the computer program codes run on an electronic device, the computer program codes enable the electronic device to perform the method of any one of claims 1 to 6.

Images