Image processing method and electronic equipment

Abstract

The invention discloses an image processing method and electronic equipment, and the method comprises the steps: carrying out the drawing, rendering and synthesis operation of a first image frame and a second image frame of a first application through the electronic equipment in a process from the start to exit of the first application of the electronic equipment. The application process draws and renders the first image frame in a drawing and rendering period of the first image frame, the obtained first image frame is stored in an idle cache object of a cache queue, and when the synthesis thread does not execute synthesis operation in a synthesis period of the first image frame, the synthesis thread sends a first adjustment request to the application process, therefore, the application process increases the number of idle cache objects in the cache queue, and then the application process can store the drawn and rendered second image frame into one idle cache object of the cache queue. According to the method, the frame loss condition of the application process in the drawing and rendering process of the image frame is avoided, and the problem of image frame display lagging is solved.

Description

Image processing method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on October 13, 2022, with application number 202211253545.4 and invention name “Image Processing Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to an image processing method and electronic device. Background Art

[0003] With the advancement of terminal technology, the interaction performance between various terminals (such as mobile phones) and users is becoming increasingly optimized. For example, terminals can enhance the user experience by setting animation effects corresponding to operations. Animation effects are dynamic display effects created by the continuous display of multiple frames of images. Displaying images on electronic devices usually requires a series of processes such as drawing, rendering, and synthesis.

[0004] Among them, the application process of the electronic device is responsible for drawing and rendering each image frame in the display screen, and the synthesis thread of the electronic device is responsible for synthesizing the drawn and rendered image frames and sending them for display.

[0005] However, in some cases, the application process cannot draw the image frames normally, resulting in frame loss, which in turn causes the image frames sent by the compositor to display freeze.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide an image processing method and an electronic device, which, in the process of drawing image frames of motion effects, avoids the problem of excessive displacement caused by a large displacement interval between drawing two image frames, ensures the continuous display of each image frame of the motion effect, and makes the display effect smoother and more fluent.

[0008] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0009] In a first aspect, an image processing method is provided, the method comprising:

[0010] The electronic device receives a first operation performed by a user on a touch screen of the electronic device; the electronic device starts a first application in response to the first operation; the electronic device receives a second operation performed by the user on the touch screen of the electronic device; the electronic device exits the first application in response to the second operation.

[0011] In the process from starting the first application to exiting the first application, the electronic device performs drawing, rendering, and synthesis operations on the first image frame and the second image frame of the first application.

[0012] The electronic device performs rendering and synthesis operations on the first image frame and the second image frame of the first application, including:

[0013] The application process draws and renders the first image frame within the drawing and rendering cycle of the first image frame, and stores the obtained first image frame in a free cache object in the cache queue; when the synthesis thread does not perform a synthesis operation within the synthesis cycle of the first image frame, the synthesis thread sends a first adjustment request to the application process; based on the first adjustment request, the application process increases the number of free cache objects in the cache queue, so that the application process draws and renders the second image frame within the drawing and rendering cycle of the second image frame, and stores the obtained second image frame in a free cache object in the cache queue.

[0014] The drawing and rendering cycle of the second image frame is after the drawing and rendering cycle of the first image frame, and the drawing start time of the second image frame differs from the drawing start time of the first image frame by N cycles, where N is a positive integer.

[0015] In the present application, when the synthesizer is not synthesizing, the free cache objects in the cache queue can be dynamically increased, and the application process can continue to normally obtain the free cache objects to store the rendered image frames, thereby avoiding the problem in the prior art that the application process cannot obtain the free cache objects to store the rendered image frames and does not perform the rendering operation of the next image frame, thereby causing frame loss when drawing the next image frame. Through the image processing method provided by the present application, there is always at least one free cache object in the cache queue that can be used by the application process, avoiding the problem of frame loss caused by the application process not performing the image frame drawing and rendering operation due to the lack of free cache objects in the cache queue, thereby solving the problem of visual freeze caused by frame loss after the image frame is sent for display. By ensuring that the application process has enough free cache objects to perform the storage operation of the rendered image frame, the smoothness of the dynamic display effect of the displayed image frame is guaranteed.

[0016] In a possible implementation manner of the first aspect, the first image frame and the second image frame are image frames in a startup process of the first application.

[0017] In this application, during the startup process of an electronic device executing an application, image processing is performed on the image frames of the startup animation during the startup process. When the synthesis thread does not perform synthesis operations, the number of free cache objects in the cache queue is dynamically increased, so that there is always at least one free cache object in the cache queue used by the application process. During the drawing and rendering cycle of each image frame, the application process can store the rendered image frame into the free cache object of the cache queue, avoiding the frame loss that may occur during the drawing and rendering process of the application process, solving the problem of display jamming of the image frame sent to the display due to frame loss, and improving the display smoothness of the startup animation during the application startup process.

[0018] In a possible implementation of the first aspect, the first image frame and the second image frame are image frames when the electronic device switches from a first refresh rate to a second refresh rate during the startup of a first application; the first refresh rate is less than the second refresh rate.

[0019] The first refresh rate may be 60 Hz, i.e., 60 frames of images are refreshed in 1 second, or one frame is refreshed every 16.6 milliseconds. The second refresh rate may be 90 Hz, i.e., 90 frames of images are refreshed in 1 second, or one frame is refreshed every 11.1 milliseconds.

[0020] In the present application, in the scenario of refresh rate switching, the image processing method provided in this embodiment can effectively solve the problem that during the application startup process, due to the refresh rate switching, the synthesis thread is asynchronous with the application process processing cycle, and the synthesis thread believes that there is a task backlog in the image frame to be synthesized and does not perform the synthesis operation, resulting in no cache object being released in the cache queue.

[0021] In a possible implementation manner of the first aspect, the first image frame and the second image frame are image frames after the first application is started.

[0022] In the present application, for different scenarios, after the startup of the first application is completed, the image processing method provided in this embodiment can still dynamically increase the number of free cache objects in the cache queue when the synthesis thread does not perform synthesis operations. This solves the problem of display jamming of the image frames sent for display due to frame loss during the internal image display process of the first application after the startup of the first application is completed, for example, thereby improving the display smoothness of the images displayed within the application.

[0023] In a possible implementation manner of the first aspect, the first image frame and the second image frame are image frames in the process of exiting the first application.

[0024] In this application, during the exit process of an electronic device executing an application, image processing is performed on the image frames of the exit animation during the exit process. When the synthesis thread does not perform synthesis operations, the number of free cache objects in the cache queue is dynamically increased, so that there is always at least one free cache object in the cache queue used by the application process. During the drawing and rendering cycle of each image frame, the application process can store the rendered image frame into the free cache object of the cache queue, avoiding the frame loss that may occur during the drawing and rendering process of the application process, solving the problem of display freeze of the image frame sent to the display due to frame loss, and improving the display smoothness of the exit animation during the application exit process.

[0025] In a possible implementation of the first aspect, the first image frame and the second image frame are image frames when the electronic device switches from a first refresh rate to a second refresh rate during the exit process of the first application; the first refresh rate is less than the second refresh rate.

[0026] The first refresh rate may be 60 Hz, i.e., 60 frames of images are refreshed in 1 second, or one frame is refreshed every 16.6 milliseconds. The second refresh rate may be 90 Hz, i.e., 90 frames of images are refreshed in 1 second, or one frame is refreshed every 11.1 milliseconds.

[0027] In the present application, in the scenario of refresh rate switching, the image processing method provided in this embodiment can effectively solve the problem that during the application exit process, due to the refresh rate switching, the synthesis thread is asynchronous with the application process processing cycle, and the synthesis thread believes that there is a task backlog in the image frame to be synthesized and does not perform the synthesis operation, resulting in no cache object being released in the cache queue.

[0028] In a possible implementation of the first aspect, a drawing and rendering cycle of the second image frame is a cycle next to a drawing and rendering cycle of the first image frame; and a drawing start time of the second image frame differs from a drawing start time of the first image frame by one cycle.

[0029] In the present application, the drawing start time of the second image frame differs from the drawing start time of the first image frame by 1 cycle. If the synthesis thread does not perform the synthesis operation in the synthesis cycle of the first image frame, it will affect the drawing and rendering of the second image frame. Since the second image frame and the first image frame are adjacent image frames, when the second image frame is lost, there will be obvious image frame display jamming. Based on this, when the second image frame and the first image frame are adjacent image frames, the effect of the image processing method provided by the present application is more obvious. When the synthesis thread does not perform the synthesis operation, the number of free cache objects in the cache queue is dynamically increased so that there is always at least one free cache object in the cache queue used by the application process. In the drawing and rendering cycle of the second image frame, the application process can store the rendered image frame into the free cache object of the cache queue, avoiding the frame loss that may occur in the process of the application process drawing and rendering the image frame, solving the problem of display jamming of the image frame sent for display due to frame loss, and improving the display smoothness of the image frame.

[0030] In a possible implementation of the first aspect, the first adjustment request includes a first indication value; the first indication value is used to indicate an increase in the number of cache objects, and increasing the number of idle cache objects in the cache queue includes:

[0031] The application process adds a free cache object with a first indication value to the cache queue.

[0032] In this application, the application process can increase the number of free cache objects in the cache queue according to the first indicator value. In actual applications, the first indicator value can be 1, 2, 3..., and the indicator value can be adjusted according to different situations, so that the effect of dynamically adjusting the cache queue is better.

[0033] In a possible implementation of the first aspect, the application process adds a free cache object with a first indication value to a cache queue, including:

[0034] The application process adds the address of the free cache object of the first indicator value to the cache queue in the order of enqueuing.

[0035] In this application, each cache object in the cache queue has an arrangement order. The application process adds the address of the free cache object with the first indication value to the cache queue according to the order of enqueuing, which can ensure that the arrangement order of the cache objects in the original cache queue is not disturbed.

[0036] In a possible implementation of the first aspect, the method further includes:

[0037] The composition thread queries the number of all cache objects in the cache queue; if the number of all cache objects reaches the maximum number of cache objects, the composition thread stops sending the first adjustment request for the cache object to the application process.

[0038] In the present application, due to the hardware performance requirements of the electronic device, the number of cache objects in the cache queue has a maximum number of cache objects. When the synthesis thread determines that the number of all cache objects in the cache queue has reached the maximum number of cache objects, it stops sending the first adjustment request for adding free cache objects to the application process. In this way, the normal operation of the electronic device is guaranteed and the problem of electronic device abnormality caused by the inability to add cache objects in the cache queue is avoided.

[0039] In a possible implementation of the first aspect, after storing the first image frame in an idle cache object in the cache queue, the method further includes:

[0040] The synthesis thread obtains and records the storage time when the application process stores the first image frame into the target cache object.

[0041] In this application, after the composition thread completes a frame composition operation, it can record the time when the application process stores the frame into the target cache object. The composition thread records the time of each storage and, based on the time difference between each storage time, can determine whether the application process has completed the rendering of the image frame.

[0042] In a possible implementation of the first aspect, the method further includes:

[0043] When the synthesis thread performs a synthesis operation in the synthesis cycle of the first image frame, the synthesis thread determines the time difference between the current system time and the storage time of the last recorded image frame stored in the target cache object; if the time difference is greater than or equal to the preset time threshold, the synthesis thread sends a second adjustment request for the cache object to the application process; the application process reduces the number of idle cache objects in the cache queue according to the second adjustment request.

[0044] Optionally, the composition thread may also determine the frame interval based on the current refresh rate. The preset time threshold may be M frame intervals, where M is 1, 2, 3, etc.

[0045] In the present application, after the synthesis thread completes the synthesis operation of an image frame, the synthesis thread can record the storage time when the application process stores the image frame into the target cache object. The synthesis thread can obtain the time difference between the current system time and the last storage time. If the time difference is greater than the preset time threshold, it is determined that the rendering thread has lost frames. The existence of frame loss means that the production speed of the application process is slower than the consumption speed of the synthesis thread, and there are enough free cache queues in the cache queue for it to use. Alternatively, it may be that the application process has completed the drawing of the image frame in the current scene. In these cases, the synthesis thread can generate a second adjustment request to reduce the cache queue to reduce the number of free cache objects in the cache queue. Timely release of free cache objects in the cache queue can reduce the occupancy of storage resources.

[0046] In a possible implementation of the first aspect, the second adjustment request includes a second indication value, and the second indication value is used to indicate a reduced number of cache objects; reducing the number of free cache objects in the cache queue includes: the application process reduces the free cache objects of the second indication value from the cache queue.

[0047] In the present application, the application process can reduce the number of free cache objects in the cache queue according to the second indicator value in the second adjustment request. In actual applications, the second indicator value can be 1, 2, 3, etc., and the indicator value can be adjusted according to different situations, so as to achieve a better effect of dynamically reducing the cache queue.

[0048] In a possible implementation of the first aspect, the application process reduces the free cache objects of the second indication value from the cache queue, including: the application process removes the addresses of the free cache objects of the second indication value from the cache queue in a dequeue order.

[0049] In this application, each cache object in the cache queue has an arrangement order. The application process removes the address of the idle cache object with the second indication value from the cache queue in the dequeue order, which can ensure that the arrangement order of the cache objects in the original cache queue is not disturbed.

[0050] In a possible implementation of the first aspect, the method further includes:

[0051] The composition thread queries the number of all cache objects in the cache queue; if the number of all cache objects is reduced to the minimum number of cache objects, the composition thread stops sending the second adjustment request for the cache object to the application process.

[0052] In this application, due to the rendering performance requirements of the electronic device, the number of cache objects in the cache queue has a minimum number of cache objects. When the synthesis thread determines that the number of all cache objects in the cache queue has been reduced to the minimum number of cache objects, it stops sending the second adjustment request for reducing the number of free cache objects to the application process. This ensures the normal operation of the electronic device and avoids the problem of electronic device abnormality caused by the inability to reduce the number of cache objects in the cache queue.

[0053] In a possible implementation of the first aspect, the method further includes:

[0054] If the synthesis thread performs a synthesis operation in the synthesis cycle of the first image frame, the synthesis thread obtains a target cache object from the cache queue; the target cache object stores the first image frame after drawing and rendering; the synthesis thread performs a synthesis operation on the first image frame after drawing and rendering.

[0055] In the present application, when the synthesis thread normally performs the synthesis operation of the first image frame, the synthesis thread normally obtains the cache object stored in the rendered first image frame from the cache queue, synthesizes the rendered first image frame, and realizes the display of the synthesized first image frame, as well as the space release of the cache object, so that the released free cache object can be obtained in time in the cache queue for use by the application process.

[0056] In a second aspect, an electronic device is provided, comprising a memory, a display screen, and one or more processors; the memory and the display screen are coupled to the processor; computer program code is stored in the memory, and the computer program code comprises computer instructions, which, when executed by the processor, enable the electronic device to perform a method as described in any one of the above-mentioned first aspects.

[0057] In a third aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium. When the computer-readable storage medium is executed on an electronic device, the electronic device can execute any one of the methods in the first aspect.

[0058] In a fourth aspect, a computer program product comprising instructions is provided, which, when executed on an electronic device, enables the electronic device to execute any one of the methods described in the first aspect.

[0059] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute the method of the first aspect.

[0060] It can be understood that the beneficial effects that can be achieved by the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the chip described in the fifth aspect provided above can refer to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] FIG1 is a schematic diagram of a normal display of an application startup animation on a mobile phone interface provided by an embodiment of the present application;

[0062] FIG2 is a schematic diagram of a normal display of an off-screen sliding effect on a mobile phone interface provided by an embodiment of the present application;

[0063] FIG3 is a schematic diagram of an abnormal display of an application startup animation on a mobile phone interface provided by an embodiment of the present application;

[0064] FIG4 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0065] FIG5 is a schematic diagram of a software structure of an electronic device provided in an embodiment of the present application;

[0066] FIG6 is a timing diagram of normal drawing, rendering, synthesis, and display of multiple image frames provided by an embodiment of the present application;

[0067] FIG7 is a diagram showing a state change of a cache in a cache queue provided by an embodiment of the present application;

[0068] FIG8 is a timing diagram of abnormal drawing, rendering, synthesis, and display of multiple image frames provided by an embodiment of the present application;

[0069] FIG9 is a timing diagram of multi-agent interaction during an abnormal image frame drawing, rendering, synthesis, and display process provided by an embodiment of the present application;

[0070] FIG10 is a timing diagram of changes in the number of caches in a cache queue during abnormal image frame drawing, rendering, synthesis, and display provided by an embodiment of the present application;

[0071] FIG11 is a flow chart of an image processing method in an application startup dynamic effect scenario provided by an embodiment of the present application;

[0072] FIG12 is a schematic diagram of a normal display of an application exit animation on a mobile phone interface provided by an embodiment of the present application;

[0073] FIG13 is a flow chart of an image processing method in an application exit animation scenario provided by an embodiment of the present application;

[0074] FIG14 is a timing diagram of multi-agent interaction for dynamically adjusting the number of caches in a cache queue during image frame drawing, rendering, synthesis, and display provided by an embodiment of the present application;

[0075] FIG15 is a timing diagram of dynamically adjusting the number of buffers in a buffer queue during image frame drawing, rendering, synthesis, and display provided by an embodiment of the present application;

[0076] FIG16 is a schematic structural diagram of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations on the present application. As used in the specification and claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: the situation where A exists alone, A and B exist at the same time, and B exists alone, wherein A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are a kind of "or" relationship.

[0078] References to "one embodiment" or "some embodiments" etc. described in this specification mean that the specific features, structures or characteristics described in conjunction with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in another way. The term "connected" includes direct and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0079] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0080] Electronic devices can display dynamic display effects (animation effects) based on images triggered by different operations on their displays. One type of animation effect involves changes in the display positions of different image frames. This type of animation effect is referred to as the first type of animation effect in this application. Animation effects in which the display positions of image frames do not change are referred to as the second type of animation effect.

[0081] In the first category of animations, there are animations in which the display position of the image frame is related to the system time, such as the application startup animation, application exit animation, off-screen sliding animation, etc. Among them, the application startup animation refers to the animation displayed when the application is started; the application exit animation refers to the animation displayed when the application exits; the off-screen sliding animation specifies that the user slides the screen with his finger, and the operating object continues to move after the finger leaves the screen. When drawing each image frame in this type of animation, the electronic device needs to calculate the display position of the current image frame based on the system time of drawing the current image frame. The second category of animations includes game scene animations, application internal scene animations, and hand-following effects in other scenarios.

[0082] The image processing method provided in the present application can be applied to all animation scenes. For example, in the process of an electronic device displaying the startup animation of an application, the image frames in the process of the startup animation of the application are processed based on the image processing method. For example, in the process of an electronic device displaying the exit animation of an application, the image frames in the process of the exit animation of the application are processed based on the image processing method. For example, in the process of an electronic device displaying an off-screen sliding animation, the image frames in the process of the off-screen sliding animation are processed based on the image processing method. For example, in the process of an electronic device displaying a game scene animation, the image frames in the process of the game scene animation are processed based on the image processing method. For example, in the process of an electronic device displaying an internal scene animation of an application, the image frames in the process of the internal scene animation of the application are processed based on the image processing method. For example, in other scenes of hand-following effects, electronic devices can process the image frames in the hand-following effects based on the image processing method.

[0083] Processing image frames based on the image processing method provided in this application can effectively avoid frame loss during the drawing process of image frames for displaying motion effects, and further effectively solve the problem of display freeze caused by image frames for displaying motion effects.

[0084] In some examples, different image frames have different display positions, which means that the distances from a designated vertex of the image frame (such as the upper left vertex of the interface) to the origin of the image frame are different between different image frames.

[0085] Exemplarily, FIG1 takes the electronic device as a mobile phone as an example, and shows a schematic diagram of displaying an application startup animation. Referring to FIG1 (a), the user clicks on the icon of application 5 on the mobile phone desktop. In response to the click operation, application 5 is started, and the startup animation of application 5 is displayed. The display screen of the startup animation of application 5 gradually changes from FIG1 (b) to FIG1 (f). The image frames in the startup animation of application 5 include the five image frames displayed from FIG1 (b) to FIG1 (f). It can be seen that the distance between the specified vertex of these five image frames (such as the upper left vertex of the interface) and the origin of the image frame is different. From FIG1 (b) to FIG1 (f), the distance between the specified vertex of the image frame (such as the upper left vertex of the interface) and the origin of the image frame gradually increases until the image frame fills the screen.

[0086] In some examples, different image frames have different display positions, which means that the distances from the designated vertex of the image frame (such as the upper left vertex of the interface) to the screen origin are different between different image frames.

[0087] For example, FIG2 takes the electronic device as a mobile phone as an example, and shows a schematic diagram of displaying an off-screen sliding effect. Referring to FIG2 (a), the current interface of FIG2 is page 0 of the desktop, and the current interface includes application 1, application 2, application 3, application 4, and application 5. The user swipes left on the current page interface of the mobile phone desktop. In response to the left swipe operation, the mobile phone displays the next page interface of the current interface with a left sliding effect, wherein the next page interface of the current interface is page 1 of the desktop, and the first page interface includes application 6, application 7, and application 8. The display screen of the sliding effect triggered by the mobile phone in response to the left swipe operation gradually changes from FIG2 (b) to FIG2 (f), and the image frames of the left swipe sliding effect include the 5 image frames displayed from FIG2 (b) to FIG2 (f). It can be seen that the distances between the designated vertex of each of the five image frames (e.g., the point where the hollow circle is located) and the screen origin (e.g., the point where the solid circle is located) are different. From FIG2(b) to FIG2(f), the distance between the designated vertex of the image frame and the screen origin gradually decreases until the distance is zero, and the next page interface is completely displayed on the screen. In this embodiment, the minimum distance between the designated vertex of the image frame and the screen origin can be determined based on actual conditions.

[0088] It is understandable that in some examples, different image frames have different display positions, which may also mean that the distances from the designated vertex of the image frame to the image frame origin and the distances from the designated vertex of the image frame to the screen origin are different between different image frames.

[0089] In the following embodiments of the present application, the mobile phone display application startup dynamic effect is used as an example for description. It is understandable that the image processing method provided in the embodiments of the present application is also applicable to other types of dynamic effects.

[0090] In the embodiment of the present application, the distance from a specified vertex of an image frame (such as the upper left vertex of the interface) to the origin of the image frame is called displacement. The displacement change between adjacent image frames is called displacement interval.

[0091] For example, the displacement of the current image frame in the application startup animation is the system time for drawing the current image frame / the total animation time * the total displacement, where the total animation time refers to the total time for displaying all image frames in the animation under normal circumstances; the total displacement refers to the displacement of the last frame of the animation.

[0092] In another example, the displacement y(t) of the current image frame can be calculated as y(t) = y(0) + t*n, where y(0) is the displacement of the first image frame of the animation effect. n is a preset displacement interval. t can be calculated as t = currenttime - (currenttime - t0)% frame rate, where "%" represents the remainder operation, currenttime is the current time, and t0 is the drawing time of the first image frame of the animation effect.

[0093] During the technical implementation of a display application's startup animation, the application's application process is used to draw and render each image frame in the animation. The application process further comprises the application's main thread and a rendering thread. The main thread draws the image frames, while the rendering thread renders them. A general compositing thread is used to synthesize the rendered image frames. Specifically, for each image frame in the animation, the main thread calculates the displacement required to draw the current frame based on the system time it takes to draw the current frame, and then draws the current frame based on the calculated displacement. After obtaining the drawn image frame, the rendering thread renders the drawn image frame. After obtaining the rendered image frame, the compositing thread composites the multiple layers within the rendered image frame. The compositing thread then sends the composited image frame to the display driver for display. Throughout this process, the main thread, rendering thread, compositing thread, and display driver all execute corresponding operations based on their respective trigger signals, thereby achieving the drawing, rendering, compositing, and displaying of the multiple image frames in the animation, ultimately achieving coherent display of the multiple image frames in the animation. Normally, the displacement interval between adjacent image frames is fixed. For example, as shown in FIG1 , in the five image frames from FIG1 (b) to FIG1 (f), the displacement intervals between all adjacent image frames remain unchanged, and in the process of displaying the startup animation of application 5, the display screen is coherent and smooth.

[0094] During the process of generating the application startup animation, the application main thread, rendering thread, synthesis thread and display driver will all perform corresponding operations upon receiving their respective corresponding trigger signals. The trigger signal may include a drawing signal for triggering the application main thread to perform a drawing operation, a synthesis signal for triggering the synthesis thread to perform a synthesis operation, etc. For example, the application main thread receives a drawing signal, calculates the displacement of the current image frame based on the system time for drawing the current image frame, and draws the current image frame based on the calculated displacement; after obtaining the drawn image frame, the application main thread wakes up the rendering thread to perform the rendering operation of the drawn image frame. The synthesis thread receives a synthesis signal, performs the operation of synthesizing multiple layers in the rendered image frame; after obtaining the synthesized image frame, the synthesis thread sends the synthesized image frame to the display driver for display. If any of the application main thread, rendering thread, synthesis thread and display driver does not perform the corresponding operation according to their respective corresponding trigger signals or trigger conditions, it may cause the calculated displacement results of certain image frames in the animation to deviate from the displacement of the image frame under normal circumstances. For example, the rendering thread may fail to render an already drawn image frame. Since the application main thread and the rendering thread are serial threads, the failure of the rendering thread to render the already drawn image frame will have a negative impact on the application main thread, causing the application main thread to be unable to perform the drawing operation of a new image frame. The final result is that the application main thread cannot perform the drawing operation of the current image frame after completing the drawing operation of the previous image frame. As a result, the time interval between the application main thread's execution of the drawing operation of the current image frame and the drawing operation of the previous image frame is too long. When the application main thread starts to draw the current image frame, the displacement calculated based on the system time of drawing the current image frame and the displacement calculated for the previous image frame are too large. The actual displacement of the current image frame deviates from the theoretical displacement of the next image frame of the previous image frame. The displacement interval between adjacent image frames changes, resulting in a sudden enlargement of the screen when displaying the startup animation effect, causing visual lag to the user. Figure 3 shows a schematic diagram of the process of abnormally displaying the startup animation of application 5 in an electronic device such as a mobile phone. In one example, under abnormal circumstances, because the time interval for the application main thread to draw the current image frame is too long, the displacement interval of the current image frame calculated based on the drawing system time is too large, resulting in the application main thread dropping frames, so the startup animation includes 3 image frames. Referring to Figure 3, in the same scenario as Figure 1 in which application 5 is started, the application main thread of the electronic device shown in Figure 3 does not perform the drawing and rendering of the image frame shown in (d) of Figure 1 and the image frame shown in (e) of Figure 1. After a certain time interval, the application main thread calculates the drawing displacement of the image frame shown in (f) of Figure 3 based on the system time. Therefore, the displacement interval between the displacement of the image frame shown in (f) of Figure 3 and the displacement of the image frame shown in (c) of Figure 3 is too large.Referring to Figure 3(a), a user clicks the icon of Application 5 on the display interface. In response to the user's click, Application 5 starts and displays a startup animation. The display screen of Application 5's startup animation gradually changes from Figure 3(b) to Figure 3(f). The displacement interval between adjacent image frames of Application 5's startup animation changes between Figure 3(c) and Figure 3(f). The display screen of Figure 3(c) and Figure 3(f) has a significant sudden enlargement change, resulting in a discontinuous display process and causing visual lag for the user.

[0095] The above scenarios illustrate the problems encountered during the drawing, rendering, synthesis, and display of image frames for the first type of animation. For the second type of animation, there may also be situations where, when the application process is drawing and rendering an image frame, there are no free cache objects in the cache queue due to the synthesis thread not synthesizing or other reasons. This causes the application process to be unable to perform the drawing and rendering operations, resulting in frame drops. Once an application process drops frames while drawing and rendering an image frame, it means that the display will experience freezes and other issues during the display of the image frame.

[0096] The embodiment of the present application provides an image processing method, which can be applied to an electronic device with a display screen, including image processing of the drawing, rendering, synthesis and display of image frames, and is suitable for processing scenarios of image frames in all types of dynamic effects. Through the image processing method provided by this embodiment, the application process for image frame drawing and rendering can obtain a free cache object in each drawing cycle to store the rendered image frame to perform drawing and rendering of the image frame in the next drawing cycle. In the image processing scenario for the first type of dynamic effect, through the image processing method provided by this embodiment, the application process can avoid the problem that the application process does not perform the drawing and rendering operation due to the lack of free cache objects in the cache queue, thereby avoiding the problem of visual freeze after display due to the large displacement interval between two adjacent image frames. In the image processing scenario for the second type of dynamic effect, through the image processing method provided by this embodiment, the application process can still avoid the problem that the application process does not perform the drawing and rendering operation due to the lack of free cache objects in the cache queue, thereby solving the problem of freeze in the image frame display process when the application process loses frames.

[0097] The electronic device in the embodiments of the present application can be a portable computer (such as a mobile phone), a tablet computer, a laptop computer, a personal computer (PC), a wearable electronic device (such as a smart watch), an augmented reality (AR) / virtual reality (VR) device, an in-vehicle computer, a smart TV, or other device including a display screen. The following embodiments do not impose any special restrictions on the specific form of the electronic device.

[0098] Please refer to Figure 4, which shows a block diagram of an electronic device (such as electronic device 100) provided in an embodiment of the present application. Among them, electronic device 100 may include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, an antenna 2, a radio frequency module 350, a communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, an earphone interface 370D, a sensor module 380, a button 390, a camera 391 and a display screen 392. Among them, the sensor module 380 may include a pressure sensor 380A, a touch sensor 380B, etc.

[0099] The structure shown in the embodiment of the present invention does not limit the electronic device 100. The electronic device 100 may include more or fewer components than shown, or some components may be combined or separated, or arranged differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0100] The processor 310 may include one or more processing units. For example, the processor 310 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0101] The controller is the decision-maker that directs the various components of electronic device 100 to coordinate operations according to instructions. It serves as the nerve center and command center of electronic device 100. Based on instruction opcodes and timing signals, the controller generates operational control signals to control instruction fetching and execution.

[0102] Processor 310 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 310 is a high-speed cache memory that can store instructions or data that have just been used or are being recycled by processor 310. If processor 310 needs to use the same instruction or data again, it can directly access the memory. This avoids repeated accesses, reduces processor 310 latency, and thus improves system efficiency.

[0103] In some embodiments, the processor 310 may include an interface. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface.

[0104] The interface connection relationship between the modules shown in the embodiment of the present invention is for illustrative purposes only and does not limit the structure of the electronic device 100. The electronic device 100 may adopt different interface connection methods or a combination of multiple interface connection methods in the embodiment of the present invention.

[0105] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the radio frequency module 350, the communication module 360, the modem and the baseband processor.

[0106] Electronic device 100 implements display functionality through a GPU, display screen 392, and an application processor. The GPU is a microprocessor for image processing that connects display screen 392 and application processor AP. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 310 may include one or more GPUs that execute program instructions to generate or modify display information.

[0107] Display screen 392 is used to display images, videos, etc. Display screen 392 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N display screens 392, where N is a positive integer greater than 1.

[0108] In this embodiment, the display screen can be any type of display screen, which can be a touch screen or a non-touch screen. In this embodiment, the display screen 392 can display operation-triggered animations, for example, by clicking an application icon on the display screen to trigger the display of an application launch animation; for example, by clicking an exit application control to trigger the display of an application exit animation; for example, the display screen can display manual hand effects, game scene animations, etc.

[0109] The electronic device 100 can implement a shooting function through an ISP, a camera 391, a video codec, a GPU, a display screen, and an application processor.

[0110] The external memory interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 310 via the external memory interface 320 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0111] The internal memory 321 can be used to store computer executable program codes, which include instructions. The processor 310 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 321. The memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, other volatile solid-state storage devices, a universal flash storage (UFS), etc.

[0112] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 370 , the speaker 370A, the receiver 370B, the microphone 370C, the headphone jack 370D, and the application processor.

[0113] The pressure sensor 380A is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor 380A can be provided on the display screen 392. There are many types of pressure sensors 380A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a device comprising at least two parallel plates having a conductive material. When a force acts on the pressure sensor, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to the display screen 392, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor 380A. The electronic device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 380A.

[0114] Touch sensor 380B, also known as a "touch panel," may be provided on display screen 392 to detect touch operations applied thereto or in the vicinity thereof. The detected touch operations may be communicated to an application processor to determine the type of touch event and provide corresponding visual output via display screen 392.

[0115] Keys 390 include a power button, a volume button, and the like. Keys 390 may be mechanical keys or touch-sensitive keys. Electronic device 100 receives input from keys 390 and generates key signal input related to user settings and function control of electronic device 100.

[0116] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0117] Figure 5 is a block diagram of the software structure of electronic device 100 according to an embodiment of the present invention. A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other via software interfaces. In some embodiments, the Android system is divided into four layers: the application layer, the application framework layer, the Android runtime and system libraries, the hardware abstraction layer, and the kernel layer.

[0118] The application layer can include a series of application packages. As shown in Figure 5, the application package can include camera, gallery, calendar, phone, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0119] Each application consists of a main application thread and a rendering thread. The main application thread is used to draw the corresponding image frame when a drawing signal arrives. The rendering thread is used to render the drawn image frame.

[0120] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0121] As shown in FIG5 , the application framework layer may include a desktop launcher, a window manager, a content provider, an image composition system, a view system, an input manager, an activity manager, and a resource manager, etc.

[0122] In this embodiment, the desktop launcher is configured to receive a first operation by a user on the touch screen of an electronic device and, in response to the first operation, launch a first application; and further configured to receive a second operation by the user on the touch screen of the electronic device and, in response to the second operation, exit the first application. The first application may be any application included in the application layer.

[0123] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0124] Content providers are used to store and retrieve data and make it accessible to applications. This data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0125] The image synthesis system is used to control image synthesis and generate a vertical synchronization (Vsync) signal. The image synthesis system may be a surface flinger.

[0126] The image synthesis system includes a synthesis thread and a Vsync thread. The synthesis thread triggers the synthesis of multiple layers in a rendered image frame when a Vsync signal arrives. The Vsync thread generates the next Vsync signal based on the Vsync signal request and sends the Vsync signal to the corresponding other threads.

[0127] The view system includes visual controls, such as controls for displaying text and images. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface that includes a text notification icon can include a view for displaying text and a view for displaying images. A display interface can be composed of one or more views.

[0128] The input manager is used to manage input devices. For example, the input system can determine input operations such as mouse clicks, keyboard inputs, and touch swipes.

[0129] The Activity Manager is used to manage the lifecycle of each application and the navigation back function. It is responsible for creating the Android main thread and maintaining the lifecycle of each application.

[0130] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0131] The Android runtime includes the core library and the virtual machine. The Android runtime is responsible for scheduling and management of the Android system.

[0132] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0133] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0134] The system library can include multiple functional modules, such as image rendering library, image synthesis library, input library, surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0135] Image rendering library, used for rendering two-dimensional or three-dimensional images.

[0136] Image synthesis library for compositing 2D or 3D images.

[0137] In a possible implementation, the application renders the image through the image rendering library, and then sends the rendered image to the application's cache queue, so that the image synthesis system sequentially obtains a frame of image to be synthesized from the cache queue, and then performs image synthesis through the image synthesis library.

[0138] The input library is used to process input devices and can implement mouse, keyboard, and touch input processing, etc.

[0139] The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as static image files.

[0140] The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0141] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0142] The hardware abstraction layer (HAL) can contain multiple library modules, such as the hardware compositor (HWComposer, HWC) and the camera library module. The Android system can load the corresponding library modules for the device hardware, enabling the application framework layer to access the device hardware. Device hardware can include displays and cameras in electronic devices.

[0143] HWC is the HAL layer module for window synthesis and display in Android. The image synthesis system provides HWC with a complete list of all windows, allowing HWC to decide how to handle these windows based on its hardware capabilities. HWC will mark the synthesis method for each window, for example, whether it is synthesized by GPU or HWC. SurfaceFlinger is responsible for synthesizing all windows marked as GPU synthesis into an output buffer, and then handing this output buffer along with the other windows to HWC, allowing HWC to complete the synthesis and display of the remaining windows.

[0144] The kernel layer is the layer between hardware and software. The kernel layer includes at least the touch panel (TP) driver, display driver, camera driver, audio driver, and camera driver.

[0145] Hardware can be audio devices, Bluetooth devices, camera devices, sensor devices, etc.

[0146] Combined with the software structure of the electronic device provided in Figure 5 above, the reasons for visual freezes when displaying adjacent image frames of the animation due to frame loss in the animation or excessive drawing time interval and displacement interval between adjacent frames in the animation are analyzed in the process of displaying animation in the prior art.

[0147] For example, let's take the example of an electronic device displaying an application's startup animation. This involves the interaction between the electronic device's application, image synthesis system, and display driver. The image synthesis system can be a compositor. Each application's application process includes an application main thread and a rendering thread; the compositor includes a synthesis thread and a Vsync thread. The Vsync thread generates a Vsync signal and sends it to corresponding threads to wake them up to perform corresponding operations. For example, if a user performs a touch operation on an electronic device to launch an application, the electronic device's display driver sends the input event corresponding to the touch operation to the input thread of the system service, which then sends the input event to the application's main thread. After receiving the input event, the application's main thread requests a Vsync signal from the synthesis thread to draw an image frame. Upon receipt of the Vsync signal, the application's main thread performs operations such as drawing the current image frame of the application's startup animation, resulting in a drawn image frame. The rendering thread renders the drawn image frame, resulting in a rendered image frame. Upon receipt of the Vsync signal, the synthesis thread synthesizes multiple layers of the rendered image frame, resulting in a synthesized image frame. Furthermore, the synthesis thread is also responsible for sending the synthesized image frames to the HWC, and the HWC displays the image frames through the display driver.

[0148] The Vsync signals generated by the Vsync thread include the Vsync_APP signal, the Vsync_SF signal, and the HW_Vsync signal. The Vsync thread generates the Vsync_APP signal and sends the Vsync_APP signal to the application main thread. When the Vsync_APP signal arrives, the application main thread performs the drawing operation of the current image frame. The Vsync thread generates the Vsync_SF signal and sends the Vsync_SF signal to the synthesis thread. When the Vsync_SF signal arrives, the synthesis thread obtains the rendered image frame and performs the image frame synthesis operation. The Vsync thread generates the HW_Vsync signal and sends the HW_Vsync signal to the display driver of the electronic device. When the HW_Vsync signal arrives, the display driver refreshes the display image frame.

[0149] The period in which the Vsync thread generates the Vsync signal is related to the frame rate of the electronic device. The frame rate refers to the number of frames that refresh the image in 1 second, which can also be understood as the number of times the graphics processor in the electronic device refreshes the screen per second. A high frame rate can produce smoother and more realistic animations. The more frames per second, the smoother the displayed action will be. For example, a frame rate of 60Hz means refreshing 60 frames of images in 1 second, that is, refreshing one frame of image every 16.6 milliseconds. Accordingly, the period in which the Vsync thread generates the Vsync signal is 16.6 milliseconds. For example, a frame rate of 90Hz means refreshing 90 frames of images in 1 second, that is, refreshing one frame of image every 11.1 milliseconds. Accordingly, the period in which the Vsync thread generates the Vsync signal is 11.1 milliseconds.

[0150] Figure 6 shows a timing diagram of each thread processing job in an electronic device when the frame rate is 60Hz. In a display scenario with a frame rate of 60Hz, the total number of frames included in the animation is 6, the total distance of the animation is 96, and the total time of the animation is 99.6ms. The Vsync thread generates a VSYNC_APP signal according to a cycle of 16.6ms and sends it to the main thread of the application to wake up the main thread of the application and the rendering thread to perform drawing and rendering operations. Under normal circumstances, according to the signal cycle, the time interval and displacement interval for the main thread of the application to draw each image frame in the animation remain unchanged. In this embodiment, the drawing time interval is 16.6ms, and the drawing displacement interval is 16.

[0151] Figure 6 shows the correspondence between the frame interval and the frame rate. For a 60Hz frame rate, the frame interval is 16.6ms. The timestamp records the time it takes the application's main thread to draw each image frame. The image drawing offset corresponds to the frame interval; here, for a 16.6ms frame interval, the offset is 16. VSYNC_APP ID is the cycle number of the VSYNC_APP signal received by the application's main thread. Drawing and rendering refers to the drawing and rendering operations performed by the application's main thread and rendering thread. The buffer queue in Figure 6 is used to store rendered image frames. The rendering thread can store rendered image frames in an empty buffer in the buffer queue; the compositing thread can retrieve rendered image frames from the buffer queue for compositing. In other words, the rendering thread is the producer of the buffer queue, and the compositing thread is the consumer of the buffer queue. The buffer queue has a maximum number of buffers. In the example in Figure 6, the maximum number of buffers in the buffer queue is 4. The compositing thread in the figure shows the compositing thread performing compositing operations. Display refers to the display driver displaying image frames. Figure 6 also includes the offset of each displayed image frame and the time interval between adjacent displayed image frames.

[0152] An electronic device can create a buffer queue (buffer queue), the producer of which is a rendering thread and the consumer is a synthesis thread. The buffer queue can include multiple buffers (buffers). In the initial state of the buffer queue, each buffer is an empty buffer (free buffer), and an empty buffer is a buffer that is not occupied by a rendering thread or a synthesis thread. Generally, the maximum number of buffers (MaxBufferCount) in the buffer queue is determined by the frame rate of the electronic device. For example, when the frame rate of the electronic device is 60Hz, the MaxBufferCount of the buffer queue can be 10. The MaxBufferCount of the buffer queue is an empirical value.

[0153] The following briefly describes the cache usage mechanism in the cache queue, combining the producers and consumers of the cache queue. See Figure 7. The rendering thread is the producer of the cache queue, and the compositing thread is the consumer of the cache queue. The process of drawing, rendering, compositing, and displaying an image frame includes:

[0154] (1) When the VSYNC_APP signal arrives, the main thread of the application executes the drawing operation of the current image frame and obtains the drawn image frame.

[0155] The rendering thread performs rendering operations on the drawn image frames, and if there is an empty buffer (free buffer) in the buffer queue, it dequeues an empty buffer (free buffer) from the buffer queue to store the rendered image frame. At this time, the status of the buffer is updated to dequeued, indicating that the buffer is in a state of being obtained by the rendering thread for corresponding operations.

[0156] Among them, the process of dequeueing the buffer includes: the rendering thread sends a request to dequeue the free buffer to the application main thread, and the application main thread determines whether the number of buffers in the cache queue with a status of dequeued has reached the maximum number that can be dequeued. If the number of buffers in the dequeued state is less than the maximum number that can be dequeued, it means that there are still free buffers in the current cache queue. At this time, the application main thread searches for a free buffer in the order of free buffers and marks the state of the buffer as dequeued. After marking the state of the buffer, the cache information of the buffer is returned to the rendering thread, and the rendering thread performs the storage operation of the rendered image frame based on the cache information. The cache information includes the cache address, cache status identifier, etc.

[0157] (2) After completing the storage operation of the rendered image frame, the rendering thread queues the cache of the rendered image frame into the cache queue. At this time, the status of the cache is updated to queued, indicating that the cache is in a state of waiting to be synthesized.

[0158] The queue buffer process includes: the rendering thread sends a queue buffer request to the application main thread, which carries the buffer cache information. The application main thread updates the cache status to queued based on the cache information.

[0159] (3) When the Vsync_SF signal arrives, the synthesis thread requests (acquires) a cache that stores the rendered image frame from the cache queue to perform the synthesis operation of the image frame layer. At this time, the status of the cache is updated to "acquired", indicating that the cache is in a state of being acquired by the synthesis thread for synthesis.

[0160] The acquire buffer process includes: the compositing thread sends an acquire buffer request to the application main thread. The application main thread determines whether the number of acquired buffers in the cache queue is greater than or equal to the maximum number of synthesizable buffers. If the number of acquired buffers in the current cache queue is less than the maximum number of synthesizable buffers, the application main thread sends the cache information of the first queued buffer in the order of the queued buffers to the compositing thread and marks the buffer as acquired. The compositing thread then performs compositing operations on the rendered image in the buffer based on the cache information.

[0161] (4) After completing the image frame synthesis operation, the synthesis thread can send the synthesized image frame to the HWC and display driver for display. After completing the sending of the synthesized image frame, the display driver releases the cache. At this time, the cache status is updated to free.

[0162] The buffer release process involves the display driver releasing the buffer used by the composite image frame displayed in the previous display cycle during the current display cycle and returning the buffer information to the compositing thread. The compositing thread then returns the buffer information to the application main thread. The application main thread then updates the buffer status to free based on the buffer information.

[0163] Optionally, the application main thread may also notify the rendering thread that there is an empty buffer in the buffer queue, so that the rendering thread obtains the empty buffer in the current rendering cycle or the next rendering cycle to store the rendered image frame.

[0164] It should be noted that the cache queues mentioned in this embodiment are all cache queues stored in the application. The application's main thread is responsible for acquiring information and updating the status of each cache in the cache queue. The rendering thread needs to interact with the application's main thread to implement dequeue and queue buffer operations; the compositing thread also needs to interact with the application's main thread to implement acquire and release buffer operations.

[0165] Based on the above cache usage mechanism in the cache queue, the timing diagram of the application main thread, rendering thread, composition thread and display driver processing operation shown in Figure 6 is explained. The MaxBufferCount of the cache sequence in Figure 6 is 4.

[0166] 1. During the period when Vsync-APP ID is 1, when the Vsync_APP signal arrives, the application main thread draws image frame 4. By default, the initial displacement of the first image frame 4 is 0. After obtaining the drawn image frame 4, the rendering thread renders the drawn image frame 4. By interacting with the application main thread, it obtains a free buffer from the cache queue and stores the rendered image frame 4. The application main thread then updates the status of the cache storing the rendered image frame 4 to dequeued.

[0167] When the Vsync_SF signal arrives, the compositing thread determines to perform compositing. It interacts with the application's main thread to retrieve rendered frame 1 from the cache queue for compositing. The application's main thread then updates the status of the buffer occupied by frame 1 to "acquired." The cache queue also includes the buffers storing rendered frame 2 and rendered frame 3, both of which are in the queued state. At this point, there are no free buffers in the cache queue.

[0168] 2. During the Vsync-APP ID cycle 2, the compositing thread completes the compositing operation on the rendered image frame 1 and sends it to the HWC for display. The HWC displays the composite image frame 1 through the display driver and releases the cache occupied by the composite image frame 1 before the end of the current cycle. The cache information is returned to the compositing thread, which returns the cache information to the main application thread. The main application thread updates the cache status based on the cache information. At this point, there is a free buffer in the cache queue.

[0169] When the Vsync_APP signal arrives, the application main thread draws image frame 5. The calculated displacement of image frame 5 is 16, and the displacement interval between image frame 5 and image frame 4 is 16. After obtaining the drawn image frame 5, the rendering thread renders the drawn image frame 5. By interacting with the application main thread, it obtains a free buffer from the cache queue and stores the rendered image frame 5. The application main thread then updates the status of the cache storing the rendered image frame 5 to dequeued.

[0170] When the Vsync_SF signal arrives, the compositing thread determines to perform the compositing operation. The compositing thread interacts with the application's main thread to retrieve rendered image frame 2 from the cache queue for compositing. The application's main thread updates the status of the buffer occupied by image frame 2 to acquired. The cache queue also includes the buffer storing rendered image frame 3 and the buffer storing rendered image frame 4, both of which are queued. The cache queue also includes the buffer currently storing rendered image frame 5, which is dequeued. At this point, there are no free buffers in the cache queue.

[0171] 3. During the Vsync-APP ID cycle 3, the compositing thread completes compositing of rendered frame 2 and sends it to the HWC for display. The HWC displays frame 2 via the display driver and, before the end of the current cycle, releases the buffer occupied by frame 2 and returns the buffer information to the compositing thread. The compositing thread returns the buffer information to the main application thread, which then updates the buffer status based on the buffer information. At this point, there is a free buffer in the buffer queue.

[0172] When the Vsync_APP signal arrives, the main application thread draws image frame 6. The calculated displacement of image frame 6 is 32, and the displacement interval between image frame 6 and image frame 5 is 16. After obtaining the drawn image frame 6, the rendering thread renders it. By interacting with the main application thread, it obtains a free buffer from the cache queue and stores the rendered image frame 6. The main application thread then updates the status of the cache storing the rendered image frame 6 to dequeued.

[0173] When the Vsync_SF signal arrives, the compositing thread determines to perform the compositing operation. By interacting with the application's main thread, the compositing thread retrieves rendered image frame 3 from the cache queue for compositing. The application's main thread updates the status of the cache occupied by image frame 3 to "acquired." The cache queue also includes the buffer storing rendered image frame 4 and the buffer storing rendered image frame 5, both of which are in the queued state. The cache queue also includes the buffer currently storing rendered image frame 6, which is in the dequeued state. At this point, there are no free buffers in the cache queue.

[0174] In subsequent cycles, the rendering and compositing threads perform similar operations to the first three cycles. The compositing thread performs compositing operations normally during each cycle and displays the composite image frame in the next cycle after the composite image frame is rendered. The display driver releases the buffer for the composite image frame before the end of the current display cycle and returns the cache information of the released buffer to the compositing thread. The compositing thread then sends this buffer information to the main application thread, which updates the cache status based on the information. Simultaneously, the main application thread notifies the rendering thread of the existence of a free buffer in the cache queue, allowing the rendering thread to retrieve the last free buffer in the cache queue and store it in the next rendered image frame during each cycle. Because the calculated displacement interval between the image frame rendered by the main application thread and the previous image frame remains constant at 16, the displacement interval of 16 between adjacent composite image frames after rendering, compositing, and display also remains constant, resulting in a coherent display of multiple frames.

[0175] However, in practice, various reasons can prevent the rendering thread from accessing a free buffer to store rendered images. Since the rendering thread and the main application thread are serial threads, a failure to render in the rendering thread will affect the main application thread's ability to draw the next frame. This results in a significant time gap between the time the main application thread draws the previous frame and the time it draws the current frame. Consequently, the displacement calculated by the main application thread differs significantly from the displacement of the previous frame. When adjacent frames with a significant displacement gap are rendered, synthesized, and displayed, visual jitter may occur during the display process due to the large displacement gap.

[0176] Figure 8 shows an example where the compositing thread fails to perform frame composition within the corresponding cycle, preventing the rendering thread from obtaining a free buffer from the cache queue to store the rendered frame. This impacts the application's main thread's calculation and rendering of the next frame, resulting in a large shift gap between adjacent frames. Referring to Figure 8, the buffer sequence's maxbuffercount is 3.

[0177] 1. During the period when Vsync_APP ID is 1, when the Vsync_APP signal arrives, the application main thread calculates the displacement of drawn image frame 1 based on the timestamp and total displacement distance. At 16.6ms, the displacement of drawn image frame 1 is 0. After obtaining the drawn image frame 1, the application main thread wakes up the rendering thread to render image frame 1. The rendering thread interacts with the application main thread to obtain a free buffer from the cache queue and store the rendered image frame 1. The application main thread then updates the status of the buffer storing the rendered image frame 1 to dequeued.

[0178] When the Vsync_SF signal arrives, the composition thread determines not to perform the composition operation. At this time, the cache queue includes one dequeued buffer and two free buffers.

[0179] 2. During the Vsync_APP ID 2 cycle, the rendering thread completes the operation of storing rendered image frame 1 in the cache. By interacting with the application main thread, the rendering thread stores the rendered image frame in the cache queue. The application main thread updates the cache status to queued. At this time, the cache queue contains one queued buffer and two free buffers.

[0180] When the Vsync_APP signal arrives, the application main thread calculates the displacement of drawn image frame 2 based on the timestamp and total displacement distance. The displacement interval is 16, and the time interval is 16.6mm. At 33.2ms, the calculated displacement of image frame 2 is 16. After obtaining the drawn image frame 2, the application main thread wakes up the rendering thread to render image frame 2. The rendering thread interacts with the application main thread to obtain a free buffer from the cache queue and store the rendered image frame 2. The application main thread updates the status of the buffer storing the rendered image frame 2 to dequeued.

[0181] When the Vsync_SF signal arrives, the composition thread determines not to perform the composition operation. At this time, the buffer queue includes a queued buffer, a dequeued buffer, and a free buffer.

[0182] 3. During the Vsync_APP ID 3 cycle, the rendering thread completes the operation of storing rendered image frame 2 in the cache. By interacting with the application main thread, the rendering thread stores the rendered image frame in the cache queue. The application main thread updates the cache status to queued. At this time, the cache queue contains two queued buffers and one free buffer.

[0183] When the Vsync_APP signal arrives, the application's main thread calculates the displacement of rendered image frame 3 based on the timestamp and total displacement distance. The displacement interval is 16, and the time interval is 16.6 mm. At 49.8 ms, the calculated displacement of image frame 3 is 32. After obtaining the rendered image frame 3, the application's main thread wakes up the rendering thread to render image frame 3. The rendering thread interacts with the application's main thread, obtains a free buffer from the cache queue, and stores the rendered image frame 3. The application's main thread then updates the status of the buffer storing the rendered image frame 3 to dequeued.

[0184] When the Vsync_SF signal arrives, the compositing thread determines to perform compositing. It interacts with the application's main thread, requesting a queued buffer from the cache queue for compositing. Following the cache order, the application's main thread returns the cache information corresponding to rendered image frame 1 to the compositing thread, which then performs compositing based on the cache information.

[0185] At this point, the cache queue includes an acquired buffer, a queued buffer, and a dequeued buffer.

[0186] 4. During the period when Vsync-APP ID is 4, when the Vsync_APP signal arrives, since there is no free buffer in the cache queue, the rendering thread cannot obtain a free buffer from the cache queue to store the rendered image frame. The rendering thread does not perform rendering operations, and the application main thread does not perform drawing operations.

[0187] When the Vsync_SF signal arrives, the composition thread determines not to perform composition operations.

[0188] The display driver displays the received synthesized image frame 1 .

[0189] 5. During the period when Vsync-APP ID is 5, when the Vsync_SF signal arrives, the composition thread determines not to perform the composition operation. Since there is no new image to be displayed, the display driver still displays the composed image frame 1.

[0190] There is still no free buffer in the cache queue, so the application main thread and rendering thread do not perform drawing and rendering operations during this cycle.

[0191] 6. During the period when Vsync-APP ID is 6, when the Vsync_SF signal arrives, the composition thread determines not to perform the composition operation. Since there is no new image to be displayed, the display driver still displays the composed image frame 1.

[0192] There is still no free buffer in the cache queue, so the application main thread and rendering thread do not perform drawing and rendering operations during this cycle.

[0193] 7. During the Vsync-APP ID 7 cycle, when the Vsync_SF signal arrives, the compositing thread determines to execute the compositing operation. By interacting with the application main thread, it retrieves the rendered image frame 2 from the cache queue for compositing. Since there are no new images to be displayed during the current cycle, the display driver continues to display the composited image frame 1.

[0194] Before the end of this display cycle, the display driver releases the cache occupied by the synthesized image frame 1 and returns the cache information of the cache to the synthesis thread. The synthesis thread returns the cache information to the application main thread. The application main thread updates the status of the cache according to the cache information and updates the cache status to free.

[0195] At this point, the cache queue includes an acquired buffer, a queued buffer, and a free buffer.

[0196] When the Vsync_APP signal arrives, the application main thread calculates the displacement of the drawn image frame 4 based on the timestamp and total displacement distance. At this point, three rendering cycles have passed, and the current cycle is the fourth cycle, with a time interval of 16.6ms*(1+3)=66.4ms. Therefore, the calculated displacement interval should be 16*(1+3)=64, and the final displacement of the drawn image is 32+16*4=96, with the displacement of image frame 4 being 96. After obtaining the drawn image frame 4, the application main thread wakes up the rendering thread to render image frame 4. The rendering thread interacts with the application main thread to obtain the last free buffer from the cache queue and store it in the rendered image frame 4. The application main thread updates the status of the cache storing the rendered image frame 4 to dequeued.

[0197] At this point, the cache queue includes an acquired buffer, a queued buffer, and a dequeued buffer.

[0198] 8. During the period when Vsync-APP ID is 8, there is no free buffer in the cache queue, so the application main thread and rendering thread do not perform drawing and rendering operations.

[0199] When the Vsync_SF signal arrives, the composition thread determines to perform the composition operation and obtains the rendered image frame 3 from the cache queue for composition by interacting with the application main thread.

[0200] In this cycle, the synthesis thread sends the synthesized image frame 2 to the display, and the display driver displays the received synthesized image frame 2.

[0201] Before the end of this cycle display, the display driver releases the cache occupied by the synthesized image frame 2 and returns the cache information of the cache to the synthesis thread. The synthesis thread returns the cache information to the application main thread, and the application main thread updates the status of the cache according to the cache information.

[0202] At this point, the cache queue includes a free buffer, an acquired buffer, and a queued buffer.

[0203] Since the current cycle timestamp has exceeded the total time of the animation, the animation is considered to have been drawn, so the application main thread and rendering thread do not perform drawing and rendering operations.

[0204] 9. During the cycle with Vsync-APP ID 9, when the Vsync_SF signal arrives, the synthesis thread determines to perform the synthesis operation. By interacting with the application main thread, image frame 4 is obtained from the cache queue for synthesis. The synthesized image frame 3 is sent for display, and the display driver displays the received synthesized image frame 3. Before the end of the cycle for displaying image frame 3, the display driver releases the cache occupied by the synthesized image frame 3 and returns the cache information of the cache to the synthesis thread. The synthesis thread returns the cache information to the application main thread, and the application main thread updates the status of the cache based on the cache information. Since the current cycle revelation timestamp has exceeded the total time of the animation, the application main thread and rendering thread do not perform drawing and rendering operations.

[0205] 10. During the Vsync-APP ID cycle 10, the synthesized image frame 4 is sent for display, and the display driver displays the received synthesized image frame 4. Before the end of the cycle for displaying image frame 4, the display driver releases the cache occupied by synthesized image frame 4 and returns the cache information to the compositing thread. The compositing thread returns the cache information to the main application thread, which updates the cache status based on the cache information. Because the current cycle revelation timestamp has exceeded the total duration of the animation effect, the main application thread and rendering thread do not perform any drawing or rendering operations.

[0206] From steps 1-10 above, it can be seen that during cycles with Vsync-APP IDs 4, 5, and 6, the compositing thread did not perform any compositing operations, and no corresponding buffers were released. Consequently, the rendering thread was unable to obtain a free buffer from the cache queue during cycle 4 to store the rendered image frame 4, resulting in the main application thread being unable to calculate and draw image frame 4. Normally, when the main application thread draws image frame 4 during cycle 4 with Vsync-APP ID 4, the resulting displacement interval between image frame 4 and image frame 5 should be 16. However, in the example shown in Figure 6, the buffer for image frame 1 was released before the end of cycle 7 with Vsync-APP ID 7. The main application thread then updated the status of the released buffer during cycle 7 with Vsync-APP ID 7, thereby waking up the rendering thread to obtain a free buffer for drawing image frame 4. At this point, the calculated displacement of image frame 4 is 96, and the displacement interval between image frame 4 and image frame 3 is 64, which differs from the normal displacement interval of 16.

[0207] In the case where the displacement interval changes and the displacement interval between adjacent frames is large, in the process of display driving image frame 4 and image frame 3, since the displacement interval between image frame 4 and image frame 3 is too large, the displacement changes of adjacent image frames will be discontinuous, and there will be obvious visual freezes when displaying the motion effect.

[0208] To better understand the problems in the existing technology, we will provide an example based on the process of rendering and compositing each image frame in the animation, combining the interaction between the application, the compositor, and the HWC. As shown in Figure 9, it includes the following stages:

[0209] Phase 1: triggering the application drawing and rendering phase:

[0210] S101 : The Vsync thread of the synthesizer sends a Vsync_APP signal to the main application thread of the application.

[0211] The Vsync thread of the synthesis thread generates a Vsync_APP signal and sends the Vsync_APP signal to the application main thread of the application. After the application main thread receives the Vsync_APP signal, it starts to execute operations such as drawing and rendering of the current image frame.

[0212] S102: The main thread of the application starts measuring, laying out, and drawing.

[0213] The main thread of the application can obtain the system time for drawing the current image frame. Based on the motion effect curve and the system time, it measures and calculates the displacement of the current frame image, lays out the layout, and draws the image, thereby obtaining the drawn image frame. For example, the image frame here can be image frame 1.

[0214] S103: The main thread of the application wakes up the rendering thread of the application to perform rendering operations.

[0215] The application main thread wakes up the rendering thread to perform rendering operations on the drawn image frame 1.

[0216] S104: The rendering thread dequeues an empty buffer from the buffer queue through the application main thread.

[0217] After completing the rendering operation of image frame 1, the rendering thread requests an empty buffer from the buffer queue through the application main thread to store the rendered image frame 1.

[0218] S105: The last empty buffer in the buffer queue is occupied by the rendering thread.

[0219] The rendering thread obtains the last empty buffer and stores the rendered image frame 1 into the buffer.

[0220] S106: The rendering thread stores the rendered image frame into a cache, and updates the state of the cache through the application main thread.

[0221] The rendering thread enqueues the cache of the rendered image frame 1 into the cache queue through the application main thread, and the application main thread updates the status of the cache, so that the synthesis thread can obtain the rendered image frame from the cache queue for synthesis operation during the synthesis cycle.

[0222] Phase 2: The synthesis thread does not execute the synthesis phase:

[0223] S201 . The Vsync thread of the synthesizer sends a Vsync_SF signal to the synthesis thread.

[0224] The Vsync thread generates a Vsync_SF signal and sends the Vsync_SF signal to the synthesis thread. After the Vsync_SF signal arrives, the synthesis thread determines whether to perform a synthesis operation of the image frame.

[0225] S202: The synthesis thread does not perform the synthesis operation.

[0226] The synthesis thread determines not to perform the synthesis operation. For example, the situations in which the synthesis thread determines not to perform the synthesis operation include abnormal performance of the synthesis thread itself, which causes the synthesis thread to run too long, misses the display signal and causes frame loss; or, due to frame cutting, the interval between two adjacent image frames is too large, and the synthesis thread cannot wait for the display signal, resulting in non-synthesis based on the back pressure mechanism.

[0227] The consequence of the compositing thread not performing compositing operations is that the buffers in the application's cache queue containing rendered image frames cannot be consumed by the compositing thread. If the compositing thread does not composite the image frames in the cache queue, it will not perform the subsequent display and cache release process. As a result, the number of empty buffers in the cache queue will continue to decrease until there are no empty buffers in the cache queue. At this time, the rendering thread will be unable to obtain empty buffers to store rendered image frames.

[0228] Phase three: application triggers the rendering phase:

[0229] S301 : The Vsync thread of the synthesizer sends a Vsync_APP signal to the main application thread.

[0230] The Vsync thread generates a Vsync_APP signal and sends the Vsync_APP signal to the application main thread. After the Vsync_APP signal arrives, the application main thread starts to execute the current image frame drawing and rendering operation.

[0231] S302: The main application thread starts measuring, laying out, and drawing.

[0232] The main thread of the application can obtain the system time for drawing the current image frame, measure and calculate the displacement of the current frame image based on the motion effect curve and the system time, layout, and draw the image, thereby obtaining the drawn image frame. For example, the image frame here can be image frame 2.

[0233] S303: The main application thread wakes up the rendering thread to perform rendering operations.

[0234] The application main thread wakes up the rendering thread to perform rendering operations on the drawn image frame 2.

[0235] S304: The rendering process requests an empty cache from the cache queue through the application main thread.

[0236] After completing the rendering operation of image frame 1, the rendering thread requests an empty buffer from the buffer queue through the application main thread to store the rendered image frame 2.

[0237] S305: There is no empty buffer in the buffer queue, and the rendering thread waits for the synthesis thread to release the buffer.

[0238] Because the compositing thread did not perform the compositing operation in S202, the buffers in the buffer queue containing the rendered image frames were not consumed, and no buffers were released. Therefore, after the last empty buffer in the buffer queue was used in S105, there were no empty buffers in the buffer queue. The rendering thread could not obtain an empty buffer and remained in a waiting phase.

[0239] In the fourth stage, the synthesis thread performs synthesis and interacts with HWC and applications:

[0240] S401 : The Vsync thread sends a Vsync_SF signal to the composition thread.

[0241] The Vsync thread generates a Vsync_SF signal and sends the Vsync_SF signal to the synthesis thread. After the Vsync_SF signal arrives, the synthesis thread determines whether to perform a synthesis operation.

[0242] S402: The synthesis thread starts synthesis, and after synthesis is completed, the synthesized image frame is sent to the HWC for display.

[0243] The synthesis thread determines to perform the synthesis operation, obtains the rendered image frame 1 from the cache queue through the application main thread for synthesis operation, and sends the synthesized image frame 1 to the HWC for display.

[0244] S403. The HWC returns the cache information of the empty cache that has been displayed to the composition thread.

[0245] After displaying the synthesized image frame 1, the HWC releases the cache of the synthesized image frame 1 in the next display cycle and returns the cache information of the cache to the synthesis thread.

[0246] S404: The synthesis thread returns the cache information of the cache to the main application thread.

[0247] After obtaining the cache information of the cache, the composition thread returns the cache information of the cache to the main thread of the application.

[0248] S405: The application main thread updates the status of the cache in the cache queue according to the cache information, increases the number of empty caches in the cache queue by 1, and wakes up the rendering thread for rendering.

[0249] After obtaining the cache information, the main thread of the application increases the number of empty caches in the cache queue by one according to the cache information, and wakes up the rendering thread waiting for the empty cache to perform rendering operations.

[0250] S406: The rendering thread dequeues an empty buffer from the buffer queue through the application main thread to perform a storage operation on the rendered image frame, and updates the status of the buffer through the application main thread.

[0251] After receiving the wake-up message from the application main thread, the rendering thread dequeues an empty buffer from the buffer queue through the application main thread to store the rendered image frame 2, and updates the status of the buffer through the application main thread.

[0252] Obviously, in the above process, the rendering thread of the application is unable to obtain an empty cache from the cache queue in time, resulting in the rendering thread being in a waiting state (S305). The rendering thread does not perform rendering operations, which affects the drawing operations of the main thread of the application, thereby causing the displacement interval between adjacent image frames drawn by the main thread of the application to be too large.

[0253] Alternatively, Figure 10 shows a timing diagram of how the MaxBufferCount of a buffer queue changes during image frame drawing, rendering, synthesis, and display. Referring to Figure 10 , the cycle in which frame 11 occurs is the first cycle. In this example, the MaxBufferCount in the buffer queue is 4.

[0254] During the first cycle, the number of queued buffers in the cache queue is 2, and there are 2 free buffers and 2 queued buffers in the cache queue. The application main thread draws image frame 11, and the calculated displacement interval of image frame 11 is 16. After the application main thread completes drawing image frame 11, it wakes up the rendering thread to perform rendering of image frame 11. After the rendering thread completes rendering of image frame 11, the application main thread dequeues a free buffer from the cache queue to store the rendered image frame 11. At this time, the number of queued buffers in the cache queue increases by 1, and the number of free buffers decreases by 1.

[0255] During this cycle, the synthesis thread interacts with the application main thread to request a queued buffer from the cache queue for synthesis operation. According to the order of image frames, the synthesis thread obtains image frame 10 and performs synthesis operation.

[0256] During the second cycle, the number of queued buffers in the cache queue is 3, with 1 free buffer and 3 queued buffers. The application's main thread draws image frame 12, calculating a displacement interval of 16. After the application's main thread completes drawing frame 12, it wakes up the rendering thread to render frame 12. After rendering frame 12, the rendering thread, through the application's main thread, dequeues the last free buffer from the cache queue to store the rendered frame 12. At this point, the number of queued buffers in the cache queue increases by 1, and the number of free buffers decreases by 1.

[0257] The synthesis thread does not synthesize during this cycle.

[0258] During the third cycle, all buffers in the cache queue are occupied, and the number of queued buffers reaches the maximum, equal to MaxBufferCount (4). There are no free buffers in the cache queue for the rendering thread to use. After the application main thread completes drawing frame 13 based on the displacement interval of 16, the rendering thread cannot obtain a free buffer to store the rendered frame, and the rendering thread enters a waiting state.

[0259] The synthesis thread does not synthesize during this cycle.

[0260] During the fourth cycle, all buffers in the cache queue are occupied. The rendering thread is in a waiting state and does not perform any rendering operations. The application's main thread is also unable to draw the next image frame.

[0261] During this period, the synthesis thread does not synthesize. The display driver displays the received synthesized image frame 9, and the displacement interval of the image frame 9 is 16.

[0262] In the fifth cycle, the synthesis thread interacts with the application main thread and requests a queued buffer from the cache queue for synthesis operation. In the order of image frames, the synthesis thread obtains the rendered image frame 11 to perform synthesis operation. In this cycle, the number of queued buffers in the cache queue is reduced by one. The display driver displays the received synthesized image frame 10, and the displacement interval of the image frame 10 is 16. Before the display of the synthesized image frame 10 ends, the buffer of the synthesized image frame 9 displayed in the previous cycle is released, and the cache information of the buffer is returned to the synthesis thread. The synthesis thread returns the cache information of the buffer to the application main thread, and updates the status of the cache in the cache queue through the application main thread. At this time, there is a free buffer in the cache queue.

[0263] The application main thread wakes up the rendering thread to perform the operation of storing the rendered image frame. The rendering thread interacts with the application main thread to dequeue a free buffer from the cache queue to perform the storage operation of the rendered image frame 13.

[0264] In the sixth cycle, the main application thread draws the image frame 14 , and the calculated displacement interval of the image frame 14 is 48. After the main application thread completes the drawing of the image frame 14 , the rendering thread is awakened to perform the rendering operation of the image frame 14 .

[0265] During this cycle, the compositing thread interacts with the application's main thread to request a queued buffer from the cache queue for compositing. Following the order of the image frames, the compositing thread retrieves rendered image frame 12 for compositing. The display driver displays the received composite image frame 11, which has a shift interval of 16.

[0266] Clearly, the displacement interval for frame 14 drawn by the main thread during this cycle differs from the calculated displacement interval for frame 13. Frame 13's displacement interval is 16, while frame 14's is 48, calculated based on the time and motion effect curves for two cycle durations. The displacement between adjacent frames is excessively large, causing visual stuttering between frames 13 and 14.

[0267] Combined with the buffer usage mechanism in the cache queue described above, this paper analyzes the reasons why the rendering thread cannot obtain free buffers from the cache queue in time during the drawing and rendering synthesis process of time-related animation effects.

[0268] The rendering thread stores rendered image frames in the free buffers in the cache queue. The number of free buffers in the cache queue gradually decreases, while the number of queued buffers gradually increases. In this case, the synthesis thread does not consume the queued buffer in a timely manner. That is, the compositor does not request the queued buffer in a timely manner to perform synthesis operations on the rendered image frames, resulting in no buffer being sent to the display and released to a free state. The number of free buffers in the cache queue becomes smaller and smaller, and the synthesis thread does not perform synthesis operations until there are no free buffers in the cache queue. The rendering thread can no longer obtain free buffers from the cache queue to store rendered image frames, causing the rendering thread to be in a waiting state. The rendering thread cannot continue to perform rendering operations, which affects the application main thread's drawing of the next image frame. The time interval for the application main thread to wait causes the displacement interval between the two adjacent image frames it draws.

[0269] For example, the reason why the synthesis thread does not perform the synthesis operation may be that the performance of the synthesizer where the synthesis thread is located is abnormal, resulting in too long running time and missing the display signal, resulting in frame loss and no synthesis operation is performed; it may also be due to frame cutting causing the interval between two adjacent image frames to be too large, and the synthesis thread cannot wait for the display signal, resulting in the failure to perform the synthesis operation based on the back pressure mechanism.

[0270] Among them, the backpressure mechanism refers to the fact that the synthesis thread believes that there is a task accumulation of image frames to be synthesized (rendered image frames), which causes the synthesis thread to misjudge that it does not need to perform synthesis operations at present, thereby causing the synthesis task of the synthesis thread to lag.

[0271] The compositing thread's mechanism is to send the current frame directly to the HWC when GPU compositing is in progress, without waiting for the previous frame to be displayed. The HWC maintains an asynchronous buffer queue, which serially composes the frames sent by the compositing thread. Because the asynchronous buffer queue allows for backlogs, the compositing thread may not execute compositing tasks if it determines that there is a backlog of frames to be composited (already rendered frames).

[0272] For example, in the process of displaying image frames by an electronic device, there may be a frame rate switching situation, and different frame rate switching may also cause the synthesizer to not perform the synthesis operation, especially in the scenario of switching from a low frame rate to a high frame rate. For example, at the beginning, the frame rate of the electronic device is 60Hz, and the application process, synthesis thread and display driver all perform corresponding drawing rendering, synthesis and display sending operations according to the cycle corresponding to the frame rate of 60Hz. In a certain cycle, the frame rate of the electronic device is switched to 90Hz, and the application process, synthesis thread and display driver all perform corresponding drawing rendering, synthesis and display sending operations according to the cycle corresponding to the frame rate of 90Hz. The cycle of the frame rate of 90Hz is shorter than the cycle of the frame rate of 60Hz, that is, when the frame rate is 90Hz, the application process, synthesis thread and display driver execute the processing speed of the image frames of each cycle faster, and when the frame rate is 60Hz, the application process, synthesis thread and display driver execute the processing speed of the image frames of each cycle slower. When the display driver displays synthesized frames at a 60Hz frame rate and a 16.6ms period, the application process has already started rendering frames at a 90Hz frame rate and a 11.1ms period. This causes the display driver to display frames slower than the application process can render frames and the compositing thread can synthesize frames. This causes a backlog of synthesized frames, which can lead the compositing thread to mistakenly believe that compositing is not currently necessary.

[0273] The non-composition of the composition thread results in no consumer consumption in the cache queue, which further causes the rendering thread to be unable to dequeue the free buffer from the cache queue, thereby blocking the normal operation of the rendering thread and the main thread of the application, resulting in the above problem.

[0274] In addition to the reasons described in the above embodiments that may cause the composition thread to fail to perform composition operations, other situations may occur in the normal operation of some electronic devices that may cause the composition thread to fail to perform image composition. For example, the composition thread may fail to perform composition operations due to the composition thread entering a dead state; or due to an abnormality in the electronic device's input / output (I / O) access; or due to an abnormality in the electronic device's central processing unit (CPU) resources.

[0275] This embodiment provides an image processing method, which can effectively avoid the situation where the synthesis thread does not perform a synthesis operation and there is no free buffer in the cache queue, resulting in the rendering thread being unable to dequeue the free buffer from the cache queue to store the rendered image frame, thereby affecting the application main thread from rendering the next image frame, resulting in frame loss in the image frame drawn by the application main thread, and thus causing the image frame sent to the display to be stuck.

[0276] Based on the hardware structure of the electronic device and the software architecture of the electronic device shown in Figures 4 and 5, an image processing method is provided, taking the electronic device 100 as an example of executing the embodiment of the present disclosure. For example, Figure 11 shows an example of a user performing a first operation on the display screen of an electronic device, and the electronic device launching a first application in response to the first operation. During the process of launching the first application, the application process of the electronic device interacts with the synthesis thread to implement an example of the image processing method. This includes:

[0277] S501: The electronic device receives a first operation of a user on a touch screen of the electronic device.

[0278] The execution subject here can be a desktop application of an electronic device, for example, a desktop launcher of an electronic device, and the launcher is used to receive a first operation of the user on the touch screen of the electronic device. Among them, the first operation can be a single-click operation, a double-click operation, etc. of the user on the touch screen. The first operation is a selection operation of the user for the desktop application of the electronic device. For example, the first operation is a single-click operation of the user on the touch screen of the first application on the desktop of the electronic device. Referring to (a) of Figure 1, the first operation can be a single-click operation of the user on the touch screen of the mobile phone for application 5, which is used to start application 5.

[0279] S502: The electronic device starts a first application in response to a first operation.

[0280] In response to the first operation, the launcher starts the desktop application corresponding to the first operation. Referring to Figure 1, the user performs a single-click operation on the touch screen of the mobile phone for application 5, and the launcher starts application 5 in response to the single-click operation.

[0281] During the launch of Application 5, the launcher displays all image frames of the launch animation of Application 5 on the phone's desktop. For example, the image frames of the launch animation of Application 5 include five image frames. The display process of the launch animation of Application 5 can be referred to from Figure 1(b) to Figure 1(f). All image frames in the launch animation have a time sequence.

[0282] During the startup of the first application, the following steps are performed:

[0283] S503: The application process draws and renders the first image frame within the drawing and rendering cycle of the first image frame, and stores the obtained first image frame in an idle cache object in the cache queue.

[0284] Before displaying the five image frames of the startup animation of the startup application 5, the electronic device needs to draw, render, and synthesize these image frames, and then send the synthesized image frames for display to present the final display effects of Figure 1 (b) to Figure 1 (f).

[0285] Generally, an application process draws and renders image frames. Specifically, an application main thread in an application process draws image frames, a rendering thread in the application process renders the drawn image frames, and a compositing thread composites the rendered image frames.

[0286] The first image frame is an image frame of the startup animation during the startup process of application 5.

[0287] During the startup process of application 5, the application main thread of the application process draws the first image frame within the drawing and rendering cycle of the first image frame, and the rendering thread of the application process renders the drawn first image frame to obtain a rendered first image frame. If there are free cache objects in the cache queue, the rendering thread stores the rendered first image frame in a free cache object in the cache queue. Accordingly, after the rendered first image frame is stored in a free cache object in the cache queue, the number of free cache objects in the cache queue decreases by one.

[0288] S504: When the synthesis thread does not perform a synthesis operation within the synthesis period of the first image frame, the synthesis thread sends a first adjustment request to the application process.

[0289] The composition cycle of the first image frame is after the drawing and rendering cycle of the first image frame. If the composition thread does not perform a composition operation within the composition cycle of the first image frame, that is, the composition thread does not perform a composition operation on the rendered first image frame, the cache queue has no consumption, and there may be no free cache objects in the cache queue. In this case, the composition thread sends a first adjustment request to the application process.

[0290] S505. The application process increases the number of free cache objects in the cache queue based on the first adjustment request, so that the application process draws and renders the second image frame within the drawing and rendering cycle of the second image frame and then stores the obtained second image frame in a free cache object in the cache queue.

[0291] The first adjustment request may carry a first indication value, which is used to indicate an increase in the number of cache objects and to increase the number of free cache objects in the cache queue. The free cache objects are free buffers in the cache queue.

[0292] The application process increases the number of free cache objects in the cache queue based on the first indicator value in the first adjustment request. For example, if the first indicator value is 1, the application process increases the number of free cache objects in the cache queue by 1. For example, if the first indicator value is 2, the application process increases the number of free cache objects in the cache queue by 2.

[0293] After increasing the number of free cache objects in the cache queue, it is ensured that there is always at least one free cache object in the cache queue available for use by the application process. In other words, there is always at least one free cache object in the cache queue that can be used by the application process's rendering thread to store the image frame obtained in the next image frame drawing and rendering cycle, for example, the second image frame obtained in the second image frame drawing and rendering cycle. Here, the second image frame is an image frame in the startup animation during the startup process of application 5.

[0294] The drawing and rendering cycle of the second image frame is after the drawing and rendering cycle of the first image frame, and the drawing start time of the second image frame differs from the drawing start time of the first image frame by N cycles, where N is a positive integer.

[0295] The image frames in the startup animation have a time sequence, and the drawing and rendering cycle of the second image frame is located after the drawing and rendering cycle of the first image frame. Optionally, the drawing and rendering cycle of the second image frame can be the next cycle of the drawing and rendering cycle of the first image frame; or the drawing and rendering cycle of the second image frame can be the next N cycles of the drawing and rendering cycle of the first image frame. For example, the drawing and rendering cycle of the second image frame can be the second cycle after the drawing and rendering cycle of the first image frame.

[0296] Optionally, the first image frame and the second image frame may also be image frames after the electronic device has completed launching the first application, for example, multiple image frames in an internal display screen of the first application.

[0297] Optionally, the first image frame and the second image frame may also be image frames when the electronic device switches from a first refresh rate to a second refresh rate during startup of the first application, wherein the first refresh rate is lower than the second refresh rate.

[0298] Here, the refresh rate refers to the frame rate of the electronic device. The first refresh rate can be 60Hz, that is, 60 frames are refreshed in 1 second, and one frame is refreshed every 16.6 milliseconds. The second refresh rate can be 90Hz, that is, 90 frames are refreshed in 1 second, and one frame is refreshed every 11.1 milliseconds.

[0299] In the scenario of refresh rate switching, the image processing method provided in this embodiment can effectively solve the problem that during the application startup process, due to the refresh rate switching, the synthesis thread is asynchronous with the application process processing cycle, and the synthesis thread believes that there is a task backlog in the image frame to be synthesized and does not perform the synthesis operation, resulting in no cache object being released in the cache queue.

[0300] In this embodiment, during the startup process of an application executed by an electronic device, image processing is performed on the image frames of the startup animation during the startup process. When the synthesis thread does not perform synthesis operations, the number of free cache objects in the cache queue is dynamically increased, so that there is always at least one free cache object in the cache queue used by the application process. During each image frame drawing and rendering cycle, the application process can store the rendered image frame in the free cache object of the cache queue, avoiding the frame loss that may occur during the drawing and rendering process of the application process, solving the problem of display freeze of the image frame sent for display due to frame loss, and improving the display smoothness of the startup animation during the application startup process.

[0301] Optionally, referring to FIG12 , the user may also perform a second operation based on the display screen of the electronic device, and the electronic device may exit the first application in response to the second operation. Referring to FIG13 , FIG13 shows an example of an image processing method implemented by the interaction between the application process of the electronic device and the composition thread during the process of exiting the first application. The method includes:

[0302] S601: The electronic device receives a second operation of the user on the touch screen of the electronic device.

[0303] Optionally, the second operation may be a swipe-up operation.

[0304] Accordingly, the execution subject here can be a desktop application of an electronic device, for example, a desktop launcher of an electronic device, and the launcher is used to receive a second operation of the user on the touch screen of the electronic device. Among them, the second operation can be a sliding operation of the user on the touch screen, etc. The first operation is the user's exit operation for the application of the electronic device. For example, the second operation is the user's swiping up to exit the first application on the desktop on the touch screen. Referring to Figure 12 (a), the first operation can be a swiping up operation of the user on the touch screen of the mobile phone for application 5, which is used to exit application 5 and return to the desktop of the mobile phone.

[0305] S602: The electronic device exits the first application in response to the second operation.

[0306] In response to the second operation, the launcher exits the current interface of the application corresponding to the second operation and returns to the mobile phone's desktop. Referring to Figure 12, the user swipes up on the touch screen of the mobile phone to select application 5. In response to the swipe up, the launcher exits application 5 and returns to the mobile phone's desktop display interface.

[0307] During the process of exiting Application 5, the launcher displays all image frames of the exit animation for Application 5 on the phone's desktop. For example, the image frames for the exit animation for Application 5 include five frames. The display process of the exit animation for Application 5 can be seen from Figures 12(b) to 12(f). All image frames in the exit animation have a time sequence.

[0308] During the exit process of the first application, the electronic device performs the following steps:

[0309] S503: The application process draws and renders the first image frame within the drawing and rendering cycle of the first image frame, and stores the obtained first image frame in an idle cache object in the cache queue.

[0310] Before displaying the five image frames of the exit animation of exiting application 5, the electronic device needs to draw, render, and synthesize these image frames, and then display the synthesized image frames to present the final display effects of Figure 12 (b) to Figure 12 (f).

[0311] The first image frame is an image frame in the exit animation during the exit process of application 5.

[0312] During the startup process of application 5, the application main thread of the application process draws the first image frame within the drawing and rendering cycle of the first image frame, and the rendering thread of the application process renders the drawn first image frame to obtain a rendered first image frame. If there are free cache objects in the cache queue, the rendering thread stores the rendered first image frame in a free cache object in the cache queue. Accordingly, after the rendered first image frame is stored in a free cache object in the cache queue, the number of free cache objects in the cache queue decreases by one.

[0313] S504: When the synthesis thread does not perform a synthesis operation within the synthesis period of the first image frame, the synthesis thread sends a first adjustment request to the application process.

[0314] The composition cycle of the first image frame is after the drawing and rendering cycle of the first image frame. If the composition thread does not perform a composition operation within the composition cycle of the first image frame, that is, the composition thread does not perform a composition operation on the rendered first image frame, the cache queue has no consumption, and there may be no free cache objects in the cache queue. In this case, the composition thread sends a first adjustment request to the application process.

[0315] S505. The application process increases the number of free cache objects in the cache queue based on the first adjustment request, so that the application process draws and renders the second image frame within the drawing and rendering cycle of the second image frame and then stores the obtained second image frame in a free cache object in the cache queue.

[0316] The first adjustment request may carry a first indication value, and the first indication value is used to indicate an increased number of cache objects, thereby increasing the number of idle cache objects in the cache queue.

[0317] The application process increases the number of free cache objects in the cache queue based on the first indicator value in the first adjustment request. For example, if the first indicator value is 1, the application process increases the number of free cache objects in the cache queue by 1. For example, if the first indicator value is 2, the application process increases the number of free cache objects in the cache queue by 2.

[0318] By increasing the number of free cache objects in the cache queue, it is ensured that there is always at least one free cache object in the cache queue available for use by the application process. In other words, there is always at least one free cache object in the cache queue that can be used by the application process's rendering thread to store the image frame obtained in the next image frame rendering cycle, for example, the second image frame obtained in the second image frame rendering cycle. Here, the second image frame is an image frame in the exit animation during the exit process of application 5.

[0319] The drawing and rendering cycle of the second image frame is after the drawing and rendering cycle of the first image frame, and the drawing start time of the second image frame differs from the drawing start time of the first image frame by N cycles, where N is a positive integer.

[0320] The image frames in the exit animation have a time sequence, and the drawing and rendering cycle of the second image frame is located after the drawing and rendering cycle of the first image frame. Optionally, the drawing and rendering cycle of the second image frame can be the next cycle of the drawing and rendering cycle of the first image frame; or the drawing and rendering cycle of the second image frame can be the next N cycles of the drawing and rendering cycle of the first image frame. For example, the drawing and rendering cycle of the second image frame can be the second cycle after the drawing and rendering cycle of the first image frame.

[0321] Optionally, the first image frame and the second image frame may also be image frames when the electronic device switches from the first refresh rate to the second refresh rate during the process of exiting the first application, wherein the first refresh rate is lower than the second refresh rate.

[0322] Here, the refresh rate refers to the frame rate of the electronic device. The first refresh rate can be 60Hz, that is, 60 frames are refreshed in 1 second, and one frame is refreshed every 16.6 milliseconds. The second refresh rate can be 90Hz, that is, 90 frames are refreshed in 1 second, and one frame is refreshed every 11.1 milliseconds.

[0323] In the scenario of refresh rate switching, the image processing method provided in this embodiment can effectively solve the problem that during the application exit process, due to the refresh rate switching, the synthesis thread is asynchronous with the application process processing cycle, and the synthesis thread believes that there is a task backlog in the image frame to be synthesized and does not perform the synthesis operation, resulting in no cache object being released in the cache queue.

[0324] In this embodiment, during the process of exiting an application executed by an electronic device, image processing is performed on the image frames of the exit animation during the exit process. When the synthesis thread does not perform synthesis operations, the number of free cache objects in the cache queue is dynamically increased, so that there is always at least one free cache object in the cache queue used by the application process. During each image frame drawing and rendering cycle, the application process can store the rendered image frame into the free cache object in the cache queue, avoiding the frame loss that may occur during the drawing and rendering process of the application process, solving the problem of display freeze of the image frame sent for display due to frame loss, and improving the display smoothness of the exit animation during the application exit process.

[0325] It is understandable that the image processing method performed by the electronic device in steps S503 to S505 can also be applied in the process from starting to exiting the first application.

[0326] Optionally, the image processing method of steps S503-S505 performed by the electronic device can also be applied in other scenarios. For example, the image processing scenario of the image frame of the internal scene dynamic effect of the electronic device, the image processing scenario of the image frame of the game scene dynamic effect of the electronic device, the image processing scenario of the image frame of the off-screen sliding dynamic effect of the electronic device, or the image processing scenario of the image frame of other hand-following dynamic effect of the electronic device, etc. In these scenarios, the problem of the main thread of the application dropping frames due to the synthesis thread of the electronic device not performing the synthesis operation, which causes the image frame to be jammed when being displayed, can be solved, thereby optimizing the display smoothness of the image frame.

[0327] In addition, since the synthesis thread is more likely to not perform synthesis operations in the scenarios of application startup of the electronic device (refer to Figure 1), application exit of the electronic device (refer to Figure 12), and off-screen sliding of the electronic device (refer to Figure 2), the effect of the image processing method provided by this embodiment is more obvious in these scenarios, and the optimized motion effect display effect is smoother.

[0328] In one example, an image processing method is provided for the interaction between the application main thread, rendering thread, compositor composition thread, compositor Vsync thread, and HWC during the drawing, rendering, composition, and display of dynamic image frames. Referring to the flowchart of the method shown in FIG14 , the method includes the following stages:

[0329] Phase 1: Application drawing and rendering phase:

[0330] S1101 . The Vsync thread of the synthesizer sends a Vsync_APP signal to the main application thread of the application.

[0331] In this embodiment, the synthesizer includes a Vsync thread and a synthesis thread. The Vsync thread is used to generate a Vsync signal. The Vsync signal includes a Vsync_APP signal and a Vsync_SF signal. The Vsync_APP signal is used to trigger the application main thread to perform image frame drawing operations. The Vsync_SF signal is used to trigger the synthesis thread to perform image frame synthesis operations.

[0332] The Vsync thread determines the signal period based on the electronic device's frame rate. For example, if the electronic device's frame rate is 60 and the image frame interval is 16.6ms, the Vsync thread generates a Vsync_APP signal every 16.6ms and sends this Vsync_APP signal to the application main thread. The Vsync thread also generates a Vsync_SF signal every 16.6ms and sends this Vsync_SF signal to the composition thread.

[0333] S1102: The main application thread starts measuring, laying out, and drawing.

[0334] The application's main thread can obtain the system time for drawing the current image frame. Based on the motion curve and system time, it measures and calculates the image displacement of the current frame, lays out the layout, and draws the image, thereby obtaining the rendered image frame. For example, the image frame here could be image frame 2. In this case, it can be assumed that the application's main thread has completed drawing image frame 1, and the rendering thread has completed rendering image frame 1.

[0335] After the Vsync_APP signal arrives, the application's main thread executes the current frame drawing operation in the current cycle. The drawn frame is frame 2. Before drawing the image frame, the application's main thread needs to measure and lay out the image frame, that is, it needs to calculate the image frame's displacement.

[0336] This embodiment is aimed at drawing the image frame of the first type of dynamic effect whose displacement is related to the system time. For example, the displacement of the image frame can be calculated as follows: y(t) = t / t total *y total

[0337] Where t is the current time, t total is the total display time of the animation, y total The displacement distance between the first and last image frames of the animation.

[0338] Alternatively, the displacement of the image frame can also be calculated as: y(t)=y(0)+t*n

[0339] Among them, y(0) is the displacement of the first image frame of the animation; t is the calculation time; and n is the preset displacement interval.

[0340] Alternatively, t can be calculated as: t = t c -(t c -t0)%q

[0341] Among them, t c is the current time; t0 is the drawing time of the first image frame of the animation; q is the frame rate of the electronic device.

[0342] After the Vsync_APP signal arrives, the main application thread measures, lays out, and draws the image frame 2 according to a preset calculation method for image frame displacement.

[0343] S1103: The application main thread wakes up the application rendering thread to perform rendering operations.

[0344] After the application main thread completes the measurement, layout, and drawing of the image frame 2 , the application main thread wakes up the application rendering thread to perform a rendering operation on the drawn image frame 2 .

[0345] S1104: The rendering process dequeues an empty buffer from the buffer queue through the application main thread.

[0346] In this embodiment, after completing the rendering operation of image frame 2, the rendering thread interacts with the application main thread to dequeue a free buffer from the cache queue to store the rendered image frame 2.

[0347] S1105: The last empty buffer in the buffer queue is occupied by the rendering thread.

[0348] If there is a free buffer in the cache queue, the rendering thread can interact with the application main thread to dequeue the free buffer and store the rendered image frame 1.

[0349] Optionally, the rendering thread may dequeue a free buffer from the cache queue in a first-in-first-out (FIFO) manner; or, the rendering thread may dequeue a free buffer from the cache queue in other agreed ways.

[0350] After the rendering thread dequeues the free buffer, the application main thread updates the cache status to dequeued.

[0351] S1106: The application main thread updates the cache status and sends a response to the composition thread.

[0352] The rendering thread may enqueue the buffer of rendered image frame 2 into the buffer queue according to a first-in-first-out (FIFO) acquisition method; alternatively, the rendering thread may enqueue the buffer of rendered image frame 2 into the buffer queue according to other agreed methods. After the rendering thread stores rendered image frame 2 into the buffer, it enqueues the buffer containing rendered image frame 2 into the buffer queue and, through interaction with the application main thread, updates the status of the buffer to queued. After updating the buffer status, the application main thread sends a response to the compositing thread, causing the compositing thread to request the queued buffer for compositing operations.

[0353] Phase 2: The synthesis thread does not execute the synthesis phase:

[0354] S1201. The Vsync thread of the synthesizer sends a Vsync_SF signal to the synthesis thread.

[0355] In this embodiment, the Vsync thread of the synthesizer generates a Vsync_SF signal according to a frame interval and sends the Vsync_SF signal to the synthesis thread. The synthesis thread determines whether to perform a synthesis operation of the image frame.

[0356] S1202: The synthesis thread does not perform synthesis operations.

[0357] In this embodiment, upon arrival of the Vsync_SF signal, the composition thread determines that no composition operation will be performed in the current composition cycle. The composition thread does not composite the image frames in the cache queue, meaning that the queued buffers in the cache queue have not been consumed and no acquired buffers have been released. The composition thread's current composition cycle is the cycle immediately following the application main thread's drawing cycle for frame 2.

[0358] S1203: The synthesis thread determines to increase the maximum cache quantity of the cache queue by 1.

[0359] In this embodiment, if the composition thread does not perform the composition operation, it will inevitably cause the queued buffer in the cache queue to not be consumed. In this case, if the composition thread determines that the composition operation will not be performed in the current cycle, the maximum buffer count (MaxBufferCount) of the cache queue is increased by 1.

[0360] Optionally, the composition thread can use a timer to determine whether a composition operation has been executed within a period of time. If no composition operation has been executed, the maximum buffer size of the cache queue is increased by 1. The period of time can also be the length of N cycles corresponding to the current frame rate; where N is 1, 2, 3, ... k (an integer). To more effectively adjust the maximum buffer size of the cache queue, N should not be too large. It should be noted that when N is 2, it means that the composition thread has not executed a composition operation for two consecutive cycles.

[0361] Optionally, the synthesis thread can send a request to the application main thread to increase MaxBufferCount by 1, so that the application main thread performs corresponding operations based on the request. For example, after receiving the synthesis thread's request to increase MaxBufferCount by 1, the application main thread increases the maximum buffer quantity of the cache queue. The synthesizer can first query the MaxBufferCount of the cache queue, and then determine the amount to increase based on MaxBufferCount. Optionally, the synthesizer can call a preset query function to query the application main thread for the MaxBufferCount of the cache queue. Exemplarily, the preset query function can be IGrapgicBufferConsumer, and the getMaxBufferCount interface is added to IGrapgicBufferComsumer to dynamically query the maximum value. When querying, the synthesizer acts as a consumer and calls the IGrapgicBufferComsumer::getMaxBufferCount() function through the thread Binder to query the application main thread.

[0362] Optionally, the application main thread can obtain a cache from other caches not used by the composition thread and the rendering thread, and set the cache as the cache that can be used by the composition thread and the rendering thread, thereby increasing the number of cache queues.

[0363] S1204: The application main thread adds an available cache to the cache queue, so that the maximum cache quantity of the cache queue increases by 1.

[0364] In this embodiment, the caches in the electronic device are occupied by various threads to perform corresponding operations. Some caches can be used by the synthesis thread and the rendering thread to implement the drawing, rendering and synthesis operations of the image frame. These caches form the cache queue of this embodiment. These caches are not allowed to be used by the synthesis thread and the rendering thread. These caches are called unavailable caches in this embodiment. The unavailable caches include empty caches and occupied caches. The main thread of the application can obtain an empty cache from the unavailable cache and add it to the cache queue of this embodiment, thereby increasing the MaxBufferCount of the cache queue, so that the MaxBufferCount in the cache queue is increased by one.

[0365] After the application main thread increases the MaxBufferCount of the buffer queue, there is at least one free buffer in the buffer queue that can be used by the rendering thread.

[0366] Phase three, application drawing and rendering phase:

[0367] S1301. The Vsync thread of the synthesizer sends a Vsync_APP signal to the application main thread.

[0368] In this embodiment, the Vsync thread generates a Vsync_APP signal according to a frame interval, and sends the Vsync_APP signal to the application main thread. When the Vsync_APP signal arrives, the application main thread starts to execute the image frame drawing and rendering operation.

[0369] S1302: The main application thread starts measuring, laying out, and drawing.

[0370] Similar to step 1102 above, the main application thread can obtain the system time for drawing the current image frame, and measure, calculate, lay out, and draw the current frame image displacement based on the motion effect curve and the system time, thereby obtaining the drawn image frame. For example, the image frame here can be image frame 3.

[0371] The application main thread draws image frame 3 during the cycle immediately following the cycle for drawing image frame 2. This cycle can be considered a normal cycle. The displacement interval of image frame 3 is the same as the displacement interval of image frame 2. Alternatively, the displacement interval of image frame 3 is equal to a preset displacement interval threshold.

[0372] S1303: The main application thread wakes up the rendering thread to perform rendering operations.

[0373] After the application main thread completes the measurement, layout, and drawing of the image frame 3 , the application main thread wakes up the application rendering thread to perform a rendering operation on the drawn image frame 3 .

[0374] S1304: The rendering process dequeues an empty buffer from the buffer queue through the application main thread.

[0375] Since the main thread increases the MaxBufferCount in the buffer queue in S1204, there is now at least one free buffer in the buffer queue that can be used by the rendering thread. The rendering thread dequeues a free buffer from the buffer queue to store the rendered image frame 3.

[0376] S1305: Get the last empty cache.

[0377] In this embodiment, there is one free buffer left in the cache queue. In this embodiment, the free buffer is a buffer added by the application main thread. Before the application main thread adds a buffer to the cache queue, all buffers in the cache queue have been occupied.

[0378] The application rendering thread obtains the last free buffer in the cache queue and stores the rendered image frame 3.

[0379] When the rendering thread dequeues a free buffer, the application main thread can update the status of the buffer in the buffer queue, updating the status of the buffer from free to dequeued.

[0380] S1306: The rendering thread stores the rendered image frame into a cache, and the application main thread sends a response to the synthesis thread.

[0381] The rendering thread, through interaction with the application's main thread, enqueues the buffer containing rendered image frame 3 into the cache queue. The application's main thread then updates the cache's status from dequeued to queued. The application's main thread sends a response to the compositing thread, allowing it to request the queued buffer from the cache queue for compositing the rendered image frame.

[0382] S1307: The synthesis thread records the time of entering the queue cache.

[0383] In this embodiment, the composition thread receives the response sent by the application main thread and records the time when the application main thread executes the queue buffer. The queue buffer time can also represent the time when the rendering thread executes the rendering operation.

[0384] Optionally, the composition thread may also obtain the queue buffer time from the application main thread, and thus record the obtained queue buffer time.

[0385] In this embodiment, the synthesis thread records the time of each queue buffer in the cache queue, and can determine whether the rendering thread performs the rendering operation normally according to the cycle corresponding to the frame rate by comparing the time of two adjacent queue buffers. If the time difference between the times of two adjacent queue buffers is consistent with the frame interval, it is considered that the rendering thread performs the rendering operation normally according to the cycle, indicating that the consumption and production of buffers in the application's buffer queue are in a balanced state, that is, the rendering thread can dequeue free buffers from the cache queue for use. If the time difference between the times of two adjacent queue buffers is greater than the frame interval, it is considered that the rendering thread has an abnormality, or the rendering thread has completed the rendering operation of all image frames of the current animation. At this time, the production of buffers in the cache queue is less than the consumption, then the synthesis thread can dynamically adjust the number of buffers in the cache queue, for example, reducing the MaxBufferCount of the cache queue.

[0386] Phase 4: The synthesizer performs synthesis and interacts with HWC and applications:

[0387] S1401. The Vsync thread of the synthesizer sends a Vsync_SF signal to the synthesis thread.

[0388] According to the frame rate of the electronic device, after a frame interval, the Vsync thread generates a Vsync_SF signal and sends the Vsync_SF signal to the synthesis thread. After receiving the Vsync_SF signal, the synthesis thread determines whether to perform an image frame synthesis operation.

[0389] S1402: The synthesis thread starts synthesis, and after synthesis is completed, the synthesized image frame is sent to the HWC for display.

[0390] In this embodiment, after receiving the Vsync_SF signal, the composition thread determines to perform a composition operation. The composition thread interacts with the application main thread to obtain the rendered image frame 1 from the cache queue, performs a composition operation on the rendered image frame 1, and sends the composed image frame 1 to the HWC for display.

[0391] Optionally, the compositing thread may acquire a rendered image frame (queued buffer) from the cache queue in a FIFO acquisition manner; or, the compositing thread may acquire a queued buffer from the cache queue in other agreed manners.

[0392] The composition thread acquires a queued buffer through the application main thread, and the application main thread can update the state of the cache from queued to acquired.

[0393] S1403. The HWC returns the cache information of the released cache to the composition thread.

[0394] In this embodiment, the HWC displays the composite image frame 1 sent by the composition thread and releases the buffer of the previous image frame before the end of displaying the composite image frame 1. After releasing the buffer occupied by the previous image frame, the HWC returns the buffer information of the buffer to the composition thread.

[0395] S1404: The synthesis thread returns the cache information to the application main thread through a callback function.

[0396] In this embodiment, the composition thread returns the cache information to the application main thread through a callback function, and the application main thread performs an update operation on the cache status in the cache queue based on the cache information.

[0397] Phase 5: The synthesis thread determines whether to adjust the number of cache queues:

[0398] S1501. The Vsync thread sends a Vsync_SF message to the composition thread.

[0399] After a frame interval, the Vsync thread generates a Vsync_SF signal and sends the Vsync_SF signal to the synthesis thread. When the Vsync_SF signal arrives, the synthesis thread determines whether to perform a synthesis operation of the image frame.

[0400] S1502: The synthesis thread starts synthesis and sends the synthesized image frame to the HWC after synthesis is completed.

[0401] In this embodiment, the composition thread determines to perform the composition operation. The composition thread interacts with the application main thread to obtain the rendered image frame 2 from the cache queue, performs the composition operation on the rendered image frame 2, and sends the composed image frame 2 to the HWC for display.

[0402] Alternatively, the composition thread may acquire a queued buffer from the cache queue in a FIFO manner; or, the composition thread may acquire a queued buffer from the cache queue in another agreed manner. In this embodiment, after performing the composition operation, the compositor stops increasing the MaxBufferCount of the cache queue.

[0403] When the compositor acquires a queued buffer, the application's main thread can update the cache's status from queued to acquired.

[0404] S1503. The synthesis thread obtains the time of the last queue entry and calculates the difference between the current system time and the time of the last queue entry. If the difference is greater than or equal to a preset threshold, the decision is made to start dynamically reducing the maximum cache size of the cache queue.

[0405] When the synthesis thread performs a synthesis operation, the synthesis thread determines whether it is necessary to adjust the MaxBufferCount of the cache queue based on the time of each queue buffer in the cache queue recorded in S1307 above. Adjusting the MaxBufferCount actually adjusts the number of free buffers in the cache queue. For example, the synthesis thread can obtain the current system time and the time of the last queue buffer, calculate the time difference between the current system time and the time of the last queue buffer, and if the time difference is greater than the interval of two frames, it is determined that the rendering thread has lost frames and has lost two image frames. In this case, the synthesis thread can generate a request to reduce the MaxBufferCount of the cache queue and send the request to the application main thread to reduce the MaxBufferCount.

[0406] Optionally, the compositor thread can also determine whether the number of queued buffers in the cache queue is increasing to determine whether to dynamically reduce the number of free buffers in the cache queue. For example, if the compositor thread determines that the number of queued buffers in the cache queue is no longer increasing, it determines that the rendering thread is not performing any rendering operations. In this case, the compositor thread can also generate a request to reduce the cache queue's MaxBufferCount and send this request to the application main thread to reduce MaxBufferCount.

[0407] Optionally, the composition thread can call a preset query function to query the application main thread for the status of each buffer in the cache queue, thereby determining whether the number of queued buffers has increased. For example, the getQueuedBufferCount interface can be added to IGrapgicBufferConsumer to dynamically adjust the number of queued buffers. When querying, the composition thread acts as a consumer and calls the IGrapgicBufferComsumer::getQueuedBufferCount() function to query the application main thread through Binder.

[0408] Stage 6: The synthesizer dynamically adjusts the number of cache queues:

[0409] S1601. The Vsync thread sends a Vsync_SF message to the composition thread.

[0410] After a frame interval, the Vsync thread generates a Vsync_SF signal and sends the Vsync_SF signal to the synthesis thread. After the Vsync_SF signal arrives, the synthesis thread determines whether to perform the synthesis operation of the image frame.

[0411] S1602: The synthesis thread starts synthesis and sends the synthesized image frame to the HWC after synthesis is completed.

[0412] In this embodiment, the composition thread determines to perform the composition operation. The composition thread interacts with the application main thread to obtain the rendered image frame 3 from the cache queue, performs the composition operation on the rendered image frame 3, and sends the composed image frame 3 to the HWC for display.

[0413] Alternatively, the compositing thread may acquire a buffer (queued buffer) of a rendered image frame from the cache queue using a FIFO acquisition method; or, the compositing thread may acquire a queued buffer from the cache queue using other agreed methods. In this embodiment, after performing the compositing operation, the compositor stops increasing MaxBufferCount.

[0414] When the compositor acquires a queued buffer, the application's main thread can update the cache's status from queued to acquired.

[0415] S1603. The HWC returns the cache information of the released cache to the composition thread.

[0416] In this embodiment, illustratively, the HWC displays the composite image frame 2 sent by the composition thread, and releases the buffer of image frame 1 before the end of displaying the composite image frame 2. After releasing the buffer occupied by image frame 1, the HWC returns the cache information of the cache to the composition thread.

[0417] S1604: The composition thread sends the cached information back to the application main thread through the callback function, and sends a request to the application main thread to reduce the maximum cache quantity of the cache queue by 1.

[0418] In this embodiment, the composition thread returns the cache information to the application main thread via a callback function, and the application main thread updates the status of the cache in the cache queue based on the cache information. Optionally, the application main thread receives a request from the composition thread to reduce the MaxBufferCount of the cache queue by one, and the application main thread reduces the MaxBufferCount of the cache queue by one.

[0419] S1605: The application main thread removes the empty cache in the cache queue and updates the removed cache to an unavailable cache.

[0420] In this embodiment, the application main thread can remove one free buffer from the cache queue. This free buffer is destroyed, the graphics buffer is released, and the removed buffer is updated to an unavailable buffer. This unavailable buffer is no longer available for use by the rendering thread or the compositing thread, so the MaxBufferCount of the cache queue in this embodiment is reduced by 1.

[0421] In this embodiment, when the composition thread is not performing composition operations, it interacts with the application main thread to increase the MaxBufferCount of the cache queue, that is, to increase the number of free buffers in the cache queue. This ensures that even when the composition thread is not performing composition (that is, the consumer of the cache queue is not consuming), the rendering thread can still obtain free buffers to store rendered image frames (the producer of the cache queue still has free buffers available), thereby achieving a balance between production and consumption of the cache queue. The rendering thread will not be unable to obtain free buffers for production due to non-consumption, which will affect the rendering thread's rendering operation and the application main thread's drawing operation. In this way, the application main thread can normally draw each image frame according to the frame interval, and the rendering thread can normally render each image frame according to the frame interval. In the scenario of animations where the displacement of rendered image frames is related to system time, based on the method provided in this embodiment, the main thread of the application can normally draw each image frame of the animation. Therefore, the problem of excessive time interval between two adjacent image frames drawn by the main thread of the application in the prior art, which causes visual stuttering, will not occur. This improves the smoothness of the animation display and avoids the problems of stuttering and frame loss.

[0422] In addition, after performing the synthesis operation, the synthesis thread determines whether the rendering thread is working normally according to the frame interval. If it is determined that the rendering thread has not performed the rendering operation for at least one cycle, it generates a request to reduce the MaxBufferCount of the cache queue, and interacts with the main thread of the application to reduce the MaxBufferCount of the cache queue. That is, the number of free buffers in the cache queue is reduced, and the redundant buffers in the cache queue can be released in time. The released buffers can be used for other operations to improve the utilization of the buffer.

[0423] Figure 15 shows a timing diagram of how the MaxBufferCount in the cache queue changes during the image frame drawing, rendering, and synthesis process in conjunction with the image processing method provided in an embodiment of the present application. Referring to Figure 15 , the dynamic adjustment process of the MaxBufferCount in the cache queue in this embodiment is further explained. For each segmented region, the cycle in which frame 11 occurs is considered the first cycle. In this example, the MaxBufferCount in the cache queue is 4.

[0424] During the first cycle, the number of queued buffers in the cache queue is 2, that is, there are 2 free buffers and 2 queued buffers in the cache queue. The application main thread draws image frame 11, and the displacement interval of image frame 11 calculated by it is 16. After the application main thread completes drawing image frame 11, it wakes up the rendering thread to perform rendering of image frame 11. After completing the rendering of image frame 11, the rendering thread interacts with the application main thread to dequeue a free buffer from the cache queue and store the rendered image frame 11. The number of queued buffers in the cache queue increases by 1, and the number of free buffers decreases by 1.

[0425] In this cycle, the synthesizer obtains the rendered image frame 10 in the order of the image frames and performs a synthesis operation on the image frame 10. The display driver displays the synthesized image frame 9 on the screen.

[0426] In the second cycle, the number of queued buffers in the cache queue is 3, that is, there are 3 free buffers and 1 queued buffer in the cache queue. The application main thread draws image frame 12, and the displacement interval of image frame 12 calculated by it is 16. After the application main thread completes the drawing of image frame 12, it wakes up the rendering thread to perform the rendering of image frame 12. After completing the rendering of image frame 12, the rendering thread interacts with the application main thread to obtain the last free buffer from the cache queue and stores it in the rendered image frame 12. The number of queued buffers in the cache queue increases by 1, and the number of free buffers decreases by 1. At this point, the number of queued buffers in the cache queue has reached MaxBufferCount.

[0427] During this cycle, the composition thread does not perform composition. The display driver displays the composite image frame 9 on the screen.

[0428] In the third cycle, the number of queued buffers in the cache queue has reached MaxBufferCount, which is 4. That is, all buffers in the cache queue are occupied. The main thread of the application draws image frame 13, and the calculated displacement interval of image frame 13 is 16.

[0429] During this cycle, the buffer queue's MaxBufferCount is dynamically increased by 1. At this point, MaxBufferCount is 5. The rendering thread can retrieve the last newly added free buffer from the buffer queue and store it in the rendered image frame 13. Both the rendering thread and the application's main thread are in a normal state. The compositor does not perform compositing during this cycle.

[0430] In the fourth cycle, the number of queued buffers in the cache queue has reached MaxBufferCount, which is 5. All buffers in the cache queue are occupied. The main thread of the application draws image frame 14, and the calculated displacement interval of image frame 14 is 16.

[0431] During this cycle, the number of buffers in the cache queue is dynamically increased, and MaxBufferCount is incremented by 1. At this point, MaxBufferCount is 6. The rendering thread can retrieve the last newly added free buffer from the cache queue and store it in the rendered image frame 14. Both the rendering thread and the application main thread are in normal state.

[0432] During this period, the composition thread does not perform composition. The display driver displays the composite image frame 9 on the screen.

[0433] In the fifth cycle, the synthesis thread obtains a queued buffer from the cache queue to perform a synthesis operation, for example, obtains image frame 11 to perform a synthesis operation. In this cycle, the number of queued buffers in the cache queue is reduced by one to 5. The display driver displays the received synthesized image frame 10, and the displacement interval of the image frame 10 is 16. Before the display of the synthesized image frame 10 is completed, the buffer of the synthesized image frame 9 is released, and the cache information of the buffer is returned to the synthesis thread. The synthesis thread returns the cache information to the application main thread, and the application main thread updates the cache status of the cache queue according to the cache information. At this time, there is a free buffer in the cache queue.

[0434] The main application thread draws image frame 15, and the calculated displacement interval of image frame 15 is 16. The rendering thread interacts with the main application thread to dequeue the free buffer and store the rendered image frame 15.

[0435] In the sixth cycle, the compositing thread retrieves a queued buffer from the cache queue for compositing. For example, it retrieves image frame 12 and performs the compositing operation. During this cycle, the number of queued buffers in the cache queue decreases by one to 5.

[0436] During this cycle, the display driver displays composite image frame 11, with a shift interval of 16. Before displaying composite image frame 11, the buffer for composite image frame 10 is released, and the buffer information is returned to the compositing thread. The compositing thread returns this buffer information to the main application thread, which updates the buffer status of the buffer queue based on the buffer information. At this point, there is one free buffer in the buffer queue.

[0437] During this cycle, the application main thread does not draw image frames, and the rendering thread does not render image frames.

[0438] The composition thread interacts with the application main thread to reduce the MaxBufferCount of the buffer queue, reducing the number of MaxBufferCount by 1. At this time, MaxBufferCount is 5.

[0439] In the seventh cycle, the composition thread obtains a queued buffer from the cache queue to perform a composition operation, for example, obtaining image frame 13 to perform a composition operation. During this cycle, the number of queued buffers in the cache queue decreases by one to 4.

[0440] During this cycle, the display driver displays composite image frame 12, with a shift interval of 16. Before displaying composite image frame 12, the buffer for composite image frame 11 is released, and the buffer information is returned to the compositing thread. The compositing thread returns this buffer information to the main application thread, which updates the buffer status of the buffer queue based on the buffer information. At this point, there is one free buffer in the buffer queue.

[0441] During this cycle, the application main thread does not draw image frames, and the rendering thread does not render image frames.

[0442] The composition thread interacts with the main application thread to reduce the MaxBufferCount of the buffer queue, reducing the number of MaxBufferCount by 1. At this time, MaxBufferCount is 4.

[0443] Clearly, in this embodiment, dynamically increasing the cache queue's MaxBufferCount during the second, third, and fourth cycles when the composition thread isn't performing composition allows the rendering thread to continuously and stably acquire free buffers for storing rendered image frames. This also ensures that the application's main thread can calculate the displacement of image frames in each cycle according to the normal frame interval. The normal operation of the application's main and rendering threads ensures that the displacement interval between adjacent image frames drawn by the application's main thread remains constant, resulting in a coherent and smooth animation of multiple adjacent image frames after composition and display, avoiding visual stuttering.

[0444] In addition, when it is determined that the application main thread and rendering thread do not perform drawing and rendering operations, dynamically reducing the MaxBufferCount of the cache queue can release redundant buffers in time and improve buffer utilization.

[0445] In this embodiment, when the synthesizer is not synthesizing, the free cache objects in the cache queue can be dynamically increased, and the application process can continue to normally obtain the free cache objects to store the rendered image frames, thereby avoiding the problem in the prior art that the application process cannot obtain the free cache objects to store the rendered image frames, and does not perform the rendering operation of the next image frame, thereby causing the next image frame to be drawn and lost. Through the image processing method provided by the present application, there is always at least one free cache object in the cache queue that can be used by the application process, avoiding the problem of the application process losing frames due to not performing the image frame drawing and rendering operation due to the lack of free cache objects in the cache queue, thereby solving the problem of visual freezes caused by frame loss after the image frame is sent for display. By ensuring that the application process has enough free cache objects to perform the storage operation of the rendered image frame, the smoothness of the dynamic display effect of the displayed image frame is guaranteed.

[0446] Some embodiments of the present application provide an electronic device that may include a memory, a display, and one or more processors. The display, memory, and processor are coupled. The memory is configured to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device may perform the various functions or steps performed by the electronic device in the above-described method embodiments. The structure of the electronic device may refer to the structure of electronic device 100 shown in FIG4 .

[0447] An embodiment of the present application also provides a chip system (e.g., a system on a chip (SoC)). As shown in Figure 16, the chip system includes at least one processor 701 and at least one interface circuit 702. The processor 701 and the interface circuit 702 can be interconnected via lines. For example, the interface circuit 702 can be used to receive signals from other devices (e.g., a memory of an electronic device). For another example, the interface circuit 702 can be used to send signals to other devices (e.g., a processor 701 or a camera of an electronic device). Exemplarily, the interface circuit 702 can read an instruction stored in a memory and send the instruction to the processor 701. When the instruction is executed by the processor 701, the electronic device can execute the various steps in the above embodiments. Of course, the chip system can also include other discrete components, which is not specifically limited in the embodiment of the present application.

[0448] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes the various functions or steps executed by the electronic device 100 in the above-mentioned method embodiment.

[0449] The present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the functions or steps executed by the electronic device 100 in the above method embodiment. For example, the computer may be the above electronic device 100.

[0450] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0451] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0452] The units described as separate components may or may not be physically separate, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0453] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0454] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0455] The above content is only a specific embodiment of this application, but the scope of protection of this application is not limited to this. Any changes or replacements within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An image processing method, characterized in that: include: The electronic device receives a first operation by a user on a touch screen of the electronic device; The electronic device launches a first application in response to the first operation; The electronic device receives a second operation performed by the user on the touch screen of the electronic device; The electronic device exits the first application in response to the second operation; wherein, during the process from the start of the first application to the exit of the first application, the electronic device performs drawing, rendering, and synthesis operations on the first image frame and the second image frame of the first application; The electronic device rendering and compositing operations on the first image frame and the second image frame of the first application include: The application process draws and renders the first image frame within a drawing and rendering cycle of the first image frame, and stores the obtained first image frame in an idle cache object in a cache queue; When the synthesis thread does not perform a synthesis operation within a synthesis period of the first image frame, the synthesis thread sends a first adjustment request to the application process; Based on the first adjustment request, the application process increases the number of free cache objects in the cache queue so that after the application process draws and renders the second image frame within the drawing and rendering cycle of the second image frame, the obtained second image frame is stored in a free cache object in the cache queue, wherein the drawing and rendering cycle of the second image frame is after the drawing and rendering cycle of the first image frame, and the drawing start time of the second image frame differs from the drawing start time of the first image frame by N cycles, where N is a positive integer.

2. The method according to claim 1, characterized in that The first image frame and the second image frame are image frames in the process of starting the first application.

3. The method according to claim 2, characterized in that The first image frame and the second image frame are image frames when the electronic device switches from a first refresh rate to a second refresh rate during the startup of the first application; The first refresh rate is less than the second refresh rate.

4. The method according to claim 1, wherein The first image frame and the second image frame are image frames after the first application is launched.

5. The method according to claim 1, wherein The first image frame and the second image frame are image frames in the process of exiting the first application.

6. The method according to claim 5, characterized in that The first image frame and the second image frame are image frames when the electronic device switches from a first refresh rate to a second refresh rate during the process of exiting the first application; The first refresh rate is less than the second refresh rate.

7. The method according to any one of claims 1 to 6, characterized in that The drawing and rendering cycle of the second image frame is the next cycle of the drawing and rendering cycle of the first image frame; and the drawing start time of the second image frame differs from the drawing start time of the first image frame by one cycle.

8. The method according to any one of claims 1 to 6, characterized in that The first adjustment request includes a first indication value; the first indication value is used to indicate an increased number of cache objects, and increasing the number of free cache objects in the cache queue includes: The application process adds a free cache object with the first indication value to the cache queue.

9. The method according to claim 8, characterized in that The application process adds the free cache object of the first indication value to the cache queue, including: The application process adds the addresses of the free cache objects of the first indicator value to the cache queue in an enqueuing order.

10. The method according to any one of claims 1 to 6, characterized in that The method further comprises: The synthesis thread queries the number of all cache objects in the cache queue; If the number of all cache objects reaches the maximum number of cache objects, the composition thread stops sending the first adjustment request for the cache object to the application process.

11. The method according to any one of claims 1 to 6, characterized in that After storing the first image frame in an idle cache object in a cache queue, the method further includes: The synthesis thread obtains and records the storage time when the application process stores the first image frame in the target cache object.

12. The method according to claim 11, characterized in that The method further comprises: In a case where the composition thread performs a composition operation in a composition period of the first image frame, the composition thread determines a time difference between a current system time and a storage time of a last recorded image frame stored in the target cache object; If the time difference is greater than or equal to a preset time threshold, the composition thread sends a second adjustment request for the cache object to the application process; The application process reduces the number of idle cache objects in the cache queue according to the second adjustment request.

13. The method according to claim 12, characterized in that The second adjustment request includes a second indication value, where the second indication value is used to indicate a reduced number of cache objects. The reducing the number of idle cache objects in the cache queue includes: The application process reduces the free cache objects of the second indication value from the cache queue.

14. The method according to claim 13, characterized in that The application process reducing the free cache objects of the second indication value from the cache queue includes: The application process removes the address of the free cache object of the second indicator value from the cache queue in a dequeue order.

15. The method according to any one of claims 1 to 6, characterized in that The method further comprises: The synthesis thread queries the number of all cache objects in the cache queue; If the number of all cache objects is reduced to the minimum number of cache objects, the composition thread stops sending the second adjustment request for the cache objects to the application process.

16. The method according to any one of claims 1 to 6, characterized in that The method further comprises: If the composition thread performs a composition operation in a composition cycle of the first image frame, the composition thread obtains a target cache object from the cache queue; the target cache object stores the first image frame after drawing and rendering; The synthesis thread performs a synthesis operation on the first image frame after drawing and rendering.

17. An electronic device, characterized in that: The electronic device includes a memory, a display screen, and one or more processors; the memory and the display screen are coupled to the processor; the memory stores computer program code, and the computer program code includes computer instructions, which, when executed by the processor, enable the electronic device to perform the method described in any one of claims 1 to 16.

18. A computer-readable storage medium, characterized in that The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 16.

Images